Hendrix Test Plan

WORKING DRAFT

Overview

This document provides a high level overview of the test areas/and scenarios to be covered by QA for the Hendrix Songbird application release. Detailed test cases will be available via our test case management database at: http://litmus.songbirdnest.com/

Supported OS Platforms

QA will test with the following OS platforms for the Hendrix project.  Tier 1 platforms will receive full new feature, regression, performance, stability and nest integration testing.  Tier 2 will receive installation testing as well as quick looks of all Functional test types:

Tier 1

  1. Windows XP SP3 (no change from 1.0 release)
  2. Windows Vista Ultimate SP1 32-bit (no change from 1.0 release)
  3. Linux Ubuntu 8.10 32-bit (upgrade from 8.04 tested with 1.0 release)
  4. Mac i686 Leopard 10.5.6 (upgrade from 10.5.5 tested with 1.0 release)

Tier 2

  1. Linux Fedora 9 64-bit (was a Tier 1 platform in 1.0)
  2. Windows Vista Ultimate SP1 64-bit
  3. Mac i686 Tiger 10.4.11

Supported Mobile Devices

Tier 1

MTP
  • Philips SA5285 (8GB)
  • Philips SA5245 (4GB)
  • Sansa e260 (4GB)
  • SanDisk Fuze (4GB)
iPod (sanity tests only)
  • iPod Classic 6th Gen (120 GB)
  • iPod Nano 4th Gen (8 or 16GB)

Tier 2 (sanity tests only)

  • Sansa Clip (1 or 2GB)
  • Creative Zen (4GB)

Installation/Upgrade Testing

Verify 1.1.0rc1/final

  • Verify 1.1.0rc1
  • Re-verify for any subsequent 1.1.0rc builds
  • Verify 1.1.0 Final

Updates (using an 1.0.0 profile)

  • Verify 1.0.0 final > 1.1.0rc1
  • Re-verify from 1.0.0 final for any subsequent 1.1.0rc builds
  • Verify 1.1.0rc1 > 1.1.0rc2
  • Verify subsequent rc upgrades as required
  • Verify 1.0.0 final >1.1.0 final

Updates must be verified as working within one week of code complete.

Install Types

Installation testing scenarios will include:

  • Partials: Updates using snippets instead of complete downloads.
  • Complete: Updating from an older version which downloads a 'complete' mar file (i.e. 1.0.0 > 1.1.0)
  • Fallback: Replacing the string from pending to 'failed' in the update.status file. Restart the app and ensure user gets another copy of the update.

First Run Bundles

First run Add-Ons for 1.1.0 will be the following, in this order:

Windows i686
  1. iPod Device Support
  2. MTP Device Support
  3. QuickTime Playback
  4. Windows Media Playback
  5. SHOUTcast Radio
  6. Concerts
  7. last.fm
  8. mashTape
Mac OSX i686
  1. iPod Device Support
  2. QuickTime Playback
  3. SHOUTcast Radio
  4. Concerts
  5. last.fm
  6. mashTape
Linux i686/Linux x86_64
  1. iPod Device Support
  2. SHOUTcast Radio
  3. Concerts
  4. last.fm
  5. mashTape

Language Support

The following foreign languages will be tested to ensure that they load/view correctly:

  • French (France)
  • Italian (Italy)
  • German (Germany)
  • Spanish (Spain)
  • Chinese (China)
  • Japanese (Japan)

New Features Functional Testing

This section outlines the new features for the Hendrix 1.1.0 Songbird application and areas where new test cases will be written to execute throughout the product development.


Fetching Album Art

Fetching Album Art Project Page

The new Album Art Fetching feature will give Songbird the ability to query multiple sources in an attempt to associate album artwork with every audio track in a user's library.  Songbird will have three built-in sources of album art: track metadata, the local filesystem, and Last.fm.  An add-on may also be created to allow fetching art from Amazon.

At a high level, testing will ensure that:

  1. The album art search is conducted through several individual "Fetchers", which are executed in a specific order:
    • Track Metadata Fetcher
    • Last.fm Fetcher
    • Add-on Fetchers (amazon)
    • Local Filesystem Fetcher (Disabled by default)

  2. Songbird stops searching for album art as soon as artwork is found.
  3. Found album art is written to the track as an id3v2 APIC tag with a type of $03.
  4. Album art is not written to the track if the "write to track" preference is disabled.

User Interface

A new, user-invoked dialog box will be added to Songbird. This dialog will display progress on an album by album detail, with the current fetcher being used to search for artwork. Progress will display as a progress bar and a cancel button is available to abort the search.  Details needed.

Preferences

songbird.metadata.artwork.enableWriting (default: true)

  • When set to true, Songbird will attempt write any artwork it locates to the headers of the associated mp3 track.
  • When set to false, Songbird will not attempt to write artwork to the headers of any mp3 tracks.

songbird.albumart.file.base_names (default: cover,folder)

  • The Local Filesystem Fetcher will look for artwork in files with any one of the names specified in this preference (when combined with the songbird.albumart.file.extensions preference).

songbird.albumart.file.extensions (default: jpg,jpeg,png,gif,bmp,tif,tiff)

  • The Local Filesystem Fetcher will combine these extensions with the file names in the base_names property to create the list of filenames it will search for.

songbird.albumart.maxsize (default: 16777216)

  • The maximum size, in bytes, for an album artwork image.

songbird.albumart.file.enabled (default: false)
  • Enable or disable the File fetcher.

songbird.albumart.file.priority
(default: -1)
  • The priority of the File fetcher.

songbird.albumart.metadata.enabled (default: true)
  • Enable or disable the Metadata fetcher.

songbird.albumart.metadata.priority
(default: 0)
  • The priority of the Metadata fetcher.

songbird.albumart.lastfm.enabled (default: true)
  • Enable or disable the Last.FM fetcher.

songbird.albumart.lastfm.priority
(default: 1)
  • The priority of the Last.FM fetcher.
Priorities
  • Priorities are ordered from lowest number to highest, so a fetcher with a priority of 1 will be used before a fetcher with a priority of 5.
  • A priority of less than 0 indicates that there is no priority for the fetcher, commonly we will use -1 to indicate this. A fetcher with this priority will be appended to the end of the fetcher list.
Track Metadata Fetcher
  • Images already embedded in a track's metadata will be located and used as the track's art. 
  • Songbird will not look any further for art if it is located in the track.
  • Songbird will look for art marked as "front cover" ($03) or "other" ($00). Any other embedded art will be ignored.
  • If artwork is found for both "front cover" and "other", the "front cover" art will be the art used by Songbird.
  • If artwork is found in "other" ($00) and the enableWriting preference is set to true, Songbird will add another copy of the cover art to the track and mark it as "front cover" ($03) data.
  • Songbird does not enter a failure mode when encountering corrupted data in an APIC tag.
  • If the Fetcher does not locate art in the track metadata, the next Fetcher will begin searching for art.
Local Filesystem Fetcher
  • The Local Filesystem Fetcher is not used by default to find artwork, it must be enabled to allow it to be used during a scan. This is because we get unexpected results when a user does not have an organized library on the file system.
  • Using the comma-delimited values in the songbird.albumart.file.base_names preference as one set and the comma-delimited values in the songbird.albumart.file.extensions preference as another set, the Local Filesystem Fetcher will search through the Cartesian Product of the two sets to find a matching file, i.e.

    {cover,photo} x {jpg,png} = {cover.jpg,cover.png,photo.jpg,photo.png}
     
  • The Fetcher will stop searching for artwork when it locates a matching filename with valid artwork.
  • Found artwork will be written to an APIC tag in the track's id3v2 header with a type of $03 (front cover).
  • The Fetcher will continue searching for artwork if it locates a matching filename with invalid artwork.
  • Songbird does not enter a failure mode when the Fetcher locates a matching filename with invalid artwork.
  • If the Fetcher does not locate art in the track metadata, the next Fetcher will begin searching for art.
Last.FM Fetcher
  • The Last.FM Fetcher will be used by default if artwork is not found using the Metadata Fetcher.
  • We use the Last.FM REST api to locate a tracks artwork under the track.getInfo method. This method is not a write method so we do not require the user to log into thier Last.FM account.
  • The Fetcher will search the images returned in the following order ['Large', 'Medium', 'Small'] and stop searching for artwork when it locates a url to an image.
  • The Fetcher will download images it finds to the profile cache folder, if this fails then the search fails and it will move on to the next fetcher.
  • If the Fetcher does not locate art online the next Fetcher will begin searching for art.

Watch Folders

Watch Folders Project Page

This feature...


Video Playback

Video Playback Project Page

Full video support is currently planned for Isan, not Hendrix.  In this release, known issues in video support will be fixed.  Those issues are listed on the Video Playback project page (linked above).  The test effort during Hendrix will be limited to verifying fixes for the bugs within the scope of the project.


Normalization of Playback Volume

Normalisation Project Page

Songbird will use volume normalization information, when available, to adjust playback volume.  Normalization information may be available in the form of Replay Gain data and/or iTunes Sound Check metadata.  In instances where both types of data are available, Replay Gain takes precedence.

Testing will ensure that volume normalization data, when present, is read and seems to be used properly.  The testing effort will likely not extend beyond simple listening tests with files that we've modified to be either extremely loud or extremely quiet.

Replay Gain information can be stored in (at least) three different ways:

TXXX ID3v2 header (mp3)

foobar2000 stores Replay Gain information using the id3v2.3 TXXX field.

replaygain_album_gain (lower numbers = softer playback)
replaygain_album_peak
replaygain_track_gain (lower numbers = softer playback)
replaygain_track_peak

See also: TXXX (user-defined text information frame, see section 4.2.6 at http://www.id3.org/id3v2.4.0-frames)

Vorbis comments (flac, ogg)

vorbisgain, flac 1.1.3, and probably other encoders/Replay Gain calculators store Replay Gain information for Ogg and FLAC files in Vorbis comments fields.  The fields are the same as those for mp3s, except in all caps:

REPLAYGAIN_TRACK_GAIN
REPLAYGAIN_TRACK_PEAK
REPLAYGAIN_ALBUM_GAIN
REPLAYGAIN_ALBUM_PEAK

It appears that there are deprecated tags named RG_RADIO, RG_PEAK, and RG_AUDIOPHILE.  These are probably not implemented and will not be tested.

The source code for vorbisgain defines NO_PEAK as -1.f and NO_GAIN as -10000.f.  It appears that values greater than the NO_PEAK and NO_GAIN are valid peak and gain values.

All of these assumptions are based on the source code found here: http://sjeng.org/vorbisgain.html

ID3v2.* Relative volume adjustment tags (RVAD/RGAD/XRVA/RVA/RVA2)

RVA: Implemented in id3v2.0.  Allows per-channel relative volume adjustment for the left and right channels only.

RVAD: Implemented in id3v2.3.  Allows per-channel relative adjustment for left, right, center, left back, right back, and bass channels.  Testing will only cover adjustments to the left and right channels.  See Section 4.12 of the id3v2.3.0 specification at http://www.id3.org/id3v2.3.0#head-d3...2bb676653d3604.

RGAD: Labeled as an "Unofficial Frame Seen in the Wild" by id3.org.  See http://www.id3.org/Replay_Gain_Adjustment and http://replaygain.hydrogenaudio.org/...mat_id3v2.html.  If this is in use in the wild, its prevalence is very limited.

XRVA: Labeled as an "Unofficial Frame Seen in the Wild" by id3.org.  See http://www.id3.org/Experimental_RVA2 and http://normalize.nongnu.org/.  This tag is created by the normalize program and is used to back-port RVA2 data from id3v2.4 tags to id3v2.3 tags.

RVA2: Implemented in id3v2.4.  See section 4.11 of http://www.id3.org/id3v2.4.0-frames.
LAME header information stored within the Xing tag

Replay Gain information can also be stored within a LAME header inside an Xing header.  The authoritative page seems to be this one: http://gabriel.mp3-tech.org/mp3infot...tml#replaygain.

iTunes Sound Check

Songbird will use iTunes Sound Check information, when available, to normalize playback volume.  iTunes stores Sound Check data inside an id3v2.2 COMM (comment) tag using the tag iTunNorm.


Sorting

Sorting Project Page

The overhaul of Songbird's sorting mechanism in Hendrix will rely primarily on Unicode Collation to create industry-standard sorted lists within Songbird's library (and similar media lists, such as Playlists and filtered library views).  Collation algorithms will sort on a per-locale basis, so testing must cover not only sorted lists on machines in the US English locale, but also in the locales of our Tier 1 languages (Chinese, French, German, Italian, Japanese, and Spanish).

We will also be introducing special rules to Songbird's sorting functionality to be more intelligent about sorting punctuation, whitespace, and definite and indefinite articles.

Testing of Special Rules

In addition to the introduction of Unicode Collation, Songbird will begin omitting certain characters and words when sorting.  Testing will ensure that the following characters/words, when appearing at the beginning of a name, do not determine the sort order of that particular item:

  • From English: the words "A", "An", "The" (case-insensitive)
  • From French: the words "Un", "Une", "L'", "La", "Le", "Les" (case-insensitive)
  • From Spanish: the words "El", "La", "Los", "Las", "Un", "Una", "Unos", "Unas" (case-insensitive)
  • From German: the words "Der", "Die", "Das", "Ein", "Eine" (case-insensitive)
  • From Italian: the words "Lo", Il", "L'" (also in Fr), "La" (also in Fr), "Gli", "Un" (also in Fr), "Una", "Un'" (case-insensitive)
  • From Portugese: the words "O", "A" (also in En), "Os", "As", "Um", "Uma", "Uns", "Umas" (case-insensitive)
  • Punctuation characters
  • Whitespace characters

Items beginning with digits should sort numerically.  The exception to this rule is negative numbers; items beginning with a negative number will sort as though the number is positive.

Secondary Sort

Secondary Sort rules determine how items in the library are sorted based upon which column is the Primary Sort column.  See the Secondary Sort Column Rules section of the Hendrix Sorting feature document for a full list of secondary sorting behavior.

Unicode Collation Testing

Code charts from Unicode should be used to test correct sorting.  Note that we are using OS-specific implementations of the Unicode Collation algorithm, so sorting should be tested thoroughly on all Tier 1 platforms. The test effort should cover the following:

Unicode Collation Charts for English, French, German, Italian, and Spanish:
Basic Latin
Latin-1

Language-specific Collation Charts (locale-specific subsets of Latin and Latin-1):
English: http://developer.mimer.com/charts/english.htm
French: http://developer.mimer.com/charts/french.htm
German: http://developer.mimer.com/charts/german.htm
Italian: http://developer.mimer.com/charts/italian.htm
Spanish: http://developer.mimer.com/charts/spanish.htm

Unicode Collation Charts for Japanese:

Hiragana
Katakana

Unicode Collation Chart for Chinese and Japanese:

CJK Unified Ideographs

In addition to the Tier 1 Languages, the community has expressed gratitude for the fact that Songbird sorts Hebrew and Cyrillic properly.  A spot-check should be performed on these languages to ensure we didn't regress.

Hebrew
Cyrillic

N.B.: The Swedish language will be a good way to quickly verify that locale-specific sorting is occurring, because the characters å, ä, and ö should be sorted at the end of the alphabet.  See http://developer.mimer.com/charts/swedish.htm for a full Swedish collation chart.

Product Impact

Search: Songbird's library search functionality currently piggybacks off of the existing sorting data.  We should expect search to be affected by the modification to the existing sorting data.

Database: Part of the upgrade from the 1.0 product will be a database migration that updates the existing sorting data. See bug 14271 for details.
 


Invoking Songbird from URLs

Invoking Songbird from URLs Project Page


Gapless Playback

Gapless Playback Project Page



Content Type Detection

Content Type Detection Project Page

Mediacore Maintenance Work


Mediacore Maintenance Work Project Page

ACM mp3 Decoder

ACM mp3 Decoder (Bugzilla)

Mediacore Preferences

Mediacore Preferences Project Page

DirectSound Sink Rewrite


blah blah blah...

Regression Functional Testing

QA will conduct a full test pass of all of the following functional areas on all supported Tier 1 OS platforms to ensure the following legacy code does not regress with the Hendrix Songbird application release:

  • Playback of all supported audio types
  • Menubars
  • Default Service Pane nodes & their child nodes as well as context menus
  • Library playlist functionality
  • Downloads list functionality
  • Web Playlist functionality (sites in Songbird GUI links only; scraping sites)
  • New Playlist/Smart Playlist functionality
  • Full/Mini Feathers Views (Gonzo, YABS and one other popular feather addon)
  • Media Views
  • Display Panes
  • All Options/Pref Panes and their full functionality
  • Metadata Editor
  • Tabbed browsing
  • WebPage API (run all test file/page, use example apps from music blog/store)
  • Add-Ons/Get Extensions install functionality (via Songbird install as well as drag/drop)
  • iTunes Libary Importer functionality
  • iPod Device Support and Summary Page functionality
  • MTP Device Support and Summary Page functionality
  • SHOUTcast Add-on
  • mashTape Add-on
  • Concerts Add-on
  • Last.fm Add-on
  • Performance
    • Cold Start
    • Scan for Media - General
    • Scan for Media - Large Libraries
    • Library Database

Nest Integration Testing

App Update

  • verify previous release > RC1 update
  • verify RC1 > RC2 update
  • re-verify for any subsequent RC builds
  • verify previous release > current release
  • verify partial mar file is applied by default
  • verify failover to complete mar file
  • verify download progress bar works correctly
  • verify time remaining counter works correctly
  • verify download size counter works correctly
  • verify minor update dialog link points to correct release notes
  • verify major update iframe renders HTML correctly

Add-ons Installation

  • install addon using install button on a listing page
  • install addon using install button on Other versions page
  • verify clicking install button returns a dialog with correct URL
  • verify compatibilty message for incompatible extension

Add-ons Update

  • using find updates from the Addons dialog, verify a new addon version can be found and installed correctly

Metrics Integration

  • verify metrics are being collected
  • verify metrics post at expected interval

Bookmarks

  • verify bookmark favicons render correctly
  • verify bookmark urls match what is in bookmarks.xml
  • verify bookmark urls are valid
  • verify bookmark location and title match

Birdhouse

  • verify that birdhouse.songbirdnest.com is viewable in the bird
  • verify that birdhouse.songbirdnest.com redirects to www.songbirdnest.com in any other user agent
  • verify that legacy versions (more than 2 releases old) bring up "get a new version" prompt

Locales

  • verify L10n bundles download and install correctly
  • verify L10n bundles update correctly
     

Out of Scope

The following items are out of scope of this testing effort:

Tag page
FileSizeDateAttached by 
 U0000.pdf
Basic Latin
159.78 kB13:40, 6 Jan 2009aliActions
 U0080.pdf
Latin-1
138.69 kB13:40, 6 Jan 2009aliActions
 U0400.pdf
Cyrillic
171.09 kB13:57, 6 Jan 2009aliActions
 U0590.pdf
Hebrew
77.96 kB13:57, 6 Jan 2009aliActions
 U3040.pdf
Hiragana
105.71 kB13:40, 6 Jan 2009aliActions
 U30A0.pdf
Katakana
96.39 kB13:54, 6 Jan 2009aliActions
 U4E00.pdf
CJK Unified Ideographs
4.83 MB13:57, 6 Jan 2009aliActions
Viewing 1 of 1 comments: view all
Languages: 2 sets of tests.

Restart w/ different languages in English OS.
Test import and sorting in native OSes for 2 non-English languages (German and Chinese) edited 16:03, 16 Jan 2009
Posted 15:59, 16 Jan 2009
Viewing 1 of 1 comments: view all
You must login to post a comment.