Template:Test Plan
Table of contents
- 1. Resources and Coverage
- 2. Supported OS Platforms
- 3. Supported Mobile Devices
- 4. Application Installation and Update Testing
- 5. New Features Functional Testing
- 5.1. Feature
- 5.1.1. Feature Subset 1
- 5.1.1.1. Feature Subset 1 Section 1
- 5.1.1.2. Feature Subset 1 Section 2
- 5.1.2. Feature Subset 2
- 5.1.1. Feature Subset 1
- 5.2. Feature
- 5.1. Feature
- 6. Regression Functional Testing
- 6.1. Priority 1
- 6.2. Priority 2
- 6.3. Priority 3
- 7. Performance Testing
- 8. Nest Integration Testing
- 8.1. Add-ons Installation
- 8.2. Add-ons Update
- 8.3. Locales
- 9. Out of Scope
{{wiki.toc("/QA/Releases/Project_Test_Plan")}}
Resources and Coverage
In addition to the people listed in the Project Definition table, n other people are available to help with verifications, device testing, and regression testing.
We plan to cover 100% of the testing effort as outlined below. If we start to run short on time, testing will be truncated in the Priority 3 regression tests.
Supported OS Platforms
Tier 1
- Windows XP SP3 32-bit
- Windows Vista Home Premium SP2 32-bit
- Linux Ubuntu 9.04 32-bit
- Mac OS X 10.5.8
Tier 2
- Windows Vista Home Premium SP2 64-bit
- Linux Fedora 9 64-bit
Supported Mobile Devices
Tier 1
- Philips GoGear 52xx (MTP)
- Philips GoGear Mix (MSC)
- Sansa e260
Tier 2
- iPod Classic 6th gen
- iPod Nano 4th gen
- iPod Shuffle 3rd gen
Application Installation and Update Testing
New Installations
The Songbird installer for Windows will be exercised on Tier 1 Windows platforms using the pre-defined regression tests in Litmus (full list of tests).
First Run Add-Ons
The following add-ons will be offered to users, in the following order, during the Songbird first run wizard:
- Last.fm
- mashTape
- Concerts
- 7digital
- SHOUTcast
- MSC (Windows only)
- MTP (Windows only)
- QuickTime Playback (Windows and Mac)
- Windows Media Playback (Windows only)
Updates to Existing Installations
The following update scenarios, using both partial and complete updates, will be tested on all supported platforms:
- Previous final to each beta for the new release
- Previous final to new release final
- Each new release beta to the subsequent new release beta
- The last new release beta to the new release final
New Features Functional Testing
Feature
Text
Feature Subset 1
Text
Feature Subset 1 Section 1
Text
Feature Subset 1 Section 2
Text
Feature Subset 2
Text
Feature
Regression Functional Testing
The existing suite of regression tests will be split up into three priority tiers to ensure that the highest priority features are thoroughly tested. A list of the full set of regression tests, plus the execution status of those tests, is available in Litmus (http://litmus.songbirdnest.com/test_...t_run_id=xxxxx).
Regression areas fall into priority tiers (listed below), with Priority 1 being the highest priority and Priority 3 being the lowest. We intend to execute 100% of the regression tests; however, if we run low on time during the testing phase, we may choose to execute only minimal testing against Priority 3 areas. Priority 1 tests will be executed before Priority 2 tests during their scheduled week to ensure adequate time for bug fixing (see the xxxx Test Schedule for details).
Priority 1
- Audio Playback
- Audio Transcoding
- Device Support (MSC and MTP)
- Installation
- Localization
- Media Import
- Metadata
- Video Playback
Priority 2
- Album Art
- APIs
- CD Rip
- iTunes Import/Export
- OS Integration
- Playlists
- Search
- UI
Priority 3
- Browser
- Media Download
- Profile Management
- Surveys
Performance Testing
Performance Testing will ensure that Songbird remains within 20% of the following baseline performance values on Windows:
| Test Task | Testcase ID | Test Setup/Description | Performance Metrics Goal | ||
| Application Startup | |||||
| First Run - Final Win setup wizard screen to --> first SB setup screen - w/ Network | TBD | Clean, full install | TBD | ||
| First Run - Final SB setup screen to first app window UI shown - w/ Network | TBD | No First Run Addons installed No files imported | TBD | ||
| First Run - Final Win setup wizard screen to --> first SB setup screen - w/o Network | TBD | Clean, full install | TBD | ||
| First Run - Final SB setup screen to first app window UI shown - w/o Network | TBD | No First Run Addons installed No files imported | TBD | ||
| Cold Startup - w/ Network | 3488 | No First Run Addons installed 10,000 local tracks in library | < 30s | ||
| Cold Startup - w/o Network | TBD | No First Run Addons installed 10,000 local tracks in library | < 30s | ||
| First Quit/Relaunch immediately after online install - w/ Network | TBD | No First Run Addons installed 10,000 local tracks in library | < 8s | ||
| First Quit/Relaunch immediately after online install - w/o Network | TBD | No First Run Addons installed 10,000 local tracks in library | < 8s | ||
| Warm Startup - Subsequent average relaunches w/ Network | 3489 | No First Run Addons installed 10,000 local tracks in library | < 8s | ||
| - Subsequent average relaunches w/o Network | TBD | No First Run Addons installed 10,000 local tracks in library | < 8s | ||
| Media Import | |||||
| Import 10,000 audio items | 3491 | Reference 10,000 local tracks, no feature extraction | 2m40s | ||
| Import XXX video items | TBD | Reference XX files | TBD | ||
| Device - MTP | |||||
| Mount time | 3494 | Time for the icon of a clean device to appear in the Service Pane | < 5s | ||
| Full Sync (1K/10K) Audio | 3492 | Transfer 1000 songs from library of 10,000 songs to clean device (MTP speed of ~4MB/s and tracks size of ~5MB) | 25m | ||
| Partial Sync Audio | 3496 | Transfer 20 songs from library of 10,000 songs to device with 1000 songs (MTP speed of ~4MB/s and tracks size of ~5MB) | < 30s | ||
| Device library display | 3519 | Display of 1K device library in list view | < 20s | ||
| Sync video | TBD | Reference XX files | TBD | ||
| Transcode video | TBD | Reference XX files | TBD | ||
| Device - MSC | |||||
| Mount time | 3491 | Time for an icon for the device to appear in the service pane; clean device | < 5s | ||
| Full Sync (1K/10K) Audio | 3493 | Transfer 1000 songs from library of 10,000 songs to clean device (MTP speed of ~4MB/s and tracks size of ~5MB) | 25m | ||
| Partial Sync Audio | 3495 | Transfer 20 songs from library of 10,000 songs to device with 1000 songs (MTP speed of ~4MB/s and tracks size of ~5MB) | < 30s | ||
| Device library display | 3520 | Display of 1K device library in list view | < 20s | ||
| Sync video | TBD | Reference XX files | TBD | ||
| Transcode video | TBD | Reference XX files | TBD | ||
| Usage | |||||
| Memory usage | 3521 | Application idle after a cold start | ~ 80MB | ||
| Memory usage | 3522 | Import 10,000 local tracks | Not over 120MB | ||
| Memory usage | 3523 | Transfer 1000 tracks to clean device | Not over 120MB | ||
| Memory usage | 3524 | Transfer 20 tracks to device with 1000 tracks | 80M -120MB | ||
| CPU usage during playback | 3525 | Audio track playback only | < 5% | ||
Nest Integration Testing
Add-ons Installation
- install addon using install button on a listing page
- install addon using install button on Other versions page
- verify clicking install button returns a dialog with correct URL
- verify compatibilty message for incompatible extension
Add-ons Update
- using find updates from the Addons dialog, verify a new addon version can be found and installed correctly
Locales
- verify L10n bundles download and install correctly
- verify L10n bundles update correctly
Out of Scope
The following items will not be tested in this release:
- Tag page
- What links here
| Images 0 | ||
|---|---|---|
| No images to display in the gallery. |
