T14 November 7, 2002 2:30 PM F EATURE R ISK A NALYSIS : W HAT D O I - - PDF document

BIO PRESENTATION International Conference On Software Testing Analysis & Review November 4-8, 2002 Anaheim, CA USA

T14

November 7, 2002 2:30 PM

FEATURE RISK ANALYSIS: WHAT DO I TEST FIRST? AND LAST?

Steve Tolman PowerQuest Corporation

Steve Tolman

Steve Tolman is a development manager at PowerQuest Corporation in Orem, Utah. He has been with PowerQuest for over four years as a Manager of Software Quality as well as his current job as Development Manager. Steve has more than 14 years of experience in software testing, development, and management. Steve holds a B.S. from Brigham Young University in Information Management.

Feature Risk Analysis Feature Risk Analysis

What Do I Test First? And Last? What Do I Test First? And Last?

Steve Tolman Development Manager PowerQuest Corporation

PowerQuest Corporation 2

Feature Risk Analysis

Overview Feature Risk Analysis is a quick, valuable method designed to determine which features need the most testing, and which need the least.

• Quantify and qualify testing effort
• Define scope of testing
• Know when you are done

PowerQuest Corporation 3

Feature Risk Analysis

What Is It?

• Feature risk analysis is:
• A tool used to help identify acceptance (i.e., smoke test), basic, and

advanced functionality levels of testing.

• A means of prioritizing testable items
• Analyzing risk should be an ongoing effort started early

in the project life cycle. As the product matures, the analysis of risk may change.

PowerQuest Corporation 4

Feature Risk Analysis

Assumptions

• Team allowed flexibility to deliver quality product.
• The definition of quality is well understood by the team.
• Organization processes are defined, understood, and followed.
• Project is scoped and minimally defined.
• FRA should be updated through life of project.

PowerQuest Corporation 5

Feature Risk Analysis

Guidelines

• An odd number of testers (I recommend three or five) will meet and

review all known functionality of a product.

• Each functional piece (testable item) is listed so that all can see

and discuss it.

• Once all testable items are listed, they are sorted into high, medium,

and low priority groups.

• Within each grouping, each testable item is sorted by most

important to least important.

– This is represented with a number. Example, within the high priority group, H1 is the first item, H2 is the second, H3 is the third, etc. until the last item Hn is listed.

PowerQuest Corporation 6

Feature Risk Analysis

Guidelines

• Once all functional items are listed, grouped, and prioritized the

testers identify all possible operating environments (operating system platforms, file systems, languages, hardware, etc.) where the product will operate.

• These environments are combined, where it makes sense, to

derive the smallest set of environments with which the product must be tested.

• When these steps have been completed, the team can then create

a test matrix and test cases based on these prioritized lists and environments.

PowerQuest Corporation 7

Feature Risk Analysis

Guidelines

• This test matrix and test cases should be made available for

review by other members of the development team.

PowerQuest Corporation 8

Feature Risk Analysis

Example

Windows Display Properties Settings dialog

PowerQuest Corporation 9

Feature Risk Analysis

Step 1: List Features

• Dialog controls | System menu (Alt-Space) | Move
• Dialog controls | System menu (Alt-Space) | Close
• Dialog controls | Close ‘X’ button
• Dialog controls | “What’s this” help button
• Dialog controls | OK button
• Dialog controls | Cancel button
• Dialog controls | Apply button
• Dialog controls | Apply button | Confirm dialog
• Dialog controls | ESC supported?
• Dialog controls | Tab order
• Dialog controls | Ctrl-Tab
• Dialog controls | Mouse-over help
• Dialog controls | Right click menu context sensitivity
• Dialog controls | Mnemonics
• Dialog controls | Space bar selection
• Dialog controls | Alt-Down arrow on drop down
• Dialog controls | Localization?
• Dialog controls | Timing?
• Dialog controls | Visually correct – Spacing/layout
• Monitor icons | Gray background w/ simulated LCD display
• Monitor icons | Blended background w/ simulated monitor
• Monitor icons | Single monitor
• Monitor icons | Multi-monitor
• Monitor icons | External monitors
• Display drop-down | Correctly display all known monitors
• Display drop-down | Each display is selectable
• Screen resolution | Monitor type effects
• Screen resolution | Video adapter effects
• Screen resolution | Min resolution
• Screen resolution | Max resolution
• Screen resolution | Correct display of target resolution as slider moves
• Color quality | Dropdown has selectable palettes
• Color quality | Color bar adjusts as different palettes are selected
• Checkbox: Use this device as the primary monitor | Enabled/disabled appropriately
• Checkbox: Extend my Windows desktop onto this monitor checkbox | Enabled/disabled appropriately
• Identify button | Correctly identifies each monitor in use
• Troubleshoot button | Correctly open troubleshooting dialog
• Advanced button | Correctly opens advanced dialog

PowerQuest Corporation 10

Feature Risk Analysis

Step 1: List Features (Combine)

• Dialog | Parent controls System menu (Alt-Space), Close ‘X”,

“What’s this?” help, OK, Cancel, Apply, ESC supported?)

• Dialog | Client controls (Tab, Ctrl-Tab, Mouse-over help, Right

click menu context sensitivity, Mnemonics, Alt-Down arrow on drop down, checkboxes)

• Dialog | Presentation (Timing, Visually correct – Spacing/layout)
• Monitors | Monitors (single, multiple, external)
• Screen resolution
• Color quality
• Identify button
• Troubleshoot button
• Advanced button

PowerQuest Corporation 11

Feature Risk Analysis

Step 2: Prioritize Features

• High
• Advanced button
• Screen resolution
• Monitors
• Medium
• Troubleshoot button
• Color quality
• Dialog | Presentation

PowerQuest Corporation 12

Feature Risk Analysis

Step 2: Prioritize Features

• Low
• Dialog | Parent controls System menu (Alt-Space), Close ‘X”,

“What’s this?” help, OK, Cancel, Apply, ESC supported?)

• Identify button
• Dialog | Client controls (Tab, Ctrl-Tab, Mouse-over help, Right click

menu context sensitivity, Mnemonics, Alt-Down arrow on drop down, checkboxes)

PowerQuest Corporation 13

Feature Risk Analysis

Step 3: Prioritize Within Each Group

• High

H1 Dialog | Advanced button H2 Dialog | Screen resolution H3 Dialog | Monitors

• Medium

M1 Color quality M2 Dialog | Presentation M3 Troubleshoot button

PowerQuest Corporation 14

Feature Risk Analysis

Step 3: Prioritize Within Each Group

• Low

L1 Dialog | Parent controls L2 Dialog | Identify button L3 Dialog | Client controls Note: By the time you get to the lows, it doesn’t really matter how they are prioritized – they are all low priority. If you find some low priority items whose priority matters, perhaps they are not really low priority.

PowerQuest Corporation 15

Feature Risk Analysis

List Valid Environments

• Languages
• English, French, German, Japanese, Spanish
• Operating Systems
• Windows 98, 98SE, ME , 2000 Pro, XP

PowerQuest Corporation 16

Feature Risk Analysis

Test Matrix

Risk Boundary

PowerQuest Corporation 17

Feature Risk Analysis

Summary

Benefits:

• FRA can be broken up by product components (i.e., install, main

.EXE, utilities, networking, video display, etc.)

• Allows test team to identify risk boundaries in product.
• Easy way to show work completed vs. work remaining.
• Helps evaluate can/can’t ship situations
• Team members become very proficient at this very quickly
• Significantly reduces stress

PowerQuest Corporation 18

Feature Risk Analysis

Overview

Questions?