T14 November 7, 2002 2:30 PM F EATURE R ISK A NALYSIS : W HAT D O I - PDF document

BIO PRESENTATION T14 November 7, 2002 2:30 PM F EATURE R ISK A NALYSIS : W HAT D O I T EST F IRST ? A ND L AST ? Steve Tolman PowerQuest Corporation International Conference On Software Testing Analysis & Review November 4-8, 2002 Anaheim, CA USA

Steve Tolman Steve Tolman is a development manager at PowerQuest Corporation in Orem, Utah. He has been with PowerQuest for over four years as a Manager of Software Quality as well as his current job as Development Manager. Steve has more than 14 years of experience in software testing, development, and management. Steve holds a B.S. from Brigham Young University in Information Management.

Feature Risk Analysis Feature Risk Analysis What Do I Test First? And Last? What Do I Test First? And Last? Steve Tolman Development Manager PowerQuest Corporation

Feature Risk Analysis Overview Feature Risk Analysis is a quick, valuable method designed to determine which features need the most testing, and which need the least. • Quantify and qualify testing effort • Define scope of testing • Know when you are done PowerQuest Corporation 2

Feature Risk Analysis What Is It? •Feature risk analysis is: • A tool used to help identify acceptance (i.e., smoke test), basic, and advanced functionality levels of testing. • A means of prioritizing testable items •Analyzing risk should be an ongoing effort started early in the project life cycle. As the product matures, the analysis of risk may change. PowerQuest Corporation 3

Feature Risk Analysis Assumptions •Team allowed flexibility to deliver quality product. •The definition of quality is well understood by the team. •Organization processes are defined, understood, and followed. •Project is scoped and minimally defined. •FRA should be updated through life of project. PowerQuest Corporation 4

Feature Risk Analysis Guidelines •An odd number of testers (I recommend three or five) will meet and review all known functionality of a product. •Each functional piece (testable item) is listed so that all can see and discuss it. •Once all testable items are listed, they are sorted into high, medium, and low priority groups. •Within each grouping, each testable item is sorted by most important to least important. – This is represented with a number. Example, within the high priority group, H1 is the first item, H2 is the second, H3 is the third, etc. until the last item H n is listed. PowerQuest Corporation 5

Feature Risk Analysis Guidelines •Once all functional items are listed, grouped, and prioritized the testers identify all possible operating environments (operating system platforms, file systems, languages, hardware, etc.) where the product will operate. • These environments are combined, where it makes sense, to derive the smallest set of environments with which the product must be tested. • When these steps have been completed, the team can then create a test matrix and test cases based on these prioritized lists and environments. PowerQuest Corporation 6

Feature Risk Analysis Guidelines •This test matrix and test cases should be made available for review by other members of the development team. PowerQuest Corporation 7

Feature Risk Analysis Example Windows Display Properties Settings dialog PowerQuest Corporation 8

Feature Risk Analysis • Dialog controls | System menu (Alt-Space) | Move • Dialog controls | System menu (Alt-Space) | Close Step 1: List Features • Dialog controls | Close ‘X’ button • Dialog controls | “What’s this” help button • Dialog controls | OK button • Dialog controls | Cancel button • Dialog controls | Apply button • Dialog controls | Apply button | Confirm dialog • Dialog controls | ESC supported? • Dialog controls | Tab order • Dialog controls | Ctrl-Tab • Dialog controls | Mouse-over help • Dialog controls | Right click menu context sensitivity • Dialog controls | Mnemonics • Dialog controls | Space bar selection • Dialog controls | Alt-Down arrow on drop down • Dialog controls | Localization? • Dialog controls | Timing? • Dialog controls | Visually correct – Spacing/layout • Monitor icons | Gray background w/ simulated LCD display • Monitor icons | Blended background w/ simulated monitor • Monitor icons | Single monitor • Monitor icons | Multi-monitor • Monitor icons | External monitors • Display drop-down | Correctly display all known monitors • Display drop-down | Each display is selectable • Screen resolution | Monitor type effects • Screen resolution | Video adapter effects • Screen resolution | Min resolution • Screen resolution | Max resolution • Screen resolution | Correct display of target resolution as slider moves • Color quality | Dropdown has selectable palettes • Color quality | Color bar adjusts as different palettes are selected • Checkbox: Use this device as the primary monitor | Enabled/disabled appropriately • Checkbox: Extend my Windows desktop onto this monitor checkbox | Enabled/disabled appropriately • Identify button | Correctly identifies each monitor in use • Troubleshoot button | Correctly open troubleshooting dialog • Advanced button | Correctly opens advanced dialog PowerQuest Corporation 9

Feature Risk Analysis Step 1: List Features (Combine) • Dialog | Parent controls System menu (Alt-Space), Close ‘X”, “What’s this?” help, OK, Cancel, Apply, ESC supported?) • Dialog | Client controls (Tab, Ctrl-Tab, Mouse-over help, Right click menu context sensitivity, Mnemonics, Alt-Down arrow on drop down, checkboxes) • Dialog | Presentation (Timing, Visually correct – Spacing/layout) • Monitors | Monitors (single, multiple, external) • Screen resolution • Color quality • Identify button • Troubleshoot button • Advanced button PowerQuest Corporation 10

Feature Risk Analysis Step 2: Prioritize Features •High •Medium •Advanced button •Troubleshoot button •Screen resolution •Color quality •Monitors •Dialog | Presentation PowerQuest Corporation 11

Feature Risk Analysis Step 2: Prioritize Features •Low •Dialog | Parent controls System menu (Alt-Space), Close ‘X”, “What’s this?” help, OK, Cancel, Apply, ESC supported?) •Identify button •Dialog | Client controls (Tab, Ctrl-Tab, Mouse-over help, Right click menu context sensitivity, Mnemonics, Alt-Down arrow on drop down, checkboxes) PowerQuest Corporation 12

Feature Risk Analysis Step 3: Prioritize Within Each Group • High • Medium H1 Dialog | Advanced button M1 Color quality H2 Dialog | Screen resolution M2 Dialog | Presentation H3 Dialog | Monitors M3 Troubleshoot button PowerQuest Corporation 13

Feature Risk Analysis Step 3: Prioritize Within Each Group •Low L1 Dialog | Parent controls L2 Dialog | Identify button L3 Dialog | Client controls Note: By the time you get to the lows, it doesn’t really matter how they are prioritized – they are all low priority. If you find some low priority items whose priority matters, perhaps they are not really low priority. PowerQuest Corporation 14

Feature Risk Analysis List Valid Environments •Languages • English, French, German, Japanese, Spanish •Operating Systems • Windows 98, 98SE, ME , 2000 Pro, XP PowerQuest Corporation 15

Feature Risk Analysis Test Matrix Risk Boundary PowerQuest Corporation 16

Feature Risk Analysis Summary Benefits : • FRA can be broken up by product components (i.e., install, main .EXE, utilities, networking, video display, etc.) • Allows test team to identify risk boundaries in product. • Easy way to show work completed vs. work remaining. • Helps evaluate can/can’t ship situations • Team members become very proficient at this very quickly • Significantly reduces stress PowerQuest Corporation 17

Feature Risk Analysis Overview Questions? PowerQuest Corporation 18