WG1 5/15/2002 9:15:00 AM I DENTIFYING T ESTING P RIORITIES T HROUGH - - PDF document
P R E S E N T A T I O N BIO PRESENTATION WG1 5/15/2002 9:15:00 AM I DENTIFYING T ESTING P RIORITIES T HROUGH R ISK A NALYSIS Rick Craig Softw are Quality Engineering International Conference On Software Testing Analysis & Review May 13-
P R E S E N T A T I O N BIO PRESENTATION International Conference On Software Testing Analysis & Review May 13- May 17, 2002 Orlando, FL USA
5/15/2002 9:15:00 AM
IDENTIFYING TESTING PRIORITIES THROUGH RISK ANALYSIS
Rick Craig Softw are Quality Engineering
STQE Publishing STQE Publishing
Rick D Craig
Rick D. Craig is an experienced test manager, consultant, and lecturer who's helped hundreds of companies throughout Europe, Asia, Australia, and the Americas improve their testing. He's been a featured speaker at testing conferences since 1983, and is the former American editor of Software Quality Management Magazine. Rick is also a colonel in the United States Marine Corps Reserve and a technical editor for the StickyMinds.com Web site.
1
2
RISK – “the chance of injury, damage or loss; A dangerous chance, a hazard
3
– To determine what to test, the testing priority and the depth of testing
– To identify potential (test) schedule disruptions and to formulate and agree upon contingencies
4
Type of Risk Analysis Key Activities Results Risk Analysis
Planning Risks And Contingencies Identify Potential Disruptions to Schedule (Planning Risks)
Contingencies Testing Priorities
Software Risk Analysis (Business and Technical) Identify Impact of Failure Identify Likelihood Of failure
5
Numerical Values
Brainstorming Team
List of Features
The Likelihood
The Impact
The Features
Mitigation
Modify the Values
The Risk Priority
The “Cut line”
6
Step 1
Include end-users, developers, testers, marketers and business analysts.
Form a Brainstorming Team Step 2
Compile a system-wide list of features and attributes.
Compile a List of Features
7
Features Attributes
8
Step 3
Assign a relative value for Likelihood of
feature or attribute will fail to operate correctly?
Determine The Likelihood
9
Medium
Security
Low
Performance
Medium
Usability
Low
Make a loan payment
High
Purchase stamps
Medium
Transfer funds
Low
Check account Balance
Medium
Deposit cash
High
Withdraw cash
Attributes Features
Likelihood ATM Software Likelihood of Failure for ATM Features/Attributes
Based on our current knowledge of the system what is the likelihood that this feature or attribute will fail or fail to
10
Step 4
Assign a relative value for the impact. What would be the impact on the user if this feature or attribute failed?
Determine The Impact
11
High Medium
Security
Medium Low
Performance
High Medium
Usability
Medium Low
Make a loan payment
Low High
Purchase stamps
Medium Medium
Transfer funds
Medium Low
Check account Balance
High Medium
Deposit cash
High High
Withdraw cash
Attributes Features
Impact Likelihood ATM Software Impact of Failure for ATM Features/Attributes
What would be the impact on the user if this feature or attribute failed to operate correctly?
12
Step 5
Assign numerical values corresponding to the relative values assigned in steps 3 and 4 above.
Assign Numerical Values
Step 6
Add together the values assigned to likelihood of failure and impact of failure.
Compute The Risk Priority
13
Risk Priority Likelihood Of Failure
High (3) Low (1) Medium (2) Medium (2) Low (1)
Impact Of Failure
High (3)
Likelihood + Impact = Risk Priority
14
Priority
5 High Medium
Security
3 Medium Low
Performance
5 High Medium
Usability
3 Medium Low
Make a loan payment
4 Low High
Purchase stamps
4 Medium Medium
Transfer funds
3 Medium Low
Check account Balance
5 High Medium
Deposit cash
6 High High
Withdraw cash
Attributes Features
Impact Likelihood ATM Software Summed Priorities for ATM Features/Attributes
15
Step 7
Review/modify priorities based on complexity, Pareto analysis, new or modified features, development methodology, environmental accessibility, usability and team history, etc.
Review/ Modify the Values
16
features
technology
accessibility
17
Step 8
Reorganize the list of features and attributes in order of risk priority.
Prioritize The Features
18
3 Medium Low
Make a loan payment
3 Medium Low
Check account Balance
3 Medium Low
Performance
Priority
4 Low High
Purchase stamps
4 Medium Medium
Transfer funds
5 High Medium
Security
5 High Medium
Usability
5 High Medium
Deposit cash
6 High High
Withdraw cash
Attributes Features
Impact Likelihood ATM Software Sorted Priorities for ATM Features/Attributes
19
Step 9
Establish a “cut line” which separates features “to be tested” and features “not to be tested”.
Determine The “Cut line”
20
Priority
3 Medium Low
Performance
3 Medium Low
Check account Balance
Not to be tested (or tested less) 3 Medium Low
Make a loan payment
4 Low High
Purchase stamps
4 Medium Medium
Transfer funds
To Be Tested 5 High Medium
Security
5 High Medium
Usability
5 High Medium
Deposit cash
6 High High
Withdraw cash
Attributes Features
Impact Likelihood ATM Software “Cut-Line” for ATM Features/Attributes
21
Step 10
Decide which risks (if any) could be reduced by adding resources, changing the development methodology, etc.
Consider Mitigation
22
3 Medium Low
Make a loan payment
3 Medium Low
Check account Balance
Mitigation Priority
3 Medium Low
Performance
4 Low High
Purchase stamps
4 Medium Medium
Transfer funds
5 High Medium
Security
Early User Feedback 5 High Medium
Usability
Early Prototype 5 High Medium
Deposit cash
Code Inspection 6 High High
Withdraw cash
Attributes Features
Impact Likelihood ATM Software Mitigated List of Priorities for ATM Features/Attributes
23
24
late activities that may jeopardize the testing schedule.
– Unrealistic delivery dates – Staff availability – Budget shortfall – Environmental options – Tool inventory – Acquisition schedule – Participant buy-in – Training needs – Scope of testing – Lack of requirements – Risk assumptions – Usage assumptions – Resources – Feature creep – Poor quality s/w
25
26
– The user introduces a major requirements change late in the software life-cycle.
– Ask the user group to contribute more users to the testing effort (i.e., add more resources).
– Decide not to implement a low priority feature until a later release (e.g., reduce the scope).
– Delay Implementation
– Decide not to test (or at least to test less) some of the low risk features identified in the course of the software risk analysis (i.e., reduce quality processes).
27
Priority
3 Medium Low
Performance
3 Medium Low
Check account Balance
Not to be tested (or tested less) 3 Medium Low
Make a loan payment
4 Low High
Purchase stamps
4 Medium Medium
Transfer funds
To Be Tested 5 High Medium
Security
5 High Medium
Usability
5 High Medium
Deposit cash
6 High High
Withdraw cash
Attributes Features
Impact Likelihood ATM Software “Cut-Line” Adjustments
Contingency #4 - Do not test some low risk features, i.e. move the cut line up.
28
– You can’t test everything – Software risk analysis helps determine where to focus the testing
– The best laid plans of mice and men… – Identifying planning risks early allows the entire engineering staff the
contingencies for the inevitable changes that occur in every project.
29
“If you do not actively attack risks, they will actively attack you
Tom Gilb