WG1 5/15/2002 9:15:00 AM I DENTIFYING T ESTING P RIORITIES T HROUGH - - PDF document

wg1
SMART_READER_LITE
LIVE PREVIEW

WG1 5/15/2002 9:15:00 AM I DENTIFYING T ESTING P RIORITIES T HROUGH - - PDF document

P R E S E N T A T I O N BIO PRESENTATION WG1 5/15/2002 9:15:00 AM I DENTIFYING T ESTING P RIORITIES T HROUGH R ISK A NALYSIS Rick Craig Softw are Quality Engineering International Conference On Software Testing Analysis & Review May 13-


slide-1
SLIDE 1

P R E S E N T A T I O N BIO PRESENTATION International Conference On Software Testing Analysis & Review May 13- May 17, 2002 Orlando, FL USA

WG1

5/15/2002 9:15:00 AM

IDENTIFYING TESTING PRIORITIES THROUGH RISK ANALYSIS

Rick Craig Softw are Quality Engineering

slide-2
SLIDE 2

STQE Publishing STQE Publishing

Rick D Craig

Rick D. Craig is an experienced test manager, consultant, and lecturer who's helped hundreds of companies throughout Europe, Asia, Australia, and the Americas improve their testing. He's been a featured speaker at testing conferences since 1983, and is the former American editor of Software Quality Management Magazine. Rick is also a colonel in the United States Marine Corps Reserve and a technical editor for the StickyMinds.com Web site.

slide-3
SLIDE 3

1

Identifying Testing Priorities Through Risk Analysis

By Rick D. Craig Software Quality Engineering

slide-4
SLIDE 4

2

RISK – “the chance of injury, damage or loss; A dangerous chance, a hazard

RISK Likelihood Impact

slide-5
SLIDE 5

3

  • Purpose of Software Risk Analysis

– To determine what to test, the testing priority and the depth of testing

  • Purpose of Planning Risk Analysis

– To identify potential (test) schedule disruptions and to formulate and agree upon contingencies

slide-6
SLIDE 6

4

Type of Risk Analysis Key Activities Results Risk Analysis

Planning Risks And Contingencies Identify Potential Disruptions to Schedule (Planning Risks)

Contingencies Testing Priorities

Software Risk Analysis (Business and Technical) Identify Impact of Failure Identify Likelihood Of failure

slide-7
SLIDE 7

5

Software Risk Analysis Process Overview

  • 5. Assign

Numerical Values

  • 1. Form a

Brainstorming Team

  • 2. Compile a

List of Features

  • 3. Determine

The Likelihood

  • 4. Determine

The Impact

  • 8. Prioritize

The Features

  • 10. Consider

Mitigation

  • 7. Review/

Modify the Values

  • 6. Compute

The Risk Priority

  • 9. Determine

The “Cut line”

slide-8
SLIDE 8

6

Step 1

Include end-users, developers, testers, marketers and business analysts.

Form a Brainstorming Team Step 2

Compile a system-wide list of features and attributes.

Compile a List of Features

Process Overview ….

slide-9
SLIDE 9

7

List Features of an ATM

  • Withdraw cash
  • Deposit cash
  • Check account balance
  • Transfer funds
  • Purchase stamps
  • Make a loan payment
  • Usability
  • Performance
  • Security

Features Attributes

slide-10
SLIDE 10

8

Step 3

Assign a relative value for Likelihood of

  • Failure. What is the likelihood that this

feature or attribute will fail to operate correctly?

Determine The Likelihood

Process Overview ….

slide-11
SLIDE 11

9

Medium

Security

Low

Performance

Medium

Usability

Low

Make a loan payment

High

Purchase stamps

Medium

Transfer funds

Low

Check account Balance

Medium

Deposit cash

High

Withdraw cash

Attributes Features

Likelihood ATM Software Likelihood of Failure for ATM Features/Attributes

Based on our current knowledge of the system what is the likelihood that this feature or attribute will fail or fail to

  • perate correctly?
slide-12
SLIDE 12

10

Step 4

Assign a relative value for the impact. What would be the impact on the user if this feature or attribute failed?

Determine The Impact

Process Overview ….

slide-13
SLIDE 13

11

High Medium

Security

Medium Low

Performance

High Medium

Usability

Medium Low

Make a loan payment

Low High

Purchase stamps

Medium Medium

Transfer funds

Medium Low

Check account Balance

High Medium

Deposit cash

High High

Withdraw cash

Attributes Features

Impact Likelihood ATM Software Impact of Failure for ATM Features/Attributes

What would be the impact on the user if this feature or attribute failed to operate correctly?

slide-14
SLIDE 14

12

Step 5

Assign numerical values corresponding to the relative values assigned in steps 3 and 4 above.

Assign Numerical Values

Process Overview ….

Step 6

Add together the values assigned to likelihood of failure and impact of failure.

Compute The Risk Priority

slide-15
SLIDE 15

13

Risk Priority Likelihood Of Failure

High (3) Low (1) Medium (2) Medium (2) Low (1)

Impact Of Failure

High (3)

Likelihood + Impact = Risk Priority

slide-16
SLIDE 16

14

Priority

5 High Medium

Security

3 Medium Low

Performance

5 High Medium

Usability

3 Medium Low

Make a loan payment

4 Low High

Purchase stamps

4 Medium Medium

Transfer funds

3 Medium Low

Check account Balance

5 High Medium

Deposit cash

6 High High

Withdraw cash

Attributes Features

Impact Likelihood ATM Software Summed Priorities for ATM Features/Attributes

slide-17
SLIDE 17

15

Step 7

Review/modify priorities based on complexity, Pareto analysis, new or modified features, development methodology, environmental accessibility, usability and team history, etc.

Review/ Modify the Values

Process Overview ….

slide-18
SLIDE 18

16

Modify Likelihood using “Failure Indicators”

  • Team history
  • Complexity
  • Usability
  • New or modified

features

  • New techniques or

technology

  • Pareto analysis
  • Environmental

accessibility

slide-19
SLIDE 19

17

Step 8

Reorganize the list of features and attributes in order of risk priority.

Prioritize The Features

Process Overview ….

slide-20
SLIDE 20

18

3 Medium Low

Make a loan payment

3 Medium Low

Check account Balance

3 Medium Low

Performance

Priority

4 Low High

Purchase stamps

4 Medium Medium

Transfer funds

5 High Medium

Security

5 High Medium

Usability

5 High Medium

Deposit cash

6 High High

Withdraw cash

Attributes Features

Impact Likelihood ATM Software Sorted Priorities for ATM Features/Attributes

slide-21
SLIDE 21

19

Step 9

Establish a “cut line” which separates features “to be tested” and features “not to be tested”.

Determine The “Cut line”

Process Overview ….

slide-22
SLIDE 22

20

Priority

3 Medium Low

Performance

3 Medium Low

Check account Balance

Not to be tested (or tested less) 3 Medium Low

Make a loan payment

4 Low High

Purchase stamps

4 Medium Medium

Transfer funds

To Be Tested 5 High Medium

Security

5 High Medium

Usability

5 High Medium

Deposit cash

6 High High

Withdraw cash

Attributes Features

Impact Likelihood ATM Software “Cut-Line” for ATM Features/Attributes

slide-23
SLIDE 23

21

Step 10

Decide which risks (if any) could be reduced by adding resources, changing the development methodology, etc.

Consider Mitigation

Process Overview ….

slide-24
SLIDE 24

22

3 Medium Low

Make a loan payment

3 Medium Low

Check account Balance

Mitigation Priority

3 Medium Low

Performance

4 Low High

Purchase stamps

4 Medium Medium

Transfer funds

5 High Medium

Security

Early User Feedback 5 High Medium

Usability

Early Prototype 5 High Medium

Deposit cash

Code Inspection 6 High High

Withdraw cash

Attributes Features

Impact Likelihood ATM Software Mitigated List of Priorities for ATM Features/Attributes

slide-25
SLIDE 25

23

Planning Risks and Contingencies

slide-26
SLIDE 26

24

Planning Risks

  • Planning risks are unscheduled events or

late activities that may jeopardize the testing schedule.

  • Some common planning risks include:

– Unrealistic delivery dates – Staff availability – Budget shortfall – Environmental options – Tool inventory – Acquisition schedule – Participant buy-in – Training needs – Scope of testing – Lack of requirements – Risk assumptions – Usage assumptions – Resources – Feature creep – Poor quality s/w

slide-27
SLIDE 27

25

Contingencies

  • Reduce the scope
  • Delay implementation
  • Add resources
  • Reduce quality processes
slide-28
SLIDE 28

26

Sample Planning Risk and Contingencies

  • Sample Planning Risk

– The user introduces a major requirements change late in the software life-cycle.

  • Sample Contingency #1

– Ask the user group to contribute more users to the testing effort (i.e., add more resources).

  • Sample Contingency #2

– Decide not to implement a low priority feature until a later release (e.g., reduce the scope).

  • Sample Contingency #3

– Delay Implementation

  • Sample Contingency #4

– Decide not to test (or at least to test less) some of the low risk features identified in the course of the software risk analysis (i.e., reduce quality processes).

slide-29
SLIDE 29

27

Priority

3 Medium Low

Performance

3 Medium Low

Check account Balance

Not to be tested (or tested less) 3 Medium Low

Make a loan payment

4 Low High

Purchase stamps

4 Medium Medium

Transfer funds

To Be Tested 5 High Medium

Security

5 High Medium

Usability

5 High Medium

Deposit cash

6 High High

Withdraw cash

Attributes Features

Impact Likelihood ATM Software “Cut-Line” Adjustments

Contingency #4 - Do not test some low risk features, i.e. move the cut line up.

slide-30
SLIDE 30

28

Bottom Line:

  • Software risk analysis:

– You can’t test everything – Software risk analysis helps determine where to focus the testing

  • Planning Risk

– The best laid plans of mice and men… – Identifying planning risks early allows the entire engineering staff the

  • pportunity to develop and agree on

contingencies for the inevitable changes that occur in every project.

slide-31
SLIDE 31

29

“If you do not actively attack risks, they will actively attack you

Tom Gilb