T15 September 22, 2005 1:30 PM D EFECT P REDICTION WITH R ELIABILITY - - PDF document

BIO PRESENTATION SUPPLEMENTAL

BETTER SOFTWARE CONFERENCE & EXPO 2005 September 19-22, 2005 Hyatt Regency San Francisco Airport San Francisco, California, USA

T15

September 22, 2005 1:30 PM

DEFECT PREDICTION WITH RELIABILITY GROWTH MODELING

Michael Allegra GXS

Michael Allegra

Michael Allegra has devoted most of his 14 year career in software testing to leading and managing teams at IBM’s Global Services division. He was responsible for delivering high quality software for Fortune 500 customers and managing a billing test group responsible for $250 million in annual revenue. Michael has led process improvement activities for his organization involving CMM and industry best practices. He recently left IBM and joined GXS where he manages a test group for the leading provider of messaging applications and services.

Defect Prediction with Reliability Growth Modeling Defect Prediction with Reliability Growth Modeling

Michael Allegra IBM/GXS Michael Allegra IBM/GXS

Agenda Agenda

• How will this technology help me?
• Overview of Reliability Growth Models
• Describe a model applicable to most software

development

• CASRE Tool setup (its free!)
• Entering the data and format with provided

templates

• Interpreting the results
• Summary

Why should I care? Why should I care?

• Software reliability models can tell us;

– Number of latent defects – ‘Good Enough’ testing – Effectiveness of current test cycle – Service level/Availability target validation – Help Desk staffing needs

Reliability Growth Modeling

Typical formulas

Reliability Growth Modeling

Typical formulas

Reliability Growth Modeling

Typical formulas

Reliability Growth Modeling

Typical formulas

Fahgetaboutit! We aren’t covering that!

Basic Concepts Basic Concepts

• Skip the Rocket Science stuff!
• Reliability Growth Modeling is a subset of

the Software Reliability Engineering methodology.

• SRE covers a wide range of technical topics

that cannot be covered in this brief tutorial.

• RGM can be useful without all the SRE

steps by following some basic principles

Model Types Model Types

• Defect prediction models Static or

Dynamic

– Static

• Defects per KLOC, Function Point, Class, etc
• Based on historical projects
• Example, test team historically finds 5 defects/FP and 1 defect

escapes per every 7 found in test

– 21FP in release = 105 defects found in test phase and 15 escapes

• r latent defects.
• Useful for ballpark resource planning.
• Not always accurate due to assumptions
• Proven best for model level predictions

Dynamic models Dynamic models

• Uses defect data from actual project as

it moves through development lifecycle

• Best suited for product or service level

reliability, not modules

• Two basic models

– Full-life cycle: Rayleigh model – Qualify phase – reliability growth models

Dynamic Models;Rayleigh Dynamic Models;Rayleigh

• A type of Weibull distribution
• Models entire development life cycle

Dynamic Models; Reliability Growth Dynamic Models; Reliability Growth

• Reliability growth is present when failure

intensity decreases as test time increases

• Used during Independent Test phase

– Function Test, System Test, etc – When testing most closely resembles end- user activity

• Two types of Reliability Growth Models;

– Time Between Failures (when is next failure) – Fault Count (how many failures exist)

Recommended Model Recommended Model

• Yamada S-shaped model
• Fault detection rate starts flat and builds reflecting tester

learning curve and test environment stability

Yamada S-shaped model

10 20 30 40 50 60 70 80 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 test interval defects

Model assumptions Model assumptions

• Software is tested in a similar manner as

customers

• Unknown number of defects at start of

formal testing

• Fixes are clean
• Discovered defects are quickly fixed and

not present in future test intervals

• Finite number of faults

Recommended Project Types Recommended Project Types

• At least 10 test intervals
• Expecting at least 50 defects
• One month or greater test time
• For iterative development:

– Must allocate new functions delivered with new defects – See John Musa’s book for details

Team must agree on defect definition! Team must agree on defect definition!

• The IBM definition of a defect is ‘any non-conformance to the

agreed requirements, acceptance criteria, specifications, plans, standards or other input requirements.’

• Defects that are ‘opened’ and subsequently ‘cancelled’ by a

tester should not be included.

• Any behavior of the product that would result in a customer

reporting a defect after the product is released should be considered a defect.

• Consideration should also be given to the following types of

defects:

– Documentation – Build – Packaging – Installation – Performance – Usability

CASRE CASRE

• Computer Aided Software Reliability

Estimation

• Developed by Jet Propulsion Laboratory
• Yes, its Free!
• Primarily developed for scholars,

researchers, etc.

• Demo will cover basic scenario step by

step

CASRE CASRE

• Requires MS Windows
• Download CASRE 3.0 from:

– www.openchannelfoundation.org/projects/CASRE_3.0 – You will need to register and fax back a license agreement.

• Installation instructions are included on

web site.

• Verify the software starts without error.

– Note: make sure the ‘casrev3.ini’ file is in your windows directory (c:\winnt)

Process flow, high level Process flow, high level

2) Import to CASRE 4) Import results then Format 5) Interpret Results 3) Select/Run Model 1) Update data file

EXCEL CASRE Prediction per test interval

Create the data file Create the data file

• Start this after you have about 5 test intervals.
• Before you begin, gather your defect list that

contains the following information

– Date defect opened (sort list by date) – Number of defects per severity per date – Interval size per increment (hours/day, days/week, etc)

• Format data file as shown in template
• Save as type ‘internal#.dat’ (Oct18.dat)
• Isolate yourself from interruption while

creating!

Create defect data file Create defect data file

• Test interval time
• Test interval

# of Defects Test Time Severity

CASRE CASRE

• Start CASRE, open .dat file
• Verify data imported correctly
• Setup and Execute Model

– Select Data Range – Select future Prediction Intervals – Choose model, ‘Yamada s-Shaped’

• Select model results
• Check ‘Goodness to Fit’

CASRE CASRE

DEMO IN CASRE

Import and Format Results Import and Format Results

Model Results; Predicted vs Actual Defects Model Results; Predicted vs Actual Defects

Total Defects Predicted vs. Actual

10 20 30 40 50 60 9/26 10/3 10/10 10/17 10/24 10/31 11/7 11/14 11/21 11/28 12/5 Test Day Defects Predicted Cumulative failures Actual Cumulative Failures

Predicted defects uncovered w ith 5 additional test days 56 53

If there are significant differences past the first 1/3 of the test cycle then the testing effectiveness must be reevaluated for the following:

• If the actuals continue to be well below the prediction, the test coverage is probably

insufficient or there are not enough people testing given the current rate of execution.

• If the actuals are above the prediction then you may need to re-check your parameters in your

data file and in CASRE.

Model Results; Predicted failure rate Model Results; Predicted failure rate

Push for additional test time if Predicted Failure rate:

• still increasing
• has not started to decline significantly
• remains at a level that is unacceptable to the team and business organization

Consider reducing test time if Predicted Failure rate:

• Is at a level that is acceptable to the business, i.e. ‘we will not find enough defects to justify the cost of the

remaining test intervals’

• has leveled-off at a low rate and the test coverage has hit all major functions, therefore rate is not likely to

increase.

Predicted Failure Rate by Test lnterval

0.00 0.10 0.20 0.30 0.40 0.50 9/26 10/3 10/10 10/17 10/24 10/31 11/7 11/14 11/21 11/28 12/5 Estimation at each Test Iteration Failures per test hour

Model Results; Total Defects Predicted Model Results; Total Defects Predicted

After a flattening of the estimated total defects, if the rate begins to rise again it can mean the scope of the testing is not sufficient and test coverage should be increased.

Estimated Total Defects in Product as testing progressed

(95% Confidence Interval & Most Likely Estimate) 20 40 60 80 100 120 1 / 2 1 / 2 7 1 1 / 3 1 1 / 1 1 1 / 1 7 1 1 / 2 4 1 2 / 1 Estimation at each Test Iteration Defect ects

Low 53 Most Likely 61 High 79

Model Results; Defects Predicted vs. Actual Model Results; Defects Predicted vs. Actual

• Exited test with 53 uncovered defects
• Model predicted 61 defects most likely
• 62 defects total after 2 months release.
• No further defects reported to date. Model proved to be very accurate!

Actual Latent Defects

45 50 55 60 65 70 75 80 12/16/03 12/23/03 12/30/03 1/6/04 1/13/04 1/20/04 1/27/04 Defects Post GA defects

HIGH LOW

Most Likely

Conclusion Conclusion

• Reliability growth modeling is another

useful tool for test practioners

• Demonstrated:

– In-progress testing scope adjustment – Failure rate trend analysis – Latent defects prediction

• Presented a ‘quick start’ procedure that

hopefully encourages further exploration.

References References

• ‘Software Reliability Engineering’, John

Musa; http://members.aol.com/johndmusa/

• ‘Metrics and Models in Software

Quality Engineering’, Stephen Kan

• CASRE:www.openchannelfoundation.org/pro

jects/CASRE_3.0

• See included whitepaper and excel template

Predicting Software Defects with Dynamic Reliability Modeling

An introduction for the common test practioner

Michael Allegra June 1, 2005 Abstract How does a test practioner know when they have tested enough to recommend proceeding with a software release? How do they know if their testing is uncovering defects at an acceptable rate? Dynamic reliability modeling is a powerful technique that can help answer these questions that are critical to business success. This author’s view of the field of software defect prediction is that it is mostly practiced by statisticians, mathematicians or the top technical folks at places like NASA. Indeed, if you open a book on the subject you’ll come across some advanced mathematical symbols and statistical formulas that you likely haven’t seen since college(and maybe not even then!). However, most test practioners are very competent in their profession and can implement modern test techniques without having a PhD. The target audience of this paper is the professional software testers that are involved with real world software development projects and are interested in practical techniques to increase their

• rganization’s quality capabilities. This paper is an introduction to the topic and the

reader is encouraged to learn more about reliability modeling so they can make customizations specific to their organization. The goals of this paper are:

• I. Provide basic understanding of software reliability modeling
• II. Describe a prediction model that is applicable to most software projects
• III. Instruct how to download and setup CASRE, a tool for defect prediction modeling
• IV. Walkthrough CASRE setup and execution
• V. Show how to create easy to read charts from CASRE output
• VI. Interpret charts and describe decisions points

What is Software Reliability modeling?

Reliability models are mathematical and statistical functions that have been developed

• ver the last couple decades by many mature software development organizations to

assess the number of total defects in an application and to quantify the application’s reliability over time. This paper will not break down the mathematics of these functions since they are impractical to implement without a dedicated modeling tool. You may ask ‘What is the practical application of these predictions?’

• 1. Foremost, is the predicted reliability of an application is useful for establishing

realistic service availability targets and acceptable levels of quality to customer delivered products. For example, if a service has a target of 99.99% availability for the first 3 months of its release but, the defect prediction shows dozens of high severity defects still present after the last test phase then a case can be made for lowering that availability target.