F1 Friday, May 19, 2006 10:00AM T HE L AST P RESENTATION ON T EST E - - PDF document

BIO PRESENTATION

International Conference On Software Testing Analysis and Review May 15-19, 2006 Orlando, Florida USA

F1

Friday, May 19, 2006 10:00AM

THE LAST PRESENTATION ON TEST ESTIMATION YOU WILL EVER NEED TO ATTEND

Geoff Horne Geoff Horne Testing

Geoff Horne

Geoff Horne is based in New Zealand and has founded and run two testing consultancy companies which grew to enjoy an international clientele. He has over 28 years experience in IT including software development, sales and marketing and IT and project management. In 1994, almost by accident, he found himself involved in testing a complex fault management system, which led to further testing and QA assignments covering a wide range of applications and tools. Geoff’s companies were subsequently founded to bring a full range of testing consultancy services to the IT industry. Latterly, Geoff has focused on a few select clients running complex test projects in a programme test management capacity. Geoff has written a variety of white papers on the subject of software testing and has been a regular speaker at the Star testing conferences. He is married with four children and in his spare time (which there is not a lot of) enjoys writing and recording

• riginal contemporary Christian music.

www.ghtest.com

The Last Presentation on Test Estimation You Will Ever Need to Attend

(I think!)

Geoff Horne - Principal

www.ghtest.com

Why estimate testing?

To endeavour to quantify:

• Timetable
• Resourcing
• Budget
• Environments

www.ghtest.com

Why is estimating so hard?

So many variables…

• Product - complexity, availability, stability
• Test environment - availability, stability
• Issues - volume, severities, time to fix, regression
• Personnel - suitability, experience
• Test stages - analysis, design, development, execution
• Evolving project – re/descoping, priority shifts

www.ghtest.com

And what’s the usual response?

Something like, and I quote...

• “We don’t have that luxury”
• “We don’t have time for best practice”
• “We’re building a Holden not a Rolls Royce”
• “I think you’re over-complicating the exercise”
• “We brought you in to save time”
• “Ha ha ha” (sic: maniacal laughter)

www.ghtest.com

Test estimation

Two types:

• “Guesstimation”
• “Testimation”

www.ghtest.com

Examples of “Guesstimation”

Based on previous exercises and prior knowledge:

• Percentages - of development efforts
• Comparative size - to previous or similar exercises
• Application metrics - no. of screens, functions etc.
• And the one we all love…..

you’ve got n people and n weeks!

www.ghtest.com

“Guesstimation:”

Good starting point however…

www.ghtest.com

“Guesstimation:”

Only a starting point!

• Educated “finger in the wind”
• Beware of getting locked into a “guesstimation”
• You don’t know what you don’t know
• Watch out for the monkey on your back!

www.ghtest.com

“Guesstimation” to “Testimation”

Applying some science

• Start with assumptions
• Add what you do know
• Allow for what you don’t
• Refine as “unknowns” become “knowns”
• Develop “testimation” models

www.ghtest.com

“Guesstimation” to “Testimation”

Key: Gaining buy-in to the process, from:

• Project managers
• Test team
• Other project team leaders
• Sponsors
• Developers
• Business

www.ghtest.com

“Testimation” models:

Simple model

• Based on averages
• Assumes some sort of previous metrics to work from
• Provides “lump sum” result
• Good for structured testing initiatives

www.ghtest.com

“Testimation” models:

Simple model

www.ghtest.com

“Testimation” models:

Simple model OK however…

• Usually not 8 productive work hours in a day
• Test effort cannot always be divided evenly
• Test team productivity not always linear
• No allowance for complexity or priority
• Tester productivity differences
• Time estimates are flat

www.ghtest.com

“Testimation” models:

Simple model OK however…

• Time models need to allow for:
• Development
• Requirements analysis
• Test case design
• Test script development
• Reviews and other overheads

www.ghtest.com

“Testimation” models:

Simple model OK however…

• Time models need to allow for:
• Execution
• Pretest – test data setup, prerequisites etc.
• Execution of script
• Retesting
• Wait time for issue repair
• Documentation and other overheads

www.ghtest.com

“Testimation” models:

More comprehensive model…

• Still based on averages
• Still requires some sort of metrics to start, however…
• Allows for: different levels of tester

multiple complexities multiple time models productivity diminishing returns variables work hours

www.ghtest.com

“Testimation” models:

More comprehensive model…

www.ghtest.com

Test Lifecycle:

Sou rce D ocu m e n t s Te st St ra t e g y Te st Pla n Re q u ire m e n t s Te st Re q u ire m e n t s

( re vie w )

Te st Ca se s

( re vie w )

Te st Scrip t s

( re vie w )

I n cid e n t Te st Log s

( re vie w )

Re p ort s AN ALYSE D ESI GN D EVELO P EX ECU TE SI GN O FF R EPAI R A N A L Y S E R E V I E W M E A U S U R E R E P O R T D E V E L O P E X E C U T E R E V I E W R E V I S E

www.ghtest.com

Test estimation; “Testimation”

An evolutionary process:

Strategy Plan Development Design Execution Stage View Method Activity 40,000’ 20,000’ 10,000’ 5,000’ sea level

Rough estimate based on % of development effort Better estimate based on expected no. of test cases and modelling Estimate Estimate Monitor and manage Monitor and manage

Viscosity

Better estimate still, based on test case design and schedules Estimate

www.ghtest.com

“Guesstimation” vs “Testimation”

Spot the difference:

• “Guesstimation” is a good starting point however…
• Not what you want to get locked in to
• “Testimation” provides quantifiable estimates
• Can be used to create buy-in
• Refining the “testimation” provides basis for test plan

www.ghtest.com

“Testimation”

Feedback loop:

• Capture test metrics as plan is executed
• Regularly measure against estimates
• Full analysis at test completion
• Use to measure test efficiency
• Document for future projects
• Learn!

www.ghtest.com

Summary

• Estimating test effort is an evolutionary process
• Modelling provides tangible estimates
• Gaining buy-in allows estimate to be refined
• Reviewing estimates vs actuals measures accuracy
• Feeding back lessons learned improves process

www.ghtest.com

www.ghtest.com