1 Bug Triage Regression Testing Decide which bugs to fix Good: - PDF document

What is a bug? � Formally, a “software defect” A Bug’s life � SUT fails to perform to spec � SUT causes something else to fail � SUT functions, but does not satisfy Finding and Managing Bugs usability criteria CSE 403 � If the SUT works to spec and someone wants it changed, that’s a feature Lecture 23 request What are the contents of a What makes a good bug bug report? report? � Repro steps – how did you cause the failure? � Clear, descriptive title � Observed result – what did it do? � Accurate description of the problem � Expected result – what should it have done? � Environment description � Any collateral information: return � Steps to reproduce the problem values/output, debugger, etc. � Ideally, a minimal set of steps � Environment � Test platforms must be reproducible � “It doesn’t do it on my machine” Ranking bugs A Bug’s Life � Severity � Priority � Sev 1: crash, hang, � Pri 1: Fix data loss immediately � Sev 2: blocks � Pri 2: Fix before next feature, no release outside team workaround � Pri 3: Fix before ship � Sev 3: blocks feature, workaround � Pri 4: Fix if nothing available better to do ☺ � Sev 4: trivial (e.g. cosmetic) 1

Bug Triage Regression Testing � Decide which bugs to fix � Good: rerun the test that failed � Reasons NOT to fix bugs � Or write a test for what you missed � Ambiguous status of the bug � Better: rerun related tests (e.g. � Cost of fix vs. benefit to the product component level) � Risks that fixing it will cause other � Best: rerun all product tests problems � Automation can make this feasible! � Incorrect fix � Other portions of the code depend on incorrect behavior Tracking Bugs When can I ship? � Raw bug count � Test coverage sufficient � Slope is useful predictor � Bug slope, find vs. fix lead to � Ratio by ranking convergence � How bad are the bugs we’re finding? � Severity mix is primarily low-sev � Find rate vs. fix rate � One step forward, two back? � Priority mix is primarily low-pri � Management choices � Load balancing � Review of development quality Milestones BUGs vs. Time � Feature complete � All features are present � Code complete � Coding is done, except for the bugs � Code Freeze � No more coding � Release Candidate � I think it’s ready to ship � It’s out the door Feature Code Code Release Complete Complete Freeze Candidate 2

Basic entomology Errors in testing � Some facts about bugs (from Code Complete) � Test cases can have errors too! � The scope of most errors is limited � Spending hours looking for bugs in code � 85% could be fixed by modifying a single routine � Errors in assignment statements are common that turn out to be in the test case � 41% of errors in assignment statements � Most errors easy to fix � Test cases more prone to errors than � 85% a few hours the code � 14% a few hours to a few days � 1% multiple days � Especially when the developer writes the � Old Microsoft Data (1992) test case � 10 to 20 defects per 1000 lines of code in test � 0.5 defects per 1000 lines of released code When do you fix bugs? � Write the code first, have someone else fix the bugs � Fix blocking bugs, but save the minor stuff until code complete � Aggressively fix bugs as they are found 3