Inspection Versus Testing (1) � AT&T classifies inspection as a form of testing � In CISC 323, we use the term "testing" to mean dynamic testing: executing software to find faults � inspection does not replace dynamic testing � software process should include both inspection and testing CISC 323, winter 2003, software quality 29
Inspection Versus Testing (2) � difficult to inspect for qualities such as performance, timing, load � inspection can be applied earlier than testing � inspection can be applied to any software product, not just code CISC 323, winter 2003, software quality 30
Defects vs. Faults � Inspection finds defects: problems in software � Testing finds faults: incorrect behavior � When you find a fault by testing: • must debug to trace the fault to a defect in the code � Often more cost-effective to find defects via Inspection CISC 323, winter 2003, software quality 31
Analysis Tools � inspection is time-consuming (and therefore expensive) so inspect for things tools can’t find � static analysis tools can find some types of defects • compilers find undeclared variables, syntax errors • other static analysis tools may check for: • unreachable code • unused variables • memory leaks • "unsafe" programming practices • misspelled words, some grammatical errors � not cost-effective to inspect for these if tools exist CISC 323, winter 2003, software quality 32
Inspection Metrics � to be classified as “inspection”, metrics must be kept � record: • defects found in product under inspection • where defect found in product • time taken to perform inspection • size of product inspected � may also record • category of defect (e.g.,major/minor) • what prompted detecting the defect • suggested improvements • failure counts collected during testing and in the field CISC 323, winter 2003, software quality 33
Use of Metrics � Data collected can tell you: • total defects per line of code or per time unit • percentage of defects found at different stages • e.g. 60% of defects found by Inspection, 40% by testing • percentage of time spent on different activities: • design, coding, documentation, inspection, testing, etc. � Data can be used to: • justify time spent inspecting & testing • suggest improvements in development process • evaluate changes in process CISC 323, winter 2003, software quality 34
Inspection Process from Software Inspection , Tom Gilb & Dorothy Graham CISC 323, winter 2003, software quality 35
People Involved in Inspection � Inspection is a team process: ideally 2-6 people � Inspection Leader: • organizes & manages Inspection process • moderates meetings • should have special training (multi-day course, certification) � checkers (inspectors) • spend time reading/analyzing the product • may include author(s), leader • short training: 1-day course or "on the job" • technical knowledge relating to product CISC 323, winter 2003, software quality 36
Entry � entry criteria set up for products to be inspected � a lot of controversy about when to inspect code • before or after testing • before or after compile � if entry criteria not met, product not inspected • e.g., too many typing errors • e.g., source code does not compile cleanly � inspections are expensive • don’t waste inspectors time � author offers up product for inspection � leader decides if it's ready -- quick examination CISC 323, winter 2003, software quality 37
Kickoff � initial meeting for all participants � organized by leader � familiarize everyone involved with task at hand � define everyone’s roles � hand out materials needed � deal with general questions � set goals � discuss overall plan, timetable � long documents divided into "chunks": separate inspection & logging for each chunk CISC 323, winter 2003, software quality 38
Individual Checking (1) � each checker works alone using documents handed out in kick-off meeting � aims to find maximum number of unique major potential defects � mainly by looking for discrepancies between source documents and products being inspected, i.e.: • requirements to design • design to code • requirements to test plan CISC 323, winter 2003, software quality 39
Individual Checking (2) � usually uses a checklist � kickoff meeting suggests amount of time to spend: • typical rate is 1 hour per document page � checker records actual time taken � keeps notes of issues found • description, location, possibly impact � "issue" = matter requiring attention • possible defect • "question of intent" • suggestion CISC 323, winter 2003, software quality 40
Logging Meeting (1) � 3 purposes: • log issues identified by each checker • work as group to find more issues • suggestions for process improvement (inspection process and overall software process) � controlled by moderator (usually the leader) • must be very skilled in handling people and issues • maintain time discipline and issue discipline � no evaluation of issues • just log issues, like brainstorming CISC 323, winter 2003, software quality 41
Logging Meeting (2) � must not last more than 2 hours � not for explaining, defending, suggesting fixes, or training � identify issues, then log them � issues can be questions to product author and suggestions for improvements • not discussed in meeting � NOT an attack or evaluation of product author � management personnel should NOT be in attendance CISC 323, winter 2003, software quality 42
Edit � log of issues given to author to resolve � author can • classify or reclassify issues (which are really defects) • voluntarily improve product as per issues raised • request rules or checklist be changed CISC 323, winter 2003, software quality 43
Follow-up � inspection leader makes sure satisfactory closure has taken place on all issues • actions to correct all defects (change or change request) • suggestions for process changes reported � if major changes, product may need to be inspected again CISC 323, winter 2003, software quality 44
Exit � need exit criteria � need closure on all issues in writing � record metrics for inspection process (for process improvement) � product now considered “safe” to pass on to next phase in software development CISC 323, winter 2003, software quality 45
The V-model Requirements System Test Inspection Architecture Integration Test Inspection Unit Test Design Inspection Inspection Code CISC 323, winter 2003, software quality 46
Costs of Inspection � Inspection takes time � Typically 10-15% of development budget � Also start-up costs: • establishing procedures • training leaders, inspectors � Is it worth it? CISC 323, winter 2003, software quality 47
Benefits of Inspection (1) � Defects found early save time and money � many defects found before testing • so testing is much quicker � total development time is reduced • even when counting time for inspection � in the long run, inspections increase productivity CISC 323, winter 2003, software quality 48
Benefits of Inspection (2) � Defects found early result in better product � inspection results in • cleaner design • better documentation • better code • fewer defects • defects that remain will be easier to fix when found CISC 323, winter 2003, software quality 49
Benefits of Inspection (3) � feedback for mangement about software process: • inspector's comments plus metrics � defects found earlier means fewer deadline surprises � inspection trains inspectors: • learn from good code and documentation of others • learn from mistakes of others CISC 323, winter 2003, software quality 50
Success Stories (1) � IBM Federal Systems Division • kept metrics for similar projects before and after introducing Inspection into development process • delivered lines of code per work-month: • 144 without Inspection • 300 with Inspection CISC 323, winter 2003, software quality 51
Success Stories (2) � ICL (in U.K.): • 57.7% of defects found by Inspection • cost of finding defect by Inspection: 1.58 work hours • cost of finding without Inspection: 8.47 work hours • only 6% of development time devoted to Inspection CISC 323, winter 2003, software quality 52
Success Stories (3) � Large IBM project: • 500,000 lines of code (networked operating system) • based on past experience without inspection, expected to find about 800 bugs during tests at trial site • used inspection at 11 stages of devopment • found 8 bugs at trial site! CISC 323, winter 2003, software quality 53
Software Inspection Research (1) � Research into inspection • inspection process • techniques for individual inspectors CISC 323, winter 2003, software quality 54
Software Inspection Research (2) � research at Lucent Technologies (University of Maryland) • experiment where they varied • number of inspectors • number of teams inspecting the same code • teams working in parallel and teams working in tandem (where bugs were fixed before 2nd inspection took place) • one of the conclusions • most significant improvements in inspection comes from improving the way the inspectors do their individual work CISC 323, winter 2003, software quality 55
Software Inspection Research (3) � new techniques to help guide inspectors in their work • scenario-based reading • eg. put yourself in the role of a tester • have a series of questions and scenarios to work with as you inspect CISC 323, winter 2003, software quality 56
Checklists � common guidance used for inspections � good checklists are hard to make up • rely on past experience • can get too long and unwieldy • need to be updated as new situations arise • inspectors have a tendency to follow the checklist and do no more • can be misinterpreted • may not cover everything necessary to the right level of detail CISC 323, winter 2003, software quality 57
Approach for Checklists � (general approach for inspections as well) � determine what qualities are important for the software system � determine what software design/code characteristics contribute to those qualities � determine what you want the inspectors to look for • also determines practices you want the software developers to follow � checklist used for Assignment 4 is based on McCall's Quality Model CISC 323, winter 2003, software quality 58
McCall's Quality Model CISC 323, winter 2003, software quality 59
Testing � Now we move to the subject of testing � Recall that software process should include both testing and inspection � Inspecting documents & code before testing results in fewer defects for testing to find – speeds up testing � A bit of informal testing before code inspection might save time – get rid of obvious bugs quickly CISC 323, winter 2003, software quality 60
Testing vs. Debugging � Textbook says: • "Precisely speaking, the elimination of the syntactical bug is the process of debugging, whereas the detection and elimination of the logical bug is the process of testing" � We disagree. � Testing = finding faults (situations in which the program does not perform correctly) � Debugging = fixing faults (tracing a fault to defects in the software and fixing them – hopefully without intruducing new ones!) � In common use, sometimes "debugging" includes informal testing by programmer CISC 323, winter 2003, software quality 61
Comments About Debugging � We will not be discussing debugging in detail in this course � However, debugging is made easier by many of the techniques we have discussed to improve readability and modularity � Also made easier with good design documentation CISC 323, winter 2003, software quality 62
Testing is Hard! � hits budget and schedule limits � may include heavy debugging • may even include product (re-)development � bugs slip through to customers • user executes untested code • order in which statements are executed in actual use differ from that in testing • user applies combination of untested input values • user’s operating environment is never tested CISC 323, winter 2003, software quality 63
Skills of a Tester (1) � Not all testers are developers • e.g. testers at Corel who are photographers (Corel Draw testing) • e.g. testing buddies at Microsoft � testers must have • ability to select good test cases • ability to withstand the most severe deadline and resource pressures • ability to organize and analyze large volumes of data • ability to sort out complex version control issues CISC 323, winter 2003, software quality 64
Skills of a Tester (2) � More skills employers look for: • software engineering skills • understanding the rules of software engineering • computer programming • operating system level knowledge • communication skills • organizational skills • hands-on experience CISC 323, winter 2003, software quality 65
Attitude For Testers � destructive creativity � detective skills � understanding the product as the sum of its parts � appreciating the customer’s perspective � requirements change � skeptical but not hostile attitude � ability to be the bearer of bad news and remain objective � an eagerness to embrace new technologies CISC 323, winter 2003, software quality 66
Definitions of Testing (1) � Myers, Glenford J., The Art of Software Testing, Wiley, 1979 • “The Process of executing a program with the intent of finding errors” � Hetzel, Bill, A Complete Guide to Software Testing, QED Information Sciences Inc., 1988 • "Testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results." CISC 323, winter 2003, software quality 67
Definitions of Testing (2) � IEEE Standard 610.12-1990 • “The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component” � IEEE Standard 829-1983 • “The process of analyzing a software item to detect the differences between existing and required conditions (that is , bugs) and to evaluate the features of the software items” CISC 323, winter 2003, software quality 68
Purposes of Testing � To find bugs - including bugs in the tests! � To estimate reliability in use by tracking the execution time interval between failures � To estimate defects remaining by tracking the defects found per person-hour of testing and debugging time � To decide on when to release: e.g. by deciding that the remaining known defects are acceptable in the target market � To learn where process problems are by classifying and counting defects found CISC 323, winter 2003, software quality 69
What Testing Can't Do � Testing can't guarantee code is correct � You can't try every possible combination of inputs, every environment � Rigorous testing increases confidence that software runs correctly, or that few bugs remain � For some critical software, formal mathematical methods used to "prove" correctness – but that's another topic for other courses CISC 323, winter 2003, software quality 70
Stages of Testing (1) � Requirements and Design testing • when requirements and design are written in special languages which permit "execution" to simulate actual system � User Interface testing • Often done at prototyping stage � Unit/Module Testing � Component/subsystem Testing CISC 323, winter 2003, software quality 71
Stages of Testing (2) � Integration Testing • combining components incrementally and testing � Daily builds and Sanity testing � System Testing • Performance and Load/Stress Testing • Usability Testing, but this is a late stage to be doing it! CISC 323, winter 2003, software quality 72
Stages of Testing (3) � Alpha and Beta Testing • alpha = testing full system in-house • beta = outside users � Acceptance Testing � Installation/Compatibility Testing � Platform/Configuration/Port testing CISC 323, winter 2003, software quality 73
Stages of Testing (4) � Integrity and Release testing • documentation consistent with product • media not infected with viruses • intended version is what is actually released,... � Reliability/Certification Testing • estimates MTTF (mean time to failure) with reference to operational profiles • sometimes for certification of conformance to a standard - e.g. ACVC (Ada), ISO protocol test suites � Security/Fault Tolerance/Recovery testing � Regression Testing CISC 323, winter 2003, software quality 74
Black Box Testing � pick test cases based on requirements only � don't use knowledge of implementation � view code as a black box: insides not visible � can be used at any stage: • unit, subsystem, system testing CISC 323, winter 2003, software quality 75
Black-Box Testing Strategies � look at specifications for inputs � test normal use: • sampling of "normal" inputs (random values) � test abnormal use: • different kinds of erroneous/illegal input • combinations of errors � test boundary conditions: • values close to boundary between normal and abnormal CISC 323, winter 2003, software quality 76
Example /** * Raises a number to an integral power. * * Parameters: * base the number to be raised to a power * pow the power to which base is to be raised. * Must not be negative. * Return value base raised to the pow-th power */ static double power(double base, int pow) { � error inputs: negative values for pow � boundary cases: pow = 0, pow = 1, pow = -1 � normal cases: • pow >= 2, range of values • base: negative, 0, positive • combinations of these CISC 323, winter 2003, software quality 77
Possibilities For Automation � Ideal: tool to generate set of test cases and expected results from specifications � Reality: needs very precise, mathematical specifications, usually not available � So usually generate test cases & expected results by hand � Automated execution is still possible CISC 323, winter 2003, software quality 78
Test Harness � Takes set of test inputs & expected results � Runs program on each test input, compares results with expected � Reports any discrepancies � Complicated to write for GUIs, embedded systems, but often possible � Takes lots of drudgery out of testing � Makes testers willing to run and re-run large test suites CISC 323, winter 2003, software quality 79
White Box Testing � Pick test cases based on the actual software � also called "clear box" or "structural" testing � key word is coverage : • do the test cases cover all of the code? � different measures of coverage CISC 323, winter 2003, software quality 80
Statement Coverage � Every statement in the software must be executed at least once by the test cases � Quote from text (Murray): • "Testing less than this for new software is unconscionable and should be criminalized" � NASA story CISC 323, winter 2003, software quality 81
Branch Coverage � Every branch alternative must be covered by at least one test case � More demanding than statement coverage: if (x > 3) y = 14; // continue -- no else part � We need at least one test case where x > 3 is true and at least one where x > 3 is false. CISC 323, winter 2003, software quality 82
Path Coverage � Every possible path through the program must be covered by at least one test case void tst(int x) { if (x > 0) pos = pos+1; if (x % 2 == 0) even = even+1; return; x > 0 x <= 0 } � Needs 4 test cases: • all combinations of alternatives √ √ x is even √ √ x is odd CISC 323, winter 2003, software quality 83
Example void tst(int x) { if (x > 0) pos = pos+1; if (x%2 == 0) even = even+1; } � Statement Coverage: tst(2) � Branch Coverage: tst(-1),tst(2) � Path Coverage: tst(-2),tst(-1),tst(1),tst(2) CISC 323, winter 2003, software quality 84
What About Loops? � May require test cases which make each loop execute 0 times, 1 time, many times � for (int i = 0; i < n; i++) {...} • Test cases should include: • n <= 0 (no times around the loop) • n = 1 (one time around the loop) • n > 1 (several times around the loop) CISC 323, winter 2003, software quality 85
More About Loops � real time software generally runs inside an infinite loop • need to carefully define what you mean by path or loop coverage CISC 323, winter 2003, software quality 86
Path Coverage: Impossible Combinations � Not all combinations are possible: if (x > 2) ... else ... if (x > 4) ... else ... � impossible to have first loop false, second true – no test case for that combination CISC 323, winter 2003, software quality 87
Path Coverage: Combinatorial Explosion � Large method with many branches and loops: • many combinations, lots of test cases • difficult to manage • May catch special inputs that trigger an error � Full path coverage only practical for unit testing CISC 323, winter 2003, software quality 88
Example static double power(double base, int pow) { double answer = 1; while (pow > 0) { if (pow % 2 == 0) { // pow is even pow = pow / 2; base = base * base; } else { // pow is odd pow = pow - 1; answer = answer * base; } // end if } // end while return answer; } // end power White box checklist: pow = 0, 1, 2, bigger (so while loop executed 0, 1, many times) even and odd powers to exercise both parts of if statement CISC 323, winter 2003, software quality 89
Data Flow Testing � Another kind of white box testing � exercise (in order of weakest to strongest testing) • all input and output variables • both truth values of all predicates • all definitions of variables (eg variables on LHS of assignments) • all uses of variables that are defined • all loops CISC 323, winter 2003, software quality 90
Possibilities For Automation � Ideal: tool to generate set of test cases satisfying coverage requirements • very difficult � Can't generate expected output from code! � Some help: tools to help verify coverage: • which statements were executed by this test case, or this set of test cases? • which branch alternatives were executed? • which paths? • test harness still useful, or above can be integrated into harness CISC 323, winter 2003, software quality 91
White Box Alone � White box alone is not a good way to generate test cases. • may lead to focus on less important parts • intrinsically leads to incomplete testing • if coder overlooked possibilities, test cases will not detect them CISC 323, winter 2003, software quality 92
Black Box Alone � Black box alone may not cover the code adequately • some special cases are not obvious from specifications, depend on algorithms � Studies examined black box test suites developers considered extremely thorough • only exercised 1/3 to 1/2 of code! CISC 323, winter 2003, software quality 93
Another Example: Background � Heron's formula for area of triangle with sides a, b and c: + + a b c • let s = 2 − − − • area = s(s a)(s b)(s c) � Example program takes three sides of triangle as command- line arguments (string form) � checks to make sure it's an equilateral triangle • if not, prints error message • if yes, computes and prints area CISC 323, winter 2003, software quality 94
Example Code with two Branches public class Triangle { public static void main(String args[]) { int sideA = Integer.parseInt(args[0]); int sideB = Integer.parseInt(args[1]); int sideC = Integer.parseInt(args[1]); if ((sideA == sideB) && (sideA == sideC)) { double s = 0.5 * (sideA + sideB + sideC); double area = Math.sqrt(s / (s - sideA) * (s - sideB) * (s - sideC)); System.out.println("area = " + area); } else System.out.println("not equilateral"); } // end main } // end class CISC 323, winter 2003, software quality 95
Branch Coverage � one if statement, so two test cases needed � false case: pick sides that aren't equal • run Triangle 2 3 4 • output: not equilateral � true case: pick equal sides • run Triangle 2 2 2 • output: area = 1.7320508075688772 � output is correct, looks like all is well CISC 323, winter 2003, software quality 96
Two Errors not Caught by Test Cases public class Triangle { public static void main(String args []) { int sideA = Integer.parseInt(args[0]); int sideB = Integer.parseInt(args[1]); should be 2 int sideC = Integer.parseInt(args[1]); if ((sideA == sideB) && (sideA == sideC)) { double s = 0.5 * (sideA + sideB + sideC); double area = Math.sqrt(s / (s - sideA) * (s - sideB) * (s - sideC)); System.out.println("area = " + area); should be * } else System.out.println("not equilateral"); } // end main } // end class CISC 323, winter 2003, software quality 97
Combining the Two Approaches � pick a set of test cases from the specification (black box) � then look at code (white box) and see if test cases cover the code adequately � it not, add more to complete coverage � power example: • black box alone didn't tell us it was important to make sure we tested both odd and even powers • suggests additional test cases: pow = 16, pow = 15 CISC 323, winter 2003, software quality 98
Testing at Different Levels � Unit testing: • dual black box/clear box approach • level of smallest granularity: want to look at code � Above unit testing level: gray box • awareness of structure, but not all details, � System testing: primarily black box CISC 323, winter 2003, software quality 99
A True Story From CISC 121 � Fall term, 2002: Margaret thought of an "improvement" to quicksort partitioning algorithm � Implemented it, tested it and it worked � Released it to students for an assignment, immediate complaints that it didn't work! CISC 323, winter 2003, software quality 100
Recommend
More recommend
