Software Quality Engineering - - PDF document
– Software does what is wanted and expected – On budget and on schedule – Users are happy – Level of bugginess is tolerable – Changes are not too expensive – Software is cost-effective in use and supports
things like it (a wine of excellent quality); of high quality, general excellence
needs.
enable it to perform its intended end use.
is a factor
cost is the dominant factor.
cost of repairs.
roads)
ride for lower cost)
vehicle weight)
– Timeliness – Functionality – Cost – Correctness – Reliability – Maintainability
resources)
functionality / lower cost)
maintainability)
acceptable for different software types
that must be embedded in the product
throughout the development lifecycle
quality
– reliability – performance – usability
source: http://www.cse.dcu.ie/essiscope/sm2/9126ref.html
– Measurements chosen depend on type of product (e.g. code, design, …) – may not ensure quality of external characteristics
– Corresponds to user/manufacturer views – measurements chosen depend on product (e.g. OS, spreadsheet, …)
development
– cost vs. reliability – cost vs. functionality
– high volume products have a higher perceived value
– potential savings/productivity enhancements – profit opportunities
– always a compromise
✁ ✂✄ ☎ ✆✞✝ ✟ ✠ ✡ ✠ ✆☞☛ ✌ ✍ ☎ ✎ ✏ ✎ ✂ ✍ ☎ ✑ ✄ ☎ ✒ ✝ ✓✔ ✍ ✁ ✎ ✍ ✔ ✏ ☎ ✠ ✆☞☛ ✌ ✍ ☎ ✎ ✏ ✎ ✑ ✝ ✏ ✡ ✆ ✆ ✄ ✡ ✍ ☎ ✝ ✓ ✔ ✍ ✁ ✒ ✄ ✕ ✏ ✡ ✝ ☎ ✠ ✆☞☛ ✌ ✍ ☎ ✎ ✏ ✎ ✍ ✑ ✑ ✠ ✔ ✠ ✍ ✓ ✔ ☛accuracy, usability
after the fact
needed qualities in the products, up front
the programmer can do to ensure quality
– link these actions to high level qualities
statements must be measurable
design, identify characteristics of design that would embed those qualities
process, e.g.:
– by using quality checklists during inspection, design – by determining testing criteria up front – by following a documented design process
– ease with which software can be made to demonstrate its faults through execution based testing – related to architectural documentation, separation
development
Remainder of material about quality focuses on two topics related to software quality:
– Testing: finding defects by executing software and examining results (and then fixing the defects) – Inspection: a formal way of analyzing software and other documents
software product and record metrics related to that product.
"reviews":
– formal process – products inspected at greater level of detail – metrics
"...a formal evaluation technique in which software requirements, design, or code are examined in detail by a person or group other than the author to detect faults, violations of development standards, and other problems..."
✁ ✁ ✂✄ ✄ ✄ ☎ ✆✞✝ ✟ ✠ ✝ ✡ ✠ ☛ ☞✍✌ ✎ ✎ ✝ ✡✏ ✌ ✑ ☎ ✌ ✑ ✆ ✒ ✝ ✡ ✓ ✄ ✟✔ ✕ ✟ ✓ ✓ ✡ ✕ ✟ ✔ ✖ ✓ ✡ ✗ ✕ ✟ ✌ ☞✍✌ ✔ ✏ – Inspecting the product of a manufacturing process – Physical product, evaluate by looking, manipulating – Often put through rigorous tests, keep metrics
– used similar processes to inspect software (code) – landmark paper in 1976 – often called "Fagan Inspections" – Inspection process has been extended & refined since, ongoing research
(code)
design documents, drawings, etc.
mean dynamic testing: executing software to find faults
inspection and testing
performance, timing, load
product, not just code
software
– must debug to trace the fault to a defect in the code
Inspection
defects
– compilers find undeclared variables, syntax errors – other static analysis tools may check for:
from Software Inspection, Tom Gilb & Dorothy Graham
inspection & logging for each chunk
potential defects
documents and products being inspected, i.e.:
✥ ✦ ✧★ ✩ ✪ ✦ ✧ ✫ ✧✬ ✭✯✮ ✭✯✰ ✱ ✧ ✮ ✪✳✲ ✬ ✥ ✱ ✧ ✮ ✪ ✲ ✬ ✭ ✰ ✴✰ ✱ ✧ ✥ ✦ ✧★ ✩ ✪ ✦ ✧ ✫ ✧✬ ✭✯✮ ✭✯✰ ✭ ✧ ✮ ✭✯✵ ✶✸✷ ✬ – log issues identified by each checker – work as group to find more issues – suggestions for process improvement (inspection process and overall software process)
– must be very skilled in handling people and issues – maintain time discipline and issue discipline
– just log issues, like brainstorming
training
suggestions for improvements
✤ ✥✦ ✧ ★ ✩ ✪✫ ✬ ✪ ✪✭ ★ ✩ ✥ ✮ ✭ ✭ ✧ ✩ ✥✯ – classify or reclassify issues (which are really defects) – voluntarily improve product as per issues raised – request rules or checklist be changed
closure has taken place on all issues
– actions to correct all defects (change or change request) – suggestions for process changes reported
inspected again
process improvement)
next phase in software development
Requirem ents Cost of running inspections is about 10-15% of cost of development budget. Inspection Architecture Inspection Design Inspection Code Inspection Unit Test Integration Test System Test
– establishing procedures – training leaders, inspectors
– testing is much quicker
– even when counting time for inspection
– cleaner design – better documentation – better code – fewer defects – defects that remain will be easier to fix when found
process:
– inspector's comments plus metrics
surprises
– learn from good code and documentation of
– learn from mistakes of others
– kept metrics for similar projects before and after introducing Inspection into development process – delivered lines of code per work-month:
– 57.7% of defects found by Inspection – cost of finding defect by Inspection: 1.58 work hours – cost of finding without Inspection: 8.47 work hours – only 6% of development time devoted to Inspection
– 500,000 lines of code (networked operating system) – based on past experience without inspection, expected to find about 800 bugs during tests at trial site – used inspection at 11 stages of devopment – found 8 bugs at trial site!
– inspection process – techniques for individual inspectors
– experiment where they varied
✁ ✂ ✄ ☎ ✆✞✝ ✟ ✠ ✡ ☛ ✂☞ ✌✝ ✍ ✎ ✠ ✟ ☞ ✏ ✂ ✄ ☎ ✆✞✝ ✟ ✠ ✡ ✎ ✝ ✑ ☎ ☞ ☛ ✂ ☞ ✌✝ ✍ ✎ ☛ ✂✒ ✎ ✓ ✝ ☞ ✑ ☎ ✝ ✍ ✠ ✔ ✝ ✏ ✎ ✝ ✑ ☎ ☞ ✕ ✠ ✟ ✖ ☛ ✂✒ ☛ ✂ ✌ ✑ ✟ ✑ ✗ ✗ ✝ ✗ ✑ ✂ ✔ ✎ ✝ ✑ ☎ ☞ ✕ ✠ ✟ ✖ ☛ ✂✒ ☛ ✂ ✎ ✑ ✂ ✔ ✝ ☎ ✘ ✕ ✓ ✝ ✟ ✝ ✆ ✄ ✒ ☞ ✕ ✝ ✟ ✝ ✡ ☛ ✙ ✝ ✔ ✆✞✝ ✡ ✠ ✟ ✝ ✚ ✂ ✔ ☛ ✂☞ ✌✝ ✍ ✎ ☛ ✠ ✂ ✎ ✠ ✠ ✖ ✌ ✗ ✑ ✍ ✝ ✛– one of the conclusions
✏ ☎ ✠ ☞ ✎ ☞ ☛ ✒ ✂ ☛ ✡ ☛ ✍ ✑ ✂ ✎ ☛ ☎ ✌ ✟ ✠ ✜ ✝ ☎ ✝ ✂ ✎ ☞ ☛ ✂ ☛ ✂☞ ✌ ✝ ✍ ✎ ☛ ✠ ✂ ✍ ✠ ☎ ✝ ☞ ✡ ✟ ✠ ☎ ☛ ☎ ✌ ✟ ✠ ✜ ☛ ✂ ✒ ✎ ✓ ✝ ✕ ✑ ✢ ✎ ✓ ✝ ☛ ✂ ☞ ✌✝ ✍ ✎ ✠ ✟ ☞ ✔ ✠ ✎ ✓ ✝ ☛ ✟ ☛ ✂ ✔ ☛ ✜ ☛ ✔ ✄ ✑ ✗ ✕ ✠ ✟ ✖ their work
– scenario-based reading
✏ ✝ ✒ ✥ ✌ ✄ ✎ ✢ ✠ ✄ ✟ ☞ ✝ ✗ ✡ ☛ ✂ ✎ ✓ ✝ ✟ ✠ ✗ ✝ ✠ ✡ ✑ ✎ ✝ ☞ ✎ ✝ ✟ ✏ ✓ ✑ ✜ ✝ ✑ ☞ ✝ ✟ ☛ ✝ ☞ ✠ ✡ ✦ ✄ ✝ ☞ ✎ ☛ ✠ ✂ ☞ ✑ ✂ ✔ ☞ ✍✝ ✂ ✑ ✟ ☛ ✠ ☞ ✎ ✠ ✕ ✠ ✟ ✖ ✕ ☛ ✎ ✓ ✑ ☞ ✢ ✠ ✄ ☛ ✂ ☞ ✌✝ ✍ ✎ – rely on past experience – can get too long and unwieldy – need to be updated as new situations arise – inspectors have a tendency to follow the checklist and do no more – can be misinterpreted – may not cover everything necessary to the right level of detail
software system
contribute to those qualities
Quality Model
McCall's Quality Model
both testing and inspection
results in fewer defects for testing to find – speeds up testing
inspection might save time – get rid of
– may even include product (re-)development
– user executes untested code – order in which statements are executed in actual use differ from that in testing – user applies combination of untested input values – user’s operating environment is never tested
– software engineering skills
✥ ✗ ✓ ✙ ✁ ✞ ✝ ✆ ✟✓ ✙ ✒✔✓ ☎ ✆ ✍ ✁ ✞ ✗ ☞ ✁ ✝ ✡ ✜ ✝ ✡ ✜ ✆ ✌ ✟ ✞ ✁ ✁ ✓ ☎ ✒ ✓ ✁ ✁ ✞ ✒✔✓ ☎ ✥ ✛ ✡ ✤ ✎ ✗ ✆ ✁ ✞ ✎ ✞ ✡ ☎ ✞ ✟ ✤ ✤ ✒✔✓ ☎ ✥ ✡ ✎ ✁ ✞ ✟ ✆ ✒✔✓ ☎ ✝ ✢ ✝ ✆ ✁ ✤ ☞ ✁ ✥ ✁ ☞ ✦ ✓ ✡ ✌ ☞ ✁ ✙ ☎ ✁– communication skills – organizational skills – hands-on experience
Testing, Wiley, 1979
– “The Process of executing a program with the intent of finding errors”
Testing, QED Information Sciences Inc., 1988
– "Testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results."
– “The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component”
– “The process of analyzing a software item to detect the differences between existing and required conditions (that is , bugs) and to evaluate the features of the software items”
execution time interval between failures
found per person-hour of testing and debugging time
the remaining known defects are acceptable in the target market
and counting defects found
inputs, every environment
software runs correctly, or that few bugs remain
mathematical methods used to "prove" correctness – but that's another topic for
– when requirements and design are written in special languages which permit "execution" to simulate actual system
– Often done at prototyping stage
– Incremental Testing
usual version
integration testing.
– Performance and Load/Stress Testing – Usability Testing, but this is a late stage to be doing it!
– alpha = testing full system in-house – beta = outside users
– unit, subsystem, system testing
– sampling of "normal" inputs (random values)
– different kinds of erroneous/illegal input – combinations of errors
– values close to boundary between normal and abnormal
/** * Raises a number to an integral power. * * @param base the number to be raised to a power * @param pow the power to which base is to be raised. * Must not be negative. * @return base raised to the pow-th power */ static double power(double base, int pow) {
– pow >= 2, range of values – base: negative, 0, positive – combinations of these
expected results from specifications
specifications, usually not available
results by hand
results with expected
systems, but often possible
test suites
– do the test cases cover all of the code?
executed at least once by the test cases
– "Testing less than this for new software is unconscionable and should be criminalized"
at least one test case
if (x > 3) y = 14; // continue -- no else part
and at least one where x > 3 is false.
covered by at least one test case
void tst(int x) { if (x>0) pos = pos+1; if (x%2==0) even = even+1; return; }
– all combinations alternatives
void tst(int x) { if (x>0) pos = pos+1; if (x%2==0) even = even+1; }
execute 0 times, 1 time, many times
– Test cases should include:
infinite loop
– need to carefully define what you mean by path or loop coverage
if (x > 2) ... else ... if (x > 4) ... else ...
true – no test case for that combination
loops:
– many combinations, lots of test cases – difficult to manage – May catch sepcial inputs that trigger an error
testing
static double power(double base, int pow) { double answer = 1; while (pow > 0) { if (pow % 2 == 0) { // pow is even pow = pow / 2; base = base * base; } else { // pow is odd pow = pow - 1; answer = answer * base; } // end if } // end while return answer; } // end power
White box checklist: pow = 0, 1, 2, bigger (so while loop executed 0, 1, many times) even and odd powers to exercise both parts of if statement
– all input and output variables – both truth values of all predicates – all definitions of variables (eg variables on LHS of assignments) – all uses of variables that are defined – all loops
coverage requirements
✁ ✂ ✄ ☎ ✆ ✝ ✞ ✟ ✟ ✞ ✠ ✡ ☛ ☞ generate test cases.
– may lead to focus on less important parts – intrinsically leads to incomplete testing – if coder overlooked possibilities, test cases will not detect them
adequately
– some special cases are not obvious from specifications, depend on algorithms
developers considered extremely thorough
– only exercised 1/3 to 1/2 of code!
– let s = – area =
command-line arguments (string form)
2 c b a + + ) )( )( ( c s b s a s s − − −
public class Triangle { public static void main(String args []) { int sideA = Integer.parseInt(args[0]); int sideB = Integer.parseInt(args[1]); int sideC = Integer.parseInt(args[1]); if ((sideA == sideB) && (sideA == sideC)) { double s = 0.5 * (sideA + sideB + sideC); double area = Math.sqrt(s / (s - sideA) * (s - sideB) * (s - sideC)); System.out.println("area = " + area); } else System.out.println("not equilateral"); } // end main } // end class
– run Triangle 2 3 4 – output: not equilateral
– run Triangle 2 2 2 – output: area = 1.7320508075688772
public class Triangle { public static void main(String args []) { int sideA = Integer.parseInt(args[0]); int sideB = Integer.parseInt(args[1]); int sideC = Integer.parseInt(args[1]); if ((sideA == sideB) && (sideA == sideC)) { double s = 0.5 * (sideA + sideB + sideC); double area = Math.sqrt(s / (s - sideA) * (s - sideB) * (s - sideC)); System.out.println("area = " + area); } else System.out.println("not equilateral"); } // end main } // end class
should be 2 should be *
(black box)
cover the code adequately
– black box alone didn't tell us it was important to make sure we tested both odd and even powers – suggests additional test cases: pow = 16, pow = 15
– dual black box/clear box approach – level of smallest granularity: want to look at code
– awareness of structure, but not all details,
– generally least effective – very difficult to find bugs – avoid this!
– test all modules individually (unit testing) – put together gradually & test larger pieces
– finally test whole system
– start with tested modules in either case – planning of increments starts early
– top down requires stubs – bottom up requires drivers – 'sandwich' variants exist
– bug fix, change or extension
another – module interactions change
plan, test cases.
✥ ✦✤✧ ★✪✩ ✫ ★✪✬ ✭ ✫✯✮ ✩ ✫ ✰✱ ✩ ✮ ✩ ✲ ✱ ✳ ✴ ✮ ✵ ✮ ✶ ✩ ✱ ✴✷ ✮ ✸ ✵ ✮ ✫✯✮ ✩ ✫✯✱ ✴✷ ✮ ✸ ✹ ✵ ✹ ✴ ✩ ✹ ✷ ✮ ✫✯✮ ✺ ✻ ✼ ✮ ✵ ✮ ★✪✩ ✱ ✫ ✵ ✱ ✽ ✮✿✾ ✹ ❀ ❀ ✴ ✮ ✫❂❁ ✮ ✮ ✬ ❃ ✹ ✩ ✩ ★ ✴ ✷ ✮ ✵ ✮ ✽ ✶ ✬ ✽ ✱ ✬ ✫ ✫✯✮ ✩ ✫ ★ ✬ ✭ ✰ ✹ ✩ ✫ ✩ ✱ ✬ ✽ ✫ ✼ ✮ ✰ ✹ ✩ ✫ ✩ ✹ ❀ ✽ ✮ ✰ ★ ✽ ★ ✬ ✭ ❁ ✼ ★ ✰ ✼ ✫✯✮ ✩ ✫ ✵ ✮ ✩ ✶ ✷ ✫ ✩ ✰✱ ✬ ✴ ✮ ✵ ✮ ✶ ✩ ✮ ✽ ✱ ✬ ✽ ❁ ✼ ★ ✰ ✼ ✫ ✮ ✩ ✫ ✰ ✱ ✩ ✮ ✩ ✰ ✱ ✬ ✴ ✮ ✵ ✮ ✵ ✶ ✬❅❄ ✺ ❆ ✹ ✲ ✮ ✫✯✮ ✩ ✫ ✰ ✱ ✩ ✮ ✩ ✲ ✱ ✳ ✬ ✮ ✮ ✽ ✫ ✹ ✴ ✮ ✽ ✵ ✹ ❃ ❃ ✮ ✽ ✺ ✬ ✮ ❁ ✫✯✮ ✩ ✫ ✰ ✱ ✩ ✮ ✩ ✲ ✱ ✳ ✴ ✮ ✬ ✮ ✮ ✽ ✮ ✽ regression testing if the original specifications were in error
specified early as criteria for accepting product
software may pass tests but not do much else (Ada compilers)
detailed prior specification of the tests, but depends on precise, complete and correct system specifications.
popularity of OOP
code?
– OO testing is unnecessary. OO promotes iteration and reuse, so we have a more effective way to develop trustworthy classes.
– Human error is as likely as ever. We have to check class functionality and interactions between classes
– testing is a structured waterfall idea and isn’t consistent with incremental OO development
– tests can be designed and exercised at many points in the process – paradigm of “design a little, code a little” becomes “design a little, code a little, test a little” – Extreme Programming (XP) promotes early testing (more later)
– testing is too expensive
– pay me now or pay me MUCH more later – failures in operational systems can cause severe secondary problems – proper testing is very cheap by comparison, even when done manually – efficient testing of large or complex systems needs some automated support
– testing is the same as conventional software – all you need is black-box system testing
– OO code structure matters – polymorphism, inheritance, encapsulation present
– conventional testing is useless for objects
– there is a large body of knowledge about testing – basic testing techniques continue to apply with necessary changes – if scope of testing is small, ie classes and collaborations
✘ ✟ ✂ ✏ ✟ ✆ ✝ ✗ ✟ ✂ ✕ ✒ ✝ ✆ ✤ ✓ ✂ ✏ ✡ ✠ ✂ ✑ ✂ ✠ ☞ ✥ ✥ ✏ ✔ ✂ ✕ ✆ ☎ ✆ ✕ ✦ ✖ ✍ ✠ ✂ ☛✄✡ ✟ ✂ ✠– as scope of testing gets larger
✘ ✟ ✂ ✏ ✟ ✆ ✝ ✗ ✟ ✂ ✕ ✒ ✝ ✆ ✤ ✓ ✂ ✏ ✎ ✂ ✕ ✍ ✖✂ ✖ ✍ ✠ ✂ ✟ ✠ ✡ ✁ ✆ ✟ ✆ ✍ ✝✡ ☛ – inheritance means never having to say you are sorry – specializing from a trusted superclass means the inherited part of subclasses will also be correct – we don’t need to retest inherited features
– subclasses create new ways to misuse inherited features – different test cases are needed for each subclass (more later)
– reuse means never having to say you are sorry – reusing a trusted class means behaviour of server
tested
– nothing prevents a new client class from using the server object incorrectly – all client class use of a server needs to be exercised
– black box testing is sufficient – designing test cases using class interface and specification assures the class is fully exercised – white box testing violates encapsulation
– studies show black-box test suites typically only exercise 1/3 to 1/2 of the statements (let alone paths or states) – typically miss abnormal paths, exception and error-handling
– focus on testing in order to give chance to think about what code will do, independently of how it is implemented – write every test he/she can think of that won’t run immediately on existing code – consider tests as giving confidence that things are working
✥ ✦ ✧ ★✪✩ ✫ ✧✬ ✭ ✮ ✯ ✧ ✬ ✰ ✧✱ ✭ ★ ✧ ✲ ✩ ✳ ✳ ✮ ✱ ✳– consider testing as a type of metric:
✴ ✬ ✵ ✶ ✭✸✷ ✹ ✦ ✧ ✶ ✧ ✹ ✭ ✩ ✦ ✧ ✬ ✭ ✺ ✹ ✭ ✫ ✹ ✱ ✻ ✭ ✲ ✧ ★ ✧ ✯ ✵✱ ✬ ✭ ✦ ✹ ✭ ✧ ★ ✲✪✼ ✹ ✩ ✭ ✵ ✯ ✹ ✭ ✧ ★ ✭ ✧ ✬ ✭ ✬✾✽ ★ ✵✱ ✻ ✭ ✧ ✿ ✮ ✬ ✭ – pick up first test case – implement code to get test case running – pick up next test case – refactor code to get simplest implementation to get both test cases running – repeat for rest of test cases
– focus on testing up front is interesting
– we will look at unit testing only
software system uses
– human interfaces
✥ ✦✧ ★ ✩✫✪ ✬✫✭ ✮ ✧ ✪ ✮ ✯✫✧ ✰ ✭ ✧ ★ ✪ ✱ ✪ ✦ ✦ ✰ ✲ ✳ ✱✴ ★ ✧ ✵ ✳ ★ ✩ ✭ ✪ ✶ ★ ✵ ✴ ✷ ✧ ✸ ✦ ✰ ✭ ★ ✱ ✪ ✲✭ ✳ ✬✫✧ ✷ ✩✫✪ ✵ ★ ✪ ★ ✧ ✭ ★ ✳ ✲ ✮ ✰ ★ ✶ ✷ ✪ ✦ ✬ ✳ ✶ ✶ ✧ ✷ ✧ ✲ ★ ✬✫✧ ✹ ✳ ✱ ✧ ✭– software interfaces
✸ ✩✫✪ ✵ ✭ ✪ ✶ ★ ✵ ✴ ✷ ✧ ✰ ✭ ✧ ✭ ✪ ✮ ✧ ✷ ✴ ★ ✳ ✲✺ ✭ ✻ ✭ ★ ✧ ✦✽✼ ✬ ✴ ★ ✴ ✾✫✴ ✭ ✧ ✼ ✷ ✰ ✲ ★ ✳ ✦✧ ✯ ✳ ✾ ✷ ✴ ✷ ✻ ✸ ✦ ✰ ✭ ★ ★ ✧ ✭ ★ ✧ ✿ ✮ ✧ ✱ ★ ✧ ✬ ✴ ✲ ✬ ✰ ✲ ✧ ✿ ✮ ✧ ✱ ★ ✧ ✬ ✴ ✱ ★ ✳ ✹ ✳ ★ ✳ ✧ ✭ ✪ ✶ ✭ ✧ ✷ ✹ ✳ ✱ ✧ ✱ ✯ ✴ ✭ ✭ ✧ ✭ – file system interfaces
✸ ✷ ✧ ✴ ✬✫✭ ✪ ✷ ✵ ✷ ✳ ★ ✧ ✭ ★ ✪ ✧ ✿ ★ ✧ ✷ ✲ ✴ ✯ ✶ ✳ ✯✫✧ ✭ ✸ ✦ ✰ ✭ ★ ✱ ✩✫✧ ✱ ✥ ★ ✪ ✭ ✧ ✧ ✳ ✶ ✬ ✴ ★ ✴ ✶ ✷ ✪ ✦ ★ ✩✫✧ ✶ ✳ ✯ ✧ ✳ ✭ ✴ ✮ ✮ ✷ ✪ ✮ ✷ ✳ ✴ ★ ✧ ✳ ✲ ✱ ✪ ✲ ★ ✧ ✲ ★ ✴ ✲ ✬ ✶ ✪ ✷ ✦ ✴ ★ ★ ✳ ✲✺– communication interfaces
✸ ✬ ✳ ✷ ✧ ✱ ★ ✴ ✱ ✱ ✧ ✭ ✭ ★ ✪ ✮ ✩ ✻ ✭ ✳ ✱✴ ✯ ✬✫✧ ✹ ✳ ✱ ✧ ✭ ✦ ✧ ✺ ✧ ★ ✷ ✳ ✹ ✧ ✷ ✭ ✼ ✱ ✪ ✲ ★ ✷ ✪ ✯ ✯✫✧ ✷ ✭ ✼ ✪ ★ ✩ ✧ ✷ ✧ ✦ ✾ ✧ ✬ ✬✫✧ ✬ ✭ ✻ ✭ ★ ✧ ✦ ✭ ✩ ✸ ✦ ✰ ✭ ★ ✾ ✧ ✴ ✾ ✯ ✧ ★ ✪ ✺ ✧ ✲✧ ✷ ✴ ★ ✧ ✹ ✴ ✯ ✳ ✬ ✴ ✲ ✬ ✳ ✲ ✹ ✴ ✯ ✳ ✬ ✮ ✷ ✪ ★ ✪ ✱ ✪ ✯ ✭ ★ ✷ ✧ ✴ ✦✭ ✼ ✧ ✺ ✮ ✴ ✱ ✥ ✧ ★ ✭ – no guarantee that C calls A properly – we still have to test C's use of A
super int A(int x) int B(int x) sub int B(int x) int C(int x)
super int A(int x) int B(int x) sub int B(int x)
goals of uncovering errors in results, state of
invocation of object
– methods can be invoked in any random order leading to a large number of combinations
intensive and error prone
– automate as appropriate
✁ ✂ ✄ ☎✆ ✝ ✞ ✄✟ ✠ ✝ ✄ ✡ ✝ ✄ ☛ ☞✍✌ ☎ ✄ ☞ ✆ ✎ ☞ ✆ ✝ ☞ ✝ ✌ ✏ ✑✒✡ ✓ ✔ ✏ ✎ ✕ ✖ ✗ ✌ ✓ ✆ ✘ ✘ ✌ ✘ ✝ ✙ ✚✛✜ ✢ ✛ ✣ ✤ ✥ ✦ ✤ ✦ ✧ ★ ✜ ✩✫✪ ✤ ✬ ✥ ✭ ✮ ✯ ✰ ✱ ✱– often include extra code to provide information about variables, object properties, etc during test runs – write code for testability: controllability and
– comparison of software’s actual output to its expected output – often difficult to do
✥ ✦✧ ★ ✩✤✪ ✫✬ ✭✮ ✯ ✬ ✰ ✱ ✦ ✲ ✳✵✴ ✪ ✬ ✱ ✱✷✶ ✫ ✮ ✭ ✳ ✸ ✮ ✯ ✳ ✭ ✧ ✴ ✰ ✯ ✦ ✦ ✭ ✮ ✪ ✸ ✹ ✪ ✸ ✬ ✭ ✺ ✹ ✬ ✳ ✭ ✴ ✸ ✬ ✻ ✳ ✭ ✧ ✱ ✶ ✬ ✭✬ ✱ ✶ ✼ ✳ ✭ ✧ ✺✽✬ ✸ ✬ ✾ ✿✤❀ ❁ ❂ ❃ ❂ ❄❆❅ ✿❈❇ ❉❊ ❋● ❊ ❋❍ ■ ❀ ❏ ❑ ▲ ▼ ❏ – when to stop testing
✥ ✦ ✧ ✦ ★ ✩ ✩✫✪ ✬ ✭✯✮ ✰ ✪ ✱ ✦✲ ✦ ✰ ✱ ✦ ✳ ✱ ✴ ✳ ✵✷✶ ✮ ★ ✰ ✸✹ ✱ ✲ ✲ ✮ ✧ ✱ ✦ ✲ ✺ ✮ ✧– when the product is ready to release
✥ ✻ ★ ✧ ✮ ✸ ✱ ✰ ✻ ✦ ✧ ✵ ✰ ✮ ✧ ✧ ✼ ✦ ✸✯✽ ✮ ✶ ✮ ✰ ✳– estimation of number of bugs left in software
✥ ✻ ★ ✧ ✮ ✸ ✱ ✰ ✾ ✲ ★ ✳ ✮ ★ ✳ ✬ ✭ ✵ ✺ ✭ ✻ ✦ ✽ ✧ ★ ✲ ✮ ✧ ✳ ✵ ✩ ✩ ✻ ✮ ✵ ✰✽ ✴ ✱ ✦ ✰ ✸ ✾ ✬ ✭✯✮ ✲ ✮ ✵ ✰ ✧ ✱ ✴ ✳ ✬ ★ ✲ ✮ ✻ ✦ ✽ ✧ ★ ✲ ✮ ✻ ✮ ✵ ✰✽ ✴ ✱ ✦ ✰ ✸ ✾ ✲ ★ ✳ ✮ ✱ ✴ ✻ ✦ ✽ ✴ ✵✤✿ ✵ ✰✽