Software Quality Engineering - PDF document

� ☎ ✕ ✖ ✖ ✂ ✗ ✚ ✞ ✘ ✛ ✄ ☎✆ ☞ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ ✁✒ ✁ ✔ ✗ ☞ ☛ ✄ ☎ ☎✆ ✝ ✞ ✟ ✓ ✠ ✡ ☎ ✆ ✞ ✏ ✠ ✁✒ ✎ ✑ ✁ ✎ ✍ ✌ ☞ Value-based view • helps to decide trade-offs during development – cost vs. reliability – cost vs. functionality • cost to consumer is affected by scale – high volume products have a higher perceived value • value is defined by – potential savings/productivity enhancements – profit opportunities ✖✣✢ ✗✙✘

� ✏ ✆ ✡ ✏ ✝ ✑ ✎ ✏ ✎ ☎ ✍ ✌ ✠ ☎ ✔ ✄ ✍ ✎ ✁ ✍ ✓✔ ✝ ✒ ☎ ✄ ✑ ☎ ✍ ✂ ✎ ✆ ✡ ✎ ✍ ☛ ✔ ✓ ✍ ✠ ✔ ✠ ✑ ✑ ✍ ✎ ✏ ✎ ☎ ✌ ✍ ✠ ☎ ✝ ✡ ✏ ✕ ✄ ✒ ✁ ✍ ✔ ✓ ✝ ☎ � ✏ ☎ ✌ ✖ ✖ ✕ ✔ ☞ ☞ ✓ ✁ ✁✒ ✎ ✑ ✏ ✎ ✍ ☞ ✍ ✠ ✞ ☛ ☎ ✡ ✠ ✆ ✟ ✞ ✝ ☎✆ ☎ ✄ ✂ ✗ ✗ ✚ ✠ ✠ ✡ ✠ ✟ ☎ ✂✄ ✁ ✁ ✁✒ ✚ ✞ ✌ ✚ ☎✆ ✞ ✘ ✛ ✄ ☎ ✞ ✘ ✄ ✚✜ Problems with Qualities • qualities are at odds with one another – always a compromise ✆✞✝ ✆☞☛ ✆☞☛ ✆☞☛ • difficulty in trying to measure qualities e.g., accuracy, usability • what does quality really mean? ✖✣✢ ✗✙✘

� ☎ ✖ ✗ ✁ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✠ ✡ ☎ ✆ ✞ ✏ ✒ ✠ ✎✒ ✑ ✓ ✎ ✍ ✌ ☞ Proposed solution 1: Focus on Product • most dimensions of quality are measured after the fact • need to identify actions that can embed needed qualities in the products, up front • implementation quality model: identify what the programmer can do to ensure quality – link these actions to high level qualities ✗✤✣ ✘✚✙

✓ � ☎✆ ✂ ✗ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎ ✞ ✕ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✖ ✗ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ✡ ☞ ☛ ☎ ✠ ✑ ✔ ✒ ✓ ☞ ✎✒ ✏ ✎ ✍ ✌ Proposed Solution 1 (cont’d) • requirements quality model: all requirement statements must be measurable • design quality model: define qualities for design, identify characteristics of design that would embed those qualities ✗✤✣ ✘✚✙

✓ ☎ ✖ ✗ � ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✔ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ Proposed solution 2: Focus on Process • integrate quality decisions into the design process, e.g.: – by using quality checklists during inspection, design – by determining testing criteria up front – by following a documented design process ✗✤✣ ✘✚✙

✓ ☎ ✖ ✓ ✗ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✔ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ Qualities Discernable at Runtime • performance • security • availablity • functionality • usability ✗✤✣ ✘✚✙

✓ ☎ ✖ ✗ ✗ ✘ ✒ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✘ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✠ ✡ ☎ ✆ ✞ ✏ ✒ ✠ ✎✒ ✑ ✓ ✎ ✍ ✌ ☞ Qualities Not Discernable at Runtime • modifiability • portability • reusability • testability – ease with which software can be made to demonstrate its faults through execution based testing – related to architectural documentation, separation of concern, information hiding, incremental development ✗✤✣ ✘✚✙

� ☎ ✕ ✁ ✖ ✗ ✗ ✚ ✞ ✘ ✛ ✄ ☎✆ ☞ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ ✒ � ✒ ✔ ✖ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ✓ ☎ ✡ ✞ ✏ ✒ � ✎✒ ✠ ✑ ✎ ✍ ✌ ☞ Business Qualities • time to market • cost • projected lifetime of system • targeted market • integration with existing legacy systems ✖✣✢ ✗✙✘

� ☎ ✕ � ✖ ✗ ✗ ✚ ✞ ✘ ✛ ✄ ☎✆ ☞ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ ✒ � ✒ ✔ ✖ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✓ ✡ ☎ ✠ ✞ ✏ ✒ � ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ What's Ahead Remainder of material about quality focuses on two topics related to software quality: – Testing: finding defects by executing software and examining results (and then fixing the defects) – Inspection: a formal way of analyzing software and other documents ✖✣✢ ✗✙✘

� ☎✆ ✖ ✖ ✂ ✗ ✚ ✞ ✘ ✛ ✄ ☎ ✞ ✔ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ ✒ � ✒ ✕ ✗ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ☞ ✡ ☎ ☛ ✠ ✠ ✑ ✓ ☞ � ✎✒ ✒ ✏ ✎ ✍ ✌ What is Software Inspection? • Formal process to statically analyze a software product and record metrics related to that product. • Differs from informal "walk-throughs" or "reviews": – formal process – products inspected at greater level of detail – metrics ✖✣✢ ✗✙✘

� ✡ ☎ ✑ ✌ ✡✏ ✝ ✎ ✎ ☛ ✠ ✝ ✑ ✠ ✟ ☎ ✄ ✄ ✂✄ ✁ ✁ � ✒ ✌ ✆ ✒ ✟ ✔ ✌ ✟ ✕ ✗ ✡ ✓ ✖ ✔ ✕ ✒ ✡ ✓ ✓ ✟ ✕ ✟✔ ✄ ✓ ✡ ✝ � ✏ ✚ ✠ ☞ ✓ ✒ � ✎✒ ✑ ✏ ✎ ✍ ✌ ☞ ✞ ✔ ☛ ☎ ✡ ✠ ✆ ✟ ✞ ✝ ☎✆ ☎ ✄ ✂ ☞ ✕ ✞ ✄ ✠ ✚ ✚✜ ✘ ✄ ✖ ☎✆ ☎ ✞ ✛ ✗ ✘ ✞ ✚ ✖ ✗ IEEE Definition "...a formal evaluation technique in which software requirements, design, or code are examined in detail by a person or group other than the author to detect faults, violations of development standards, and other problems..." ☞✍✌ ☞✍✌ ✆✞✝ ✖✣✢ ✗✙✘

�✁ ☎✆ ✖ ✖ ✗ ✂ ✚ ✞ ✘ ✛ ✄ ☎ ✞ ✔ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ ✒ � ✒ ✕ ✗ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ☞ ✠ ☎ ☛ ✡ ✠ ✑ ✓ ✒ ☞ ✎✒ � ✏ ✎ ✍ ✌ History • Traditional meaning of inspection: – Inspecting the product of a manufacturing process – Physical product, evaluate by looking, manipulating – Often put through rigorous tests, keep metrics • Michael Fagan at IBM: – used similar processes to inspect software (code) – landmark paper in 1976 – often called "Fagan Inspections" – Inspection process has been extended & refined since, ongoing research ✖✣✢ ✗✙✘

�✁ ✮ ✰ ✹ ✦ ✪ ✮ ✴ ✳ ✪ ✲ ✦ ✱ ✮ ✮ ✭ ✫ ✰ ✮✯ ✪ ✫ ✪ ✥ ✥✦ ✱ ✦ ✒ ✮ ✲ ✮ ✳ ✪ ✧ ✥ ✰ ✥✦ ✱ ✭ ✪ ✲ ✰ ✳ ✪ ✥✦ ✧ ✪ ✮ ✲ ✳ ✪ ✂ ✤ � ☞ ☞ ☞ ✓ ✒ � ✎✒ ✑ ✏ ✎ ✍ ✌ ✠ ✖ ✞ ☛ ☎ ✡ ✠ ✆ ✟ ✞ ✝ ☎✆ ☎ ✄ ✔ ✕ ✘ ✚ ✠ ✚ ✒ ✚✜ ✖ ✄ ✞ ☎✆ ☎ ✞ ✄ ✛ ✘ ✞ ✚ ✗ ✗ What Gets Inspected? • Inspection process initially used to inspect software (code) • Extended to include all documents: ✧✩★ ✬✩✭ ✴✸✷ ✫✶✵ ✲✻✺ ✹✽✼ • Above includes diagrams, prose, pseudo-code, etc. • Some other industries have adopted Inspection for design documents, drawings, etc. ✖✣✢ ✗✙✘

�✁ ☎ ✕ ✖ ✂ ✗ ✗ ✚ ✞ ✘ ✛ ✄ ☎✆ ☞ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ �✒ � ✔ ✖ ☞ ☛ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✓ ✡ ☎ ✠ ✞ ✏ � �✒ ✠ ✑ ✎ ✎ ✍ ✌ ☞ Inspection Versus Testing (1) • AT&T classifies inspection as a form of testing • In CISC 323, we use the term "testing" to mean dynamic testing: executing software to find faults • inspection does not replace dynamic testing • software process should include both inspection and testing ✖✣✢ ✗✙✘

� ☎ ✁ ✖ ✖ ✗ ✗ ✚ ✞ ✘ ✛ ✄ ☎✆ ☞ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ �✒ � ✔ ✕ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ✓ ☎ ✡ ✞ ✏ � �✒ ✎ ✠ ✑ ✎ ✍ ✌ ☞ Inspection Versus Testing (2) • difficult to inspect for qualities such as performance, timing, load • inspection can be applied earlier than testing • inspection can be applied to any software product, not just code ✖✣✢ ✗✙✘

�✒ ☎ ✕ ✖ ✂ ✗ ✗ ✚ ✞ ✘ ✛ ✄ ☎✆ ☞ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ �✒ � ✔ ✖ ☞ ☛ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✓ ✡ ☎ ✠ ✞ ✏ � ✠ ✎ ✑ �✒ ✎ ✍ ✌ ☞ Defects vs. Faults • Inspection finds defects: problems in software • Testing finds faults: incorrect behavior • When you find a fault by testing: – must debug to trace the fault to a defect in the code • Often more cost-effective to find defaults via Inspection ✖✣✢ ✗✙✘

� ✏ ✔ ☎ ✄ ✡ ✗ ✄ ✆ ✏ ✂ ✎ ✂ ✗ ✕ ✄ ✆ ✝ ✌✆ ✁ ✎ ✏ ✆ ✟✠ ☎ ✌ ☞ ✏ ☎ ✡ ✓ ✄ ✑ ✠ ☎✆ ✒ ✌ ☞ ✘ ✝ � ✎ ✗ ✄ ✆ ✏ ✏ ✆ ✘ ✝✆ ✏ ✠ ☎ ✄ ✄ ✡ ✄ ✌ ☎ ✡ ☎ ☎ ✌ ☞ ✏ ✎ ✌ ✌ ✕ ✠ ✌ ✠ ☎ ☛ ✙ ✡ ✄ ☛ ✎ ✄ ✆ ✕ ✎ �✒ � ✓ ☞ ☞ ✔ ✖ ✏ ✖ ✗ ✗ ✚ ✞ ✘ ✛ ✑ ✎ ✍ ✆ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✠ ✍ ✡ ☎ ☛ ✞ ✠ ☞ ✌ ✄ ☎ ☎✆ ✝✡ ✄ ☎✆ ✝ ✞ ✆ ✟✠ ☎ ☛ ✁ ☎ ☞ ✁ ✂ ✁ ✌ ☎ ☛ ✂ � ✞ �✒ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ � Analysis Tools • static analysis tools can find some types of defects – compilers find undeclared variables, syntax errors – other static analysis tools may check for: ✓✖✕ ✌✛✚ • should inspect for things tools can’t find ✖✣✢ ✗✙✘

� ✴ ✩ ✯ ✩ ✦ ★ ✱ ✩ ✲ ✫ ✲✳ ✴ ✧ ✭ ✷ ✬ ✯ ✭ ✺ ✻ ✼ ✽ ✧ ✭ ✬ ✴ ✾ ✰ ✰ ✩ ✦ ✰ ✬ ✫ ✡ ✕ ☎ ✝ ✘ ✁ ✡ ✂ ✤ ✌ ☎ ✡ ✔ ✕ ✄ ☛ ✭✮ ✝ ✫✬ ✦✧ ✤ ☛ ☎ ✘ ☎ ✁ ✕ ✌ ✂ ✎ ✘ ✝ ★ ★ ✂ ❀ ✦ ✬ ❀ ✹ ★ ✿ ✦ ✬ ❂ ❂ ✩ ✦ ✰ ✰ ✭ ✭ ✫ ✿ ★ ✫ ✧ ✹ ✰ ★ ✽ ✩ ✯ ✸ ✩ ❂ ✩ ❀ ✽ ✰ ✩ ✰ ✩ ✯ ✩ ✦ ★ ✻ ✿ ❀ ✫ ✫ ✩ ✿ ✸ ❂ ✹ ✸ ✧ ✯ ✻ ✿ ★ ✩ ✴ ✴ ✩ ❁ ✬ ✭ ✾ ✌ ✎ ✎ ✤ ✛ ✄ ☎ ☎✆ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ �✒ � ☛ ✞ ✁ ✄ ✕ ✏ ✎ ☛ ✂ ✡ ☎ ✔ ✌ ✘ ✝ ☎ ✔ ✘ ✚ ☛ ☞ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ✡ ☎ ☛ ✞ ✠ ✌ ✗ ☞ ✗ ✖ ✖ ✕ ✔ ☞ ✓ ✍ � �✒ ✎ ✑ ✏ ✎ ✡ ✂ ✰ ✂ ✝ ✁ ☛ ✡ ✄ ✕ ✎ ✤ ☛ ✂ ✁ ✡ ✁ ✘ ✘ ✘ ☎ ✡ ✄ ✡ ✔ ✄ ☎ ✕ ✘ ✎ ✂ ☎ ✒ ✆ ✘ ☎ ✏ ✝ ✔ ✔ ✎ ✝ ✘ ✁ ✂ ☛ ☎ ✄ ✎ ✂ ✌ ✕ ☎ ☎ ✘ ✝ ✄ ✂ ☎ ☎ ✞ ✤ ✙ ✡ ☛ • to be classified as “inspection”, metrics must be kept ✸✶✹ Inspection Metrics ✸✶✹ ✖✣✢ ✸✶✹ ★✪✩ ✵✶✬ ✸✶✹ ✗✙✘ ✸✶✹ ★✪✩ ★✪✩ • may also record ★✪✩ ★✪✩ • record: ★✪✾ ★✪✩ ✎ ✣✥

� ✫ ✹ ✩ ✴ ❀ ✦ ✬ ✰ ✳ ✰ ✧ ✬ ✦ ✳ ✹ ✫ ✸ ✿ ✩ ★ ★ ✤ ✬ ★ ✩ ✳ ✫ ★ ✿ ✳ ✹ ✸ ✸ ★ ✦ ✩ ✾ ✿ ✳ ✹ ✬ ✰ ✫ ✲ ✬ ✩ ✾ ✿ ✩ ✴ ✸ ★ ✯ ✩ ★ ✫ ✧ ★ ✹ ✩ ✁ ✭ ✩ ✹ ✬ ✿ ✧ ✩ ✸ ★ ✸ ❁ ✸ ★ ✦ ★ ✹ ✹ ✩ ✭ ✩ ✯ ✯ ✸ ✰ ✦ ✻ ✻ ✬ ✩ ✦ ✬ ✭ ✹ ✩ ✴ ✾ ❂ ✿ ✩ ❁ ✩ ✰ ✿ ★ ✹ ✩ ✿ ✻ ✩ ✫ ✿ ✩ ✦ ✬ ✭ ✾ ✿ ✩ ✹ ✩ ✧ ✽ ✦ ✧ ❀ ❂ ✧ ❁ ✴ ❁ ✵ ✩ ✿ ✹ ✸ ★ ✹ ✩ ✾ ✿ ✴ ✩ ✸ ★ ✮ ✯ ✸ ★ ✿ ❀ ✾ ✦ ✬ ✫ ✭ ✾ ✴ ✸ ★ ✿ ✩ ✫ ❀ ★ ✿ ✻ ✫ ★ ✿ ✬ ✫ ✹ ✸ ✾ ✦ ✫ ✧ ✿ ★ ✦ ✩ ✯ ✩ ✰ ❂ ★ ✩ ✬ ★ ✻ � �✒ ✚ ✞ ✠ ✾ ✭ ✚✜ ✬ ❀ ✩ ✸ ★ ✭ ✩ ✾ ✭ ✩ ❂ ✰ ✬ ✦ ✯ ✬ ✩ ✹ ✸ ✚ ✘ ✸ ✡ ✎ ✍ ✌ ☞ ✠ ✞ ☛ ☎ ✠ ✑ ✆ ✟ ✞ ✝ ☎✆ ☎ ✄ ✂ ✏ ✎ ✄ ✗ ✞ ☎✆ ☎ ✄ ✛ ✘ ✞ ✚ ✗ �✒ ✖ ✖ ✕ ✔ ☞ ☞ ✓ � ✹ ✴ ★ ✥✦ ✦ ✩ ✯ ✩ ✰ ✯ ✬ ✧ ✲ ✿ ✫ ✲ ✩ ✤ ✿ ✩ ✫ ✻ ✧ ★ ✯ ✿ ✬ ★ ✿ ✮ ★ ✧ ✦ ✪ ✳ ✹ ✸ ✬ ★ ✦ ✩ ✾ ✿ ✮ ✰ ★ ✹ ❀ ★ ✿ ★ ✩ ★ ✹ ✩ ✯ ✩ ✰ ✯ ✬ ✫ ✯ ✧ ★ ✹ ✩ ✦ ✭ ✩ ✾ ✿ ✦ ✬ ✰ ✩ ✯ ✯ ✸ ✩ ✭ ✧ ★ ✰ ✹ ❀ ✸✶✹ ★✪✩ ✸✶✹ ★✪✩ ★✪✾ ✸✶✹ ✸✶✹ ✩ ✶✹ ✖✣✢ ★✪✩ Use of Metrics • Data collected can tell you: ✸✶✹ ✗✙✘ • Data can be used to: ✸✶✹ ✸✶✹ ★✪✩

� ☎ ✕ ✖ ✖ ✗ ✗ ✚ ✞ ✘ ✛ ✄ ☎✆ ☞ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ �✒ � ✂ ✔ ☞ ✓ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ✡ ☎ ☛ ✞ ✠ ☞ ✌ ✍ ✎ ✏ ✑ ✎ �✒ � Inspection Process from Software Inspection , Tom Gilb & Dorothy Graham ✖✣✢ ✗✙✘

� ✴ ✸ ✹ ✫ ★ ✽ ✩ ✾ ✭ ✬ ✰ ❀ ✦ ★ ✻ ✧ ✮ ✧ ✱ ✭ ✬ ✽ ★ ❀ ✩ ✮ ✰ ❀ ❂ ✦ ✹ ✸ ✁ ❂ ✺ ★ ✿ ✻ ✺ ✹ ✬ ✸ ✦✧ ✩ ✸ ✯ ✸ ★ ✭ � ✾ ✹ ✧ ✰ ✹ ✧ ✷ ✫ ✹ ✸ ✧ ✰ ✩ ✭ ✩ ✴ ✸ ★ ✿ ✳ ✳ ✰ ✝ ✻ ✦ ✽ ✹ ✸ ✦✧ ❂ ✞ ✹ ✬ ✼ ❂ ✩ ✫ ✩ ✬ ✦ ❀ ✰ ✬ ✭ ✾ ★ ✩ ✫ ★ ✧ ❂ ✩ ✭ ★ ✽ ❂ ✽ ✧ ✭ ★ ★ ✭ ✬ ✿ ✫ ✻ ✭ ✩ ✰ ✧ ✩ ✫ ✆ ★ ✿ ✹ ✬ ✝ ✭ ✬ ✩ ✭ ✄ ❀ ✬ ✦ ✮ ✧ ✰ ✦ ✩ ✩ � ✘ ✛ ✄ ☎ ☎✆ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ �✒ ✻ ✚ ✬ ✭ ✫ ✧ ✹ ✩ ✿ ✬ ✴ ✧ ✹ ✿ ✫ ✩ ✞ ✗ ✿ ☞ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ✡ ☎ ☛ ✞ ✠ ✌ ✗ ✍ ✎ ✏ ✑ ✎ �✒ � ✓ ☞ ☞ ✔ ✕ ✖ ✖ ✿ ✧ ✾ ❁ ✸ ✦ ✩ ✾ ✿ ✩ ✧ ❂ ✽ ✰ ❂ ❀ ✩ ✽ ✧ ★ ✻ ✰ ✭ ❀ ✬ ✦ ✮ ✧ ★ ✭ ❂ ❀ ✴ ✱ ✫ ✧ ✿ ✬ ✿ ✬ ✫ ✻ ✿ ✿ ✩ ✦ ✭ ✩ ✾ ✹ ✬ ✸ ★ ✦ ✰ ✴✬ ★ ✴ ★ ✩ ✩ ✿ ✧ ✭ People Involved in Inspection • Inspection is a team process: ideally 2-6 people ✵✶✬ ✸ ☎✄ ✖✣✢ ✸✶✹ ✸✶✹ ✗✙✘ ✸✶✹ ✩ ✶✹ • checkers (inspectors) • Inspection Leader: ✸✶✹ ✸✶✹ ✸✶✹ ★✪✩ ✸ ✂✁ ★✪✩

� ✫ ✭ ✭ ✬ ✭ ✿ ✻ ✩ ✲ ✲✳ ✫ ✿ ✬ ❀ ✭ ✦ ✩ ✦ ✬ ✩ ✾ ✩ ✫ ✂ ✾ ✸ ❂ ✩ ✻ ✩ ✲ ✲✳ ✮ ★ ✬ ✬ ✴ ✧ ✹ ✮ ★ ✰ ✰ ✦ ✩ ✹ ✁ ★ ✼ ✧ ✿ ✿ ✾ ✦ ✰ ★ ✬ ✭ ✿ ★ ✸ ✴ ✩ ✬ ✻ ✬ ✾ ✩ ✿ ✹ ✬ ★ ✦ ✬ ✴ ✸ ✮ ❂ ✩ ✦ ❂ ✩ ✧ ✹ ❂ ✬ ✴ ✭ ✗ ✓ ☞ ☞ ✔ ✕ ✖ ✖ ✗ �✒ ✚ ✞ ✘ ✛ ✄ ☎ ☎✆ ✞ � ✎ ✚✜ ✡ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ☎ ✑ ☛ ✞ ✠ ☞ ✌ ✍ ✎ ✏ ✄ ✘ ✚ ✿ ✯ ✭ ✬ ✩ ✭ ★ ✩ ★ ✭ ✸ ✹ ✫ ✭ ✬ ✻ ★ ✧ ✬ ✯ ✩ ✠ ✯ ✞ ✚ �✒ � ✩ ✻ ★ ✧ ✩ ✯ ✬ ✭ Entry • entry criteria set up for products to be inspected • a lot of controversy about when to inspect code ★✪✩ ★✪✩ • if entry criteria not met, product not inspected ✸✶✹ • inspections are expensive ★✪✩ ✸✶✹ • author offers up product for inspection • leaders decides if it's ready -- quick examination ✖✣✢ ✗✙✘

� ☎ ✕ ✖ ✖ ✂ ✗ ✚ ✞ ✘ ✛ ✄ ☎✆ ☞ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ �✒ � ✔ ✗ ☞ ☛ ✄ ☎ ☎✆ ✝ ✞ ✟ ✓ ✠ ✡ ☎ ✆ ✞ ✏ ✠ �✒ ✎ ✑ � ✎ ✍ ✌ ☞ Kickoff • initial meeting for all participants • organized by leader • familiarize everyone involved with task at hand • define everyone’s roles • hand out materials needed • deal with general questions • set goals • discuss overall plan, timetable • long documents divided into "chunks": separate inspection & logging for each chunk ✖✣✢ ✗✙✘

� ✦ ✥ ✬ ✮ ✧ ✱ ✧✬ ✫ ✧ ✪ ✧ ✩ ✧★ ✦ ✥ ✁ ✓ ✒ ✛ ✞ ✱ ✮ ✛ ✩ ✮ ✧ ✭ ✧✬ ✫ ✧ ✦ ✪ ✧★ ✪ ✦ ✥ ✧ ✱ ✴✰ ✰ ✭ ✬ ✲ ✠ ✒ ✬ ✞ ✔ ✒ ✓ ✎✒ ✑ ✏ ✎ ✍ ✌ ☞ ✠ ☛ ☞ ☎ ✡ ✠ ✆ ✟ ✞ ✝ ☎✆ ☎ ✄ ✂ ✛✢ ☞ ✕ ✞ ✄ ☎ ☎✆ ✜ ✙ ✞ ✛ ✘ ✘ ✗ ✗ ✄ ✙ ✖ Individual Checking (1) • each checker works alone using documents handed out in kick-off meeting • aims to find maximum number of unique major potential defects • mainly by looking for discrepancies between source documents and products being inspected, i.e.: ✭✯✮ ✭✯✰ ✪✳✲ ✶✸✷ ✭✯✮ ✭✯✰ ✭✯✵ ✗✤✣ ✘✚✙

� ✆ ✬ ☎ ✵ ✰ ✮ ✮ ✪ ✶ ✪ ✁ ✪ ✫ ✵ ✷ ✴ ✭ ✥ ✰ ✭ ✰ ✪ ✧ ✥ ✱ ✧ ✮ ✴ ✦ ✵ ✷ ✭ ✪ ✰ ✬ ☎ ✶ ✰ ✴ ✵ ✮ ✂ ✥ ✝ ✪ ✬ ✭ ✧✬ ✭ ✞ ✮ ✬ ✩ ✲ ✲ ✧ ✮ ✭ ✪ ✰ ✰ ✰ ✮ ✧ ✪ ✆ ✶ ✧ ✱ ✧ ✝ ✴ ✪ ✭ ✥ ✞ ★ ✩ ✧ ✮ ✭ ✲ ✷ ✬ ✘ ✔ ☞ ☞ ✕ ✖ ✗ ✗ ✘ ✓ ✛ ✞ ✙ ✜ ✄ ☎ ☎✆ ✞ ✒ ✎✒ ✙ ✡ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ☎ ✑ ☛ ✞ ✠ ☞ ✌ ✍ ✎ ✏ ✄ ✛✢ ✧✬ ✩ ✭ ✧ ✪ ✮ ✂ ✄ ✰ ✦ ✶ ✵ ✧ ✦ ✱ ✰ ✴ ✩ ✫ ✷ ✦ ✷ ✒ ✴ ✪ ✵ ✁ ✭ ✥ ✛ ✠ ✞ ✛ ✒ ✓ Individual Checking (2) • usually uses a checklist • kickoff meeting suggests amount of time to spend: ✭✯✵ • checker records actual time taken • keeps notes of issues found • "issue" = matter requiring attention ✗✤✣ ✘✚✙

� ☎ ✖ ✗ ✗ ✓ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✘ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✆ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ Logging Meeting (1) • 3 purposes: – log issues identified by each checker – work as group to find more issues – suggestions for process improvement (inspection process and overall software process) • controlled by moderator (usually the leader) – must be very skilled in handling people and issues – maintain time discipline and issue discipline • no evaluation of issues – just log issues, like brainstorming ✗✤✣ ✘✚✙

� ✚✜ ✁ ✁✒ ✚ ✞ ✠ ✚ ✘ ✥✦ ✄ ✞ ☎✆ ✁ ✄ ✛ ✤ ✧ ✞ ✥ ✥✯ ✩ ✧ ✭ ✭ ✮ ✩ ★ ★ ✪✭ ✪ ✬ ✪✫ ✩ ✘ ☎ ✚ ✞ ☎ ✎ ✍ ✌ ☞ ✠ ☛ ☎✆ ☎ ✡ ✠ ✆ ✟ ✞ ✄ ✝ ✂ ✖ ✗ ✗ ✏ ✑ ✎ ✁✒ ✁ ✓ ☞ ☞ ✔ ✕ ✖ Logging Meeting (2) • must not last more than 2 hours • not for explaining, defending, suggesting fixes, or training • identify issues, then log them • issues can be questions to product author and suggestions for improvements • NOT an attack or evaluation of product author • management personnel should NOT be in attendance ✖✣✢ ✗✙✘

� ✄ ✕ � ✖ ✗ ✗ ✚ ✝ ✘ ✛ ✂ ✄☎ ☛ ✝ ✂ ✘ ✚✜ ✚ ✟ ✝ ✚ ✑ ✒ ✑ ✔ ✖ ☛ ✡ ✁ ✂ ✄ ✄☎ ✆ ✝ ✞ ☎ ✓ ✠ ✄ ✟ ✝ ✎ ✑ ✒ ✟ ✏ ✍✑ ✍ ✌ ☞ ☛ Edit • log of issues given to author to resolve • author can – classify or reclassify issues (which are really defects) – voluntarily improve product as per issues raised – request rules or checklist be changed ✖✣✢ ✗✙✘

� ✄☎ ✖ ✁ ✗ ✗ ✚ ✝ ✘ ✛ ✂ ✄ ✝ ✔ ✂ ✘ ✚✜ ✚ ✟ ✝ ✚ ✑ ✒ ✑ ✕ ✖ ☛ ✝ ✂ ✄ ✄☎ ✆ ✝ ✞ ☎ ✟ ☛ ✄ ✡ ✠ ✟ ✏ ✓ ✑ ☛ ✍✑ ✒ ✎ ✍ ✌ ☞ Follow-up • inspection leader makes sure satisfactory closure has taken place on all issues – actions to correct all defects (change or change request) – suggestions for process changes reported • if major changes, product may need to be inspected again ✖✣✢ ✗✙✘

� ✄ ✕ � ✖ ✗ ✗ ✚ ✝ ✘ ✛ ✂ ✄☎ ☛ ✝ ✂ ✘ ✚✜ ✚ ✟ ✝ ✚ ✑ ✒ ✑ ✔ ✖ ☛ ✡ ✁ ✂ ✄ ✄☎ ✆ ✝ ✞ ☎ ✓ ✠ ✄ ✟ ✝ ✎ ✑ ✒ ✟ ✏ ✍✑ ✍ ✌ ☞ ☛ Exit • need exit criteria • need closure on all issues in writing • record metrics for inspection process (for process improvement) • product now considered “safe” to pass on to next phase in software development ✖✣✢ ✗✙✘

�✁ ☎ ✗ ✗ ✘ ✘ ✂ ✛ ✞ ✙ ✜ ✄ ☎✆ ✕ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✖ ☞ ✒ ☛ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ✡ ☎ ✞ ☞ ✠ ☞ ✌ ✍ ✎ ✏ ✑ ✎✒ ✓ ✒ ✔ The V-model Requirem ents System Test Inspection Architecture Integration Test Inspection Unit Test Design Inspection Inspection Code Cost of running inspections is about 10-15% of cost of development budget. ✗✤✣ ✘✚✙

� ☎ ✖ � ✗ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✔ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ Costs of Inspection • Inspection takes time • Typically 10-15% of development budget • Also start-up costs: – establishing procedures – training leaders, inspectors • Is it worth it? ✗✤✣ ✘✚✙

� ☎ ✖ � ✗ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✔ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ Benefits of Inspection (1) • Defects found early save time and money • many defects found before testing – testing is much quicker • total development time is reduced – even when counting time for inspection • inspections increase productivity ✗✤✣ ✘✚✙

� ☎ ✖ ✗ ✁ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✠ ✡ ☎ ✆ ✞ ✏ ✒ ✠ ✎✒ ✑ ✓ ✎ ✍ ✌ ☞ Benefits of Inspection (2) • Defects found early result in better product • inspection results in – cleaner design – better documentation – better code – fewer defects – defects that remain will be easier to fix when found ✗✤✣ ✘✚✙

� ☎✆ ✗ ✗ ✂ ✘ ✛ ✞ ✙ ✜ ✄ ☎ ✞ ✕ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✖ ✘ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ☞ ✡ ☎ ☛ ✠ ✠ ✑ ✔ ☞ ✓ ✎✒ ✒ ✏ ✎ ✍ ✌ Benefits of Inspection (3) • feedback for mangement about software process: – inspector's comments plus metrics • defects found earlier means fewer deadline surprises • inspection trains inspectors: – learn from good code and documentation of others – learn from mistakes of others ✗✤✣ ✘✚✙

� ✓ ☎ ✄ ✂ ✂ ✁ � ✒ ✒ ✠ ✛ ✞ ✠ ✛ ✓ ✙ ✄ ✆ ✆ ☎✆ ☎ ✞ ☎ ✆ ✎✏ ✌✍ ✝ ✆ ✄ ✌✍ ✒ ✑✒ � ☛ ✞ ☎ ✆ ✎✏ ✞ ✛✢ ☎ ☎ ✏ ✎ ✍ ✌ ☞ ✠ ✞ ☛ ✡ ✄ ✠ ✆ ✟ ✞ ✝ ☎✆ ☎ ✄ ✂ ✑ ✎✒ ✓ ✒ ✜ ✙ ✞ ✛ ✘ ✘ ✗ ✗ ✖ ☞ ✕ ☞ ✔ ☛ Success Stories (1) • IBM Federal Systems Division – kept metrics for similar projects before and after introducing Inspection into development process – delivered lines of code per work-month: ✝✟✞ ✡☞☛ ✡☞☛ ✗✤✣ ✘✚✙

� ☎ ✕ ✖ ✁ ✗ ✗ ✚ ✞ ✘ ✛ ✄ ☎✆ ☞ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ ✁✒ ✁ ✔ ✖ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✓ ✠ ✡ ☎ ✆ ✞ ✏ ✁ ✠ ✎ ✑ ✁✒ ✎ ✍ ✌ ☞ Success Stories (2) • ICL (in U.K.): – 57.7% of defects found by Inspection – cost of finding defect by Inspection: 1.58 work hours – cost of finding without Inspection: 8.47 work hours – only 6% of development time devoted to Inspection ✖✣✢ ✗✙✘

� ☎ ✕ ✖ ✂ ✗ ✗ ✚ ✞ ✘ ✛ ✄ ☎✆ ☞ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ ✁✒ ✁ ✔ ✖ ☞ ☛ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✓ ✡ ☎ ✠ ✞ ✏ ✁ ✠ ✎ ✑ ✁✒ ✎ ✍ ✌ ☞ Success Stories (3) • Large IBM project: – 500,000 lines of code (networked operating system) – based on past experience without inspection, expected to find about 800 bugs during tests at trial site – used inspection at 11 stages of devopment – found 8 bugs at trial site! ✖✣✢ ✗✙✘

� ✄ � ✖ ✖ ✗ ✗ ✚ ✝ ✘ ✛ ✂ ✄☎ ☛ ✝ ✂ ✘ ✚✜ ✚ ✟ ✝ ✚ ✑ ✒ ✑ ✔ ✕ ☛ ✡ ✁ ✂ ✄ ✄☎ ✆ ✝ ✞ ☎ ✟ ✓ ✄ ✠ ✝ ✎ ✑ ✒ ✍✑ ✟ ✏ ✍ ✌ ☞ ☛ Software Inspection Research (1) • Research into inspection – inspection process – techniques for individual inspectors ✖✣✢ ✗✙✘

� ✍ ☞ ✠ ☎ ✏ ✛ ✝ ✑ ☞ ✗ ✌ ✖ ✠ ✠ ✎ ✎ ☛ ✠ ✎ ✜ ✠ ✟ ✌ ☎ ☛ ✂ ✒ ✑ ✍ ☛ ✡ ☛ ✂ ✂ ☛ ☎ ✟ ✕ ☞ ✒ ✄ ✆ ✝ ✝ ✟ ✓ ✕ ✘ ☎ ✝ ✔ ✝ ✝ ✎ ✚ ✍ ✌✝ ✂☞ ☛ ✔ ✂ ✝ ✡ ✟ ✠ ✡ ✔ ✝ ✙ ☛ ✝ ✝ � ✍ ✠ ✔ ☞ ✟ ✠ ✎ ✌✝ ✓ ☞ ✂ ☛ ✝ ✓ ✎ ✎ ✝ ✑ ✔ ✟ ✠ ✕ ✗ ✑ ✄ ☛ ☛ ✜ ☛ ✔ ✂ ☛ ✟ ✢ ✕ ✂ ✝ ✍ ✂ ✠ ☛ ✎ ✍ ✌ ☎ ✂☞ ☛ ✂ ☛ ☞ ✎ ✠ ✝ ✝ ✠ ✓ ✎ ✒ ✂ ☛ ✜ ✟ ☞ ✌ ☎ ☛ ☎ ✠ ✟ ✡ ✂ ✑ ✎ ✑ ✛ ✂ ✄ ✄☎ ✝ ✂ ✘ ✚✜ ✚ ✟ ✝ ✚ ✑ ✒ ✁ ✝ ✌✝ ✏ ✂ ✟ ✠ ✎ ✍ ✂☞ ✂ ☛ ✡ ✠ ✟ ☎ ✄ ✘ ✚ ✄ ☛ ✁ ✂ ✄ ✄☎ ✆ ✝ ✞ ☎ ✟ ✠ ✄ ✡ ✝ ✟ ☞ ✗ ☛ ✗ ✖ ✖ ✕ ✔ ☛ ✓ ✌ ✑ ✒ ✍✑ ✏ ✎ ✍ ✂ ☞ ☎ ✂✒ ✑ ✟ ✑ ✌ ✂ ☛ ☛ ✗ ✖ ✟ ✠ ✕ ☞ ☎ ✗ ✝ ✝ ✕ ☛ ✂✒ ☛ ✖ ✟ ✠ ☞ ✗ ☎ ✑ ✝ ✎ ✔ ✂ ✑ ✑ ✎ ✏ ☞ ✎ ✍ ✌✝ ☞ ✂ ☛ ☎ ✂✒ ✑ ✝ ✎ ✡ ✠ ✟ ☛ ✖ ☞ ✍ ✝ ✎ ✔ ✠ ✓ ✝ ✑ ☎ ✝ • research at Lucent Technologies (University Software Inspection – experiment where they varied ✖✣✢ ✆✞✝ Research (2) – one of the conclusions ✗✙✘ of Maryland) ✆✞✝ ✆✞✝

�✁ ☛ ✜ ✝ ✑ ☞ ✝ ✟ ✝ ✓ ☞ ✠ ✡ ✦ ✄ ✝ ✑ ✏ ✎ ✠ ✓ ✝ ✟ ✠ ✗ ✝ ✡ ✟ ✑ ✎ ✝ ☞ ✎ ✝ ☞ ☛ ✂ ✢ ✕ ☛ ✎ ✓ ✑ ☞ ✠ ✟ ✄ ☛ ✂ ☞ ✌✝ ✍ ✖ ✠ ✠ ✍✝ ✂ ☞ ✑ ✂ ✔ ☞ ✂ ✕ ✑ ✟ ☛ ✠ ☞ ✎ ✠ ✎ ☛ ✂ ☞ ✑ ✎✒ ✓ ✒ ✔ ☞ ✕ ✎ ✖ ✗ ✗ ✘ ✘ ✛ ✏ ✍ ✙ ✆ ✄ ☎ ☎✆ ✝ ✞ ✟ ✠ ✌ ✡ ☎ ☛ ✞ ✠ ☞ ✞ ✜ ✡ ✄ ✗ ✝ ☞ ✟ ✄ ✠ ✢ ✎ ✄ ✌ ✥ ✒ ✝ ✏ ✎ ✛ ✄ ☎ ☎✆ ✓ ✒ ✛ ✞ ✠ ✒ ✞ ✛✢ ✙ • new techniques to help guide inspectors in Software Inspection ✗✤✣ – scenario-based reading Research (3) ✘✚✙ their work

� ☎ ✖ ✗ ✗ � ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✘ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✆ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ Checklists • common guidance used for inspections • good checklists are hard to make up – rely on past experience – can get too long and unwieldy – need to be updated as new situations arise – inspectors have a tendency to follow the checklist and do no more – can be misinterpreted – may not cover everything necessary to the right level of detail ✗✤✣ ✘✚✙

� ✮ ✰ ✭ ✦✴ ✮ ✳ ★ ✬ ✪ ✯ ✬ ✬ ✭ ✹ ✪ ✪ ✦ ✥ ✧ ✧ ✁ ✒ ✴ ★ ✒ ✷ ✳ ✪ ✧ ✬ ✺ ★ ✬ ✮ ✦ ✭ ✵ ✹ ✪ ★ ✭ ✭ ✭ ✦✱ ✷ ✶ ✪ ✮ ✓ ✛ ✞ ✒ ✓ ✎✒ ✑ ✏ ✎ ✍ ✌ ☞ ✠ ☛ ☞ ☎ ✡ ✠ ✆ ✟ ✞ ✝ ☎✆ ☎ ✄ ✂ ✔ ☞ ✞ ✄ ✠ ✛ ✪ ✛✢ ✙ ✄ ✞ ☎✆ ✕ ☎ ✜ ✙ ✞ ✛ ✷ ✘ ✘ ✗ ✗ ✖ ✬ Approach for Checklists • (general approach for inspections as well) • determine what qualities are important for the software system • determine what software design/code characteristics contribute to those qualities • determine what you want the inspectors to look for ✧✩★ ✫✩✬ ✰✲✱ ✸✩✬ ✫✩✬ • checklist used for Assignment 4 is based on McCall's Quality Model ✗✤✣ ✘✚✙

McCall's Quality Model

� ☎ ✖ ✗ ✗ ✁ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✘ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✠ ✡ ☎ ✆ ✞ ✏ ✒ ✠ ✎✒ ✑ ✓ ✎ ✍ ✌ ☞ Testing • Now we move to the subject of testing • Recall that software process should include both testing and inspection • Inspecting documents & code before testing results in fewer defects for testing to find – speeds up testing • A bit of informal testing before code inspection might save time – get rid of obvious bugs quickly ✗✤✣ ✘✚✙

�✁ ☎✆ ✖ ✖ ✂ ✗ ✚ ✞ ✘ ✛ ✄ ☎ ✞ ✔ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ ✒ ✁ ✒ ✕ ✗ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ☞ ✡ ☎ ☛ ✠ ✠ ✑ ✓ ☞ ✁ ✎✒ ✒ ✏ ✎ ✍ ✌ Testing is Hard! • hits budget and schedule limits • may include heavy debugging – may even include product (re-)development • bugs slip through to customers – user executes untested code – order in which statements are executed in actual use differ from that in testing – user applies combination of untested input values – user’s operating environment is never tested ✖✣✢ ✗✙✘

� ✟ ✡ ✡ ✆ ✒ ☞ ✒ ✖ � ☎ ✝ ✁ ✞ ✗ ✝ ✝ ✁ ✞ ✟ ✎ ✢ ✁ ☎ ✞ ✟ ☞ ✁ ✦ ☞ ✓ ✟ ✓ ✟ ✙ ✓ ✟ ✁ ✞ ✁ ✡ ✙ ✆ ✝ ✡ ✤ ✁ ✍ ✆ ✟✓ ✁ ✆ ✝ ✍ ✆ ✒ ✌ ✡ ✝ ✥ ✛ ✓ ✞ ✗ ✡ ✝ ✁ ✞ ✙ ✟ ✁ ✁ ☞ ✙ ✟ ✁ ✙ ✁ ✞ ✥ ☞ ✒ ✁ ✆ ✝ ✗ ☞ ✎ � ✝ ✗ ✁ ✝ ✝ ✒ ☞ ✡ ✞ ✆ ✍ ✟ ✡ ✙ ✁ ✎ ✡ ☞ ✁ ✥ ✁ ✟ ✖ ✜ ✡ ✁✝ ✒ ✆ ✒ ☞ ✒ ✓ ✛ ✗ ✟ ✆ ✒ ☞ ✒ ✖ ✟ � ✆ ✝ ✟ ✙ ✜ ✡ ✝ ✁ ✤ ✡ ✡ ✓ ✁ ✡ ✒ ✝ ✞ ✁ ✥ ✧ ☞ ✞ ✎ ✤ ✡ ✛ ✆ ✗ ✡ ✆ ✒ ✆ ☞ ✁ ✆ ✁✝ ✆ ✂ ☎ � ✒ ✒ ✞ ✚ ✞ ✠ ✚ ✚✜ ✘ ✄ ✁ ✝ ☎✆ ✞ ✟ ☎ ✡ ✆ ✡ ✍ ✁✎ ✟ ✟ ✡ ✍ ✌ ☞ ✁ ✞ ✆ ✞ ☎ ✍ ✠ ✌ ☞ ✠ ✞ ☛ ☎ ✡ ✆ ✎ ✟ ✞ ✝ ☎✆ ☎ ✄ ✂ ✍ ✏ ✄ ✖ ✛ ✘ ✞ ✚ ✗ ✗ ✖ ✕ ✑ ✔ ☞ ☞ ✓ ✒ ✁ ✎✒ ✎ ✞ ✁ � ✡ ✆ ✒ ☞ ✒ ✖ ✟ ✆ ✁ ✜ ✡ ✝ ✡ ✞ ✚ ✆ ✝ ☞ ✝ ✛ ✒ ✖ ✟ � ✁✝ ✝ ✟ ✆ ✁ ✁✝ ✆ ✙ ✡ ✡ ☎ ✆ ✛ ✞ ✟ ✞ ✌ ✕ ☎ ✆ ✝ ✁ ✆ ✟ ☎ ✞ ✑ ☞ ✁ ✞ ✏ ✝ ✁ � ✂ ✆ ✒ ✙ ✆ ✁✝ ✙ ☎ ✠☛✡ ✒✔✓ • testers who are developers must have Skills of a Tester (1) ✖✣✢ • Not all testers are developers ✒✔✛ ✗✙✘ ✠☛✡ ✒☛✦ ✖✘✗ ✒✔✓ ✆✣✢ ✆✣✢ ✆✣✢ ✆✣✢ ✒✔✓ ✁✄✂ ✁✄✂

� ✡ ✁ ✁ ✞ ☎ ✥ ✛ ✤ ✒ ✎ ✗ ✆ ✁ ✞ ✎ ✓ ☎ ✡ ✡ ✁ ✁ ✝ ✡ ✜ ✝ ✜ ✓ ✆ ✌ ✟ ✞ ✁ ✁ ✞ ☎ ✞ ✓ ☞ ✁ ✥ ✁ ☞ ✦ ✡ ✁ ✌ ☞ ✁ ✙ ☎ ✁ ✤ ✆ ✞ ✎ ✟ ✤ ✤ ☎ ✥ ✡ ✁ ✝ ✞ ✟ ✆ ☎ ✝ ✢ ✗ ☞ ✁ ☎ ✛ ✍ ✘ ✘ ✗ ✄ ✗ ✖ ✕ ☞ ☞ ✔ ✒ ✓ ✎✒ ✙ ☛ ✞ ✟ ✆ ✠ ✡ ☎ ✞ ✑ ✠ ☞ ✌ ✍ ✎ ✏ ✞ ✜ ☎✆ ✄ ✆ ☎ ✂ ✙ ✟✓ ✆ ✝ ✞ ✁ ✙ ✓ ✗ ✥ ✝ ✒ ✓ ☎ ☎✆ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✒✔✓ Skills of a Tester (2) • More skills employers look for: ✗✤✣ – software engineering skills – hands-on experience – communication skills ✒✔✓ ✘✚✙ – organizational skills ✒✔✓ ✒✔✓

� ☎ ✖ ✗ ✁ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✠ ✡ ☎ ✆ ✞ ✏ ✒ ✠ ✎✒ ✑ ✓ ✎ ✍ ✌ ☞ Attitude For Testers • destructive creativity • detective skills • understanding the product as the sum of its parts • appreciating the customer’s perspective • requirements change • skeptical but not hostile attitude • ability to be the bearer of bad news and remain objective • an eagerness to embrace new technologies ✗✤✣ ✘✚✙

� ☎ ✖ ✗ ✗ � ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✘ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✠ ✡ ☎ ✆ ✞ ✏ ✒ ✠ ✎✒ ✑ ✓ ✎ ✍ ✌ ☞ Definitions of Testing (1) • Myers, Glenford J., The Art of Software Testing, Wiley, 1979 – “The Process of executing a program with the intent of finding errors” • Hetzel, Bill, A Complete Guide to Software Testing, QED Information Sciences Inc., 1988 – "Testing is any activity aimed at evaluating an attribute or capability of a program or system and determining that it meets its required results." ✗✤✣ ✘✚✙

� ☎ ✖ ✗ ✗ � ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✘ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✆ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ Definitions of Testing (2) • IEEE Standard 610.12-1990 – “The process of operating a system or component under specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component” • IEEE Standard 829-1983 – “The process of analyzing a software item to detect the differences between existing and required conditions (that is , bugs) and to evaluate the features of the software items” ✗✤✣ ✘✚✙

� ☎✆ ✗ ✗ ✘ ✂ ✛ ✞ ✙ ✜ ✄ ☎ ✞ ✕ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✖ ✘ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ☞ ✠ ☎ ☛ ✡ ✠ ✑ ✔ ✒ ☞ ✎✒ ✓ ✏ ✎ ✍ ✌ Purposes of Testing • To find bugs - including bugs in the tests! • To estimate reliability in use by tracking the execution time interval between failures • To estimate defects remaining by tracking the defects found per person-hour of testing and debugging time • To decide on when to release: e.g. by deciding that the remaining known defects are acceptable in the target market • To learn where process problems are by classifying and counting defects found ✗✤✣ ✘✚✙

� ☎✆ ✗ ✗ ✘ ✂ ✛ ✞ ✙ ✜ ✄ ☎ ✞ ✕ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✖ ✘ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ☞ ✠ ☎ ☛ ✡ ✠ ✑ ✔ ✒ ☞ ✎✒ ✓ ✏ ✎ ✍ ✌ What Testing Can't Do • Testing can't guarantee code is correct • You can't try every possible combination of inputs, every environment • Rigorous testing increases confidence that software runs correctly, or that few bugs remain • For some critical software, formal mathematical methods used to "prove" correctness – but that's another topic for other courses ✗✤✣ ✘✚✙

� ☎ ✖ ✗ � ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✠ ✡ ☎ ✆ ✞ ✏ ✒ ✠ ✎✒ ✑ ✓ ✎ ✍ ✌ ☞ Stages of Testing (1) • Requirements and Design testing – when requirements and design are written in special languages which permit "execution" to simulate actual system • User Interface testing – Often done at prototyping stage • Unit/Module Testing • Component/subsystem Testing ✗✤✣ ✘✚✙

�✁ ☎✆ ✗ ✗ ✘ ✂ ✛ ✞ ✙ ✜ ✄ ☎ ✞ ✕ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✖ ✘ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ☞ ✠ ☎ ☛ ✡ ✠ ✑ ✔ ✒ ☞ ✎✒ ✓ ✏ ✎ ✍ ✌ Stages of Testing (2) • Integration Testing – Incremental Testing -the usual version of integration testing. • Daily builds and Sanity testing • System Testing – Performance and Load/Stress Testing – Usability Testing, but this is a late stage to be doing it! ✗✤✣ ✘✚✙

� ☎ ✖ ✓ ✗ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✔ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ Stages of Testing (3) • Alpha and Beta Testing – alpha = testing full system in-house – beta = outside users • Acceptance Testing • Installation/Compatibility Testing • Platform/Configuration/Port testing ✗✤✣ ✘✚✙

� ✠ ✡ ✑ ✞ ✏ ✂ ✂ ✞ ☞ ☎ ✟ ✞ ✑ ✞ ✓ ✠ ✂ ✑ ✌ ✌ ✝✞ ☞ ✠ ✝✞ ✂ ✒ ✟ ✌ ☞ ✓ ✂ ✑ ✡ ✞ ✎ ✂ ✠ ✤ ✣ ✢ ✜✢ ✌ ✞ ✡ ✡ ✝ ☞ ✠ ✌ ✞ ✒ ✝✞ ✟ ✠ ✗ ☞ ✍ ✥ ✞ ✑ ✆ ☞ ✠ ✡ ✓ ✂ ✠ ✞ ✝ ☞ ✓ ✑ ✛ ★ ✬✭ ✫ ✙ ✥ ✡ ✁ ✤ ✑ ✩ ✪ ✩ ★ ✛ ✧ ✏ ✂ ✞ ✠ ✌ ✞ ✠ ☞ ✆ ✌ ✌ ✠ ✞ ✠ ✗ ✂ ☎ ✂ ✛ ✦ ☎ ✡ ✓ ✂ ✟ ✂ ☞ ✠ ☎ ✂ ☞ ✓ ☞ ✠ ✑ ✞ ☎ ✟ ✁ ✂ ✑ ✡ ✁ ✟ ✌ ✡ ✠ ✓ ✞ ☎ ✟ ✡ ✝ ✑ ✂ ✒ ✛ ✞ ✛ ✒ ✓ ✒ ✛ ✞ ✠ ✛✢ ☎ ✙ ✄ ✞ ☎✆ ☎ ✄ � ✆ ✙ ✟✌ ✍ ✠ ✞ ✠ ✌ ☞ ✂ ✝✞ ☎ ✟ ✂ ☞ ✠ ✟ ✜ ✞ ✠ ✆ ✠ ✞ ☛ ☎ ✡ ✠ ✟ ✌ ✞ ✝ ☎✆ ☎ ✄ ✂ ☞ ✍ ✛ ☞ ✘ ✘ ✗ ✗ ✖ ✕ ☞ ✎ ✔ ✒ ✓ ✎✒ ✑ ✏ ☞ ✟ ✞ ✞ ✑ ✆ ✌ ✑ ✌ ✒ ☞ ✟ ✞ ✠ ✞ ✟ ✑ ✑ ✡ ✌ ☞ ✂ ✟ ☞ ✌ ✍ ✎ ✡ ✠ ✗ ✡ ☎ ✠ ☞ ✗ ✠ ✞ ✂ ✁ ✆ ☎ ✠ ✒ ✝✞ ✌ ✁ ☞ ✡ ✟ ✡ ✂ ✆ ✠ ☎ ☞ ✍ ✁ ✞ ☞ ✠ ✞ ✓ ✟ • Security/Fault Tolerance/Recovery testing ✠☛✡ ✁✚✙✛ Stages of Testing (4) • Reliability/Certification Testing ✗✕✘ • Integrity and Release testing ✎✄✏ ✗✤✣ ☞✖✌ ✎✕✔ ✘✚✙ • Regression Testing ✗✄✞ ✗✄✏ ✠☛✡ ✁✕✔ ✁✄✞ ✁✄✂

�✁ ☎✆ ✂ ✗ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎ ✞ ✕ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✖ ✗ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ✡ ☞ ☛ ☎ ✠ ✑ ✔ ✒ ✓ ☞ ✎✒ ✏ ✎ ✍ ✌ Black Box Testing • pick test cases based on requirements only • don't use knowledge of implementation • view code as a black box: insides not visible • can be used at any stage: – unit, subsystem, system testing ✗✤✣ ✘✚✙

�✁ ☎✆ ✗ ✗ ✂ ✘ ✛ ✞ ✙ ✜ ✄ ☎ ✞ ✕ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✖ ✘ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ☞ ✡ ☎ ☛ ✠ ✠ ✑ ✔ ☞ ✓ ✎✒ ✒ ✏ ✎ ✍ ✌ Black-Box Testing Strategies • look at specifications for inputs • test normal use: – sampling of "normal" inputs (random values) • test abnormal use: – different kinds of erroneous/illegal input – combinations of errors • test boundary conditions: – values close to boundary between normal and abnormal ✗✤✣ ✘✚✙

� ☎ ✖ ✗ ✗ ✘ ✘ � ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✛ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✔ ✟ ✆ ✠ ✡ ☎ ✞ ✞ ✒ ☞ ✌ ✍ ✠ ✎ ✏ ✑ ✎✒ ✓ Example /** * Raises a number to an integral power. * * @param base the number to be raised to a power * @param pow the power to which base is to be raised. * Must not be negative. * @return base raised to the pow-th power */ static double power(double base, int pow) { • error inputs: negative values for pow • boundary cases: pow = 0, pow = 1, pow = -1 • normal cases: – pow >= 2, range of values – base: negative, 0, positive – combinations of these ✗✤✣ ✘✚✙

� ☎ ✖ ✗ � ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✔ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ Possibilities For Automation • Ideal: tool to generate set of test cases and expected results from specifications • Reality: needs very precise, mathematical specifications, usually not available • So usually generate test cases & expected results by hand • Automated execution is still possible ✗✤✣ ✘✚✙

� ☎ ✖ ✗ ✗ � ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✘ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✠ ✡ ☎ ✆ ✞ ✏ ✒ ✠ ✎✒ ✑ ✓ ✎ ✍ ✌ ☞ Test Harness • Takes set of test inputs & expected results • Runs program on each test input, compares results with expected • Reports any discrepancies • Complicated to write for GUIs, embedded systems, but often possible • Takes lots of drudgery out of testing • Makes testers willing to run and re-run large test suites ✗✤✣ ✘✚✙

� ☎ ✖ � ✗ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ✔ ☎ ✡ ✞ ✏ ✒ ✓ ✎✒ ✠ ✑ ✎ ✍ ✌ ☞ White Box Testing • Pick test cases based on the actual software • also called "clear box" or "structural" testing • key word is coverage : – do the test cases cover all of the code? • different measures of coverage ✗✤✣ ✘✚✙

�✁ ☎✆ ✗ ✂ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎ ✞ ✕ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✖ ✗ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ☞ ☎ ☛ ✡ ✠ ✑ ✔ ✒ ☞ ✎✒ ✓ ✏ ✎ ✍ ✌ Statement Coverage • Every statement in the software must be executed at least once by the test cases • Quote from text (Murray): – "Testing less than this for new software is unconscionable and should be criminalized" • NASA story ✗✤✣ ✘✚✙

� ☎ ✖ ✗ ✁ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✠ ✡ ☎ ✆ ✞ ✏ ✒ ✠ ✎✒ ✑ ✓ ✎ ✍ ✌ ☞ Branch Coverage • Every branch alternative must be covered by at least one test case • More demanding than statement coverage: if (x > 3) y = 14; // continue -- no else part • We need at least one test case where x > 3 is and at least one where x > 3 is false. ✗✤✣ ✘✚✙

� ✓ ✙ ✜ ✄ ☎ ☎✆ ✞ ✄ ✓ ✛✢ ✛ ✠ ✞ ✛ ✒ ✒ ✛ � ☎ ✆ ✆ � ✝ ✞ ✝✟ � ✠ ✡ ☛ � ☞ ✞ ✙ ☛ ✌ ☛ ✞ ✠ ✠ ✆ ✟ ✞ ✝ ☎✆ ☎ ✄ ✘ ✡ ☞ ☎ ✍ ✓ ✘ ✗ ✗ ✖ ✕ ☞ ☞ ✔ ✒ ✂ ✎ ✎✒ ✑ ✏ Path Coverage • Every possible path through the program must be covered by at least one test case void tst(int x) { if (x>0) pos = pos+1; if (x%2==0) even = even+1; return; √ √ ✁✄✂ } • Needs 4 test cases: √ √ ✁✄✂ – all combinations alternatives ✗✤✣ ✘✚✙

�✁ ☎ ✕ ✖ ✖ ✂ ✗ ✚ ✞ ✘ ✛ ✄ ☎✆ ☞ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ ✁ ✒ ✁ ✔ ✗ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✓ ✠ ☎ ☛ ✡ ✠ ✑ ✁ ✒ ☞ ✎ ✁ ✏ ✎ ✍ ✌ Example void tst(int x) { if (x>0) pos = pos+1; if (x%2==0) even = even+1; } • Statement Coverage: tst(2) • Branch Coverage: tst(-1),tst(2) • Path Coverage: tst(-2),tst(-1),tst(1),tst(2) ✖✣✢ ✗✙✘

� ☎ ✕ � ✖ ✗ ✗ ✚ ✞ ✘ ✛ ✄ ☎✆ ☞ ✞ ✄ ✘ ✚✜ ✚ ✠ ✞ ✚ ✁ ✒ ✁ ✔ ✖ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✓ ✡ ☎ ✠ ✞ ✏ ✁ ✒ ✠ ✎ ✑ ✁ ✎ ✍ ✌ ☞ What About Loops? • May require test cases which make each loop execute 0 times, 1 time, many times • for (int i = 0; i < n; i++) {...} – Test cases should include: • n <= 0 (no times around the loop) • n = 1 (one time around the loop) • n > 1 (several times around the loop) ✖✣✢ ✗✙✘

� ☎ ✁ ✗ ✗ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✖ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ✔ ☎ ✡ ✞ ✏ ✒ ✓ ✎✒ ✠ ✑ ✎ ✍ ✌ ☞ More About Loops • real time software generally runs inside an infinite loop – need to carefully define what you mean by path or loop coverage ✗✤✣ ✘✚✙

� ☎✆ ✗ ✗ ✘ ✂ ✛ ✞ ✙ ✜ ✄ ☎ ✞ ✕ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✖ ✘ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ☞ ✡ ☎ ☛ ✠ ✠ ✑ ✔ ☞ ✓ ✎✒ ✒ ✏ ✎ ✍ ✌ Path Coverage: Impossible Combinations • Not all combinations are possible: if (x > 2) ... else ... if (x > 4) ... else ... • impossible to have first loop false, second true – no test case for that combination ✗✤✣ ✘✚✙

� ☎ ✖ ✗ ✗ ✘ � ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✘ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✆ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ Path Coverage: Combinatorial Explosion • Large method with many branches and loops: – many combinations, lots of test cases – difficult to manage – May catch sepcial inputs that trigger an error • Full path coverage only practical for unit testing ✗✤✣ ✘✚✙

� ☎ ✖ ✗ ✗ ✘ ✘ ✛ � ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✞ ☞ ☛ ✂ ✄ ☎ ☎✆ ✔ ✞ ✟ ✆ ✠ ✡ ☎ ✝ ✞ ☞ ✌ ✍ ✠ ✎ ✏ ✑ ✎✒ ✓ ✒ Example static double power(double base, int pow) { double answer = 1; while (pow > 0) { if (pow % 2 == 0) { // pow is even pow = pow / 2; base = base * base; } else { // pow is odd pow = pow - 1; answer = answer * base; } // end if } // end while return answer; } // end power White box checklist: pow = 0, 1, 2, bigger (so while loop executed 0, 1, many times) even and odd powers to exercise both parts of if statement ✗✤✣ ✘✚✙

� ☎✆ ✗ ✗ ✂ ✘ ✛ ✞ ✙ ✜ ✄ ☎ ✞ ✕ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✖ ✘ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ☞ ✡ ☎ ☛ ✠ ✠ ✑ ✔ ☞ ✓ ✎✒ ✒ ✏ ✎ ✍ ✌ Data Flow Testing • Another kind of white box testing • exercise (in order of weakest to strongest testing) – all input and output variables – both truth values of all predicates – all definitions of variables (eg variables on LHS of assignments) – all uses of variables that are defined – all loops ✗✤✣ ✘✚✙

� ✛ ✙ ✕ ✔ ✗ ✓ ✍ ✗ ✚ ✗ ✗ ✗ ✫ ✑ ✜ ✢ ✪ ✌ ✍ ✎ ✎ ✮ ✕ ✔ ✚ ✕ ✓ ✓ ✏ � ✓ ✗ ✔ ★ ✩ ✓ ✔ ✑ ✕ ✗ ✎ ✓ ✪ ✌ ✍ ✎ ✎ ✣ ✚ ✕ ✙ ✑ ✎ ✪ ✔ ✚ ✑ ✕ ✙ ✣ ✗ ✏ ✙ ✔ ✗ ✯ ✕ ✬ ✢ ✏ ✙ ✔ ★ ✎ ✕ ✚ ✙ ✗ ✓ ✗ ★ ✌ ✏ ✓ ✔ ✎ ✕ ✚ ✙ ✗ ✓ ✓ ✓ ✔ ✫ ✣ ✫ ✜ ✓ ✗ ✩ ✜ ✫ ✦ ★ ✚ ✕ ✎ ✓ ✚ ✞ ✗ ✗ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎ ☎✆ ✄ ✕ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ★ ✂ ✖ ☞ ☎ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ✡ ☎ ✞ ☞ ✠ ☞ ✌ ✍ ✎ ✏ ✑ ✎✒ ✓ ✒ ✔ ✄ ✁ ✓ ✑ ✙ ✔ ✓ ✍ ✗ ✚ ✗ ✗ ✛ ✗ ✜ ✘ ✢ ✔ ✎ ✏ ✓ ✓ ✔ ✑ ✕ ✓ ✆ ✗ ✠ ☞ ✟ ✞ ✓ ✎ ✟ ✞ ✎ ✍ ✌ ✡ ☛ ✝ • Ideal: tool to generate set of test cases satisfying ✔✖✗ Possibilities For Automation ✗✧✦ • Can't generate expected output from code! ✔✖✗ • Some help: tools to help verify coverage: ✔✖✗ ✣✥✤ ✔✖✗ ✗✤✣ coverage requirements ✏✭✬ ✘✚✙ ✔✖✗ ✔✖✗ ✔✖✗ ✔✖✕ ✏✒✑ ✏✒✑ ✏✒✑ ✔✖✗

� ☎ ✖ ✁ ✗ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✔ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ White Box Alone • White box alone is not a good way to generate test cases. – may lead to focus on less important parts – intrinsically leads to incomplete testing – if coder overlooked possibilities, test cases will not detect them ✗✤✣ ✘✚✙

� ☎ ✖ ✗ ✓ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✔ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ Black Box Alone • Black box alone may not cover the code adequately – some special cases are not obvious from specifications, depend on algorithms • Studies examined black box test suites developers considered extremely thorough – only exercised 1/3 to 1/2 of code! ✗✤✣ ✘✚✙

� ☞ ✍✎ ✡ ✡ ☞ ✌ ✠ ☎ ✠ ✠ ✄ ✏ ✁ ✠ ✟ ✒ ✂ ✁ � ✒ ✓ ☞ ✁ ✛ ✄ ✍ ☞ ✠ ✍ ✄ ✁ ✠ ✟ ✕ ✍ ☞ ✡ ☞ ✆ ✔ ✟ ✌ ☎ ✓ ✝ ✡ ✒ ✄☎ ✞ ✠ ☞ ✔ ✒ ✓ ✎✒ ✑ ✠ ✎ ✍ ✌ ☞ ✞ ✕ ☛ ☎ ✡ ✠ ✆ ✟ ✞ ✝ ☎✆ ☎ ✄ ✂ ☞ ✏ ✖ ✜ ✛✢ ✛ ✙ ✄ ✞ ☎✆ ✄ ☎ ✙ ✞ ✛ ✘ ✘ ✗ ✗ Another Example: Background • Heron's formula for area of triangle: + + a b c – let s = 2 − − − s s a s b s c ( )( )( ) – area = • Example program takes three sides of triangle as command-line arguments (string form) • checks to make sure it's an equilateral triangle ✆✞✝ ✆☛✡ ✂✒✑ ✆☛✡ ✗✤✣ ✘✚✙

� ☎ ✖ ✗ ✗ ✘ ✘ ✁ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✛ ☞ ☛ ✂ ✄ ☎ ☎✆ ✔ ✞ ✟ ✆ ✠ ✡ ☎ ✝ ✞ ☞ ✌ ✍ ✠ ✎ ✏ ✑ ✎✒ ✓ ✒ Example Code with two Branches public class Triangle { public static void main(String args []) { int sideA = Integer.parseInt(args[0]); int sideB = Integer.parseInt(args[1]); int sideC = Integer.parseInt(args[1]); if ((sideA == sideB) && (sideA == sideC)) { double s = 0.5 * (sideA + sideB + sideC); double area = Math.sqrt(s / (s - sideA) * (s - sideB) * (s - sideC)); System.out.println("area = " + area); } else System.out.println("not equilateral"); } // end main } // end class ✗✤✣ ✘✚✙

� ☎ ✖ ✗ � ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✗ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✠ ✡ ☎ ✆ ✞ ✏ ✒ ✠ ✎✒ ✑ ✓ ✎ ✍ ✌ ☞ Branch Coverage • one if statement, so two test cases needed • false case: pick sides that aren't equal – run Triangle 2 3 4 – output: not equilateral • true case: pick equal sides – run Triangle 2 2 2 – output: area = 1.7320508075688772 • output is correct, looks like all is well ✗✤✣ ✘✚✙

�✁ ☎✆ ✗ ✗ ✘ ✘ ✛ ✞ ✂ ✜ ✄ ☎ ✞ ✕ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✖ ✙ ☞ ✞ ✄ ☎ ☎✆ ☞ ✞ ✟ ✆ ✠ ✡ ☎ ☛ ✝ ✠ ✌ ✍ ✎ ☞ ✏ ✑ ✎✒ ✓ ✒ ✔ Two Errors not Caught by Test Cases public class Triangle { public static void main(String args []) { int sideA = Integer.parseInt(args[0]); int sideB = Integer.parseInt(args[1]); int sideC = Integer.parseInt(args[1]); should be 2 if ((sideA == sideB) && (sideA == sideC)) { double s = 0.5 * (sideA + sideB + sideC); double area = Math.sqrt(s / (s - sideA) * (s - sideB) * (s - sideC)); System.out.println("area = " + area); should be * } else System.out.println("not equilateral"); } // end main } // end class ✗✤✣ ✘✚✙

� ☎ ✖ ✗ ✗ � ✘ ✛ ✞ ✙ ✜ ✄ ☎✆ ☞ ✞ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✕ ✘ ☞ ☛ ✂ ✄ ☎ ☎✆ ✝ ✞ ✟ ✔ ✆ ✡ ☎ ✠ ✞ ✏ ✒ ✓ ✠ ✑ ✎✒ ✎ ✍ ✌ ☞ Combining the Two Approaches • pick a set of test cases from the specification (black box) • then look at code (white box) and see if test cases cover the code adequately • it not, add more to complete coverage • power example: – black box alone didn't tell us it was important to make sure we tested both odd and even powers – suggests additional test cases: pow = 16, pow = 15 ✗✤✣ ✘✚✙

� ☎✆ ✗ ✂ ✘ ✘ ✛ ✞ ✙ ✜ ✄ ☎ ✞ ✕ ✄ ✙ ✛✢ ✛ ✠ ✞ ✛ ✒ ✓ ✒ ✖ ✗ ☞ ✞ ✄ ☎ ☎✆ ✝ ✞ ✟ ✆ ✠ ☞ ☎ ☛ ✡ ✠ ✑ ✔ ✒ ☞ ✎✒ ✓ ✏ ✎ ✍ ✌ Testing at Different Levels • Unit testing: – dual black box/clear box approach – level of smallest granularity: want to look at code • Above unit testing level: gray box – awareness of structure, but not all details, • System testing: primarily black box ✗✤✣ ✘✚✙

� ✛ ✄ ✂ ✁ � ✒ ✓ ✒ ✞ ✟ ✠ ✛ � ✛✢ ✞ ✄ ✞ ✞ ✠ ☎ ☎ ✂ ☎ ☞ ✆ ☎ ✑ ✆ ☞ ☎ ✏ ☞ ✎ ✍ ✄ ✡ ✂ ☎✆ ✙ ✄ ☎ ✑ ✏ ✎ ✍ ✌ ☞ ✠ ✞ ☛ ✡ ✜ ✠ ✆ ✟ ✞ ✝ ☎✆ ☎ ✄ ✂ ✎✒ ✓ ✒ ☞ ✙ ✞ ✛ ✒ ✘ ✘ ✗ ✗ ✖ ✕ ☞ ✔ ✄ Stages • Big Bang Testing: go right to System Testing – generally least effective – very difficult to find bugs – avoid this! • Better way: – test all modules individually (unit testing) – put together gradually & test larger pieces ☛✌☞ ☎✝✆ – finally test whole system ✗✤✣ ✘✚✙

� � ✬ ✫ ✩✪ ★ ✧ ✦ ✥ ✒ ✧ ✓ ✒ ✛ ✞ ✠ ✛ ✁ ✙ ✭ ✩ ✞ ✬ ✱ ✫ ✮ ✫ ✬ ✰ ✦ ✥ ✥ ✭ ★ ✪ ✯ ★ ✧ ✫ ✮ ✬ ✄ ✛✢ ☎✆ ☎ ✑ ✏ ✎ ✍ ✌ ☞ ✠ ✞ ☛ ✡ ✓ ✠ ✆ ✟ ✞ ✝ ☎✆ ☎ ✄ ✂ ✁ ✎✒ ✒ ☎ ✘ ☞ ☞ ✕ ✖ ✗ ✄ ✜ ✙ ✞ ✛ ✔ ✗ ✘ Integration Testing • Tests uses between modules - source of 40% of all errors in code. • Incremental: Top Down vs. Bottom Up – start with tested modules in either case – planning of increments starts early ✦✳✲ – top down requires stubs – bottom up requires drivers – 'sandwich' variants exist ✗✤✣ ✘✚✙