Next Topic: Software Quality Plan (5 lectures): quality in general - - PowerPoint PPT Presentation

CISC 323, winter 2003, software quality 1

Next Topic: Software Quality

Plan (5 lectures):

• quality in general
• inspection
• testing (general)
• testing OO programs

Reading:

• Bahrami, chapter 13 (testing only)
• inspection paper in courseware

Glen Russell: Experience with Inspection in Ultralarge- Scale Developments

CISC 323, winter 2003, software quality 2

What does quality mean?

Operational meanings:

• Software does what is wanted and expected
• On budget and on schedule
• Users are happy
• Level of bugginess is tolerable
• Changes are not too expensive
• Software is cost-effective in use and supports
• rganizational mandate

CISC 323, winter 2003, software quality 3

Definitions of Quality

OED: "the standard of something when compared to other things like it (a wine of excellent quality); of high quality, general excellence" ISO: "The totality of features and characteristics of a product

• r service that bear on its ability to satisfy specified or implied

needs."

• DoD: The degree to which the attributes of the software

enable it to perform its intended end use. Juran: Fitness for Purpose

CISC 323, winter 2003, software quality 4

"Absolutes of Quality Management"

• defined by Philip Crosby
• 1. Quality is defined as conformance to requirements, not as

'goodness' or 'elegance'.

• 2. The system for causing quality is prevention, not

appraisal.

• 3. The performance standard must be Zero Defects, not

"that's close enough".

• 4. The measurement of quality is the Price of

Nonconformance.

CISC 323, winter 2003, software quality 5

Cost of Quality vs. Cost of Failure

prevention

• quality planning
• formal technical

reviews

• training

appraisal

• in-process &

interprocess inspection

• testing
• test equipment (cost &

maintenance) failure - internal

• rework
• repair
• failure mode analysis

failure - external

• complaint resolution
• product return &

replacement

• help line support
• warranty work

CISC 323, winter 2003, software quality 6

Quality is Relative

Rolls Royce: quality when cost is no object Ford Taurus: quality tradeoffs when cost is a factor Mini: minimum acceptable quality when cost is the dominant factor. Costs include initial cost, cost of operation, and cost of repairs.

CISC 323, winter 2003, software quality 7

Quality -- General

is multidimensional (all aspects of a car) is subject to constraints (e.g. cars must fit on roads) is about acceptable compromises (e.g. poorer ride for lower cost) criteria are not independent (e.g. fuel use vs. vehicle weight) is not absolute (all levels of cars have quality)

CISC 323, winter 2003, software quality 8

Quality - Software

is multidimensional (portability, usability, functionality) is subject to constraints (regulations, rules, resources) is about acceptable compromises (e.g. less functionality / lower cost) criteria are not independent (e.g. efficiency vs maintainability) is not absolute (different levels of quality are acceptable for different software types)

CISC 323, winter 2003, software quality 9

Characteristics of Software

not physical needs are usually uncertain needs change platforms change (CPUs, OS, network) high expectations, especially of adaptability

CISC 323, winter 2003, software quality 10

Changing Approaches to Software Quality

increasing use of metrics increasing emphasis on process, components and intermediate products, as opposed to the end product "software crisis" persists, but progress is being made

CISC 323, winter 2003, software quality 11

Qualities Discernable at Runtime

functionality correctness performance security availablity functionality usability

CISC 323, winter 2003, software quality 12

Qualities Not Discernable at Runtime

cost timeliness modifiability portability reusability testability

• ease with which software can be made to demonstrate its

faults through execution based testing

• related to architectural documentation, separation of

concern, information hiding, incremental development

CISC 323, winter 2003, software quality 13

How can the Dimensions of Quality be assured?

by establishing and enforcing a process process must include measurement by making sure everyone understands the qualities that must be embedded in the product by keeping focused on the relevant qualities throughout the development lifecycle by determining “up front” how to achieve each quality verification and validation

CISC 323, winter 2003, software quality 14

ISO 9126

source: http://www.cse.dcu.ie/essiscope/sm2/9126ref.html

CISC 323, winter 2003, software quality 15

User View

can be highly personalized typical user views:

• reliability
• performance
• usability

CISC 323, winter 2003, software quality 16

Manufacturing View

avoid rework and associated costs

• ften leads to process focus (ISO 9001-3)

process does not guarantee good products; only guarantees uniformity of output

CISC 323, winter 2003, software quality 17

Product View

measure internal characteristics?

• Measurements chosen depend on type of product (e.g.

code, design, …)

• example: number and kinds of connections between

components to measure coupling

• may not ensure quality of external characteristics

measure external characteristics?

• Corresponds to user/manufacturer views
• measurements chosen depend on product (e.g. OS,

spreadsheet, …)

CISC 323, winter 2003, software quality 18

Value-based view

helps to decide trade-offs during development

• cost vs. reliability
• cost vs. functionality

cost to consumer is affected by scale

• high volume products have a higher perceived value

value is defined by

• potential savings/productivity enhancements
• profit opportunities

CISC 323, winter 2003, software quality 19

Problems with Qualities

qualities are at odds with one another

• always a compromise
• portability versus performance
• security versus fault tolerance
• modularity versus efficiency

difficulty in trying to measure qualities e.g., accuracy, usability what does quality really mean?

CISC 323, winter 2003, software quality 20

Proposed solution 1: Focus on Product

most dimensions of quality are measured after the fact need to identify actions that can embed needed qualities in the products, up front implementation quality model: identify what the programmer can do to ensure quality

• link these actions to high level qualities

CISC 323, winter 2003, software quality 21

Proposed Solution 1 (cont’d)

requirements quality model: all requirement statements must be measurable design quality model: define qualities for design, identify characteristics of design that would embed those qualities

CISC 323, winter 2003, software quality 22

Proposed solution 2: Focus on Process

integrate quality decisions into the design process, e.g.:

• by using quality checklists during inspection, design
• by determining testing criteria up front
• by following a documented design process

CISC 323, winter 2003, software quality 23

Business Qualities

time to market cost projected lifetime of system targeted market integration with existing legacy systems

CISC 323, winter 2003, software quality 24

What's Ahead

Remainder of material about quality focuses on two topics related to software quality:

• 1. Inspection: a formal way of reading and analyzing

software and other documents

• 2. Testing: finding defects by executing software and

examining results (and then fixing the defects)

CISC 323, winter 2003, software quality 25

What is Software Inspection?

Formal process to statically analyze a software product and record metrics related to that product. Differs from informal "walk-throughs" or "reviews":

• formal process
• products inspected at greater level of detail
• metrics

CISC 323, winter 2003, software quality 26

IEEE Definition

"...a formal evaluation technique in which software requirements, design, or code are examined in detail by a person or group other than the author to detect faults, violations of development standards, and other problems..."

• - IEEE Standard Glossary of Software Engineering Terminology

CISC 323, winter 2003, software quality 27

History

Traditional meaning of inspection:

• Inspecting the product of a manufacturing process
• Physical product, evaluate by looking, manipulating
• Often put through rigorous tests, keep metrics

Michael Fagan at IBM:

• used similar processes to inspect software (code)
• landmark paper in 1976
• often called "Fagan Inspections"
• Inspection process has been extended & refined since,
• ngoing research

CISC 323, winter 2003, software quality 28

What Gets Inspected?

Inspection process initially used to inspect software (code) Extended to include all documents:

• analysis/requirements
• design documents
• test plans
• user manuals
• etc.

Above includes diagrams, prose, pseudo-code, etc. Some other industries have adopted Inspection for design documents, drawings, etc.

CISC 323, winter 2003, software quality 29

Inspection Versus Testing (1)

AT&T classifies inspection as a form of testing In CISC 323, we use the term "testing" to mean dynamic testing: executing software to find faults inspection does not replace dynamic testing software process should include both inspection and testing

CISC 323, winter 2003, software quality 30

Inspection Versus Testing (2)

difficult to inspect for qualities such as performance, timing, load inspection can be applied earlier than testing inspection can be applied to any software product, not just code

CISC 323, winter 2003, software quality 31

Defects vs. Faults

Inspection finds defects: problems in software Testing finds faults: incorrect behavior When you find a fault by testing:

• must debug to trace the fault to a defect in the code

Often more cost-effective to find defects via Inspection

CISC 323, winter 2003, software quality 32

Analysis Tools

inspection is time-consuming (and therefore expensive) so inspect for things tools can’t find static analysis tools can find some types of defects

• compilers find undeclared variables, syntax errors
• other static analysis tools may check for:
• unreachable code
• unused variables
• memory leaks
• "unsafe" programming practices
• misspelled words, some grammatical errors

not cost-effective to inspect for these if tools exist

CISC 323, winter 2003, software quality 33

Inspection Metrics

to be classified as “inspection”, metrics must be kept record:

• defects found in product under inspection
• where defect found in product
• time taken to perform inspection
• size of product inspected

may also record

• category of defect (e.g.,major/minor)
• what prompted detecting the defect
• suggested improvements
• failure counts collected during testing and in the field

CISC 323, winter 2003, software quality 34

Use of Metrics

Data collected can tell you:

• total defects per line of code or per time unit
• percentage of defects found at different stages
• e.g. 60% of defects found by Inspection, 40% by

testing

• percentage of time spent on different activities:
• design, coding, documentation, inspection, testing,

etc. Data can be used to:

• justify time spent inspecting & testing
• suggest improvements in development process
• evaluate changes in process

CISC 323, winter 2003, software quality 35

Inspection Process

from Software Inspection, Tom Gilb & Dorothy Graham

CISC 323, winter 2003, software quality 36

People Involved in Inspection

Inspection is a team process: ideally 2-6 people Inspection Leader:

• organizes & manages Inspection process
• moderates meetings
• should have special training (multi-day course,

certification) checkers (inspectors)

• spend time reading/analyzing the product
• may include author(s), leader
• short training: 1-day course or "on the job"
• technical knowledge relating to product

CISC 323, winter 2003, software quality 37

Entry

entry criteria set up for products to be inspected a lot of controversy about when to inspect code

• before or after testing
• before or after compile

if entry criteria not met, product not inspected

• e.g., too many typing errors
• e.g., source code does not compile cleanly

inspections are expensive

• don’t waste inspectors time

author offers up product for inspection leader decides if it's ready -- quick examination

CISC 323, winter 2003, software quality 38

Kickoff

initial meeting for all participants

• rganized by leader

familiarize everyone involved with task at hand define everyone’s roles hand out materials needed deal with general questions set goals discuss overall plan, timetable long documents divided into "chunks": separate inspection & logging for each chunk

CISC 323, winter 2003, software quality 39

Individual Checking (1)

each checker works alone using documents handed out in kick-off meeting aims to find maximum number of unique major potential defects mainly by looking for discrepancies between source documents and products being inspected, i.e.:

• requirements to design
• design to code
• requirements to test plan

CISC 323, winter 2003, software quality 40

Individual Checking (2)

usually uses a checklist kickoff meeting suggests amount of time to spend:

• typical rate is 1 hour per document page

checker records actual time taken keeps notes of issues found

• description, location, possibly impact

"issue" = matter requiring attention

• possible defect
• "question of intent"
• suggestion

CISC 323, winter 2003, software quality 41

Logging Meeting (1)

3 purposes:

• log issues identified by each checker
• work as group to find more issues
• suggestions for process improvement (inspection process

and overall software process) controlled by moderator (usually the leader)

• must be very skilled in handling people and issues
• maintain time discipline and issue discipline

no evaluation of issues

• just log issues, like brainstorming

CISC 323, winter 2003, software quality 42

Logging Meeting (2)

must not last more than 2 hours not for explaining, defending, suggesting fixes, or training identify issues, then log them issues can be questions to product author and suggestions for improvements

• not discussed in meeting

NOT an attack or evaluation of product author management personnel should NOT be in attendance

CISC 323, winter 2003, software quality 43

Edit

log of issues given to author to resolve author can

• classify or reclassify issues (which are really defects)
• voluntarily improve product as per issues raised
• request rules or checklist be changed

CISC 323, winter 2003, software quality 44

Follow-up

inspection leader makes sure satisfactory closure has taken place on all issues

• actions to correct all defects (change or change request)
• suggestions for process changes reported

if major changes, product may need to be inspected again

CISC 323, winter 2003, software quality 45

Exit

need exit criteria need closure on all issues in writing record metrics for inspection process (for process improvement) product now considered “safe” to pass on to next phase in software development

CISC 323, winter 2003, software quality 46

The V-model

Requirements Inspection Architecture Inspection Design Inspection Code Inspection Unit Test Integration Test System Test

CISC 323, winter 2003, software quality 47

Costs of Inspection

Inspection takes time Typically 10-15% of development budget Also start-up costs:

• establishing procedures
• training leaders, inspectors

Is it worth it?

CISC 323, winter 2003, software quality 48

Benefits of Inspection (1)

Defects found early save time and money many defects found before testing

• so testing is much quicker

total development time is reduced

• even when counting time for inspection

in the long run, inspections increase productivity

CISC 323, winter 2003, software quality 49

Benefits of Inspection (2)

Defects found early result in better product inspection results in

• cleaner design
• better documentation
• better code
• fewer defects
• defects that remain will be easier to fix when found

CISC 323, winter 2003, software quality 50

Benefits of Inspection (3)

feedback for mangement about software process:

• inspector's comments plus metrics

defects found earlier means fewer deadline surprises inspection trains inspectors:

• learn from good code and documentation of others
• learn from mistakes of others

CISC 323, winter 2003, software quality 51

Success Stories (1)

IBM Federal Systems Division

• kept metrics for similar projects before and after

introducing Inspection into development process

• delivered lines of code per work-month:
• 144 without Inspection
• 300 with Inspection

CISC 323, winter 2003, software quality 52

Success Stories (2)

ICL (in U.K.):

• 57.7% of defects found by Inspection
• cost of finding defect by Inspection: 1.58 work hours
• cost of finding without Inspection: 8.47 work hours
• only 6% of development time devoted to Inspection

CISC 323, winter 2003, software quality 53

Success Stories (3)

Large IBM project:

• 500,000 lines of code (networked operating system)
• based on past experience without inspection, expected to

find about 800 bugs during tests at trial site

• used inspection at 11 stages of devopment
• found 8 bugs at trial site!

CISC 323, winter 2003, software quality 54

Software Inspection Research (1)

Research into inspection

• inspection process
• techniques for individual inspectors

CISC 323, winter 2003, software quality 55

Software Inspection Research (2)

research at Lucent Technologies (University of Maryland)

• experiment where they varied
• number of inspectors
• number of teams inspecting the same code
• teams working in parallel and teams working in

tandem (where bugs were fixed before 2nd inspection took place)

• one of the conclusions
• most significant improvements in inspection comes

from improving the way the inspectors do their individual work

CISC 323, winter 2003, software quality 56

Software Inspection Research (3)

new techniques to help guide inspectors in their work

• scenario-based reading
• eg. put yourself in the role of a tester
• have a series of questions and scenarios to work with

as you inspect

CISC 323, winter 2003, software quality 57

Checklists

common guidance used for inspections good checklists are hard to make up

• rely on past experience
• can get too long and unwieldy
• need to be updated as new situations arise
• inspectors have a tendency to follow the checklist and do

no more

• can be misinterpreted
• may not cover everything necessary to the right level of

detail

CISC 323, winter 2003, software quality 58

Approach for Checklists

(general approach for inspections as well) determine what qualities are important for the software system determine what software design/code characteristics contribute to those qualities determine what you want the inspectors to look for

• also determines practices you want the software

developers to follow checklist used for Assignment 4 is based on McCall's Quality Model

CISC 323, winter 2003, software quality 59

McCall's Quality Model

CISC 323, winter 2003, software quality 60

Testing

Now we move to the subject of testing Recall that software process should include both testing and inspection Inspecting documents & code before testing results in fewer defects for testing to find – speeds up testing A bit of informal testing before code inspection might save time – get rid of obvious bugs quickly

CISC 323, winter 2003, software quality 61

Testing vs. Debugging

Textbook says:

• "Precisely speaking, the elimination of the syntactical

bug is the process of debugging, whereas the detection and elimination of the logical bug is the process of testing" We disagree. Testing = finding faults (situations in which the program does not perform correctly) Debugging = fixing faults (tracing a fault to defects in the software and fixing them – hopefully without intruducing new ones!) In common use, sometimes "debugging" includes informal testing by programmer

CISC 323, winter 2003, software quality 62

Comments About Debugging

We will not be discussing debugging in detail in this course However, debugging is made easier by many of the techniques we have discussed to improve readability and modularity Also made easier with good design documentation

CISC 323, winter 2003, software quality 63

Testing is Hard!

hits budget and schedule limits may include heavy debugging

• may even include product (re-)development

bugs slip through to customers

• user executes untested code
• order in which statements are executed in actual use

differ from that in testing

• user applies combination of untested input values
• user’s operating environment is never tested

CISC 323, winter 2003, software quality 64

Skills of a Tester (1)

Not all testers are developers

• e.g. testers at Corel who are photographers (Corel Draw

testing)

• e.g. testing buddies at Microsoft

testers must have

• ability to select good test cases
• ability to withstand the most severe deadline and

resource pressures

• ability to organize and analyze large volumes of data
• ability to sort out complex version control issues

CISC 323, winter 2003, software quality 65

Skills of a Tester (2)

More skills employers look for:

• software engineering skills
• understanding the rules of software engineering
• computer programming
• operating system level knowledge
• communication skills
• organizational skills
• hands-on experience

CISC 323, winter 2003, software quality 66

Attitude For Testers

destructive creativity detective skills understanding the product as the sum of its parts appreciating the customer’s perspective requirements change skeptical but not hostile attitude ability to be the bearer of bad news and remain objective an eagerness to embrace new technologies

CISC 323, winter 2003, software quality 67

Definitions of Testing (1)

Myers, Glenford J., The Art of Software Testing, Wiley, 1979

• “The Process of executing a program with the intent of

finding errors” Hetzel, Bill, A Complete Guide to Software Testing, QED Information Sciences Inc., 1988

• "Testing is any activity aimed at evaluating an attribute
• r capability of a program or system and determining

that it meets its required results."

CISC 323, winter 2003, software quality 68

Definitions of Testing (2)

IEEE Standard 610.12-1990

• “The process of operating a system or component under

specified conditions, observing or recording the results, and making an evaluation of some aspect of the system

• r component”

IEEE Standard 829-1983

• “The process of analyzing a software item to detect the

differences between existing and required conditions (that is , bugs) and to evaluate the features of the software items”

CISC 323, winter 2003, software quality 69

Purposes of Testing

To find bugs - including bugs in the tests! To estimate reliability in use by tracking the execution time interval between failures To estimate defects remaining by tracking the defects found per person-hour of testing and debugging time To decide on when to release: e.g. by deciding that the remaining known defects are acceptable in the target market To learn where process problems are by classifying and counting defects found

CISC 323, winter 2003, software quality 70

What Testing Can't Do

Testing can't guarantee code is correct You can't try every possible combination of inputs, every environment Rigorous testing increases confidence that software runs correctly, or that few bugs remain For some critical software, formal mathematical methods used to "prove" correctness – but that's another topic for

• ther courses

CISC 323, winter 2003, software quality 71

Stages of Testing (1)

Requirements and Design testing

• when requirements and design are written in special

languages which permit "execution" to simulate actual system User Interface testing

• Often done at prototyping stage

Unit/Module Testing Component/subsystem Testing

CISC 323, winter 2003, software quality 72

Stages of Testing (2)

Integration Testing

• combining components incrementally and testing

Daily builds and Sanity testing System Testing

• Performance and Load/Stress Testing
• Usability Testing, but this is a late stage to be doing it!

CISC 323, winter 2003, software quality 73

Stages of Testing (3)

Alpha and Beta Testing

• alpha = testing full system in-house
• beta = outside users

Acceptance Testing Installation/Compatibility Testing Platform/Configuration/Port testing

CISC 323, winter 2003, software quality 74

Stages of Testing (4)

Integrity and Release testing

• documentation consistent with product
• media not infected with viruses
• intended version is what is actually released,...

Reliability/Certification Testing

• estimates MTTF (mean time to failure) with reference

to operational profiles

• sometimes for certification of conformance to a

standard - e.g. ACVC (Ada), ISO protocol test suites Security/Fault Tolerance/Recovery testing Regression Testing

CISC 323, winter 2003, software quality 75

Black Box Testing

pick test cases based on requirements only don't use knowledge of implementation view code as a black box: insides not visible can be used at any stage:

• unit, subsystem, system testing

CISC 323, winter 2003, software quality 76

Black-Box Testing Strategies

look at specifications for inputs test normal use:

• sampling of "normal" inputs (random values)

test abnormal use:

• different kinds of erroneous/illegal input
• combinations of errors

test boundary conditions:

• values close to boundary between normal and abnormal

CISC 323, winter 2003, software quality 77

Example

/** * Raises a number to an integral power. * * Parameters: * base the number to be raised to a power * pow the power to which base is to be raised. * Must not be negative. * Return value base raised to the pow-th power */ static double power(double base, int pow) {

error inputs: negative values for pow boundary cases: pow = 0, pow = 1, pow = -1 normal cases:

• pow >= 2, range of values
• base: negative, 0, positive
• combinations of these

CISC 323, winter 2003, software quality 78

Possibilities For Automation

Ideal: tool to generate set of test cases and expected results from specifications Reality: needs very precise, mathematical specifications, usually not available So usually generate test cases & expected results by hand Automated execution is still possible

CISC 323, winter 2003, software quality 79

Test Harness

Takes set of test inputs & expected results Runs program on each test input, compares results with expected Reports any discrepancies Complicated to write for GUIs, embedded systems, but

• ften possible

Takes lots of drudgery out of testing Makes testers willing to run and re-run large test suites

CISC 323, winter 2003, software quality 80

White Box Testing

Pick test cases based on the actual software also called "clear box" or "structural" testing key word is coverage:

• do the test cases cover all of the code?

different measures of coverage

CISC 323, winter 2003, software quality 81

Statement Coverage

Every statement in the software must be executed at least

• nce by the test cases

Quote from text (Murray):

• "Testing less than this for new software is

unconscionable and should be criminalized" NASA story

CISC 323, winter 2003, software quality 82

Branch Coverage

Every branch alternative must be covered by at least one test case More demanding than statement coverage: if (x > 3) y = 14; // continue -- no else part We need at least one test case where x > 3 is true and at least one where x > 3 is false.

CISC 323, winter 2003, software quality 83

Path Coverage

Every possible path through the program must be covered by at least one test case

void tst(int x) { if (x > 0)

pos = pos+1;

if (x % 2 == 0)

even = even+1;

return; }

Needs 4 test cases:

• all combinations of alternatives

√ √

x is odd

√ √

x is even x <= 0 x > 0

CISC 323, winter 2003, software quality 84

Example

void tst(int x) { if (x > 0) pos = pos+1; if (x%2 == 0) even = even+1; } Statement Coverage: tst(2) Branch Coverage: tst(-1),tst(2) Path Coverage: tst(-2),tst(-1),tst(1),tst(2)

CISC 323, winter 2003, software quality 85

What About Loops?

May require test cases which make each loop execute 0 times, 1 time, many times for (int i = 0; i < n; i++) {...}

• Test cases should include:
• n <= 0 (no times around the loop)
• n = 1 (one time around the loop)
• n > 1 (several times around the loop)

CISC 323, winter 2003, software quality 86

More About Loops

real time software generally runs inside an infinite loop

• need to carefully define what you mean by path or loop

coverage

CISC 323, winter 2003, software quality 87

Path Coverage: Impossible Combinations

Not all combinations are possible: if (x > 2) ... else ... if (x > 4) ... else ... impossible to have first loop false, second true – no test case for that combination

CISC 323, winter 2003, software quality 88

Path Coverage: Combinatorial Explosion

• Large method with many branches and loops:
• many combinations, lots of test cases
• difficult to manage
• May catch special inputs that trigger an error
• Full path coverage only practical for unit testing

CISC 323, winter 2003, software quality 89

Example

static double power(double base, int pow) { double answer = 1; while (pow > 0) { if (pow % 2 == 0) { // pow is even pow = pow / 2; base = base * base; } else { // pow is odd pow = pow - 1; answer = answer * base; } // end if } // end while return answer; } // end power

White box checklist: pow = 0, 1, 2, bigger (so while loop executed 0, 1, many times) even and odd powers to exercise both parts of if statement

CISC 323, winter 2003, software quality 90

Data Flow Testing

Another kind of white box testing exercise (in order of weakest to strongest testing)

• all input and output variables
• both truth values of all predicates
• all definitions of variables (eg variables on LHS of

assignments)

• all uses of variables that are defined
• all loops

CISC 323, winter 2003, software quality 91

Possibilities For Automation

Ideal: tool to generate set of test cases satisfying coverage requirements

• very difficult

Can't generate expected output from code! Some help: tools to help verify coverage:

• which statements were executed by this test case, or this

set of test cases?

• which branch alternatives were executed?
• which paths?
• test harness still useful, or above can be integrated into

harness

CISC 323, winter 2003, software quality 92

White Box Alone

White box alone is not a good way to generate test cases.

• may lead to focus on less important parts
• intrinsically leads to incomplete testing
• if coder overlooked possibilities, test cases will not

detect them

CISC 323, winter 2003, software quality 93

Black Box Alone

Black box alone may not cover the code adequately

• some special cases are not obvious from specifications,

depend on algorithms Studies examined black box test suites developers considered extremely thorough

• only exercised 1/3 to 1/2 of code!

CISC 323, winter 2003, software quality 94

Another Example: Background

Heron's formula for area of triangle with sides a, b and c:

• let s =
• area =

Example program takes three sides of triangle as command- line arguments (string form) checks to make sure it's an equilateral triangle

• if not, prints error message
• if yes, computes and prints area

2 c b a + + c) b)(s a)(s s(s − − −

CISC 323, winter 2003, software quality 95

Example Code with two Branches

public class Triangle { public static void main(String args[]) { int sideA = Integer.parseInt(args[0]); int sideB = Integer.parseInt(args[1]); int sideC = Integer.parseInt(args[1]); if ((sideA == sideB) && (sideA == sideC)) { double s = 0.5 * (sideA + sideB + sideC); double area = Math.sqrt(s / (s - sideA) * (s - sideB) * (s - sideC)); System.out.println("area = " + area); } else System.out.println("not equilateral"); } // end main } // end class

CISC 323, winter 2003, software quality 96

Branch Coverage

• ne if statement, so two test cases needed

false case: pick sides that aren't equal

• run Triangle 2 3 4
• output: not equilateral

true case: pick equal sides

• run Triangle 2 2 2
• output: area = 1.7320508075688772
• utput is correct, looks like all is well

CISC 323, winter 2003, software quality 97

Two Errors not Caught by Test Cases

public class Triangle { public static void main(String args []) { int sideA = Integer.parseInt(args[0]); int sideB = Integer.parseInt(args[1]); int sideC = Integer.parseInt(args[1]); if ((sideA == sideB) && (sideA == sideC)) { double s = 0.5 * (sideA + sideB + sideC); double area = Math.sqrt(s / (s - sideA) * (s - sideB) * (s - sideC)); System.out.println("area = " + area); } else System.out.println("not equilateral"); } // end main } // end class should be 2 should be *

CISC 323, winter 2003, software quality 98

Combining the Two Approaches

pick a set of test cases from the specification (black box) then look at code (white box) and see if test cases cover the code adequately it not, add more to complete coverage power example:

• black box alone didn't tell us it was important to make

sure we tested both odd and even powers

• suggests additional test cases: pow = 16, pow = 15

CISC 323, winter 2003, software quality 99

Testing at Different Levels

Unit testing:

• dual black box/clear box approach
• level of smallest granularity: want to look at code

Above unit testing level: gray box

• awareness of structure, but not all details,

System testing: primarily black box

CISC 323, winter 2003, software quality 100

A True Story From CISC 121

Fall term, 2002: Margaret thought of an "improvement" to quicksort partitioning algorithm Implemented it, tested it and it worked Released it to students for an assignment, immediate complaints that it didn't work!

CISC 323, winter 2003, software quality 101

Margaret's Testing Strategy

public static TestSort() { int A[] = new int[20]; for (int i = 0; i < 20; i++) A[i] = <random int between 1 and 100>; // print contents of A sort(A); // print contents of A again } // end TestSort Ran this test numerous times, no problem. What did this testing strategy miss?

CISC 323, winter 2003, software quality 102

Flaw in Testing Strategy

Method of generating test arrays: for (int i = 0; i < 20; i++) A[i] = <random int between 1 and 100>; This generates arrays with very low likelihood of duplicates. The bug involved what happens when duplicates occur in certain positions – multiple array elements equal to partition value Students' assignment used arrays with duplicates, triggered the bug.

CISC 323, winter 2003, software quality 103

Better Testing Strategy

Method of generating test arrays: for (int i = 0; i < 20; i++) A[i] = <random int between 1 and 10>; This generates arrays with duplicates. Would have caught bug. Moral: Picking test cases is difficult, requires creativity. Blank Box: Try to think of different properties input could have, test them all. Didn't think about property of having duplicates. White Box: Thorough white box attack on sorting algorithm would not have found the problem

CISC 323, winter 2003, software quality 104

Stages

Big Bang Testing: go right to System Testing

• generally least effective
• very difficult to find bugs
• avoid this!

Better way:

• test all modules individually (unit testing)
• put together gradually & test larger pieces
• (integration/subsystem testing)
• finally test whole system

CISC 323, winter 2003, software quality 105

Integration Testing

Tests uses between modules - source of 40% of all errors in code. Incremental: Top Down vs. Bottom Up

• start with tested modules in either case
• planning of increments starts early
• should revolve around uses hierarchy
• top down requires stubs
• bottom up requires drivers
• 'sandwich' variants exist

CISC 323, winter 2003, software quality 106

System Testing

Functional Testing (does it work?) Also:

• Timing, Load, Performance and Stress Testing
• Usability Testing
• Security Testing
• Storage testing
• Procedure and document testing
• e.g. user manuals
• Recovery testing
• Hardware configuration testing
• etc.

CISC 323, winter 2003, software quality 107

Regression Testing

after system is modified:

• bug fix, change or extension

should run all tests over again bug fix in one module can show up bugs in another – module interactions change automation is a great help here

CISC 323, winter 2003, software quality 108

Regression Testing Issues (1)

After changes to system, may need to update test plan, test cases.

• Existing test cases may be reusable, retestable, or
• bsolete
• There is a trade-off between possible redundant testing

costs and the costs of deciding which test results can be reused and which test cases can be rerun.

• Some test cases may need to be dropped
• new test cases may be needed

CISC 323, winter 2003, software quality 109

Regression Testing Issues (2)

2 types of regression testing

• progressive (specification has changed)
• corrective (specification is unchanged).

corrective modifications may require progressive regression testing if the original specifications were in error Version control is a major issue

• needed
• n

everything: code, test suites, data, documentation, level of operating system, compiler, linker, possibly hardware if changes are being made, ...

CISC 323, winter 2003, software quality 110

Acceptance Testing

Testing specified early as criteria for accepting product Can be dangerous! If tests are pre-specified, software may pass tests but not do much else (Ada compilers) Good acceptance testing does not involve detailed prior specification of the tests, but depends on precise, complete and correct system specifications.

CISC 323, winter 2003, software quality 111

Scenario-Based Testing

Can be used in addition to black-box and white-box strategies if system (or alternative) is already in use Study how users use system Capture sequences of operations and use as basis for test cases

CISC 323, winter 2003, software quality 112

Resources That Limit Testing

Budget Schedule Hardware Testers Debuggers Users (for usability testing)

CISC 323, winter 2003, software quality 113

Measuring Testing Progress

determining

• when to stop testing
• usually when you run out of time and/or resources
• when the product is ready to release
• based on business judgement

need quantitative measures

• estimation of number of bugs left in software
• based on
• rate at which bugs are still being found
• where in software bugs are being found
• rate of bug fixing

CISC 323, winter 2003, software quality 114

Object-Oriented Testing

Much wisdom about testing accumulated before popularity

• f OOP

How is the testing task different for object-oriented code? sources:

• Object-Oriented Testing: Myth and Reality
• Robert Binder, Object Magazine 1995
• http://www.rbsc.com/pages/myths.html

lectures by Dr. Terry Shepard

• Royal Military College of Canada
• Electrical and Computer Engineering

CISC 323, winter 2003, software quality 115

Myth #1

Myth:

• OO testing is unnecessary. OO promotes incremental

development and reuse, so we have a more effective way to develop trustworthy classes. Reality:

• Human error is as likely as ever. We have to check class

functionality and interactions between classes

CISC 323, winter 2003, software quality 116

Myth #2

Myth:

• testing is a structured waterfall idea and isn’t consistent

with incremental OO development Reality:

• tests can be designed and exercised at many points in the

process

• paradigm of “design a little, code a little” becomes

“design a little, code a little, test a little”

• Extreme Programming (XP) promotes early testing

(more later)

CISC 323, winter 2003, software quality 117

Myth #3

Myth:

• testing is too expensive

Reality:

• pay me now or pay me MUCH more later
• failures in operational systems can cause severe

secondary problems

• proper testing is very cheap by comparison, even when

done manually

• efficient testing of large or complex systems needs some

automated support

CISC 323, winter 2003, software quality 118

Myth #4

Myth:

• OO testing is the same as conventional software

Reality:

• polymorphism,

inheritance, encapsulation present

• pportunities for error (more later)

CISC 323, winter 2003, software quality 119

Myth #5

Myth:

• conventional testing is useless for objects

Reality:

• there is a large body of knowledge about testing
• basic testing techniques continue to apply with necessary

changes

• if scope of testing is small, ie classes and collaborations
• testing techniques are very OO specific – more later
• as scope of testing gets larger
• testing techniques become more traditional

CISC 323, winter 2003, software quality 120

Myth #6

Myth:

• inheritance means never having to say you are sorry
• specializing from a trusted superclass means the

inherited part of subclasses will also be correct

• we don’t need to retest inherited features

Reality:

• subclasses create new ways to misuse inherited features
• different test cases are needed for each subclass (more

later)

CISC 323, winter 2003, software quality 121

Complications From Inheritance (1)

test class super first, including methods A and B now test class sub new version of B – need to test in sub sub inherits A – no need to re-test in sub suppose C calls A:

• no guarantee that C calls A properly
• we still have to test C's use of A

super int A(int x) int B(int x) sub int B(int x) int C(int x)

CISC 323, winter 2003, software quality 122

Complications From Inheritance (2)

test class super first, including methods A and B now test class sub new version of B – need to test in sub might assume no need to test A inside sub but what if A calls B?? need to test A with the new B

super int A(int x) int B(int x) sub int B(int x)

CISC 323, winter 2003, software quality 123

Myth #7

Myth:

• reuse means never having to say you are sorry
• reusing a trusted class means behaviour of server object

is trustworthy and doesn’t need to be tested Reality:

• nothing prevents a new client class from using the server
• bject incorrectly
• all client class use of a server needs to be exercised

CISC 323, winter 2003, software quality 124

Myth #8

Myth:

• black box testing is sufficient
• designing

test cases using class interface and specification assures the class is fully exercised

• white box testing violates encapsulation

Reality:

• studies show black-box test suites typically only exercise

1/3 to 1/2 of the statements (let alone paths or states)

• typically miss abnormal paths, exception and error-

handling

CISC 323, winter 2003, software quality 125

Testing Objects

exercise methods provided by object with goals of uncovering errors in results, state of object, or both to test a method: consider not only parameters to method but state of object (values of instance variables) must consider patterns of method invocation of object

• methods can be invoked in any random order leading to

a large number of combinations