CSC 309 Lecture Notes Week 4 Formal Specs in Testing Intro to - - PowerPoint PPT Presentation

CSC309-S15-L4 Slide 1

CSC 309 Lecture Notes Week 4 Formal Specs in Testing Intro to Testing Techniques

CSC309-S15-L4 Slide 2

• I. Deriving and refining method specs.
• A. Testing requires that we know exactly what

constitutes valid versus invalid inputs.

• 1. Pre- and postconds answer this question.
• 2. Used to inform unit test development.

CSC309-S15-L4 Slide 3

Overview, cont’d

• B. Recap of what pre/postconds mean.
• 1. Precondition is one boolean expression

that is true before method executes.

• 2. Postcondition is one boolean expression

that is true after method completes.

CSC309-S15-L4 Slide 4

• II. Formal specs used in testing
• A. Formal method test consists of:
• 1. Inputs within legal ranges, expected output
• 2. Inputs outside legal ranges, expected output
• 3. Inputs on boundaries, expected output

CSC309-S15-L4 Slide 5

Formal specs in testing, cont’d

• B. Preconds used to determine inputs.
• C. Postconds used to determine expected output

CSC309-S15-L4 Slide 6

• III. Formal specs in formal verification
• A. To verify formally, two specs needed:
• 1. formal spec of given program
• 2. formal spec of programming language

CSC309-S15-L4 Slide 7

Formal specs in verification, cont’d

• B. Program spec is "entry ticket" to verification.
• C. Details in later lectures.

CSC309-S15-L4 Slide 8

• IV. Precondition enforcement -- "by contract"

versus "defensive programming"

• A. Precond failure means an op is "undefined".
• 1. For abstract spec, this is enough.
• 2. At imple’n level, precond must be dealt

with more concretely.

• 3. Tw
• basic approaches.

CSC309-S15-L4 Slide 9

Precond enforcement, cont’d

• B. Approach 1: Precond is guaranteed true,

before method call.

• 1. This is "programming by contract".
• 2. Precond enforced by callers.
• 3. Verified or checked at calling site.
• 4. Bottom line -- called method assumes its

precond is always true.

CSC309-S15-L4 Slide 10

Precond enforcement, cont’d

• C. Approach 2: Precond is checked by

method being called.

• 1. This is "Defensive programming".
• 2. Method includes logic to enforce its own

precondition.

• 3. Enforcement can:

CSC309-S15-L4 Slide 11

Precond enforcement, cont’d

• a. Assert unconditional failure.
• b. Return "null" value.
• c. Output error report.
• d. Throw an exception.

CSC309-S15-L4 Slide 12

Precond enforcement, cont’d

• D. In Model/View comm’n, we use exception

handling approach.

• E. We will discuss exception handling further in

upcoming lectures.

CSC309-S15-L4 Slide 13

• V. Details of deriving method specs.
• A. Start with Spest specs for 308.
• B. Update and expand based on design refine-

ments done in 309.

• C. For some details, see M3 example.

CSC309-S15-L4 Slide 14

Now on to General System Testing Techniques

CSC309-S15-L4 Slide 15

• VI. General Concepts

CSC309-S15-L4 Slide 16

• VI. General Concepts
• A. Components are independently testable.

CSC309-S15-L4 Slide 17

• VI. General Concepts
• A. Components are independently testable.
• B. Testing is thorough and systematic.

CSC309-S15-L4 Slide 18

• VI. General Concepts
• A. Components are independently testable.
• B. Testing is thorough and systematic.
• C. Testing is repeatable.

CSC309-S15-L4 Slide 19

• VII. Overall system testing styles

CSC309-S15-L4 Slide 20

• VII. Overall system testing styles
• A. Top-down

CSC309-S15-L4 Slide 21

• VII. Overall system testing styles
• A. Top-down
• 1. Top-level methods tested first.

CSC309-S15-L4 Slide 22

• VII. Overall system testing styles
• A. Top-down
• 1. Top-level methods tested first.
• 2. "Stubs" written for lower-level methods.

CSC309-S15-L4 Slide 23

Testing styles, cont’d

• B. Bottom-up

CSC309-S15-L4 Slide 24

Testing styles, cont’d

• B. Bottom-up
• 1. Lower-level methods tested first.

CSC309-S15-L4 Slide 25

Testing styles, cont’d

• B. Bottom-up
• 1. Lower-level methods tested first.
• 2. Function "drivers" written for upper-level

methods.

CSC309-S15-L4 Slide 26

Testing styles, cont’d

• C. Object-oriented

CSC309-S15-L4 Slide 27

Testing styles, cont’d

• C. Object-oriented
• 1. Methods for particular class are tested.

CSC309-S15-L4 Slide 28

Testing styles, cont’d

• C. Object-oriented
• 1. Methods for particular class are tested.
• 2. Stubs and drivers written as necessary.

CSC309-S15-L4 Slide 29

Testing styles, cont’d

• D. Hybrid

CSC309-S15-L4 Slide 30

Testing styles, cont’d

• D. Hybrid
• 1. A combination of top-down, bottom-up,

and object-oriented testing is employed.

CSC309-S15-L4 Slide 31

Testing styles, cont’d

• D. Hybrid
• 1. A combination of top-down, bottom-up,

and object-oriented testing is employed.

• 2. This is a good practical approach.

CSC309-S15-L4 Slide 32

Testing styles, cont’d

• E. Big-bang

CSC309-S15-L4 Slide 33

Testing styles, cont’d

• E. Big-bang
• 1. All compiled in one huge executable.

CSC309-S15-L4 Slide 34

Testing styles, cont’d

• E. Big-bang
• 1. All compiled in one huge executable.
• 2. Cross fingers and run it.

CSC309-S15-L4 Slide 35

Testing styles, cont’d

• E. Big-bang
• 1. All compiled in one huge executable.
• 2. Cross fingers and run it.
• 3. When big bang fizzles,

enter debugger and hack.

CSC309-S15-L4 Slide 36

• VIII. Independently testable designs
• A. Modular interfaces designed thoroughly.
• 1. Don’t fudge on method signatures,

pre/post logic.

• 2. Be clear on public and protected.

CSC309-S15-L4 Slide 37

Independently testable designs, cont’d

• B. Write stubs and drivers as necessary.

CSC309-S15-L4 Slide 38

Independently testable designs, cont’d

• B. Write stubs and drivers as necessary.
• 1. A stub is also known as a mock.

CSC309-S15-L4 Slide 39

Independently testable designs, cont’d

• B. Write stubs and drivers as necessary.
• 1. A stub is also known as a mock.
• 2. Drivers generally supplied by testing

framework, as part of its typical use.

CSC309-S15-L4 Slide 40

• IX. General approaches to testing

CSC309-S15-L4 Slide 41

• IX. General approaches to testing
• A. Black box testing

CSC309-S15-L4 Slide 42

• IX. General approaches to testing
• A. Black box testing
• 1. Each method viewed as black box.

CSC309-S15-L4 Slide 43

• IX. General approaches to testing
• A. Black box testing
• 1. Each method viewed as black box.
• 2. Function tested using spec only.

CSC309-S15-L4 Slide 44

General approaches, cont’d

• B. White-box testing

CSC309-S15-L4 Slide 45

General approaches, cont’d

• B. White-box testing
• 1. Testing based on method code.

CSC309-S15-L4 Slide 46

General approaches, cont’d

• B. White-box testing
• 1. Testing based on method code.
• 2. Inputs that fully exercise code logic.

CSC309-S15-L4 Slide 47

General approaches, cont’d

• B. White-box testing
• 1. Testing based on method code.
• 2. Inputs that fully exercise code logic.
• 3. Each control path is exercised at least
• nce by some test.

CSC309-S15-L4 Slide 48

General approaches, cont’d

• C. Runtime pre/postcond enforcement

CSC309-S15-L4 Slide 49

General approaches, cont’d

• C. Runtime pre/postcond enforcement
• 1. Code added to methods to enforce

pre/postconds at runtime.

CSC309-S15-L4 Slide 50

General approaches, cont’d

• C. Runtime pre/postcond enforcement
• 1. Code added to methods to enforce

pre/postconds at runtime.

• 2. E.g., input range checking.

CSC309-S15-L4 Slide 51

General approaches, cont’d

• C. Runtime pre/postcond enforcement
• 1. Code added to methods to enforce

pre/postconds at runtime.

• 2. E.g., input range checking.
• 3. Function returns (or throws) error if con-

dition is not met.

CSC309-S15-L4 Slide 52

General approaches, cont’d

• C. Runtime pre/postcond enforcement
• 1. Code added to methods to enforce

pre/postconds at runtime.

• 2. E.g., input range checking.
• 3. Function returns (or throws) error if con-

dition is not met.

• 4. Crudely, function could use assert.

CSC309-S15-L4 Slide 53

General approaches, cont’d

• D. Formal verification

CSC309-S15-L4 Slide 54

General approaches, cont’d

• D. Formal verification
• 1. Pre/post conds treated as math’l theorems.

CSC309-S15-L4 Slide 55

General approaches, cont’d

• D. Formal verification
• 1. Pre/post conds treated as math’l theorems.
• 2. Function body treated as math’l formula.

CSC309-S15-L4 Slide 56

General approaches, cont’d

• D. Formal verification
• 1. Pre/post conds treated as math’l theorems.
• 2. Function body treated as math’l formula.
• 3. Verification entails proving precond

implies postcond, through method body.

CSC309-S15-L4 Slide 57

• X. Functional unit test details

CSC309-S15-L4 Slide 58

• X. Functional unit test details
• A. List of test cases produced for each method.

CSC309-S15-L4 Slide 59

• X. Functional unit test details
• A. List of test cases produced for each method.
• B. This constitutes the unit test plan.

CSC309-S15-L4 Slide 60

Case No. Inputs Expected Output Remarks 1 parm 1 = ... ref parm 1 = ... ... ... parm m = ... ref parm n = ... data field a = ... data field a = ... ... ... data field z = ... data field z = ... return = ... throw = ... ... n parm 1 = ... ref parm 1 = ... ... ... parm m = ... ref parm n = ... data field a = ... data field a = ... ... ... data field z = ... data field z = ... return = ... throw = ... ...

CSC309-S15-L4 Slide 61

Unit test details, cont’d

• C. Note that

CSC309-S15-L4 Slide 62

Unit test details, cont’d

• C. Note that
• 1. Must specify all input parameters and

data fields.

CSC309-S15-L4 Slide 63

Unit test details, cont’d

• C. Note that
• 1. Must specify all input parameters and

data fields.

• 2. Must specify all ref parms, return val,

modified fields.

CSC309-S15-L4 Slide 64

Unit test details, cont’d

• C. Note that
• 1. Must specify all input parameters and

data fields.

• 2. Must specify all ref parms, return val,

modified fields.

• 3. Not mentioned assumed "don’t care".

CSC309-S15-L4 Slide 65

Unit test details, cont’d

• D. One test plan for each method.

CSC309-S15-L4 Slide 66

Unit test details, cont’d

• D. One test plan for each method.
• E. Unit test plans included a javadoc comments.

CSC309-S15-L4 Slide 67

• XI. Module, i.e., class testing

CSC309-S15-L4 Slide 68

• XI. Module, i.e., class testing
• A. Write unit test plans for each method.

CSC309-S15-L4 Slide 69

• XI. Module, i.e., class testing
• A. Write unit test plans for each method.
• B. For class as whole, write class test plan.

CSC309-S15-L4 Slide 70

• XI. Module, i.e., class testing
• A. Write unit test plans for each method.
• B. For class as whole, write class test plan.
• C. Guidelines:

CSC309-S15-L4 Slide 71

Class testing, cont’d

• 1. Start with unit tests for constructors.

CSC309-S15-L4 Slide 72

Class testing, cont’d

• 1. Start with unit tests for constructors.
• 2. Next, unit test other constructive methods.

CSC309-S15-L4 Slide 73

Class testing, cont’d

• 1. Start with unit tests for constructors.
• 2. Next, unit test other constructive methods.
• 3. Unit test selector methods.

CSC309-S15-L4 Slide 74

Class testing, cont’d

• 1. Start with unit tests for constructors.
• 2. Next, unit test other constructive methods.
• 3. Unit test selector methods.
• 4. Test certain method interleavings.

CSC309-S15-L4 Slide 75

Class testing, cont’d

• 1. Start with unit tests for constructors.
• 2. Next, unit test other constructive methods.
• 3. Unit test selector methods.
• 4. Test certain method interleavings.
• 5. Stress test.

CSC309-S15-L4 Slide 76

Class testing, cont’d

• D. Use/write a test driver that:

CSC309-S15-L4 Slide 77

Class testing, cont’d

• D. Use/write a test driver that:
• 1. executes each method test plan,

CSC309-S15-L4 Slide 78

Class testing, cont’d

• D. Use/write a test driver that:
• 1. executes each method test plan,
• 2. compares actual with expected output,

CSC309-S15-L4 Slide 79

Class testing, cont’d

• D. Use/write a test driver that:
• 1. executes each method test plan,
• 2. compares actual with expected output,
• 3. reports the differences, if any,

CSC309-S15-L4 Slide 80

Class testing, cont’d

• D. Use/write a test driver that:
• 1. executes each method test plan,
• 2. compares actual with expected output,
• 3. reports the differences, if any,
• 4. optionally records output results.

CSC309-S15-L4 Slide 81

Class testing, cont’d

• E. Concrete examples:

projects/work/calendar/testing/ implementation/source/java/ caltool/schedule/ ScheduleTest.java projects/work/calendar/testing/ implementation/source/java/ caltool/caldb/ UserCalendarTest.java

CSC309-S15-L4 Slide 82

Class testing, cont’d

• F. Java details

CSC309-S15-L4 Slide 83

Class testing, cont’d

• F. Java details
• 1. Each class X has companion testing class

named XTest.

CSC309-S15-L4 Slide 84

Class testing, cont’d

• F. Java details
• 1. Each class X has companion testing class

named XTest.

• 2. Test class may extend class it tests.

CSC309-S15-L4 Slide 85

Class testing, cont’d

• F. Java details
• 1. Each class X has companion testing class

named XTest.

• 2. Test class may extend class it tests.
• 3. Each method X.f has a companion unit test

method named XTest.testF.

CSC309-S15-L4 Slide 86

Class testing, cont’d

• 3. Comment at top of test class describes the

module test plan.

CSC309-S15-L4 Slide 87

Class testing, cont’d

• 3. Comment at top of test class describes the

module test plan.

• 4. The comment for each unit test method

describes unit test plan.

CSC309-S15-L4 Slide 88

Class testing, cont’d

• 3. Comment at top of test class describes the

module test plan.

• 4. The comment for each unit test method

describes unit test plan.

• 5. Each tested class implements dump

method for dumping test values as String.

CSC309-S15-L4 Slide 89

• XI. Integration testing

CSC309-S15-L4 Slide 90

• XI. Integration testing
• A. Once tested, modules are integrated.

CSC309-S15-L4 Slide 91

• XI. Integration testing
• A. Once tested, modules are integrated.
• B. Stubs replaced with actual methods.

CSC309-S15-L4 Slide 92

• XI. Integration testing
• A. Once tested, modules are integrated.
• B. Stubs replaced with actual methods.
• C. Test plan for top-most method(s) rerun with

integrated modules.

CSC309-S15-L4 Slide 93

• XI. Integration testing
• A. Once tested, modules are integrated.
• B. Stubs replaced with actual methods.
• C. Test plan for top-most method(s) rerun with

integrated modules.

• D. Continues until entire system is integrated.

CSC309-S15-L4 Slide 94

• D. Integration testing
• A. Once tested, modules are integrated.
• B. Stubs replaced with actual methods.
• C. Test plan for top-most method(s) rerun with

integrated modules.

• D. Continues until entire system is integrated.

CSC309-S15-L4 Slide 95

Integration testing, cont’d

• E. Concrete example:

projects/work/calendar/testing/ implementation/ integration-test-plan.html

CSC309-S15-L4 Slide 96

• 1. Integrate schedule + caldb

CSC309-S15-L4 Slide 97

• 1. Integrate schedule + caldb
• 2. Add view to schedule+caldb

CSC309-S15-L4 Slide 98

• 1. Integrate schedule + caldb
• 2. Add view to schedule+caldb
• 3. Add admin to schedule+view+caldb

CSC309-S15-L4 Slide 99

• 1. Integrate schedule + caldb
• 2. Add view to schedule+caldb
• 3. Add admin to schedule+view+caldb
• 4. Integrate caldb + caldb.server

CSC309-S15-L4 Slide 100

• 1. Integrate schedule + caldb
• 2. Add view to schedule+caldb
• 3. Add admin to schedule+view+caldb
• 4. Integrate caldb + caldb.server
• 5. Add caldb.server to schedule + ...

CSC309-S15-L4 Slide 101

• 1. Integrate schedule + caldb
• 2. Add view to schedule+caldb
• 3. Add admin to schedule+view+caldb
• 4. Integrate caldb + caldb.server
• 5. Add caldb.server to schedule + ...
• 6. Add options to schedule + ...

CSC309-S15-L4 Slide 102

• 1. Integrate schedule + caldb
• 2. Add view to schedule+caldb
• 3. Add admin to schedule+view+caldb
• 4. Integrate caldb + caldb.server
• 5. Add caldb.server to schedule + ...
• 6. Add options to schedule + ...
• 7. Add file to schedule + ...

CSC309-S15-L4 Slide 103

• 1. Integrate schedule + caldb
• 2. Add view to schedule+caldb
• 3. Add admin to schedule+view+caldb
• 4. Integrate caldb + caldb.server
• 5. Add caldb.server to schedule + ...
• 6. Add options to schedule + ...
• 7. Add file to schedule + ...
• 8. Add edit

schedule + ...

CSC309-S15-L4 Slide 104

• 1. Integrate schedule + caldb
• 2. Add view to schedule+caldb
• 3. Add admin to schedule+view+caldb
• 4. Integrate caldb + caldb.server
• 5. Add caldb.server to schedule + ...
• 6. Add options to schedule + ...
• 7. Add file to schedule + ...
• 8. Add edit

schedule + ...

• 9. Add top-level caltool classes

CSC309-S15-L4 Slide 105

• XII. Black box testing heuristics

CSC309-S15-L4 Slide 106

• XII. Black box testing heuristics
• A. Provide inputs where the precondition is true,

varying inputs to exercise precond logic.

CSC309-S15-L4 Slide 107

• XII. Black box testing heuristics
• A. Provide inputs where the precondition is true,

varying inputs to exercise precond logic.

• B. Provide inputs where the precond is false,

if not a by-contract method.

CSC309-S15-L4 Slide 108

Black box heuristics, cont’d

• B. For data ranges:

CSC309-S15-L4 Slide 109

Black box heuristics, cont’d

• B. For data ranges:
• 1. Provide inputs below, within, above each

precond range.

CSC309-S15-L4 Slide 110

Black box heuristics, cont’d

• B. For data ranges:
• 1. Provide inputs below, within, above each

precond range.

• 2. Provide inputs that produce outputs at bot-

tom, within, at top of each postcond range.

CSC309-S15-L4 Slide 111

Black box heuristics, cont’d

CSC309-S15-L4 Slide 112

Black box heuristics, cont’d

• C. With and/or logic, provide test cases that

fully exercise logic.

CSC309-S15-L4 Slide 113

Black box heuristics, cont’d

• C. With and/or logic, provide test cases that

fully exercise logic.

• 1. Provide an input that makes each clause

both true and false.

CSC309-S15-L4 Slide 114

Black box heuristics, cont’d

• C. With and/or logic, provide test cases that

fully exercise logic.

• 1. Provide an input that makes each clause

both true and false.

• 2. This means 2n test cases, where n is number
• f logical terms.

CSC309-S15-L4 Slide 115

Black box heuristics, cont’d

• D. Provide selected combinations of inputs.

CSC309-S15-L4 Slide 116

Black box heuristics, cont’d

• D. Provide selected combinations of inputs.
• 1. Combinatorially explosive in general.

CSC309-S15-L4 Slide 117

Black box heuristics, cont’d

• D. Provide selected combinations of inputs.
• 1. Combinatorially explosive in general.
• 2. Pairwise combination is practical approach.

CSC309-S15-L4 Slide 118

Black box heuristics, cont’d

• D. Provide selected combinations of inputs.
• 1. Combinatorially explosive in general.
• 2. Pairwise combination is practical approach.
• 3. Used by Spest generator.

CSC309-S15-L4 Slide 119

Black box heuristics, cont’d

• D. Provide selected combinations of inputs.
• 1. Combinatorially explosive in general.
• 2. Pairwise combination is practical approach.
• 3. Used by Spest generator.
• 4. See pairwise.org

CSC309-S15-L4 Slide 120

Black box heuristics, cont’d

• E. For collection classes:

CSC309-S15-L4 Slide 121

Black box heuristics, cont’d

• E. For collection classes:
• 1. Test empty collection.

CSC309-S15-L4 Slide 122

Black box heuristics, cont’d

• E. For collection classes:
• 1. Test empty collection.
• 2. Test with one, two elements.

CSC309-S15-L4 Slide 123

Black box heuristics, cont’d

• E. For collection classes:
• 1. Test empty collection.
• 2. Test with one, two elements.
• 3. Add substantial number of elements.

CSC309-S15-L4 Slide 124

Black box heuristics, cont’d

• E. For collection classes:
• 1. Test empty collection.
• 2. Test with one, two elements.
• 3. Add substantial number of elements.
• 4. Delete each element.

CSC309-S15-L4 Slide 125

Black box heuristics, cont’d

• E. For collection classes:
• 1. Test empty collection.
• 2. Test with one, two elements.
• 3. Add substantial number of elements.
• 4. Delete each element.
• 5. Repeat add/del sequence.

CSC309-S15-L4 Slide 126

Black box heuristics, cont’d

• E. For collection classes:
• 1. Test empty collection.
• 2. Test with one, two elements.
• 3. Add substantial number of elements.
• 4. Delete each element.
• 5. Repeat add/del sequence.
• 6. Stress test with order of magnitude greater

than expected size.

CSC309-S15-L4 Slide 127

• XIII. Function paths
• A. Control flow through method body.
• B. Branching defines path separation point.
• C. An old-school flow chart show paths clearly.
• D. Each path is labeled with a number.

CSC309-S15-L4 Slide 128

• XIV. White box testing heuristics
• A. Exercise each path at least once.
• B. For loops:
• 1. zero times (if appropriate),
• 2. one time
• 3. two times
• 4. a substantial number of times
• 5. max number times (if appro)

CSC309-S15-L4 Slide 129

White box heuristics, cont’d

• C. Provide inputs to reveal imple’n flaws:
• 1. particular operation sequences
• 2. inputs of particular size or range
• 3. inputs that may cause overflow, underflow,
• ther abnormal behavior
• 4. inputs that test well-known problems in

algorithm

CSC309-S15-L4 Slide 130

• XV. Reconciling path coverage
• A. Write purely black box tests.
• B. To ensure coverage, execute under path cov-

erage analyzer.

• C. If analyzer reports paths not being covered,

strengthen black box tests.

CSC309-S15-L4 Slide 131

Reconciling path coverage

• 1. Uncovered paths may contain useless or

dead code.

• 2. When legitimate code, add new black box

test cases.

• D. Complete "grey box" test plan can have path

column:

CSC309-S15-L4 Slide 132

Reconciling path coverage

Test No. Inputs Expected Output Remarks Path i parm 1= ref parm 1 = p ... ... parm m = ref parm n =

CSC309-S15-L4 Slide 133

• XVI. Large inputs and outputs
• A. For collections classes, i/o can grow large.
• B. Can be specified as file data.
• C. Referred to in test plans.

CSC309-S15-L4 Slide 134

• XVII. Test drivers
• A. Once defined, test must be executed.
• B. Test driver written as stand-alone program.
• 1. Executes all tests.
• 2. Records results.
• 3. Provides result differencer.

CSC309-S15-L4 Slide 135

Test drivers, con’td

• C. Automated in

projects/work/calendar/testing/ implementation/source/java/Makefile

Template in

classes/309/lib/csl-Makefiles/ testing-Makefile

• D. Perform tests initially using debugger.

CSC309-S15-L4 Slide 136

• XVIII. Testing concrete UIs
• A. Performed in the same basic manner.
• B. User input is simulated.
• C. Output screens validated initially by human.
• D. Machine-readable form of screen to compare

results mechanically.

CSC309-S15-L4 Slide 137

Testing concrete UIs, cont’d

• E. We’ll look at mechanized GUI testing

in a couple weeks.

CSC309-S15-L4 Slide 138

• XIX. Unit test is "dress rehearsal"

for integration testing ...

• A. Integration "should not" reveal further errors.
• B. From experience, it often does.
• C. In so doing, individual tests become stronger.

CSC309-S15-L4 Slide 139

• XX. Testing with large data.
• A. Suppose we have

class SomeModestModel { ... } class HumongousDatabase { ... }

CSC309-S15-L4 Slide 140

Large-data requirements, cont’d

• B. Modest amount of test data can be built pro-

grammatically, i.e., by calling constructive methods

• C. Large amount of (persistent) data can be

stored external from program, built by exter- nal means if appropriate.

• D. The latter are external test fixtures.

CSC309-S15-L4 Slide 141

• XXI. Other testing terminology
• A. The testing oracle.
• 1. Someone(thing) who knows correct

answers.

• 2. Used to define expected results.
• 3. Also used to analyze incorrect test results.
• 4. In CSC 309, oracle is defined by implemen-

tation of Spest postcondition.

CSC309-S15-L4 Slide 142

Terminology, cont’d

• 5. When building truly experimental code,

spec-based oracle may not be possible.

• a. E.g., AI systems.
• b. Need initial prototype development.

CSC309-S15-L4 Slide 143

Terminology, cont’d

• B. Regression testing
• 1. Run all tests whenever any change is made.
• 2. Must happen before release.
• 3. Ideally happens much more often.
• 4. Ongoing research on "smart" regression.

CSC309-S15-L4 Slide 144

Terminology, cont’d

• C. Mutation testing
• 1. It’s a way to test the tests.
• 2. Strategy -- mutate program, then rerun

tests.

• 3. E.g., "if (x < y)" is mutated to "if (x >= y)".

CSC309-S15-L4 Slide 145

Terminology, cont’d

• 4. With such mutation, tests should fail where

the mutated code produces bad result.

• 5. If previously successful tests do not fail, ...

?

CSC309-S15-L4 Slide 146

Terminology, cont’d

• a. The tests are too weak and need to be

strengthened.

• b. The mutated section of code was "dead"

and should be removed.

CSC309-S15-L4 Slide 147

Terminology, cont’d

• 6. Generally, the first of these is the case.
• 7. Mutation can be used systematically to:

CSC309-S15-L4 Slide 148

Terminology, cont’d

• a. Provide measure of testing effectiveness.
• b. Compare different testing strategies.

CSC309-S15-L4 Slide 149

• XXII. Testing directory structure
• A. Figure 1 in notes ...

CSC309-S15-L4 Slide 150 *.{h,C} project-specific package directories with .java files design implementation c++ diffs project-specific package directories iwth .class files JVM INTEL java ... T *.html javadoc images Makefile source executables Makefile input

• utput-good
• utput

*.o

• utput

diffs

CSC309-S15-L4 Slide 151

Test dir structure, cont’d

• B. Contents of testing subdirs:

CSC309-S15-L4 Slide 152

Directory or File Description *Test.java Implementation of class testing plans. input Test data input files used by test classes.

• utput-good

Output results from last good run of the tests.

• utput-prev-good

Previous good results, in case current results were erroneously confirmed to be good. $PLATFORM/output Current platform-specific output results. $PLATFORM/diffs Differences between current and good results. $PLATFORM/Makefile Makefile to compile tests, execute tests, and difference current results with good results. $PLATFORM/.make* Shell scripts called from the Makefile to per- form specific testing tasks. $PLATFORM/ .../*.class Test implementation object files.

CSC309-S15-L4 Slide 153