Software Testing Per Madsen Aalborg University Denmark Last - PowerPoint PPT Presentation

1 Software Testing Per Madsen Aalborg University Denmark Last revised 6 January 2004 Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Where is Aalborg? 2 Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Overview of This Lecture 3 � Why do we need software testing? � What is software testing? � Testing techniques � White box (structural testing) � Black box (functional testing) � Testing strategy � Testing tools � Testing and Eiffel Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Why Software Testing? 4 � Software bugs affect our everyday life. � Blue Screen of Death (Windows/ Office). � Also bugs in Linux / Open Office / Mozilla etc. � Traffic ☺ � Software problems cost a lot of money. � Ariane 5 � Costs resulting from insufficient software testing: 22.2 to 59.5 billion dollars! NIST, National Institute of Standards and Technology (May 2002) � Lots of other examples. Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

What Is Software Testing? 5 � Testing is the activity where we try to discover potential bugs in our program. � Testing is not: � Program proving � Program debugging � Program testing can be used to show the presence of defects, but never their absence! Dijkstra, Structured Programming (1972) Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

The Problem of Testing 6 � In a perfect world we would test every possible execution of a program. � But: � Even very simple programs have a large number of possible executions. � Programs with loops have an infinite number of executions. Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Testing Terminology 7 � Test cases � Test driver � Test stub � Test oracle Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Testing Activities 8 � Testing software typically involves: � Selecting what to test. � Building test cases. � Executing the test cases. � Comparing actual results / expected results. � Measuring execution characteristics (memory used, time consumed, etc.) � Testing is also bug prevention. Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Testability 9 � Testability: the ’price’ of testing a program � Testability is a design issue � Two key aspects of testability � Controllable � How can we control a program? � Observable � How can we observe the output of a program? Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Test Units 10 � What can be tested? � One instruction � One feature � One class � One cluster � One program � One library � We can do testing on all levels. Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Levels of Testing 11 � Unit test � Test a single unit/ module of a program. � Integration test � Put the units together and test them as a whole. � Big bang or iterative integration. � Acceptance test � Determine whether the customer can accept the program. � Regression test � Re-run selected tests after updating the software. Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Black Box Testing 12 � Testing “in the large” � Also known as functional testing � We test a program without using any knowledge about how the program is built. � The primary idea is to discover bugs related to the functionality of the unit. � Do we get the right output for a given input? � Find bugs in interfaces. � Find problems with efficiency, disk space etc. � Examples: � Equivalence testing � Boundary testing Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Equivalence Partitioning 13 � Equivalence partitioning is a black box technique based on the idea that we can split the input into a number of classes. � Members of each class should behave in a similar way. � Examples: � The input is a interval [ a..b ] � Choose one sample less than a , one between a and b and one larger than b . � The input is a set V � Choose a sample in V and one not in V . Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Eiffel Example 14 feature swap_elements ( a : ARRAY [ INTEGER ]; x : INTEGER ; y : INTEGER ) is -- Swap the position of two elements x and y in an array a . require a_not_void: a /= Void local tmp : INTEGER do tmp := a . item ( x ) a . put (a. item ( y ), x ) a . put ( tmp , y ) ensure x_is_old_y: a . item ( x ) = old a . item ( y ) y_is_old_x: a . item ( y ) = old a . item ( x ) end Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Test Cases 15 � Candidates for test cases � a is empty / not empty � x < y , x = y , y > x � x > a . size , y> a . size � x = a . size , x = a.size + 1, x= a.size -1 � Observation: � Static type checking and contracts can minimize the number of test cases we need. � e.g add a precondition � require x > 0 and x < = a . size Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

16 Break Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

White Box Testing 17 � Testing “in the small”. � Also known as structural testing. � We use our knowledge about the internal structure of the program when testing the program. � Examples: � Path testing � Focuses on the control flow. � Data flow � Focuses on critical data values. Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Eiffel Example 18 feature testing ( a : INTEGER ; b : INTEGER ) is -- Use a loop and a branch to illustrate how to make a flow chart. local c : INTEGER do from c :=0 until c = a loop if b >0 then c := c + b else c := c + 1 end end end Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Path Testing Example 19 � Path Testing: � Construct the flow chart � Make a test case for every independent path � The flow chart consists of: � Entry / exit points � Branches � Statements Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Number of Independent Paths 20 � Independent paths � 1 � 1,2,3,5,1 � 1,2,4,5,1 � The number of independent paths is equal to the number of regions in the graph. � Cyclomatic complexity = number of edges – number of nodes + 2 � CC = the number of test cases we need to cover all edges. � CC for the example 6 – 5 + 2 = 3 Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

White Box vs. Black Box 21 � The typical use of white box testing: � Used for unit testing. � Used by the programmer. � Not suitable for whole programs. � The typical use of black box testing � Can be used by the programmer and by others. � Used in integration and system testing. Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Testing Strategy 22 � How do we plan and structure the testing of a large program? � Who is testing? � Developers / special testing teams. � It is hard to test your own code. � What test levels do we need? � Unit, integration, system, acceptance test. � How do we do it in practice? � Manual testing vs. testing tools. Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Testing and OO Technology 23 � Observations � Polymorphism / dynamic binding increases the number of test cases we need. � A class is a natural unit for testing. � The sequence of feature calls is important. � Information hiding can make testing harder. � It is hard to observe non-exported attributes. � Deferred classes � Cannot be testing directly. � Inheritance � Should inherited features be re-tested? Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Testing Tools 24 � Automatic unit testing tools � JUnit for Java � Gobo EiffelTest for Eiffel � Available for all major languages � Capture and replay tools � Testing coverage analysis tools � Regression test tools � Save the result of selected feature calls and compare the results with new versions of the features. Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

Extreme Programming 25 � Extreme Programming (XP) is a light weighted software development process. � Test Driven Development � Write test cases before you write the code. � Let the test cases drive your design. � Testability is the most important design issue. � Assume simplicity � Start out with the simplest thing that can work. � Embrace change � The automated testing approach allows you to make changes. Chair of Softw are Engineering I ntroduction to Programming - January 6 2004

‘Automated’ Unit Testing 26 � Write test cases as features in the same language that you program. � Put assertions in the test cases. � Let a tool manage the test cases. � Run and re-run the test cases. � Keep track of bugs. � Re-run the test cases after each update. � Automated regression test. � This approach automates the execution and the book-keeping of test cases but in the end the developer still has to write the test cases. Chair of Softw are Engineering I ntroduction to Programming - January 6 2004