Learning objectives Introduce dimensions and tradeoff between test - PowerPoint PPT Presentation

Learning objectives • Introduce dimensions and tradeoff between test and analysis activities A Framework for Testing and • Distinguish validation from verification Analysis activities • Understand limitations and possibilities of test and analysis (c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 1 (c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 2 Verification and validation Validation and Verification • Validation: SW does the software system meets the user's real Specs Actual needs? System Requirements are we building the right software? • Verification: Validation Verification does the software system meets the Includes usability Includes testing, requirements specifications? testing, user inspections, static are we building the software right? feedback analysis (c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 3 (c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 4

Verification or validation depends on Validation and Verification Activities the specification 1 2 3 4 5 6 7 8 Actual Needs and Delivered User Acceptance (alpha, beta test ) Constraints Example: elevator response Package Review System System System Test Specifications Integration Analysis / Review Unverifiable (but validatable) spec: ... if a user Subsystem Integration Test presses a request button at floor i, an available Subsystem Design/Specs Analysis / elevator must arrive at floor i soon... Review validation Unit/ Unit/ Component Verifiable spec: ... if a user presses a request Module Test Components Specs button at floor i, an available elevator must verification arrive at floor i within 30 seconds... User review of external behavior as it is determined or becomes visible (c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 5 (c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 6 ever Getting what you need ... You can’t always get what you want Theorem proving: Perfect verification of • optimistic inaccuracy: we may Unbounded effort to arbitrary properties by verify general logical proof or exhaustive accept some programs that do properties. testing (Infinite effort) not possess the property (i.e., Property Model checking: it may not detect all Decidable but possibly intractable checking of violations). Decision simple temporal Pass/Fail properties. – testing Procedure Data flow Program analysis • pessimistic inaccuracy: it is not guaranteed to accept a Typical testing program even if the program techniques Precise analysis of simple syntactic does possess the property properties. being analyzed Correctness properties are undecidable Correctness properties are undecidable – automated program analysis the halting problem can be embedded in almost techniques • simplified properties: reduce every property of interest Simplified Optimistic properties inaccuracy the degree of freedom for Pessimistic simplifying the property to inaccuracy check (c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 7 (c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 8

Example of simplified property: Some Terminology Unmatched Semaphore Operations • Safe : A safe analysis has no optimistic original problem simplified property inaccuracy, i.e., it accepts only correct programs. Java prescribes a more restrictive, but • Sound : An analysis of a program P with respect if ( .... ) { statically checkable ... to a formula F is sound if the analysis returns construct. lock(S); true only when the program does satisfy the Static } formula. checking for ... match is • Complete : An analysis of a program P with synchronized(S) { if ( ... ) { necessarily ... respect to a formula F is complete if the ... inaccurate ... ... analysis always returns true when the program unlock(S); } actually does satisfy the formula. } (c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 9 (c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 10 Summary • Most interesting properties are undecidable, thus in general we cannot count on tools that work without human intevention • Assessing program qualities comprises two complementary sets of activities: validation (daes the software do what it is supposed to do?) and verification (does the system behave as specificed?) • There is no single technique for all purposes: test designers need to select a suitable combination of techniques (c) 2007 Mauro Pezzè & Michal Young Ch 2, slide 11