1 Measuring quality Costs of quality assurance Is it possible to - PDF document

Key Points � Many different characteristics of quality Quality Assurance and Testing � Importance of having independent quality assurance from development � The deliverable of QA is information CSE 403 � Write it down Lecture 22 � QA is not free Slides derived from a talk by Ian King QA ‘Good Practices’ Build Process � Source control � Undo the ‘oops � Centralized build � Be sure everyone is testing the same bits � Avoid platform dependencies � How often are new builds generated? � Periodic � Event-Driven � Configuration management Developer Practice Defect Process � Buddy builds � Why are defects tracked? � How are defects tracked? � Code review � What is the lifecycle of a bug? � Code analysis tools � How are defects prioritized? � Unit testing � Controlled check-ins/triage process � Defect analysis: � Defect source analysis � Root cause analysis 1

Measuring quality Costs of quality assurance � Is it possible to quantify software � Programmer Productivity quality? � 8-20 LOC / day � Building QA into the schedule Elements of Test Strategy Quality Assurance: � Test specification Test Development & Execution � Test plan � Test harness/architecture � Test case generation Developing Test Strategy � Test schedule Slides derived from a talk by Ian King Requirements feed into test Where is your focus? design � The customer � What factors are important to the customer? � The customer � Reliability vs. security � The customer � Reliability vs. performance � The customer � Features vs. reliability � The customer � Cost vs. ? � The customer � What are the customer’s expectations? � The customer � How will the customer use the � Schedule and budget software? 2

Test Specifications Test specification: example � CreateFile method � What questions do I want to answer about � Should return valid, unique handle for this code? Think of this as experiment design � initial ‘open’ for appropriate resource � In what dimensions will I ask these � subsequent calls for shareable resource � for files, should create file if it doesn’t exist questions? � Should return NULL handle and set error indicator � Functionality if resource is � Security � nonexistent device � inappropriate for ‘open’ action � Reliability � in use and not shareable � Performance � unavailable because of error condition (e.g. no disk � Scalability space) � Must recognize valid forms of resource name � Manageability � Filename, device, ? Test Plans Test plan: goals � How will I ask my questions? Think of this as � Enables development of tests the “Methods” section � Proof of testability – if you can’t design � Understand domain and range it, you can’t do it � Establish equivalence classes � Review: what did you miss? � Address domain classes � Valid cases � Invalid cases � Boundary conditions � Error conditions � Fault tolerance/stress/performance Test plan: example Performance testing CreateFile method � � Test for performance behavior Valid cases � execute for each resource supporting ‘open’ action � � opening existing device � Does it meet requirements? � opening existing file opening (creating) nonexistent file � execute for each such resource that supports sharing � � Customer requirements multiple method calls in separate threads/processes � multiple method calls in single thread/process � Invalid cases � Definitional requirements (e.g. Ethernet) � nonexistent device � file path does not exist � � Test for resource utilization in use and not shareable � Error cases � insufficient disk space � � Understand resource requirements invalid form of name � permissions violation � Boundary cases � � Test performance early e.g. execute to/past system limit on open device handles � device name at/past name length limit (MAXPATH) � Fault tolerance � � Avoid costly redesign to meet performance execute on failed/corrupted filesystem � execute on failed but present device � requirements 3

Security Testing Stress testing � Is data/access safe from those who should � Working stress: sustained operation at not have it? or near maximum capability � Is data/access available to those who should � Goal: resource leak detection have it? � How is privilege granted/revoked? � Breaking stress: operation beyond � Is the system safe from unauthorized control? expected maximum capability � Example: denial of service � Goal: understand failure scenario(s) � Collateral data that compromises security � “Failing safe” vs. unrecoverable failure or � Example: network topology data loss Globalization Test Cases � Localization � Actual “how to” for individual tests � UI in the customer’s language � Expected results � German overruns the buffers � One level deeper than the Test Plan � Japanese tests extended character sets � Automated or manual? � Globalization � Environmental/platform variables � Data in the customer’s language � Non-US values ($ vs. Euro, ips vs. cgs) � Mars Global Surveyor: mixed metric and SAE Test case: example Test Harness/Architecture � CreateFile method � Test automation is nearly always worth the time and expense � Valid cases � English � How to automate? � open existing disk file with arbitrary name and full path, � Commercial harnesses file permissions allowing access create directory ‘c:\foo’ � Roll-your-own � copy file ‘bar’ to directory ‘c:\foo’ from test server; � � Record/replay tools permissions are ‘Everyone: full access’ execute CreateFile(‘c:foo\bar’, etc.) � Scripted harness � expected: non-null handle returned � � Logging/Evaluation 4

Where The Wild Things Are: Test Schedule Challenges and Pitfalls � Phases of testing � “Everyone knows” – hallway design Unit testing (may be done by developers) � � “We won’t know until we get there” Component testing � Integration testing � � “I don’t have time to write docs” System testing � � Dependencies – when are features ready? � Feature creep/design “bugs” Use of stubs and harnesses � � Dependency on external groups � When are tests ready? Automation requires lead time � � The long pole – how long does a test pass take? 5