T19 Concurrent Session Thursday 10/25/2007 3:00 PM JUMP TO: Biographical Information The Presentation Session-Based Exploratory Session-Based Exploratory Testing—With a Twist Testing—With a Twist Presented by: Brenda Lee, Parallax, Inc. Presented at: The International Conference on Software Testing Analysis and Review October 22-26, 2007; Anaheim, CA, USA 330 Corporate Way, Suite 300 , Orange Park, FL 32043 888-268-8770 � 904-278-0524 � sqeinfo@sqe.com � www.sqe.com
Br Brenda Lee enda Lee Brenda Lee is a Senior Analyst with Parallax, Inc., in Albuquerque, New Mexico, where she is on a multi-year contract to Sandia National Labs as a System Test Lead and Requirements Analyst. She has over 15 years of experience in the software industry. Her experience includes Usability Testing, User Interface Design, and Technical Writing for companies such as IBM, Sensormatic Electronics, and Tyco Fire & Security. She received her Bachelor's Degree in Psychology from West Texas State University in 1987, and her Master's Degree in Psychology from New Mexico State University in 1991.
Session-Based Exploratory Testing— with a Twist Brenda S. Lee Parallax, I nc. (a wholly-owned subsidiary of Energy Solutions)
Agenda • System Test Charter Highlights • Introduction to Session-Based Exploratory Testing • Project Overview • Modifications made to the technique • Test Preparation/Execution • Comparison of Techniques • Lessons Learned • Conclusions / Summary
Our System Test Charter • Parallax provides system test services to Sandia National Labs • Independent evaluators of the system • Find defects prior to release • Characterize the behavior of the system • Formalized and repeatable testing
STARWest 2006 • Attended with my manager • Participated in the following tutorials: – “How to Break Software” – “Just in Time Testing” • Introduced to Session-Based Exploratory Testing (SBET) “We should try this sometime.” “I s it repeatable?”
What is Session-Based Exploratory Testing? • Developed by James Bach & Jonathan Bach • Consists of: – Charter Statement • Area of exploration – Executed by pairs of testers • Application user • Test session scribe – Designated session time
Project Overview • Web-based Asset Management system – Planned to replace a legacy system – Capabilities: Asset Management, Equipment Maintenance, and Inventory • System-Level Requirements signed in March 2005 • 1 st release completed in Spring 2006
Project Overview-continued • November/ December 2006 – Maintenance function under development – Sandia’s customer requested change in priorities • New priority: I nventory Functionality – Required to conduct a critical inventory • Deploy I nventory function by 1Q2007
Project Risks • Non-negotiable delivery date • No time to develop formalized procedures • If development slipped… – Impacts system test cycle • No time to fix issues • Known problems in production system • Could system test be completed in 2 weeks?
The Opportunity • Non-negotiable delivery date • Dynamic state of the application design • Time invested in scripted procedures— for different functionality • “Emerging Requirements” for user roles and data visibility Was this our opportunity to try Session-Based Exploratory Testing?
Obtaining Management Approval • Testing could be completed in 2 weeks if… – SBET was the primary test method – Minimal regression testing • How will the team learn the technique? – Hold SBET Workshop • How will we prove traceability to the requirements?
Here’s the twist… • Modify SBET Charters to…. – I nclude relevant requirements • Provides traceability – Provide sample test cases • Gave testers ideas about testing • Helped track test case execution
Sample Test Charter
Test Preparations • Early January: Wrote proposed test charters • Late-January: Conducted SBET Workshop • Mid-February: Held test team planning session – Reviewed Charters – Identified pairs to execute each charter – Explained dependencies between tests • March 2, 2007: System test started • March 12, 2007: System test ended
Test Execution • Test Cycle: 7 business days – 14 planned charters – 3 charters added – Regression tests • 45 requirements evaluated • 217 test cases executed • 15 issues identified Sandia’s customer said: “This had to be the shortest system test cycle ever.”
Comparison of Test Methods Test Method / Formalized Area of Analysis Procedures SBET Planning 2 months 1 month Execution 28 days 7 days Req. Validated 41/60 41/ 45 Test Cases 82/116 (est.) 193/ 217 (Pass/ Total) I ssues Logged 10 15 Testers Required 4 7 High Level Summary 5 days later 2 days later
System Deployment • Delivery date was met • New functions in use days after system test • $9 Million inventory was executed – Reconciled to Five Dollars • Team recognized for accomplishments • SBET is a valuable in our environment
Lessons Learned 1. Limit the scope of charters 2. Make the pass/fail criteria obvious 3. Conduct timely debriefing for sessions 4. Use standardized test summary templates 5. Trust testers to use good judgment 6. Trust the method
The Future • SBET used on another project • I ntend to use SBET on more projects • Question: Regression Tests and SBET? – Use same charters for regression tests – Write procedures from session notes – Combination of both
Conclusions About SBET • Useful in traditional development environments • Provides broad characterization • Repeatable due to detailed session notes • Effective at finding issues with the system • Modifications addressed traceability needs
Summary • Apply what you learn at conferences • Modify a technique to satisfy internal needs • Share what you learn with others
Acknowledgements • Management & Colleagues in Dept. 6453 at Sandia National Labs • Parallax Colleagues in Albuquerque, NM & Germantown, MD • James Bach & Jon Bach • Robert Sabourin, “Just In Time Testing” tutorial • Lee Copeland and SQE • My husband, Michael Lee
Questions & Answers THANK YOU! bslee@sandia.gov
Recommend
More recommend
