Session-Based Exploratory Session-Based Exploratory TestingWith a - - PDF document

T19

Concurrent Session

Thursday 10/25/2007 3:00 PM JUMP TO: Biographical Information The Presentation

Session-Based Exploratory Session-Based Exploratory Testing—With a Twist Testing—With a Twist

Presented by: Brenda Lee, Parallax, Inc.

Presented at: The International Conference on Software Testing Analysis and Review October 22-26, 2007; Anaheim, CA, USA 330 Corporate Way, Suite 300 , Orange Park, FL 32043 888-268-8770 904-278-0524 sqeinfo@sqe.com www.sqe.com

Br Brenda Lee enda Lee

Brenda Lee is a Senior Analyst with Parallax, Inc., in Albuquerque, New Mexico, where she is on a multi-year contract to Sandia National Labs as a System Test Lead and Requirements Analyst. She has over 15 years of experience in the software industry. Her experience includes Usability Testing, User Interface Design, and Technical Writing for companies such as IBM, Sensormatic Electronics, and Tyco Fire & Security. She received her Bachelor's Degree in Psychology from West Texas State University in 1987, and her Master's Degree in Psychology from New Mexico State University in 1991.

Session-Based Exploratory Testing— with a Twist

Brenda S. Lee

Parallax, I nc.

(a wholly-owned subsidiary of Energy Solutions)

Agenda

• System Test Charter Highlights
• Introduction to Session-Based Exploratory

Testing

• Project Overview
• Modifications made to the technique
• Test Preparation/Execution
• Comparison of Techniques
• Lessons Learned
• Conclusions / Summary

Our System Test Charter

• Parallax provides system test services to

Sandia National Labs

• Independent evaluators of the system
• Find defects prior to release
• Characterize the behavior of the system
• Formalized and repeatable testing

STARWest 2006

• Attended with my manager
• Participated in the following tutorials:

– “How to Break Software” – “Just in Time Testing”

• Introduced to Session-Based Exploratory

Testing (SBET) “We should try this sometime.” “I s it repeatable?”

What is Session-Based Exploratory Testing?

• Developed by James Bach & Jonathan Bach
• Consists of:

– Charter Statement

• Area of exploration

– Executed by pairs of testers

• Application user
• Test session scribe

– Designated session time

Project Overview

• Web-based Asset Management system

– Planned to replace a legacy system – Capabilities: Asset Management, Equipment Maintenance, and Inventory

• System-Level Requirements signed in

March 2005

• 1st release completed in Spring 2006

Project Overview-continued

• November/ December 2006

– Maintenance function under development – Sandia’s customer requested change in priorities

• New priority: I nventory Functionality

– Required to conduct a critical inventory

• Deploy I nventory function by 1Q2007

Project Risks

• Non-negotiable delivery date
• No time to develop formalized procedures
• If development slipped…

– Impacts system test cycle

• No time to fix issues
• Known problems in production system
• Could system test be completed in 2 weeks?

The Opportunity

• Non-negotiable delivery date
• Dynamic state of the application design
• Time invested in scripted procedures—

for different functionality

• “Emerging Requirements” for user roles and data

visibility

Was this our opportunity to try Session-Based Exploratory Testing?

Obtaining Management Approval

• Testing could be completed in 2 weeks if…

– SBET was the primary test method – Minimal regression testing

• How will the team learn the technique?

– Hold SBET Workshop

• How will we prove traceability to the

requirements?

Here’s the twist…

• Modify SBET Charters to….

– I nclude relevant requirements

• Provides traceability

– Provide sample test cases

• Gave testers ideas about testing
• Helped track test case execution

Sample Test Charter

Test Preparations

• Early January: Wrote proposed test charters
• Late-January: Conducted SBET Workshop
• Mid-February: Held test team planning session

– Reviewed Charters – Identified pairs to execute each charter – Explained dependencies between tests

• March 2, 2007: System test started
• March 12, 2007: System test ended

Test Execution

• Test Cycle: 7 business days

– 14 planned charters – 3 charters added – Regression tests

• 45 requirements evaluated
• 217 test cases executed
• 15 issues identified

Sandia’s customer said: “This had to be the

shortest system test cycle ever.”

Comparison of Test Methods

Test Method / Area of Analysis Formalized Procedures SBET Planning

2 months

1 month Execution

28 days

7 days

• Req. Validated

41/60

41/ 45 Test Cases (Pass/ Total)

82/116 (est.)

193/ 217 I ssues Logged

10

15 Testers Required 4

7

High Level Summary

5 days later

2 days later

System Deployment

• Delivery date was met
• New functions in use days after system test
• $9 Million inventory was executed

– Reconciled to Five Dollars

• Team recognized for accomplishments
• SBET is a valuable in our environment

Lessons Learned

• 1. Limit the scope of charters
• 2. Make the pass/fail criteria obvious
• 3. Conduct timely debriefing for sessions
• 4. Use standardized test summary templates
• 5. Trust testers to use good judgment
• 6. Trust the method

The Future

• SBET used on another project
• I ntend to use SBET on more projects
• Question: Regression Tests and SBET?

– Use same charters for regression tests – Write procedures from session notes – Combination of both

Conclusions About SBET

• Useful in traditional development environments
• Provides broad characterization
• Repeatable due to detailed session notes
• Effective at finding issues with the system
• Modifications addressed traceability needs

Summary

• Apply what you learn at conferences
• Modify a technique to satisfy internal needs
• Share what you learn with others

Acknowledgements

• Management & Colleagues in Dept. 6453 at

Sandia National Labs

• Parallax Colleagues in Albuquerque, NM &

Germantown, MD

• James Bach & Jon Bach
• Robert Sabourin, “Just In Time Testing”

tutorial

• Lee Copeland and SQE
• My husband, Michael Lee

Questions & Answers

THANK YOU!

bslee@sandia.gov