M9 Thursday, March 8, 2001 4:30 PM R EQUIREMENTS -D RIVEN A - - PDF document

International Conference On Software Test Automation March 5-8, 2001 San Jose, CA, USA P R E S E N T A T I O N Thursday, March 8, 2001 4:30 PM

REQUIREMENTS-DRIVEN AUTOMATED TESTING

Jeff Tatelman

Spherion Technology Architects

M9

Presentation Paper Bio

Requirements Driven A utomated Testing

Do you have your license to drive?

Jeff Tatelman SQuAD November 14th , 2000

• Business challenges
• Business solution
• Building the requirements driven

testing process

• Building the automated regression

testing process

• Q uestions & discussion

A genda

Testing can no longer be the last step...it must be an on-going part of the development process. Testing can no longer be the last step...it must be an on-going part of the development process.

Build 1 Build 1 Build 2 Build 2 Build 3 Build 3 Build 4 Build 4 App spec Test Test Test Test Test Test Test Test

Iterative Development Challenge

Challenge O f L inking Requirements To Test Cases

Development Life Cycle

Test Automation Test Management Requirements Management Test Automation

Changes

Challenge O f How To G et Started?

• How many test cases do I need?
• W hat type of test cases should I

automate ?

• W hat process should I use?
• How can I take advantage of reusability?

Building the Requirements Driven Testing Process

Business Solution

Iterative T esting

Test Scope & Strategy Test Planning Test Case Construction

Design Construction Analysis Implement

Test Execution/ Evaluation

O bject O riented T esting Approach

Requirements

Unit Testing Integration Testing Systems Testing Regression Testing

Build The Testing A pproach

Requirements

Business Events Traceablility Matrix Develop Test Plan Develop Test Cases

• 1. D eterm ine Business Events

Requirements

Shipping Inventory Maintenance Order Processing Customer Service Example of an Inventory Control System

TC1 Add customer TC2 Add Order TC3 Modify Order TC4 Change Address TC5 Ship Order TC6 Approve Order TC7 Bill Order

Test Case Repository

Example of an Inventory Control System

R1 Customer Service R2 Order Processing R3 Order Change R4 Shipping

TC2 add order TC3 Mod Order TC4 Chg Addr

Run1 Run2

TC5 Ship

Run3

TC6 Approve order

Test Planning Function Coverage Matrix

Requirements Repository

• 2. Develop Traceability M atrix

TC1 Add customer

Traceability M atrix A dvantages

• Validation of requirements
• Ensure coverage efficiency
• Knowledge of what is being testing
• Knowledge of what is not being tested
• Reduce the number of test cases
• Track changes:
• requirements to testing
• testing to requirements
• Reusability for regression/automation

Test Runs: Test Run 1 Description: Add a customer and regular

• rder. Do not approve the order.

Assumptions Customer service, products, prices, records predefined. Order number will be systems generated. Inputs UI objects- Order Service, Customer Accounts Customer John Smith, Products X23, AR232 Output Reports - Orders by Customer, Billing Signoff Run Date: Assigned:

Example of an Inventory Control System

• 3a. Develop Test Plan (Runs)

Day 1 (test cycle) Objective: To add initial orders and new customers Test Run 1 Description: Add a customer and regular

• rder. Do not approve the
• rder.

Test Run 3 Description: Add an inter-company

• rder with five items.

Approve the order and ship in the same day. Example of an Inventory Control System

• 3b. Develop the Test Plan (C ycles)

• 4. Develop T est C ases Detail

Design

Test Data Test Assumptions Test Flow Expected Results

• 4. Develop T est C ases

TEST CASE SPECIFICATION FORM

Project No: 1965 Project Name: ABC Inventory Ordering System Page 1 of 2 Test Case Description: Add a customer. Build No: 1 Case No: 2 Execution Retry No: Written By: Ruby Tuesday Date: 7/25/00 Requirement No: 1, 6, Executed By: Amanda Jones Date: 7/30/00 Setup For Test: Access to ABC Inventory ordering system website Step Action Expected Results Pass / Fail Action Taken if Step Fails 2.1 Click on ‘click here to continue’ ABC Search Customer Page becomes active page. 2.2 Click text field to enter Customer name Cursor appears and text box is enabled

Business Solution

Building t he Aut om at ed Reg ression Testing Pr ocess

• The testing of software after a

modification has been made to ensure the reliability of each software release

• The objective is to prevent defects in

production by building and executing an automated testing process that covers the critical business events.

Regression T esting Definition

Test Test How to

How To Develop A utomated Regression T esting

• Build a defined and repeatable process
• U se the matrix to help control the scope
• Se t up reusable production test data
• Se t up a separate test environment

• Create an automation test plan
• use a subset of systems test cases
• review systems test cases with users
• separate test cases into cycles
• Document screens and test files

How To Develop A utomated Regression T esting (cont.)

Run Run Run

W hy Is The Event/Scenario A pproach So Important For T est A utomation?

• Managing your tests
• Business events to test automation
• Ease of maintenance
• Supports data driven approach
• Reusability (& modularization)
• Same test event “add customer” is

reused in several automated test scenarios

K ey Components To Receiving Y our License To D rive

• Requirements management process

– treating requirements as objects using a repository

• Test management process

– testing repository – events/ scenarios

• Automated testing standards

– test plan – reusability

• Resources/

Training

• Business challenges
• Iterative development
• Changes in life cycle
• How to get started
• Business solution
• Traceability matrix
• Business events/

scenarios

• Reuse test cases for automated

regression testing

Summary

• 2. Develop trace
• 1. Gather requirements 3. Develop manual process
• 4. Manage test cases
• 5. Brainstorm
• 7. Analyze
• 8. Design

Develop test case matrix using business/functions/events to determine what to test Gather business functions from requirement repository, Documents or application. Develop manual test cases and plan Enter test cases into a repository Determine which test cases to automate Set up data files in test environment Formulate design approach & identify required routines

Process Summ ary

Create data source and test scripts

• 6. Develop environment

Develop automation test environment

• 9. Develop

Test Automation Testing Conference Requirements Driven Automated Testing

Inventory Control System Traceability Matrix Example

Function Name Run1 Run2 Run3 Run4 Run5 Run6 Run7 Run8 Run9 Customer Service New X X X Existing X X X X X Delete X Order Processing

• rder type

phone X X X X X X X mail X X

• rder status

complete X X X X X X incomplete X X X pricing taxable X X X X X X tax exempt X X X Inventory Maint. instock X X X X X X X

• ut of stock

X X replenish inventory X X Shipping bill customer approved X X X X X X X past due X X

Test Automation Testing Conference Requirements Driven Automated Testing

Inventory Control System Test Run Example Test Run Description: (Test Run #1)

Add a new customer, with an order by phone with in stock inventory for all items. The order is completed and billing is approved.

Assumptions:

1. Use default values unless specified here.

Input:

1. Customer number: D109166 2. ITEM ID QTY P30KS80 10 P30KB51 20 3UA5400-2D 10 Q220 2 NR421 2 SA1E025 2 LN1E100 5

• 3. Attach input screens for verification.

OUTPUT:

• 1. Use Account Summary Inquiry and Item Inquiry for on-line validation.
• 2. Print screen data before ENTER of data. Attach batch reports printout, and screen

data printout from inquiry screens.

SIGN OFF:

The above test case is: _______________ Approved _______________ Need to re-run ______________________ ____________________ _______________ (USER) (ANALYST) (DATE)

Test Automation Testing Conference Requirements Driven Automated Testing

Inventory Control System Test Run Example Test Run Description: (Test Run #2)

Use an existing customer, with an order by phone with in stock inventory for all items. The

• rder is completed and billing is approved.

Assumptions:

1. Customer must exist on the test database.

Input:

1. Customer number: S122906 2. ITEM ID QTY P30KS80 10 P30KB51 20 LN1E100 5 3.Attach input screens for verification

OUTPUT:

• 1. Use Account Summary Inquiry for on-line validation
• 2. Use Past Due Report for printed validation.
• 3. Print screen data before ENTER of data. Attach batch reports printout, and screen

data printout from inquiry screens.

SIGN OFF:

The above test case is: _______________ Approved _______________ Need to re-run ______________________ ____________________ _______________ (USER) (ANALYST)

Test Automation Testing Conference Requirements Driven Automated Testing

Inventory Control System Test Run Example Test Run Description: (Test Run #3)

Use an existing customer, with an order by phone with in stock inventory for all items. The order is incomplete and billing is past due.

Assumptions:

1. Customer must exist on the test database.

Input:

1. Customer number: G344109 2. ITEM ID QTY SNE400 2 RU3210 1

• 3. Attach input screens for verification

Output:

• 1. Use Account Summary Inquiry for on-line validation
• 2. Use Back Order Report for printed validation.
• 3. Print screen data before ENTER of data. Attach batch reports printout, and screen

data printout from inquiry screens.

SIGN OFF:

The above test case is: _______________ Approved _______________ Need to re-run ______________________ ____________________ _______________ (USER) (ANALYST) (DATE)

Test Automation Testing Conference Requirements Driven Automated Testing

Inventory Control System Test Plan Example

DAY 1 OBJECTIVE: Ensure the accuracy of the data converted and loaded into the Inventory System. Sanity check that all programs and database changes are installed correctly. Backup the database for benchmarking. Test Run 0 Description: Convert tax tables and history tables. Test Run 2 Description:

Use an existing customer, with an order by phone with in stock inventory for all items. The order is tax exempt, completed and billing is approved.

Test Run 5 Description:

Use an existing customer, with an order by phone with in stock inventory for all items. The order is taxable, completed and billing is approved.

Verification Process:

• Account Summary Inquiry for on-line validation
• Back Order printed report
• Item Inquiry for on-line validation
• Past Due Report for printed validation

Test Automation Testing Conference Requirements Driven Automated Testing

Inventory Control System Test Plan Example

DAY 2 OBJECTIVE: Ensure the accuracy of normal phone orders that are complete and taxable. Test approved in stock and approved out of stock orders. Test past due and in stock orders. Test Run 1 Description:

Add a new customer, with an order by phone with in stock inventory for all items. The order is completed and billing is approved.

Test Run 3 Description:

Use an existing customer, with an order by phone with in stock inventory for all items. The order is incomplete and billing is past due.

Test Run 4 Description:

Add a new customer, with an order by phone out of stock inventory for two items. The order is completed and billing is approved. .

Verification Process:

• Account Summary Inquiry for on-line validation
• Back Order printed report
• Item Inquiry for on-line validation
• Past Due Report for printed validation

Requirements-Driven Automated Testing… A License to Drive Abstract: Studies have shown that over 50% of software defects are attributed to poorly defined

• requirements. From a process improvement perspective, it is imperative that project

managers establish a more effective and efficient way of defining and tracking business

• requirements. At the same time, project managers must have a process that allows

requirements to drive the automated testing cycle. Many companies purchase automated testing tools without having a solid requirements- based testing (RBT) process for knowing what to test and when to test. These companies attempt to use the tools for unit testing when the application is still unstable, and in many cases, out of frustration, the tools end up "on the shelf." However, when business requirements drive the development of test cases and test scripts, companies realize the return on investment in their technology. This presentation demonstrates how RBT best practices coupled with automated software quality (ASQ) techniques solve the problems that plague a majority of software development projects. This presentation will describe a “how-to” approach for the development of an automated regression testing process, which tests the key requirements-based business processes and the daily business activities. It will also explain how automated test scripts will be reused for every maintenance release of the company’s application. In summary, the presentation will describe:

• Management and definition of business requirements
• Developing a matrix on what to test based on requirements
• Verification of requirements
• Reducing test cases and maximizing application coverage based upon

requirements

• Generation of test plans/cases
• Creation of automated test scripts from those requirements

Attendees will gain an understanding of how to:

• Employ an automated repository for managing requirements
• Use business events (requirements) to identify test cases
• Develop a requirements-to-test matrix showing what to test
• Develop a test plan driven by requirements
• Implement test automation for regression testing

BIOGRAPHY OF PRESENTER Jeff Tatelman CSTE, Managing Consultant, Spherion Technology Architects Jeff Tatelman brings more than 20 years of quality assurance and consulting experience in the software development industry including management of Quality Assurance teams and developing Software Quality processes. Jeff has implemented structured and automated testing, change management and tools to support the software quality process. Jeff has been responsible for implementing requirements based testing at several Fortune 1000 companies including NCR, Sprint, Coca-Cola, and SunTrust Bank. Jeff is well known in the software quality industry and has an extensive background in public speaking at various quality organizations, as well as international and local

• companies. He is also the co-founder of the Software Quality Association in Denver.

Jeff Tatelman Spherion Technology Architects 4500 Cherry Creek Drive South Suite 1050 Denver, Colorado 80246 720-524-2517

Jeff Tatelman

Jeff Tatelman brings more than twenty years of quality assurance and consulting experience in the software development industry including management of Quality Assurance teams and developing Software Quality processes. Jeff has implemented structured and automated testing, change management and tools to support the software quality process. Jeff has been responsible for implementing requirements based testing at several Fortune 1000 companies including NCR, Sprint, Coca-Cola, and SunTrust Bank. Jeff is well known in the software quality industry and has an extensive background in public speaking at various quality organizations, as well as international and local companies. He is also the co-founder of the Software Quality Association in Denver.