W8 5/16/2007 1:45:00 PM "A P ROCESS FOR S COPING AND P LANNING T - - PDF document

▶
w8
SMART_READER_LITE
LIVE PREVIEW

W8 5/16/2007 1:45:00 PM "A P ROCESS FOR S COPING AND P LANNING T - - PDF document

Mar 24, 2024 412 likes •722 views

BIO PRESENTATION W8 5/16/2007 1:45:00 PM "A P ROCESS FOR S COPING AND P LANNING T EST A UTOMATION P ROJECTS " Jennifer Seale Nationw ide Insurance International Conference On Software Test Analysis And Review May 14-18, 2007

slide-1
SLIDE 1

BIO PRESENTATION International Conference On Software Test Analysis And Review May 14-18, 2007 Orlando, FL USA

W8

5/16/2007 1:45:00 PM

"A PROCESS FOR SCOPING AND PLANNING TEST AUTOMATION PROJECTS"

Jennifer Seale Nationw ide Insurance

slide-2
SLIDE 2

Jennifer Seale Jennifer Seale

Jennifer Seale holds Master’s degrees in English and Computer Science as well as active CSTE, PMP, and Mercury CPC-QTP designations. She has seven years of experience in the software industry as a developer, tester, test coordinator, and test automation solution architect. Jennifer is a top-rated adjunct faculty member at Franklin University and has taught a variety of undergraduate programming classes over the past four years. For the past two years Jennifer has contributed to the development of the Test Automation Center of Excellence at Nationwide Insurance.

slide-3
SLIDE 3

Test Automation Centers of Excellence

Jennifer Seale

slide-4
SLIDE 4

2

Impediments to Automation Projects

  • Lack of experience and knowledge
  • Lack of resources
  • Environment instability
  • Project Process Immaturity
  • Costs

– Initial investment may be large (tools, knowledge, training) – Return on Investment may be long term – Maintenance required

slide-5
SLIDE 5

3

Nationwide’s Solution

  • Centralized Test Automation Team
  • Experts in designing automation

solutions and using automation tools

  • Partner with projects
  • Provide various services
slide-6
SLIDE 6

4

Developing a Center of Excellence

  • Set up Shop
  • Document Review Process
  • Process for engagements
  • Templates to support process
  • Process to manage requests for services
  • Process to track status
  • Continuous Improvement
  • Best Practices and Standards
slide-7
SLIDE 7

5

Customer Engagement Process (CEP) Six Phase Process

  • 1. Scoping
  • 2. Planning
  • 3. Development
  • 4. Execution
  • 5. Wrap Up
  • 6. Maintenance
slide-8
SLIDE 8

6

Templates Supporting our Process

  • Initial Screening Survey
  • Readiness Assessment
  • Complexity Assessment
  • Statement of Work (SOW)

– Schedule – Cost Benefit Analysis (CBA) – Training and Knowledge Transfer Plan

  • Proof of Concept (POC)
  • Customer Development Plan
  • Test Automation Design Document
  • Test Automation User Guide
  • Project Log
slide-9
SLIDE 9

7

CEP Phase 1: Scoping

  • Learn about project and needs
  • Complete Assessments
  • Determine scope of engagement
  • Develop preliminary schedule
  • Conduct POC
  • Get approval to continue to planning
slide-10
SLIDE 10

8

Scoping: Assessments

  • Initial Screening Survey

– Does automation make sense for the project? – Focus on goals, application, environments, builds

  • Readiness Assessment

– Is the project ready for a test automation effort? – Focus on resources, processes, artifacts, funding

  • Complexity Assessment

– How much effort will it take to automate this project? – Focus on scope, dependencies

slide-11
SLIDE 11

9

Scoping: Determine Scope We typically use the following method to determine the scope of the project:

1. Determine automation needs. 2. Break down automation needs into a list of well- defined, manageable units of work. 3. Prioritize the list in terms of functional dependencies. 4. If the prioritized list is long, divide the list into logical sections. These sections will be used during planning to allow a phased approach to automation development.

slide-12
SLIDE 12

10

Scoping: Preliminary SOW

  • Using template as base, add details as they

are revealed through conversations with the customer.

  • Helps to have information in writing
  • Will become a complete description of all the

work involved in the automation effort.

  • Need to determine preliminary schedule to

help in resource planning.

slide-13
SLIDE 13

11

Scoping: Proof of Concept (POC)

  • A POC is a prototype of the automation that

demonstrates the technical feasibility of the automation project.

  • Use Complexity Assessment results to help

determine an interesting subset of scenarios to cover in POC.

  • Automate this subset of test scenarios using

the selected automation tool(s) and report results.

slide-14
SLIDE 14

12

Scoping: Benefits of a POC

  • Increased confidence in the project
  • Early identification of challenges and risks
  • Early set up of automation environment
  • Provides useful input to estimating effort
slide-15
SLIDE 15

13

Scoping: Customer Development Plan

When we say “No” to an automation project, we offer to provide the customer with a document that

  • utlines what needs to be addressed on their project

before an automation effort can begin. Most useful when:

  • the project is interested in automation
  • automation makes sense for the project
  • ther factors exist that put the success of an automation effort

at risk

slide-16
SLIDE 16

14

CEP Phase 2: Planning

  • Refine SOW
  • Refine Schedule
  • Perform Cost Benefit Analysis
  • Training and Knowledge Transfer Plan
  • Test Automation Design Document
  • Approval for SOW
slide-17
SLIDE 17

15

Planning: Refining SOW

  • Finalize communication plan
  • Finalize details of objectives and

deliverables

  • Finalize details of schedule
  • Finalize cost estimates
slide-18
SLIDE 18

16

Planning: Cost Benefit Analysis

  • Using template as a base, tailor to fit

project specifics.

  • Estimates all costs associated with

automation effort.

  • Predicts benefits of automation effort

based on how often the project will run the automation once it is complete.

  • Predicts the breakeven point, when the

benefit is equal to the cost.

slide-19
SLIDE 19

17

Planning: Automation Costs

When planning for an automation, consider the following costs: – Staff Resources – Training – Learning curve – Initial Development – Execution and Analysis – Maintenance – Enhancements – Hardware – Support

slide-20
SLIDE 20

18

Planning: Automation Benefits

There are three types of benefit that can be realized through Test Automation:

  • Reduction –benefit realized when a test suite that is

currently being executed manually is replaced by an automated suite.

  • Coverage –benefit realized when an automated suite

is created or extended to perform testing that is not currently being executed manually due to lack to resources or funds.

  • Quality –benefit realized by detecting defects earlier

in the software development life cycle due to the automation.

slide-21
SLIDE 21

Return on Investment

Generic Automation Savings Profile

Quality Reduction Coverage

Initial Investment

  • 1000
  • 500

500 1000 1500 2000 2500 3000 3500 4000 Jan-06 Mar-06 May-06 Jul-06 Sep-06 Nov-06 Jan-07 Mar-07 May-07 Jul-07 Sep-07 Nov-07 Jan-08 Mar-08 May-08 Jul-08 Sep-08 Nov-08 Jan-09 Mar-09 May-09 Jul-09 Sep-09 Nov-09

Months Hours Break Even

slide-22
SLIDE 22

20

Planning: Training and Knowledge Transfer Plan

  • It is important that we prepare key

members of the project team to take

  • ver the automation.
  • We tailor this plan specifically to meet

the needs of the project.

  • Often this plan contains its own

schedule.

  • Typically produce a Test Automation

User Guide.

slide-23
SLIDE 23

21

Planning: Test Automation Design Document

  • Documents the automation design for

the project.

  • Provides an overview of exactly what

was automated for Test leads to use when planning future testing efforts.

  • Documents all design decisions and

workarounds that were employed.

slide-24
SLIDE 24

22

Approval of SOW

  • The completed SOW is sent to the

customer stakeholders for review.

  • We conduct a face-to-face meeting

where we walk the customer through the SOW to ensure we are in agreement.

  • Once the SOW is approved,

development begins.

slide-25
SLIDE 25

23

CEP Phase 5: Wrap Up

  • Complete Training and Knowledge

Transfer and Plan

  • Update SOW with results of

engagement

  • SOW Complete approval meeting
  • Customer Satisfaction Survey
  • Engagement Retrospective
slide-26
SLIDE 26

24

CEP Phase 6: Maintenance

  • Follow up with projects every 6 months

for life of project

  • Update CBA with actuals
  • Address noteworthy issues
slide-27
SLIDE 27

25

Managing Requests for Services

  • Spreadsheet of requests established
  • Tab per request with overview page
  • Log all pre-engagement contact
  • Used for tracking and forecasting
  • Available on shared drive, all team

members responsible for keeping up- to-date.

slide-28
SLIDE 28

26

Tracking Status: The Project Log

  • Spreadsheet with multiple tabs
  • Tab per document to track change requests
  • Tab to track technical issues
  • Tab to track resource issues
  • Tab for lessons learned
  • Always available for review
  • Wealth of historic data for future projects
slide-29
SLIDE 29

27

Continuous Improvement

  • Document Change Request (CR) System

– Survey Results – Lessons Learned – Retrospectives

  • Quarterly reviews to address CRs and

improve the process and associated templates

slide-30
SLIDE 30

28

Best Practices and Standards

  • Perfectly positioned to tailor theoretical

best practices to specific business

  • Reuse solution models across

applications

  • Develop standards for specific

technologies used by the business