W15 Track 5/6/2009 3:00:00 PM "End-to-End Testing - - PDF document

W15

Track

5/6/2009 3:00:00 PM

"End-to-End Testing "End-to-End Testing in an Enterprise Agile Environment" in an Enterprise Agile Environment"

Pr Prese esented by d by: : Billie Bell Billie Bell Intuit, Inc. Intuit, Inc.

Presented at: 330 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 · 904-278-0524 · sqeinfo@sqe.com · www.sqe.com

Billie Bell

Billie J. Bell, E2E QA Program Manager for Intuit, developed the end-to-end (E2E) QA Program Manager role after recognizing the benefits of engaging QA teams across Intuit's primarily agile project teams in understanding how cross-project integration testing influences achievement of overall business objectives. Using the rollout of specific company-wide initiatives provides a pragmatic context and motivation for development of new processes and promotes an E2E quality mindset beyond QA. Billie’s recent accomplishments include coordinating the integration of 30+ applications with a PCI compliance platform, a consolidated entitlement and software license management platform infrastructure integrating desktop and Web service applications, and a 10-layer subscription and billing SAAS initiative integrating 3rd party and internally-developed applications. Including her eight years at Intuit, Billie brings more than 20 years of experience in enterprise financial software development in QA management, product management, and program management roles in both Fortune 500 and start-up software development companies.

2/16/2009 1

Business Dem ands Com plex Technology End-to-End Testing in an Enterprise Agile Environment

Billie Bell Intuit, Inc.

A il

Technology Agile Developm ent

This presentation discusses the theory and practice of large scale integration testing An enterprise working to reach middle

Best practice

Some practical examples of mechanisms Intuit has found to be An enterprise working to reach middle ground between agile and process- based methodologies

Context

practice tip

mechanisms Intuit has found to be effective appear throughout the presentation

2/16/2009 2

Three forces in today’s business environm ent interact to create challenges for quality leaders

Accelerated Business Dem ands Com plex Agile Developm ent

Com plex Technology Developm ent

Despite our best efforts, results can som etim es com e across as grim headlines

Accelerated Business Dem ands Com plex Technology Agile Developm ent

2/16/2009 3

W e expect headlines that reflect our intentions to deliver a high quality release on tim e

Accelerated Business Dem ands Com plex Technology Agile Developm ent

Today’s challenges bring opportunities to take a fresh look at quality leadership

2/16/2009 4

Community

Quality leaders are uniquely capable to set the course for success in today’s business setting

Community

• rganizers

for quality Visibility

Visibility promoters

QA leadership role Expand quality definition Enlist all stakeholders

As a com m unity organizer for quality you can build the foundation for end-to-end m indset

QA leadership role Expand quality definition Enlist all stakeholders

2/16/2009 5

Start w ith your ow n vision

QA leadership role Expand quality definition Enlist all stakeholders

Develop shared vision

QA leadership role Expand quality definition Enlist all stakeholders

2/16/2009 6

Lead process change across the organization

QA leadership role Expand quality definition Enlist all stakeholders

A quality leader’s enhanced role should also include prom oting visibility √ Help QA and other teams see their applications in p Q pp the context of the enterprise √ Help the organization stay one step ahead by acknowledging and managing risk

QA leadership role Expand quality definition Enlist all stakeholders

2/16/2009 7

Prom oting visibility into business rationale is an elem ent of the new role QA leaders play

Business context and relative priority result in more effective testing More of the business t ti i t id

Accelerated Business Dem ands Com plex Technology Agile Developm ent

transaction is outside a single application

Agile best practices emphasize connection with the customer

QA is a business discipline, not just a set of tasks that occur in serial fashion after developm ent

Best practices to prom ote visibility bring practicality to the quality com m unity Some examples of ways to promote visibility

Best practice tip

p y p y

√ Weekly 1-hour Spotlight Sessions for overview presentation by

√ business process experts √ Enterprise and system architects

√ 2-day workshops for more complex, new systems √ Cross-project test case reviews

QA leadership role Expand quality definition Enlist all stakeholders

2/16/2009 8

Quality leaders have the perspective to lead the

• rganization in seeing the risks ahead

QA leadership role Expand quality definition Enlist all stakeholders

Risk assessm ent and m anagem ent by a cross-team netw ork prom otes accountability Use risk planning sessions to

• 1. Identify risks
• 2. Identify indicators
• 3. Plan mitigation
• 4. Identify response

k

• 5. Monitor risks

QA leadership role Expand quality definition Enlist all stakeholders

2/16/2009 9

Best practices used to m anage risk include collaboration and analysis Ways to uncover risks

Best practice tip

y

√ Pre-assessment questionnaire √ Defect analysis from previous projects √ Risk workshops

QA leadership role Expand quality definition Enlist all stakeholders

W ith a collective understanding of w here w e are, w e can exam ine how w e m easure quality

2/16/2009 10

Beyond a single Extend definition of quality Beyond a single application Across release cycles

cycles

QA leadership role Expand quality definition Enlist all stakeholders

Order System High Level Architectural Diagram

blank

blank

Order Sources Paym ent Processing and

Call Center App

W eb applications

blank

Call center

and Accounting

Processor

Legacy applications

2/16/2009 11

Order System High Level Architectural Diagram

blank

blank

Call Center App

W eb applications

blank

Call center

Processor

Legacy applications

The Purchase use case is architecturally nine layers deep

Call Center App

Processor

2/16/2009 12

The Update custom er inform ation use case spans five layers

Business Use Case

Web App A Purchase Update customer Info Bill Order Engine Web Order Processing 3rd Party CC Processor Call Center App C a l l C e n t e r s , M a i l a n d F a x

• r

d e r s Process 2 Web App B Web App C Process 1 Credit Card Authorization Cancel Bill subscription

Engine Order Source Processor IVR C Settlement Financial Accounting Customer Information

The Cancel use case is 5 layers deep

Call Center App

Processor C

2/16/2009 13

Bill Subscription use case is executed outside the application

Call Center App

Processor C

Realities of today’s enterprise technology can leave our traditional quality m onitors lacking

2/16/2009 14

W hen it com es to code coverage, the w hole is not the sum of its parts Com ponent Coverage

Com ponent 3 8 0 %

8 0 % x 8 0 % 6 4 % x 8 0 %

Com ponent 2 8 0 % Com ponent 1 8 0 %

Process Coverage

8 0 % 6 4 % 5 1 %

QA leadership role Expand quality definition Enlist all stakeholders

Payoff for integration analysis from business and technical view is m ore robust test cases

Integration analysis

Identify systems and system-to-system interfaces Identify and high level business transactions that are dependent

Best practice tip

Identify and high-level business transactions that are dependent

• n these interfaces or trigger their execution

Analyze the use of data across multiple systems to track data flow and reconciliation Simplify and communicate all of the above

Use UML diagrams to illustrate functional touchpoints Summarize and conceptualize the wealth of information Communicate the big picture, provide maps to details

Reporting test execution results by functional use case

QA leadership role Expand quality definition Enlist all stakeholders

2/16/2009 15

Onboarding applications post release extend accountability for quality

End-to-end testing becomes more like no-end-in-sight testing

QA leadership role Expand quality definition Enlist all stakeholders

Onboarding a new applications to our m odel m ay appear as a new box on the diagram

Process 2 Web App B New Web App Process 1 Web App A Order Engine Web Order Processing 3rd Party CC Processor Order Source sor Call Center App C a l l C e n t e r s , M a i l a n d F a x

• r

d e r s Process 2 1 Credit Card Authorization Settlement

Source Proces IVR Financial Accounting Customer Information

QA leadership role Expand quality definition Enlist all stakeholders

2/16/2009 16

But in reality, the picture is m ulti-dim ensional

• Backward

compatibility

Considerations

New Web App

Web Order Processing

New Web App

Process 2 Process 1 Web App A New Web App

p y

• Persistent test

infrastructure

• Technical support

mechanisms

Web Order Processing

QA leadership role Expand quality definition Enlist all stakeholders

Test autom ation and easy integration test m echanism s ensure quality post release

Automated regression tests

Automated smoke tests in pre-production environments on a frequent basis

Best practice tip

frequent basis Autolab to run build verification regression tests

Can be run on demand by development engineers

Test data mechanisms

Baseline test data sets for key data repositories Tools that verify data synchronization Dynamic data reservation systems to avoid data collisions

Dynamic data reservation systems to avoid data collisions

Examples: Test credit cards, license numbers

QA leadership role Expand quality definition Enlist all stakeholders

2/16/2009 17

Success in building an enterprise-w ide quality com m unity requires collaboration and inclusion

Enlist all stakeholders

The less obvious the obvious

QA leadership role Expand quality definition Enlist all stakeholders

2/16/2009 18

QA team s can leverage new best practices used by their developm ent partners

Certain agile best practices provide clear benefits to quality for the project at Other best practices evolve to provide even greater benefit to end-to-end quality for the project at hand

– Test-driven development can result in better unit tests – Retrospectives result in more frequent lessons- learned

benefit to end to end testing

– Business use cases created by system and enterprise architects – Imbedded developers from upstream projects

QA leadership role Expand quality definition Enlist all stakeholders

Business process experts are key contributors throughout the process I l Involve business process matter experts in all Where you find them can differ in every situation Be flexible in your expectations

QA leadership role Expand quality definition Enlist all stakeholders

phases situation

2/16/2009 19

I n sum m ary, an effective E2 E test strategy is based on shared vision and credible results

• Quality leaders as activists who

C t b d lit it – Create a broad quality community – Enable the organization to maintain sight of the big picture

• Quality defined as a continuum

– Outward-focused in addition to being inward-focused – Extending beyond the release cycle

• Operationally inclusive of all stakeholders and methodologies

– Leveraging agile development practices – Finding more ways to gain from business process expertise across the company

No single tool or m ethodology guarantees success for an end-to-end test strategy

• It is a strategy based on a combination of people, tools,

mechanisms, and best practices

• It works best when it is emergent and organic, but the seed

has to be planted and nurtured

• In many ways the best practices are the same as traditional;

what sets it apart is the focus on the end-to-end business process

It’s about people more than tools It’s about mindset more than skill set