T9 Thursday, October 30, 2003 11:15 AM T HE P ROCESS OF E FFECTING L - - PDF document

BIO PRESENTATION International Conference On Software Testing Analysis & Review October 27-31, 2003 San Jose, CA USA

T9

Thursday, October 30, 2003 11:15 AM

THE PROCESS OF EFFECTING LASTING CHANGE: MOVING TO A HIGHLY STRUCTURED TESTING

MODEL

Jon Harader & Vivek Bhatia Wells Fargo Internet Services Group QA

Jon Harader

Over 19 years experience designing, developing, integrating, delivering and testing solutions within the Fortune 50 process, Mass Transportation, and Financial Services

• industries. He is also highly skilled in sound people management techniques, project

management, process change leadership, and continuous quality improvement

• techniques. In addition, he has held a wide range of positions that have yielded his

experience base: Programmer/Analyst, Systems Analyst, Supervisor, Project/Implementation Manager, and It Management Staff Consultant at Chevron; Senior/Lead Design Engineer, Product Line Director at small startup company, Spear Technologies and most recently, Manager of QA Technical Services at Wells Fargo.

Vivek Bhatia

13 years of experience in implementing and maintaining enterprise technology solutions. He has spent the last 4 years as a “hands-on” VP/executive Director, actively involved in daily team management, as well as working with executive committees to set future strategy and goals. He has a balance of technical and managerial experience, with 8 years in project and program management, and 5 years as a DBA and application development team lead. He also has well rounded and brand name industry experience, with 4.5 years in financial services, including Wells Fargo, Fannie Mae, ABN*Amro, NASD; 3 years in retail, including Nike, Estée Lauder; 3.5 years in government, including the FAA, DOJ, Navy; and 2 years in internal professional services consulting He is well versed in formal project management, having co-wrote Deloitte Consulting’ Project Management methodology based on the internationally accepted PMBOK model published by the Project Management Institute, and then having solely written the Project Management methodology and tools for Fort Point Partners Consulting, which generated an immediate 50 percent decrease in project costs for future projects, providing consistent on-time delivery.

10/30/03 StarWest 2003 1

The Process of Effecting The Process of Effecting Lasting Change Lasting Change

Moving to a Highly Structured Testing Model Moving to a Highly Structured Testing Model

Presented By: Presented By: Vivek Bhatia Vivek Bhatia & Jon Harader & Jon Harader Wells Fargo Internet Services Group QA Wells Fargo Internet Services Group QA

10/30/03 StarWest 2003 2

What’s on Tap for Today

Context Starting Benchmarks Vision Planning Execution Mid Project Recycle Parting Wisdom

10/30/03 StarWest 2003 3

Context: Business

Internet Financial Service Applications @

www.wellsfargo.com

On Line Banking On Line Application Process Bill Payment Brokerage Etc.

10/30/03 StarWest 2003 4

Context: Internet Services Group

.COM arm of Wells Fargo that supports the

Internet Channel

Started out as an alternative customer channel In the process of moving to a more mature

• rganization

10/30/03 StarWest 2003 5

Context: Business Scope

Vast array of functions from 7 lines of business Over 46 unique products Internet Customer base (approx)

4.2M Online Banking Customers 1.3M Bill Manager Customers

Internet channel is the Banks largest “store”

10/30/03 StarWest 2003 6

Context: Technical Scope

Several thousand web pages Supported by

Large Number of Web servers Large Number of Application servers Multiple Data Centers Several Internal SORs Several 3rd party end points

10/30/03 StarWest 2003 7

Context: ISG/QA Group

Functional Testing

Manual Automated

Performance Testing

QA-Technical Services Banking Brokerage Bill Manager Public Site

10/30/03 StarWest 2003 8

Starting Benchmark: Mgmt Challenge

Unstructured Test Case Management System Dept was person-centric, not process-centric Testing not approached from a functional

perspective – hard to quantify and prove coverage

10/30/03 StarWest 2003 9

Starting Benchmark: Mgmt Challenge

Each QA functional testing area (specific

application) had its own standards and processes

No comparison criteria across testing groups Difficult to determine size or pattern of data

issues

10/30/03 StarWest 2003 10

Starting Benchmark: Staff Challenge

Staff assigned to test specific applications App-specific standards and processes resulted in an a

loss of efficiency and made it difficult to predict schedules if a non-seasoned person as was assigned

Data issues resulted in considerable manual effort

10/30/03 StarWest 2003 11

Starting Benchmark: Tech Challenge

Multiple non documented automation

methodologies

No automated linkage between test case

specification and automated test case code

Non database driven automation framework

10/30/03 StarWest 2003 12

What do we do now?

10/30/03 StarWest 2003 13

Vision: Management

Plot entire QA dept on CMM model as a single entity,

and upgrade as needed

Level 2: Define, Document new core processes for entire

dept

Level 3: Standardize across groups, ensure consistency Level 4: Define Metrics and Measure inter and intra-group

progress

Level 5: Optimize metrics

Extract and Manage Knowledge into TCMS Contain and reduce overall costs

10/30/03 StarWest 2003 14

Vision: How to Improve our CMM level

Define and Document a standardized, best-of-breed process as the core based on the best of each individual process, and enforce where appropriate in the test management tool. Dept was person-centric, not process- centric Document test cases in the test management tool and deploy using functional test case categorization. Testing not approached from a functional perspective – hard to quantify and prove coverage. Buy[not build] & Deploy a highly structured, database- driven, Test Case Management System. Unstructured Test Case Management System.

Solution Problem

10/30/03 StarWest 2003 15

Vision: How to Improve our CMM level

Create a core set of standards and processes to deploy across all groups Each QA Functional Testing Area has it

• wn set of standards and process

Common, Core standards and processes allow metrics to be generated, managed and optimized. No comparison criteria across testing groups. Integrate automation framework with test management tool to track and trend test results. No sense of size or pattern of data issues.

Solution Problem

10/30/03 StarWest 2003 16

Vision: Staff

Consistent guidance and approach Unified management team with common goals

and objectives

Predictable work load

10/30/03 StarWest 2003 17

Vision: How to Help the Staff

Common standards and process across groups allow the automation and regression staff to work efficiently, and allow everyone to smooth out the work week. App-specific standards and processes result in an a loss of efficiency and make it difficult to predict schedules Consistent guidance, common standards, and process across groups allow individuals to move laterally and vertically. Staff are assigned to specific applications Database driven automation framework integrated with a test management tool and data SOR significantly reduce the number of data issues and the time required to solve any problems. Data issues result in excessive manual effort.

Solution Problem

10/30/03 StarWest 2003 18

Vision: Technical

Data Repository TCMS SORs Data Xfer Mechanism

Applications Under Test

TestData db Regression Results

Automation Code

Automation Framework

10/30/03 StarWest 2003 19

Vision: Technical Solution

Store test case specs in attributes in TCMS, then create a db-driven framework that will retrieve specs at runtime from TCMS. No automated linkage between test case specification and automated test case. Use a single, well documented, and tightly controlled methodology and framework across all groups. Multiple non documented automation methodologies. Integrate all systems via back-end database, and make sure the framework can link into the systems. Non database driven automation framework.

Solution Problem

10/30/03 StarWest 2003 20

Planning: Main & Peripheral projects

Implement structured TCMS Db-driven automation framework

1 db for automated Test Case data 1 db for base state for Test Case data 1 db for regression results

Deploy a functional tree Create core cross-dept test case specification

standards and processes

Re-engineer cross-dept workflow to match core

standards and processes

10/30/03 StarWest 2003 21

Planning: Anticipated Paradigm Shifts

Standardization of processes and test case

specification templates within and across departments

Functional test categorization and management

Test case ownership Test case maintenance

Highly structured and controlled input of data

10/30/03 StarWest 2003 22

Planning: Executive and Line Mgmt Buy-In

For executives, we focused on:

• The # of people and $$ needed to support the current writing and coding

paradigm

• The # of people and time needed to run the test cases
• The # of hours/week needed to solve problems that shouldn’t have
• ccurred in the first place [i.e., data & process]

For line management, we focused on

• The # of hours/week their people spent communicating test case specific

information to other groups

• Their ability to quickly determine coverage and analyze results by

functional area

• The reduced # of hours required to code and run test cases for their group
• the # of hours/week needed to solve problems that shouldn’t have
• ccurred in the first place [i.e., data & process]

10/30/03 StarWest 2003 23

Execution: Implementation Process

Migrate 2-3 projects from Word into TCMS to assist with dept-specific attribute definition Define Functional Tree Pilot 2-3 projects directly entering specs into TCMS, modify as needed to meet usability, functional

• requirements. Iterate core & dept-specific

processes, standards, and attributes

Q1 Q2 Q3

proof of concept pilot stage

Create training materials, rollout to department using internal champions

rollout stage

Testing Group 1 QA-Tech Svcs

Pilot

Testing Group 2

Proof of Concept 1st Draft of core processes, test case specification standards, attributes

…

10/30/03 StarWest 2003 24

Execution: Proof of Concept Stage

Review existing test case organization & prepare strawman functional tree Set up TCMS & Import/load sample test cases into tree & load all attributes Review existing test case docs and dept template & prepare strawman DCF & reports Dept SMEs and Mgr review strawman DCF, tree, & reports and approve for pilot Dept Mgr picks one project to serve as the pilot, in which an analyst will enter info

Week 1 Week 2 Week 3 Week 4

Challenges:

• Getting consensus on the functional tree definition and extension
• Dept-specific custom field identification [DCF]
• Determining correct level of data and test case detail

Advice:

• Keep this as short as possible, but do enough due diligence to make sure you do your homework and

that the pilot is successful. Goal: Make sure TCMS works before putting it in the critical path where it could impact departmental project deadlines and budget.

Testing Group QA-TS

10/30/03 StarWest 2003 25

Execution: Pilot Stage

Analyst uses TCMS to enter & maintain test case info for the life of 1 project Analyst finds issues with defined DCFs

• r tree and reports

to implementor Compare to cross- project/cross-dept rqmts and determines whether to fix or escalate to dept SME

• r TCMS steering

committee as appropriate Either the fix is incorporated or Analyst told to accept it.

1 full project lifecycle or length of time to purge issues list

Post- mortem review by SMEs, Dept Mgr, and QATS to approve for full rollout to dept.

Challenges:

• Creation of single standard, workflow, process, and reports to fit entire testing group.
• Conflicting goals of adhering to new tree structure vs completing project
• Miscommunication/misunderstanding what’s a given and what’s possible to change during pilot

Advice:

• Be patient, go slowly, and focus on the usability of the end result. If you build a system that is more

usable, better integrated, and more robust than the current state, people will naturally come around

2 weeks

Goal: Start slow and do one project, and use lessons learned to improve the process, standards, and usability

Testing Group QA-TS

10/30/03 StarWest 2003 26

Execution: Rollout Stage

Testing Group QA-TS

Challenges:

• Staff need to digest changes and extrapolate what this means to them and how they work

Advice:

• Create customized training materials for each different application, so people can review and

understand what this means to them

• Do NOT have a single overview session for the department, as any concerns that don’t come up

because the session has too many people will result in un-resolved angst

Create a core training guide for department, with specific customized examples for each functional or application testing group that will use system Conduct individualized 2 hour training session for each functional or application group, led by internal champion, with all pilot participants present to act as teacher’s assistants After 1-2days, active and regular follow- up with each person to make sure they know how to use the system

10/30/03 StarWest 2003 27

Mid Project Recycle

Top management buy-in Standards, workflow, and process vary more than expected Line management buy-in of theory but didn’t fully realize

magnitude of impact

Usability is king Technology is a foreign language

10/30/03 StarWest 2003 28

Parting Wisdom: Managing the Change While Getting The Wash Out

Pick a mix of ground troops & management for the pilot

that can accurately speak for the group

Let the pilot take time if need be, but make sure

participants feel like they’re building something that is both much more usable and meets their needs

Be flexible but stand by the guiding principles

10/30/03 StarWest 2003 29

Parting Wisdom: Top 10 Challenges

1.

People:

1.

Acceptance of the constraints inherent in a highly structured testing model

2.

Acceptance of functional tree and extensions

3.

Acceptance of structured test definition and execution

4.

Conflicting goals of adhering to new structure vs. completing projects on time(getting wash out)

2.

Process

1.

Creating a common, core process and standards across groups

2.

Building consensus on dept-specific customized fields

3.

Level of required test case and data detail documentation

4.

Miscommunication/misunderstanding of what pilot participants can take as a given, and what they can change.

3.

Technology

1.

Out-of-box reports didn’t match needs

2.

TCMS UI is vastly different from Word or Excel, and not really customizable

10/30/03 StarWest 2003 30

Parting Wisdom: Advice

Get buy-in from key stakeholders before and

throughout the project

Change is hard Communicate, Communicate, Communicate Use an internal person for the heavy lifting and main

work