T9 Thursday, October 30, 2003 11:15 AM T HE P ROCESS OF E FFECTING L - PDF document

BIO PRESENTATION T9 Thursday, October 30, 2003 11:15 AM T HE P ROCESS OF E FFECTING L ASTING C HANGE : M OVING TO A H IGHLY S TRUCTURED T ESTING MODEL Jon Harader & Vivek Bhatia Wells Fargo Internet Services Group QA International Conference On Software Testing Analysis & Review October 27-31, 2003 San Jose, CA USA

Jon Harader Over 19 years experience designing, developing, integrating, delivering and testing solutions within the Fortune 50 process, Mass Transportation, and Financial Services industries. He is also highly skilled in sound people management techniques, project management, process change leadership, and continuous quality improvement techniques. In addition, he has held a wide range of positions that have yielded his experience base: Programmer/Analyst, Systems Analyst, Supervisor, Project/Implementation Manager, and It Management Staff Consultant at Chevron ; Senior/Lead Design Engineer, Product Line Director at small startup company, Spear Technologies and most recently, Manager of QA Technical Services at Wells Fargo . Vivek Bhatia 13 years of experience in implementing and maintaining enterprise technology solutions. He has spent the last 4 years as a “hands-on” VP/executive Director, actively involved in daily team management, as well as working with executive committees to set future strategy and goals. He has a balance of technical and managerial experience, with 8 years in project and program management, and 5 years as a DBA and application development team lead. He also has well rounded and brand name industry experience, with 4.5 years in financial services, including Wells Fargo, Fannie Mae, ABN*Amro, NASD ; 3 years in retail, including Nike, Estée Lauder ; 3.5 years in government, including the FAA, DOJ, Navy ; and 2 years in internal professional services consulting He is well versed in formal project management, having co-wrote Deloitte Consulting’ Project Management methodology based on the internationally accepted PMBOK model published by the Project Management Institute, and then having solely written the Project Management methodology and tools for Fort Point Partners Consulting, which generated an immediate 50 percent decrease in project costs for future projects, providing consistent on-time delivery.

The Process of Effecting The Process of Effecting Lasting Change Lasting Change Moving to a Highly Structured Testing Model Moving to a Highly Structured Testing Model Presented By: Presented By: Vivek Bhatia & Jon Harader & Jon Harader Vivek Bhatia Wells Fargo Internet Services Group QA Wells Fargo Internet Services Group QA 10/30/03 StarWest 2003 1

What’s on Tap for Today � Context � Starting Benchmarks � Vision � Planning � Execution � Mid Project Recycle � Parting Wisdom 10/30/03 StarWest 2003 2

Context: Business � Internet Financial Service Applications @ www.wellsfargo.com � On Line Banking � On Line Application Process � Bill Payment � Brokerage � Etc. 10/30/03 StarWest 2003 3

Context: Internet Services Group � .COM arm of Wells Fargo that supports the Internet Channel � Started out as an alternative customer channel � In the process of moving to a more mature organization 10/30/03 StarWest 2003 4

Context: Business Scope � Vast array of functions from 7 lines of business � Over 46 unique products � Internet Customer base (approx) � 4.2M Online Banking Customers � 1.3M Bill Manager Customers � Internet channel is the Banks largest “store” 10/30/03 StarWest 2003 5

Context: Technical Scope � Several thousand web pages � Supported by � Large Number of Web servers � Large Number of Application servers � Multiple Data Centers � Several Internal SORs � Several 3rd party end points 10/30/03 StarWest 2003 6

Context: ISG/QA Group � Functional Testing Bill Manager Public Site Brokerage Banking � Manual � Automated � Performance Testing QA-Technical Services 10/30/03 StarWest 2003 7

Starting Benchmark: Mgmt Challenge � Unstructured Test Case Management System � Dept was person-centric, not process-centric � Testing not approached from a functional perspective – hard to quantify and prove coverage 10/30/03 StarWest 2003 8

Starting Benchmark: Mgmt Challenge � Each QA functional testing area (specific application) had its own standards and processes � No comparison criteria across testing groups � Difficult to determine size or pattern of data issues 10/30/03 StarWest 2003 9

Starting Benchmark: Staff Challenge � Staff assigned to test specific applications � App-specific standards and processes resulted in an a loss of efficiency and made it difficult to predict schedules if a non-seasoned person as was assigned � Data issues resulted in considerable manual effort 10/30/03 StarWest 2003 10

Starting Benchmark: Tech Challenge � Multiple non documented automation methodologies � No automated linkage between test case specification and automated test case code � Non database driven automation framework 10/30/03 StarWest 2003 11

What do we do now? 10/30/03 StarWest 2003 12

Vision: Management � Plot entire QA dept on CMM model as a single entity, and upgrade as needed � Level 2: Define, Document new core processes for entire dept � Level 3: Standardize across groups, ensure consistency � Level 4: Define Metrics and Measure inter and intra-group progress � Level 5: Optimize metrics � Extract and Manage Knowledge into TCMS � Contain and reduce overall costs 10/30/03 StarWest 2003 13

Vision: How to Improve our CMM level Problem Solution Unstructured Test Case Management Buy[not build] & Deploy a highly structured, database- System. driven, Test Case Management System. Dept was person-centric, not process- Define and Document a standardized, best-of-breed centric process as the core based on the best of each individual process, and enforce where appropriate in the test management tool. Testing not approached from a functional Document test cases in the test management tool and perspective – hard to quantify and prove deploy using functional test case categorization. coverage. 10/30/03 StarWest 2003 14

Vision: How to Improve our CMM level Problem Solution Each QA Functional Testing Area has it Create a core set of standards and processes to deploy own set of standards and process across all groups No comparison criteria across testing Common, Core standards and processes allow metrics to groups. be generated, managed and optimized. No sense of size or pattern of data Integrate automation framework with test management issues. tool to track and trend test results. 10/30/03 StarWest 2003 15

Vision: Staff � Consistent guidance and approach � Unified management team with common goals and objectives � Predictable work load 10/30/03 StarWest 2003 16

Vision: How to Help the Staff Problem Solution Staff are assigned to specific Consistent guidance, common standards, and process applications across groups allow individuals to move laterally and vertically. App-specific standards and processes Common standards and process across groups allow the result in an a loss of efficiency and make automation and regression staff to work efficiently, and it difficult to predict schedules allow everyone to smooth out the work week. Data issues result in excessive manual Database driven automation framework integrated with a effort. test management tool and data SOR significantly reduce the number of data issues and the time required to solve any problems. 10/30/03 StarWest 2003 17

Vision: Technical Automation Framework Data Repository TestData db Data Xfer Mechanism Regression Results TCMS SORs Automation Code Applications Under Test 10/30/03 StarWest 2003 18

Vision: Technical Solution Problem Solution Multiple non documented automation Use a single, well documented, and tightly controlled methodologies. methodology and framework across all groups. No automated linkage between test case Store test case specs in attributes in TCMS, then create a specification and automated test case. db-driven framework that will retrieve specs at runtime from TCMS. Non database driven automation Integrate all systems via back-end database, and make framework. sure the framework can link into the systems. 10/30/03 StarWest 2003 19

Planning: Main & Peripheral projects � Implement structured TCMS � Db-driven automation framework � 1 db for automated Test Case data � 1 db for base state for Test Case data � 1 db for regression results � Deploy a functional tree � Create core cross-dept test case specification standards and processes � Re-engineer cross-dept workflow to match core standards and processes 10/30/03 StarWest 2003 20

Planning: Anticipated Paradigm Shifts � Standardization of processes and test case specification templates within and across departments � Functional test categorization and management � Test case ownership � Test case maintenance � Highly structured and controlled input of data 10/30/03 StarWest 2003 21