TK2 5/16/2007 4:30:00 PM "B UILDING THE T EST M ANAGEMENT O - - PDF document

BIO PRESENTATION International Conference On Software Test Analysis And Review May 14-18, 2007 Orlando, FL USA

TK2

5/16/2007 4:30:00 PM

"BUILDING THE TEST MANAGEMENT OFFICE"

Geoff Horne iSQA

Geoff Horne Geoff Horne

Geoff Horne is based in New Zealand and has founded and run two testing companies which have grown to enjoy an international clientele. He has over 28 years experience in IT including software development, sales and marketing and IT and project management. In 1994, almost by accident, he found himself involved in testing a complex fault management system which led to further testing and QA assignments covering a wide range of applications and tools. Geoff’s companies were subsequently founded to bring a full range of testing consultancy services to the IT industry. Of late, he has focused on a few select clients running complex test projects in a programme test management capacity. Geoff has written a variety of white papers on the subject of software testing and has been a regular speaker at the Star testing conferences. He is married with four children and in his spare time (which there is not a lot of) enjoys writing and recording

• riginal contemporary Christian music.

www.isqa.com

BUI LDI NG THE TEST MANAGEMENT OFFI CE (TMO)

Geoff Horne Managing Director

www.isqa.com

Strategy Defect management & resolution Methods, standards & processes Vendor management Process adherence Tools implementation & management Communications Environments management Status reporting Planning Release management Coverage management Issues resolution Budget & cost management Schedule management Change request management Configuration management Escalation management Support management Review & approval management Human resources management Quality assurance

www.isqa.com

The typical Test Management inter-relationship model:

www.isqa.com

Sliced:

www.isqa.com

Sliced and diced:

www.isqa.com

Strategy Defect management & resolution Methods, standards & processes Vendor management Process adherence Tools implementation & management Communications Environments management Status reporting Planning Release management Coverage management Issues resolution Budget & cost management Schedule management Change request management Configuration management Escalation management Support management Review & approval management Human resources management Quality assurance Automation

www.isqa.com

Strategies Defect management & resolution Methods, standards & processes Vendor management Process adherence Tools implementation & management Communications Environments management Status reporting Planning Release management Coverage management Issues resolution Budget & cost management Schedule management Change request management Configuration management Escalation management Support management Review & approval management Human resources management Quality assurance Automation

www.isqa.com

Defect management & resolution Tools implementation & management Environments management Coverage management Budget & cost management Configuration management Review & approval management

www.isqa.com

Defect management & resolution Tools implementation & management Environments management Coverage management Budget & cost management Configuration management Review & approval management

www.isqa.com

Defect management & resolution Tools implementation & management Environments management Coverage management Budget & cost management Configuration management Review & approval management

www.isqa.com

Defect management & resolution Tools implementation & management Environments management Coverage management Budget & cost management Configuration management Review & approval management

www.isqa.com

Environments management Defect management & resolution Tools implementation & management Budget & cost management Release management Communications Vendor management

!

Coverage management Configuration management Support management Review & approval management

www.isqa.com

What is a Test Management Office?

• A centre of governance for the testing subproject
• Support team & mechanism for Test Managers, Test Analysts etc.
• Carries responsibility for selected testing functions that may not

realistically be carried by Test Manager(s)

• Operates as a discrete team within the testing function
• TMO has an operational & delivery responsibility as opposed to

being a purely administrative function

www.isqa.com

Why use one?

• Removes some of the management overheads from Test Managers

& Analysts

• Enables test teams to focus on key testing activities
• Streamlines & spreads workloads
• Provides a central control of all test resources
• Sometimes a necessity to keep on top of testing resources, assets

and facilities

www.isqa.com

When should a TMO be used ?

• Large & complex testing projects
• Integration with other technologies eg. telecommunications

networks

• Multi-workstream testing projects
• Projects with multiple & disparate applications, platforms,

technologies, environments etc.

• No resources, infrastructures, processes within functions required

to support testing eg. environment management, configuration management, release management etc.

• Lack of expertise or commitment within same.

www.isqa.com

Which test functions can be covered by a TMO?

Strategies Defect management & resolution Methods, standards & processes Vendor management Process adherence Tools implementation & management Communications Environments management Status reporting Planning Release management Coverage management Issues resolution Budget & cost management Schedule management Change request management Configuration management Escalation management Support management Review & approval management Human resources management Automation Quality assurance

www.isqa.com

Should there be a separate TMO Manager?

• Depends on the size of the TMO and functions it covers.
• What type of person; line manager, test manager, project manager?
• Can the TMO Manager be a non-testing professional?
• Can the Test Manager also be the TMO Manager?

www.isqa.com

What type of roles are the management of each TMO function?

• Should all TMO function managers be testing professionals?
• Should any of the them be testing professionals?
• Where is the balance?

www.isqa.com

How should a TMO operate?

• Clear distinction of responsibilities between Test Manager & TMO

Manager

• When should managers operate within the TMO and when outside?
• Can the TMO Manager be responsible for one or more TMO

functions?

• Who/where should the TMO report?

www.isqa.com

Steps for building a TMO:

• Analyse requirement and determine need
• Select project functions that should come under the TMO
• Select/acquire TMO function managers
• Develop a TMO charter that clearly defines:
• What the functions are that will be under the TMO & why
• How each function will inter-relate with other project streams
• How the TMO will function with the Test Manager(s)
• Where there are overlaps with other functional teams

www.isqa.com

Analyse requirement & determine need:

Function Role Covered? How?

Test strategies

TCM

Yes Test Manager Defect management & resolution

LPM

Yes Test Manager Vendor management

LPM

No TMO Methods, standards & processes

TCM

No Engage iSQA Tools implementation & management

SPM

Yes Software Support Team Process adherence

TCM

Yes Test Manager Environments management

SPM

No TMO Release management

SPM

No TMO Communications

LPM

Yes Project Admin Status reporting

LPM

Yes Test Manager Test planning

LPM

Yes Test Manager Issues resolution

LPM

Yes Test Manager Coverage management

TCM

No TMO Test schedule development & management

LPM

No TMO Budget & cost management

SPM

Yes Project Management Office Configuration management

SPM

No TMO Change request management

LPM

Yes Software Development Team Support management

LPM

Yes Project Vendor Liaison Officer Escalation management

LPM

Yes Test Manager Human resources management

SPM

Yes HR Dept Review & approval management

LPM

No TMO Quality assurance

SPM

No Engage iSQA

Line/project manager (LPM) Test consultant/manager (TCM) Specialist manager (SPM)

www.isqa.com

Select functions for TMO:

Function Role Covered? How?

Test strategies

TCM

Yes Test Manager Defect management & resolution

LPM

Yes Test Manager

Vendor management LPM No TMO

Methods, standards & processes

TCM

No Engage iSQA Tools implementation & management

SPM

Yes Software Support Team Process adherence

TCM

Yes Test Manager

Environments management SPM No TMO Release management SPM No TMO

Communications

LPM

Yes Project Admin Status reporting

LPM

Yes Test Manager Test planning

LPM

Yes Test Manager Issues resolution

LPM

Yes Test Manager

Coverage management TCM No TMO Test schedule development & managem LPM No TMO

Budget & cost management

SPM

Yes Project Management Office

Configuration management SPM No TMO

Change request management

LPM

Yes Software Development Team Support management

LPM

Yes Project Vendor Liaison Officer Escalation management

LPM

Yes Test Manager Human resources management

SPM

Yes HR Dept

Review & approval management LPM No TMO

Quality assurance

SPM

No Engage iSQA

Line/project manager (LPM) Test consultant/manager (TCM) Specialist manager (SPM)

www.isqa.com

Resource TMO:

Function Role Fulltime? How?

Vendor management

LPM

No Second Helen to TMO Environments management

SPM

Yes Appoint Test Environment Manager Release management

SPM

Yes Appoint Release Manager Coverage management

TCM

No Engage iSQA as part of TMO Test schedule development & management

LPM

Yes Second Jim from Project X Configuration management

SPM

Yes Appoint Test Configuration Manager Review & approval management

LPM

No Second John from PMO

www.isqa.com

Develop TMO Charter:

Table of Contents

1 Introduction 1 2 TMO Team Overview 2 3 Purpose 3 Responsibilities 3 Team Structure 4 TMO Accept/Reject Processes 5 3 Test Approach 6 Test Phases 6 Objectives 7 4 Environments 8 Requesting Test Environments 8 Deployment of Code into Test Environments 9 5 Release Management 10 Release No. Structure 10 Release Processes 11 Release Notifications 13 Releases into Production 14

www.isqa.com

Example

• Local subsidiary of a worldwide mobile network operator
• $NZ50m+ project
• Frequent changes of senior project staff
• Multiple vendors
• Multiple test environments
• Multiple test workstreams including multiple technologies
• No recognition of formal testing methodologies
• Flawed test strategy
• Test phase would commence without properly completing previous

phases in order to maintain deadlines

• Totally reactive project

www.isqa.com

Logical Test Environm ents Test Configuration Coordination High Level Test Planning (Across SIT / UAT / OAT / Perf / MIT) Test Schedule Coordination Environment Coordination Test SME Coordination Release Coordination Coverage & Defect Coordination SIT Planning UAT Planning OAT Planning PRFT Planning MIT Planning

Test Planning

SME Test Resources

Physical Test Platforms Test Platform Configuration Test Applications / Libraries

Environment Status

Test Data (e.g Subscribers, Re-charge Cards, MSISDNs) Test Devices (e.g. Handsets, SIMs)

SIT Execution UAT Execution OAT Execution PRFT Execution MIT Execution

Test Execution Build & Pgm Schedule and Build Team Program Frodo TMO Processes

www.isqa.com

TMO Structure:

TMO Manager

• TMO Owner/Driver
• Test Strategy
• Testing Quality Assurance
• Test Scheduling & Co-
• rdination
• Support Management

TMO Administrator

• Reporting
• Communications

Defect & Coverage Manager

• Test Coverage Management
• Defect Management
• Issues Management

Environments Managers (2)

• Test Environments
• Test Equipment

Release Manager

• Test Releases

Configuration Manager

• Test Configuration
• Parameters

Triage Manager

• Owner of test blockages
• See blockages through to

completion

www.isqa.com

TMO Structure:

TMO Manager: senior test consultant & manager TMO Administrator: new hire, no previous testing experience Defect & Coverage Manager: experienced defect & issues manager Environments Managers: experienced environments managers,

• ne with solid testing experience

Release Manager: experienced release manager Configuration Manager: experienced configuration manager Triage Manager: experienced project manager and ace problem solver

www.isqa.com

Lessons learned & how to avoid

• Anything to do with TMO functions was thrown onto or any delays

always blamed on TMO:

• Other managers & personnel abdicating responsibility
• Our Charter was not clear enough
• Took us a while to develop an Accept/Reject process
• We lacked adequate tools to properly track issues
• We were slow to escalate issues

What could we have done better:

• Developed our Charter including all processes up front
• Ensure the Charter was properly communicated across the project and

correct expectations set as to the TMO role

• Established TMO at the project outset rather than part way through
• Flagged the needs for tools sooner

www.isqa.com

Lessons learned & how to avoid cont.

• Needed to prevent TMO becoming an unnecessary management layer &

bottleneck

• Decisions often required the Environments, Release & Configuration

Managers to concur & they where based in multiple locations

• Set up an issues “triage” function which became more of a hindrance than

benefit What could we have done better:

• Gained buy-in from external parties sooner
• Implemented a “heads-up” process
• Pushed the “triage” function out onto the relevant build teams rather than

manage within the TMO

• Endeavour to locate TMO personnel in same location if possible

www.isqa.com

Lessons learned & how to avoid cont.

• Having TMO managers hooking into existing infrastructures, teams &

personnel

• TMO managers often had difficulties in obtaining resources from relevant

infrastructure teams

• Non-project management personnel de-prioritised TMO requests as they

didn’t come from a senior project manager What could we have done better:

• Escalated blockages sooner
• Gained buy-in from external parties sooner
• Implemented a “heads-up” process
• All of the previous points as well!

www.isqa.com

Summary

• Needs to be established and clear Charter developed at the outset of a

project

• Need to carefully plan and structured to avoid:
• Dumping ground
• Blame game
• Delay agent
• Becoming an “island”
• Needs to suitably tooled & resourced
• Needs to fit into a project & not onto it
• Not the panacea of all large test project problems

However…..

www.isqa.com

Summary

• Some projects cannot run without a TMO
• Provides a central point of governance & control of all testing resources,

assets and facilities

• Removes overheads from testing personnel
• Enables test teams to focus on key testing activities
• Streamlines & spreads workloads

www.isqa.com

BUI LDI NG THE TEST MANAGEMENT OFFI CE (TMO)

Geoff Horne Managing Director