W12 September 29, 2004 3:00 PM A M ANAGER ' S G UIDE TO G ETTING THE - - PDF document

BIO PRESENTATION Better Software Conference & EXPO September 27-30, 2004 San Jose, CA USA

W12

September 29, 2004 3:00 PM

A MANAGER'S GUIDE TO GETTING THE MOST OUT OF TESTING AND QA

Brian Warren Ceridian Corporation

Brian Warren

Brian Warren is a QA Manager in Ceridian Corporation’s Human Resource Solutions division managing testing of benefits management tools and HR information systems. He has eleven years of experience in hardware and software testing and development, working in Dell’s Enterprise Systems Group, IBM’s Netfinity Group, a dot-com startup, and a niche software company that developed heads-down data entry products prior to joining Ceridian in 2003. In addition to his test management experience, Brian has worked in a wide range of development, testing, and IT roles, having served as test methodologist for Dell's Server Engineering organization. He also managed a software development and tools team responsible for test automation, load testing, and functional test tools, and worked as a database and web developer.

A Manager’s Guide to Getting the Most Out of Testing and Quality Assurance

Brian Warren, Ceridian Corp Better Software 2004

Audience Poll: Test and QA Organizations

Which of the organizations below do you

have in your company?

Quality Assurance Quality Control Testing Verification & Validation More than one of the above None of the above

Audience Poll: Test and QA Roles

How widely understood are the roles of your

Test/QA departments

Role clearly understood at all levels of the

company

Role clearly understood within Test/QA and the

teams they serve, others not so clear

Test/QA staff understand, others may not Some Test/QA staff still unclear Non-issue: No Test/QA to worry about

Understanding the role of Test and QA

Understanding of the role Test/QA play in the

corporation impacts…

How much value they are perceived to add (and how

their performance is evaluated)

How other teams interact with them How the Test/QA organizations operate

In the absence of real understanding, the labels

used bring their own expectations…

“Quality Police” or intelligence gatherers? Product experts or process gurus?

Why those expectations matter

Consider what happens then the following groups

are working off incorrect assumptions

Executive Management

Control how much is invested in Test/QA Evaluate performance of departments

Other teams (e.g.: development, marketing)

Supply critical information to Test/QA Primary customers of Test/QA outputs

Individuals within the organization

Perform the actual work

Customers

Final consumers of product and providers of corporate revenue

Job-seekers

Provide talent pool to support organizational growth

So what SHOULD it be called?

It depends on what the team is doing and

what it is trying to accomplish…

There is no single “industry standard” answer

IEEE, CMMI, RUP, and others each have their

• wn perspective

Views of testing and quality assurance

IEEE covers all the bases with two definitions for

“software quality assurance”…

(1) “a planned and systematic pattern of all actions

necessary to provide adequate confidence that an item or product conforms to established technical requirements

(2) a set of activities designed to evaluate the process by

which products are developed or manufactured”

Views of testing and quality assurance

… and definitions for “testing” and “verification and

validation”

Testing: “the process of operating a system or component under

specified conditions, observing or recording the results, and making an evaluation of some aspect of the system or component.” (IEEE, 1990)

[Verification and Validation] determines whether development

products of a given activity conform to the requirements of that activity, and whether the software satisfies its intended use and user needs. (IEEE, 1998)

Views of testing and quality assurance

CMMI provides a definition (unsurprisingly)

focused on process management…

“The purpose of SQA is to provide management with

appropriate visibility into the process being used by the software project and of the projects being built” (Paulk et al, 1995)

Views of testing and quality assurance

While RUP provides separate definitions for

“testing” and quality assurance

Test: A discipline in the software-engineering process

whose purpose is to integrate and test the system. (RUP 2003)

Quality Assurance: All those planned and systematic

actions necessary to provide adequate confidence that a product or service will satisfy given requirements for quality. (RUP 2003)

Where these definitions fail

Successful communication depends on

expressing concepts such that the entire audience understands

Technical professionals Non-technical professionals Executives and managers

Organizations, their objectives, and their

goals must be definable in business terms, not technical jargon

A more practical example…

Johanna Rothman provides two very tangible definitions…

QA staff assess the product and the system that produced the

product, to see if there are improvements the project team can make for this project or the next one.

Testers test the product, performing verification activities. They

discover where the product works and where the product doesn’t

• work. They test in whatever ways the product requires, whether

that’s for functionality, internationalization, performance, reliability, maintenance, usability, or some other attribute of the system.

When all else fails, draw a picture…

3-Dimensional Model of Testing and QA

Process Product Advisory Authoritarian Internal External

Reporting Focus Role

Based on 2-D model from Christensen and Thayer, 2001

Dimensions of Testing and QA: Focus

Product Focused Process Focused Production Focused Testing Quality Assurance Quality Control

Characteristics of Organizational Focus

A question of “how much” you can do, not “which”

you should do

All options are important and each contributes to the overall

ability to deliver the right quality to your customers

The demand for support in each of these areas varies based

• n business context

Solutions are not one-size-fits-all

Each focus represents a different set of problems and may

require different skill sets and structures

Skill Requirements by Focus

• Presentation of technical

and issue data to technical audiences and business

• wners
• Manufacturing or production

processes and operations

• Production tools,

infrastructure, and

• perations
• Production monitoring and

testing tools and techniques

• Statistical measurement

and trending analysis

• Issue characterization and

troubleshooting skills

Production-focused

• Presentation of technical

and issue data to technical audiences and business

• wners
• Presentation of process

adherence and compliance data to primarily non-technical audiences (management) Communicatio ns Skills

• Understanding of customer

and market demands and product usage

• System design lifecycle

and development processes

• Regulatory and

compliance policies Requirements and Business Analysis Skills

• Product architectures,

design, implementation, and usage

• Test design and planning

techniques

• Test (and automation) tools
• Issue characterization, and

troubleshooting skills

• To the degree needed to

review work products and determine process adherence or deviation

• Process characterization,

development, and measurement Technical Knowledge and Analytical Skills

Product-focused Process-focused Skill Set

Dimensions of Testing and QA: Role

Makes recommendations Makes decisions (gatekeeper)

Advisory Authoritative

Advisory Role Characteristics

Advisory Pro’s

Non-adversarial approach fosters cooperation and

communication

Places ownership of release decisions with those best suited

to handle them -- the business decision makers

Advisory Con’s

Can’t stop the business from making a “bad” decision should

it decide to do so

Success is heavily reliant upon the advocacy and data

presentation skills of Test/QA staff

Authoritative Role Characteristics

Authoritative Pro’s

Ability to halt a project prevents Test/QA from being overrun

by stronger personalities from other organizations

Authoritative Con’s

Sets up an adversarial relationship that can stifle

communication, cooperation, and trust

Quality of release decisions dependent upon Test/QA

• rganization’s understanding of the business – not just the

product

Test/QA organization can now be blamed regardless of the

decision made

A quote from the experts…

“Some testers dream of having veto control

• ver the release of the product. They

deserve to be punished by having their wish granted.”

Kaner, Bach, and Pettichord in Lessons Learned in Software Testing

The Bottom Line on Roles

The Advisory Role

Great approach unless your Test/QA staff are unable to

collect and present information well enough to sway the rest

• f the business

The Authoritative Role

Perfect for companies that have development managers who

make chronically poor decisions through ignoring available data and have Test/QA staff that make better business decisions than senior management

A Parting Thought:

If your testers aren’t capable of succeeding in an advisory

role, why would you trust them in an authoritative role?

Dimensions of Testing and QA: Reporting

Core Function Supporting Function Contractors, Consultants

External Internal Parallel

Internal Reporting Characteristics

Internal Pro’s

Common management between Test/QA and the teams they

support eases communications, information sharing

Single point of ownership makes accountability simple and

reduces opportunity for finger-pointing

Internal Con’s

Test/QA may be afraid to deliver bad news if it causes

problems for their management

May fragment Test/QA in large organizations, making it

difficult to maintain commonality between them

Parallel Role Characteristics

Parallel Pro’s

Independent management chain can make it easier for

Test/QA to deliver bad news

Can simplify management of very large Test/QA

• rganizations

Parallel Con’s

Greater separation can impede open communications Opportunity for finger-pointing between management chains

External Role Characteristics

External Pro’s

Staff not prone to organizational bias or indoctrinated with

corporate culture

Contract or consulting staff do not require the same long-

term investment as permanent staff

External Con’s

Staff’s culture and biases are a product of their company and

may not match yours

External agencies may tell you what you want to hear in

• rder to retain future business

Operational Impacts by Role

• Approach brings it own Talent

base

• No training: staff deliver or are

replaced

• Must hire and/or build talent

base

• Must train to grow

capabilities

• Must hire and/or build talent

base

• Must train to grow

capabilities Start-up Ease

• Unified management of

Test/QA personnel

• Common management of

Test/QA teams makes “tuning” more difficult

• Permanent headcount

reduce ability to quickly adapt staffing levels

• Sets up conflicting and/or

competing management chains

• Potential for “Us versus

Them” competition creates barriers to trust and flow of information

Parallel

• Presentation of technical and

issue data to technical audiences and business

• wners
• Difficult to manage in large
• rganizations where Test/QA

may develop as independent pockets Scalability

• Limited or no opportunity for

tuning – style and culture comes from the external

• rganization
• Headcount can typically be

rapidly modified

• Test/QA team can be “tuned”

to match its context

• Permanent headcount

reduce ability to quickly adapt staffing levels Flexibility

• Maximum resistance to trust

and access to information since staff report to a separate company

• Common management chain

minimizes “Us versus Them” mentality

• Reduces or eliminates

barriers to sharing of information Communication s, Trust, and Access to Information

External Internal Area

Corporate Governance, Testing, and QA

Methods for auditing and reporting of corporate financials

in the spotlight due to high-profile governance failures

MCI WorldCom – Arthur Andersen (Corporate Library 2002) Enron – Merrill Lynch (BBC UK, 2003)

While corporate governance is definitely not the same as

testing or quality assurance, there are parallels…

Gathering and communicating evaluations of enormously complex

systems

Potentially severe impacts (economic and/or legal) to providing

incorrect data

From Quality Crisis to Governance Crisis

• Compare the following sound bite on corporate governance

reform to statements made about the “software quality crisis”

• “Inattention to good corporate governance practices over the past

decade or more is at the heart of what has gone so terribly wrong in corporate America over the past few years. If significant steps are not taken to revisit and remodel corporate governance practices, corporate America will continue to attract anger and animosity not

• nly of disillusioned shareholders, but of a much broader cross-

section of American Society.”

William H. Donaldson Chairman, Securities and Exchange Commission March 2003

Lessons from governance reform

Lesson: Management is ultimately accountable

Senior management is responsible for the actions of their company Additional controls will not compensate for lack of ethics or

integrity on the part of leadership

Implications to Test/QA:

No Test or QA structure will reliably halt the impacts of

irresponsible management. The solution lies in developing a trustworthy management team and providing that team the data needed to make informed decisions, not in building an organization based on mistrust of management.

Lessons from governance reform

Lesson: Auditors must be both independent and singularly

focused to avoid conflicts of interest

Even highly successful auditing organizations could be tempted to

sway results due to the presence (and revenues) of other engagements with auditing clients

Implications to Test/QA:

An organization responsible for auditing process compliance must

not be the same one responsible for developing, producing, or testing product. This constitutes a conflict of interest that is not

• nly likely to be ineffective, but may provide highly biased

information.

Putting it together – a corporate strategy

The “right answer” for you depends on…

The size and scale of your organization The economic, market, and regulatory constraints of

your business context

How your organization is structured currently

But there are some common recommendations...

Recommendations

Establish and communicate a clear charter and

vision for your Testing and QA organizations)

Must be both accessible and understandable in

addition to being accurate

Not useful in educating the company if the language

• nly makes sense to professionals within the
• rganization

Recommendations

Structure Test or QA as an advisory

• rganization, not an authoritative one.

If your organization is not accustomed to this

approach, educate first! The responsibility and accountability implications of this structure must be understood from the individual level to the executive level.

Recommendations

Segment testing and QA functions by focus

where possible

Allows better support of each specific area and

reduces the breadth of skills required for the individuals in each team

Avoid combining process QA with any other

focus area.

If your organization is too small to support a

dedicated process QA organization, you should carefully re-examine whether you really need one.

Ceridian Corporation provides this information to clients and friends for general information purposes. None of this material should be construed to be offering legal advice, nor should it be relied on as specific advice to any individual or organization. Please consult your legal advisor for any specific legal advice.

References

Rothman, J., (2002), “What does your title say about your job?”, Stickyminds.com, Orange Park, FL. [Rothman, 2002] Paulk, M. C., Weber, C. V., Curtis, B., and Chrissis, M. B., (1995) The Capability Maturity Model – Guidelines for Improving the Software Process, Addison-Wesley, Reading, MA, pp. 35. [Paulk et al. 1995] Kulpa, M., and Johnson, K., Interpreting the CMMI: A process improvement approach, Auerbach Publications, New York, NY. [Kulpa and Johnson, 2003] Kaner, C., Bach, J., and Pettichord, B., (2002) Lessons Learned in Software Testing, Wiley and Sons, New York, NY, pp. 8-9. [Kaner et al, 2002] IEEE, (1990), “IEEE Standard Glossary of Software Engineering Terminology”, New York, NY. [IEEE, 1990] The Corporate Library, (2002), “Scandal Spotlight: Worldcom Scandal”, June 19-25 2002 edition, www.thecorporatelibrary.com, Portland, ME. [Corporate Library, 2002] Christensen, M. J., and Thayer, R. H., (2001), The Project Manager’s Guide to Software Engineering’s Best Practices, IEEE Computer Society Press, Los Alamitos, CA, p.281-285. [Christensen & Thayer, 2001]