T15 Thursday, October 30, 2003 3:00 PM I NCREASE THE V ALUE OF Y OUR - - PDF document
BIO PRESENTATION T15 Thursday, October 30, 2003 3:00 PM I NCREASE THE V ALUE OF Y OUR T ESTING WITH B USINESS - ORIENTED M ETRICS Claude Fenner Arsin Corporation International Conference On Software Testing Analysis & Review October
BIO PRESENTATION International Conference On Software Testing Analysis & Review October 27-31, 2003 San Jose, CA USA
Thursday, October 30, 2003 3:00 PM
INCREASE THE VALUE OF YOUR TESTING WITH BUSINESS-
ORIENTED METRICS
Claude Fenner Arsin Corporation
Claude Fenner
Claude Fenner has over 20 years experience in the software industry as an engineering manager, quality analyst, and software designer. His experience includes enterprise application software, mission-critical system software, and commercial and military embedded software. Across these industries he has transformed organizations to deliver high-quality software at lower cost through automation and globalization Arsin Corporation is a software reliability services and product company, specializing in enterprise testing, quality, and readiness assessment. Arsin provides expertise and tools to several Fortune 100 companies. Arsin also partners with major tool vendors such as Mercury Interactive and has a wholly-owned subsidiary Arsin Limited in India. Claude Fenner is responsible for driving Arsin’s tools and processes into market-ready product solutions. During his career, he has driven large-scale quality and testing programs for major quality-
led the quality teams that delivered Tandem’s high availability database and transaction processing technologies used for stock exchanges and ATM networks. At Compaq he defined and drove the Y2K process that successfully updated Compaq’s more than 1000 NonStop branded products. Recent enterprise experience includes high volume B2B and B2C middleware for financial transactions. Beyond the enterprise arena, his experience includes developing processes and building tools for assuring the reliability of commercial nuclear safety systems and embedded military systems. He earned degrees in EE/CS from the University of California Berkeley.
ARSIN Corporation
Powering Collaborative e-Business
Increase the Value of your Testing with Business-Oriented Metrics Claude Fenner – StarWest 2003
October 30, 2003
Arsin Company Confidential 2
ARSIN Corporation
StarWest 2003
A lot has changed in the last decade Now
Brand
Then
Brand
Your systems are in the hands of consumers and partners like never before Your business reputation is on the line like never before
Arsin Company Confidential 3
ARSIN Corporation
StarWest 2003
Good News – Business owners care about quality
Arsin Company Confidential 4
ARSIN Corporation
StarWest 2003
Not everyone’s perspective on quality is exactly the same
Customer thinks: “I’ve paid for a service, so give me what you promised” Service Quality Process Quality Customer Producer Business Owner Producer thinks: “I’ve been told what customers want, so that’s what I’ll build” Owner thinks: “I want customer satisfaction and I want to produce value for business stakeholders” Business-value Quality
Arsin Company Confidential 5
ARSIN Corporation
StarWest 2003
Ok, you’re visible to business owners … Now what!
Bad News: Traditional metrics don’t resonate with business managers
Arsin Company Confidential 6
ARSIN Corporation
StarWest 2003
Let’s take a closer look at a business owner
They say things like …
Are these tests really assets? Are they worth it? Why? Can I squeeze more out of them? Can I get by with less? Are things good enough for now? Can I wait later to invest? What about assets needed for future demand?
Here are things they think about …
Are our customer being served? What is the best mix of people, capital, geographical locations? What things are assets? What is an investment vs. expense?
Arsin Company Confidential 7
ARSIN Corporation
StarWest 2003
You’d better get ready to think like a business owner …
Customer Satisfaction
Threat/Risk to service quality
Business Value
TCO (Total Cost of Ownership) of testing Test asset investment and potential asset gaps over time
Hint: Quality Theory is not the answer Answer: Business-Oriented Metrics
Business-oriented metrics bridge the gap between the activities your test organization performs and the benefits to the business
Arsin Company Confidential 8
ARSIN Corporation
StarWest 2003
First, a little introduction to business-oriented metrics
The face of a business owner accountable to stakeholders and customers. Sadly for him, he didn’t know about business-oriented metrics
Arsin Company Confidential 9
ARSIN Corporation
StarWest 2003
Here are some metrics. Are they useful?
http://www.caiso.com/SystemStatus.html
Predicted potential supply Maximum supply Actual demand Predicted demand
These metrics serve the customer pretty well … but not the producer or the business owner
California Electric Supply & Demand for one day
Arsin Company Confidential 10
ARSIN Corporation
StarWest 2003
How well do these metrics serve their audience? Customer A+
Good for predicting daily outage risk and when outage might occur.
Producer D-
Doesn’t indicate anything useful about the operation of a powerplant (generation efficiency, power quality (brownouts), TTF)
Business Owner F
Too tactical: Only tells owner supply and demand for ONE DAY Big gaps are uninformative: Sense of false security. Disaster could strike without warning! (unplanned failures, demand spike, plant decommissioning) Worse, a big gap could erroneously suggest over-investment
Arsin Company Confidential 11
ARSIN Corporation
StarWest 2003
And look what happened
Lesson: Gray Davis didn’t see how increasing demand was threatened by under-investment in assets and lack
Arsin Company Confidential 12
ARSIN Corporation
StarWest 2003
How to transform these into business-oriented metrics …
coal hydro
nuclear
for assets
decay (health)
time
Revised metrics show business owner the demand for assets and asset health
Arsin Company Confidential 13
ARSIN Corporation
StarWest 2003
But don’t forget, business owners also care about value
Efficiently run businesses run close to the edge.
Business value is always critcal. No such thing as quality at any cost!
Arsin Company Confidential 14
ARSIN Corporation
StarWest 2003
Business-Oriented metrics in the Testing World
Before starting, remember what you’re trying to do
Talk to business owners in terms they understand Influence decisions by demonstrating and defending testing value Get them to participate (be a stakeholder) in your process Compete against others for scarce resources and win
Always remember the basics of metrics: It’s a sequence of steps from objectives to actions.
1.Define Objectives and Metrics 2.Collect Data 3.Reveal Trends 4.Make Decisions 5.Take Actions
Arsin Company Confidential 15
ARSIN Corporation
StarWest 2003
Business-Oriented testing metrics defined
Test Asset Trajectory metric (benefit-side)
Identifies testing assets and measures how adequate they are at meeting testing demand. Also shows asset benefit, health, and risk. The term trajectory is used to indicate that the metric is always measured over time.
Test Asset Leverage metric (cost-side)
Identifies the costs associated with the testing assets. The term leverage is used to indicate that the metric subdivides cost into components that are useful for identifying best utilization of resources.
Together they form the total asset value picture.
asset-value = benefit-side + cost-side
Note: One simple building block is key to both metrics
Arsin Company Confidential 16
ARSIN Corporation
StarWest 2003
Test Asset Trajectory metric
Manual Automated Functional Stress Demand End-to-end Performance Manual Automated Functional Assets & Types Automated Functional Performance Manual End-to-end End-to-end Performance Manual Automated Functional Automated Manual Stress Today Manual Performance Automated Functional TIme Timeframe & Events Completeness & Health Manual Automated Functional Stress End-to-end Performance Manual Automated Functional Automated Functional Performance Manual End-to-end End-to-end Performance Manual Automated Functional Automated Manual Stress Manual Performance Automated Functional
Single, clear statement of asset composition, benefit, and risk
Arsin Company Confidential 17
ARSIN Corporation
StarWest 2003
Test Asset Leverage metric
End-to-end Performance Manual Automated Functional End-to-end Performance Manual Automated Functional Automated Manual Stress Automated Functional Performance Manual End-to-end Performance Automated Functional TIme Automated Functional Stress Manual
3 5 3
People Machines Dollars Time People Machines Dollars Time People Machines Dollars Time People Machines Dollars Time
Single, clear statement of the asset cost by component
Asset Cost by component Assets Total Cost
= 11 + +
Arsin Company Confidential 18
ARSIN Corporation
StarWest 2003
Test Asset metrics building block
Asset cost (in all-or-nothing chunks) # People (by geography, by FTE/Contractor, Outsourcer) # Machines (clusters, labs, in some cases single units) Dollars for tools or testing supply Asset benefit %completed (degree to which demand is met by asset) %healthy (how much of asset is fully fit for service) Asset Type Definition Name: manual, performance, automated, etc Demand (e.g. function points, list of test items, test scenarios)
Every building block contains the same information. Demand, asset trajectory and asset leverage metrics are computed by aggregating the information from all the building blocks.
Duration asset is tied up (generally one quarter)
Arsin Company Confidential 19
ARSIN Corporation
StarWest 2003
Case Study – Software Product Company
Profile
Web services product for handling B2C and B2B financial transactions.
Major business events related to Testing
New Functionality Functional Tests Rapid Adoption Scalability & Performance Technology Changes Regression Acquisition Geographical Distribution Automation Manual Test Conversion Customer-driven Releases Rapid Regression Workforce Redistribution Employees, Contractors, Outsource
Actions and Results
Team transformed from handling a single product line to three product lines with 75% less people! Scalability for multiple thousands of users
Arsin Company Confidential 20
ARSIN Corporation
StarWest 2003
Case Study – Test Asset Trajectory (3 products 2 years)
Demand Assets & Types Timeframe & Events Completeness & Health TIme Product A release Product A dot releases Product A scalablity 5k Product A scalablity 10k Product C release Product B acquisition Technology change Financial misfortune Workforce reorg Outsourcing Stress Manual Automated Functional Performance Manual End-to-end Automated Functional Performance Manual End-to-end Automated Functional Performance Manual Automated Functional Performance Manual Automated Functional Performance Manual
A
Manual Manual Automated Functional Manual Automated Functional Manual Automated Functional
B
Automated Functional Performance Manual Automated Functional Performance Manual
C
Stress Manual Automated Functional Performance Manual End-to-end Automated Functional Performance Manual End-to-end Automated Functional Performance Manual Automated Functional Performance Manual Automated Functional Performance Manual Manual Manual Automated Functional Manual Automated Functional Manual Automated Functional Automated Functional Performance Manual Automated Functional Performance Manual
Arsin Company Confidential 21
ARSIN Corporation
StarWest 2003 Start: 8 people for 1 product End: 5 people for 3 products Due to automation investment
Case Study – Test Asset Leverage (3 products 2 years)
TIme Stress Manual Automated Functional Performance Manual End-to-end Automated Functional Performance Manual End-to-end Automated Functional Performance Manual Automated Functional Performance Manual Automated Functional Performance Manual
A
Manual Manual Automated Functional Manual Automated Functional Manual Automated Functional
B
Automated Functional Performance Manual Automated Functional Performance Manual
C
People Machines Dollars 8 10 12 18 16 10 8 5 4 5 6 7 7 7 7 6 5k 5k 10k 10k 10k 5k 5k 5k
Trends:
Arsin Company Confidential 22
ARSIN Corporation
StarWest 2003
Roadmap - Suggestions for how to proceed immediately
Choose a small, well-differentiated set of asset types.
Trade-offs are easier to make when options are distinct.
Build your model around events that create discontinuities
Upgrades, new customer deployments, configuration changes and hardware expansions, shifting organizational responsibilities, outsourcing
Predict demand accurately
Don’t present too many options. Make sure your top 3 are visible. Make points business-owners care about, especially using events Always anticipate tension and prepare to defend yourself right to the edge Appreciate your audience. KISS
These aren’t weekly metrics. Generally quarterly although sometimes non- quarterly when there are significant events or discontinuities
Arsin Company Confidential 23
ARSIN Corporation
StarWest 2003
Questions