T15 Thursday, October 30, 2003 3:00 PM I NCREASE THE V ALUE OF Y OUR - PDF document

BIO PRESENTATION T15 Thursday, October 30, 2003 3:00 PM I NCREASE THE V ALUE OF Y OUR T ESTING WITH B USINESS - ORIENTED M ETRICS Claude Fenner Arsin Corporation International Conference On Software Testing Analysis & Review October 27-31, 2003 San Jose, CA USA

Claude Fenner Claude Fenner has over 20 years experience in the software industry as an engineering manager, quality analyst, and software designer. His experience includes enterprise application software, mission-critical system software, and commercial and military embedded software. Across these industries he has transformed organizations to deliver high-quality software at lower cost through automation and globalization Arsin Corporation is a software reliability services and product company, specializing in enterprise testing, quality, and readiness assessment. Arsin provides expertise and tools to several Fortune 100 companies. Arsin also partners with major tool vendors such as Mercury Interactive and has a wholly-owned subsidiary Arsin Limited in India. Claude Fenner is responsible for driving Arsin’s tools and processes into market-ready product solutions. During his career, he has driven large-scale quality and testing programs for major quality- oriented companies including Compaq, Tandem Computers, and SAIC. At Tandem, he built and led the quality teams that delivered Tandem’s high availability database and transaction processing technologies used for stock exchanges and ATM networks. At Compaq he defined and drove the Y2K process that successfully updated Compaq’s more than 1000 NonStop branded products. Recent enterprise experience includes high volume B2B and B2C middleware for financial transactions. Beyond the enterprise arena, his experience includes developing processes and building tools for assuring the reliability of commercial nuclear safety systems and embedded military systems. He earned degrees in EE/CS from the University of California Berkeley.

ARSIN Corporation Powering Collaborative e-Business Increase the Value of your Testing with Business-Oriented Metrics Claude Fenner – StarWest 2003 October 30, 2003 ARSIN Corporation

ARSIN Corporation StarWest 2003 A lot has changed in the last decade Then Brand Brand Now Your systems are in the hands of consumers and partners like never before Your business reputation is on the line like never before Arsin Company Confidential 2

ARSIN Corporation StarWest 2003 Good News – Business owners care about quality Arsin Company Confidential 3

ARSIN Corporation StarWest 2003 Not everyone’s perspective on quality is exactly the same Owner thinks: “I want customer satisfaction and I want to produce value for Producer thinks: “I’ve been told what customers want, so that’s what I’ll build” Customer thinks: “I’ve paid for a service, so give me what you promised” business stakeholders” Business-value Quality Business Owner Customer Producer Service Quality Process Quality Arsin Company Confidential 4

ARSIN Corporation StarWest 2003 Ok, you’re visible to business owners … Now what! ?!? Bad News: Traditional metrics don’t resonate with business managers Arsin Company Confidential 5

ARSIN Corporation StarWest 2003 Let’s take a closer look at a business owner Here are things they think about … � Are our customer being served? � What is the best mix of people, capital, geographical locations? � What things are assets? What is an investment vs. expense? They say things like … � Are these tests really assets? Are they worth it? Why? � Can I squeeze more out of them? Can I get by with less? � Are things good enough for now? Can I wait later to invest? � What about assets needed for future demand? Arsin Company Confidential 6

ARSIN Corporation StarWest 2003 You’d better get ready to think like a business owner … � Customer Satisfaction � Threat/Risk to service quality � Business Value � TCO (Total Cost of Ownership) of testing � Test asset investment and potential asset gaps over time � Hint: Quality Theory is not the answer � Answer: Business-Oriented Metrics � Business-oriented metrics bridge the gap between the activities your test organization performs and the benefits to the business Arsin Company Confidential 7

ARSIN Corporation StarWest 2003 First, a little introduction to business-oriented metrics The face of a business owner Sadly for him, he didn’t know about business-oriented metrics accountable to stakeholders and customers. Arsin Company Confidential 8

ARSIN Corporation StarWest 2003 Here are some metrics. Are they useful? California Electric Supply & Demand for one day Maximum supply Predicted potential supply Predicted demand Actual demand These metrics serve the customer pretty well … but not the producer or the business owner http://www.caiso.com/SystemStatus.html Arsin Company Confidential 9

ARSIN Corporation StarWest 2003 How well do these metrics serve their audience? � Customer A+ � Good for predicting daily outage risk and when outage might occur. � Producer D- � Doesn’t indicate anything useful about the operation of a powerplant (generation efficiency, power quality (brownouts), TTF) � Business Owner F � Too tactical: Only tells owner supply and demand for ONE DAY � Big gaps are uninformative: � Sense of false security. Disaster could strike without warning! (unplanned failures, demand spike, plant decommissioning) � Worse, a big gap could erroneously suggest over-investment Arsin Company Confidential 10

ARSIN Corporation StarWest 2003 And look what happened Lesson: Gray Davis didn’t see how increasing demand was threatened by under-investment in assets and lack of asset health. Arsin Company Confidential 11

ARSIN Corporation StarWest 2003 How to transform these into business-oriented metrics … oil 1. Show assets individually coal 2. Indicate asset growth and hydro decay (health) oil 3. Add time (past and future) nuclear 4. Indicate future demand for assets time Revised metrics show business owner the demand for assets and asset health Arsin Company Confidential 12

ARSIN Corporation StarWest 2003 But don’t forget, business owners also care about value Efficiently run businesses run close to the edge. Business value is always critcal. No such thing as quality at any cost! Arsin Company Confidential 13

ARSIN Corporation StarWest 2003 Business-Oriented metrics in the Testing World � Before starting, remember what you’re trying to do � Talk to business owners in terms they understand � Influence decisions by demonstrating and defending testing value � Get them to participate (be a stakeholder) in your process � Compete against others for scarce resources and win � Always remember the basics of metrics: It’s a sequence of steps from objectives to actions. 1.Define Objectives and Metrics 2.Collect Data 3.Reveal Trends 4.Make Decisions 5.Take Actions Arsin Company Confidential 14

ARSIN Corporation StarWest 2003 Business-Oriented testing metrics defined � Test Asset Trajectory metric (benefit-side) � Identifies testing assets and measures how adequate they are at meeting testing demand. Also shows asset benefit, health, and risk. The term trajectory is used to indicate that the metric is always measured over time. � Test Asset Leverage metric (cost-side) � Identifies the costs associated with the testing assets. The term leverage is used to indicate that the metric subdivides cost into components that are useful for identifying best utilization of resources. � Together they form the total asset value picture. � asset-value = benefit-side + cost-side � Note: One simple building block is key to both metrics Arsin Company Confidential 15

ARSIN Corporation StarWest 2003 Test Asset Trajectory metric Assets & Types Today Manual Manual Demand Manual Manual Timeframe & Events Performance Performance Stress Stress Completeness & Health Automated Automated Automated Automated Functional Functional Manual Manual Manual Manual Manual Manual End-to-end End-to-end End-to-end End-to-end Manual Manual End-to-end End-to-end Automated Automated Automated Automated Automated Automated Automated Automated Functional Functional Functional Functional Functional Functional Functional Functional Performance Performance Performance Performance Performance Performance Stress Stress TIme Single, clear statement of asset composition, benefit, and risk Arsin Company Confidential 16

ARSIN Corporation StarWest 2003 Test Asset Leverage metric Manual Performance Assets Stress Asset Cost by component Automated Automated Functional Total Cost Manual Manual Manual 3 Manual End-to-end End-to-end End-to-end Automated + 5 Automated Automated Automated Functional Functional Functional Functional + 3 Performance Performance Stress Performance = TIme People People People People Machines Machines Machines Machines 11 Dollars Dollars Dollars Dollars Time Time Time Time Single, clear statement of the asset cost by component Arsin Company Confidential 17