BUILDING A CUSTOMER QUALITY DASHBOARD John Ruberto VP of Quality - PowerPoint PPT Presentation

BUILDING A CUSTOMER QUALITY DASHBOARD John Ruberto VP of Quality Engineering – Clover, a First Data Company 2

3

First, A story 4

9.86 5

6

What is 9.86? Metric First Negative Link in Google Source Lines of Code 3 Cyclomatic Complexity 19 Function Points 11 Code Coverage 6 Defect Removal Efficiency 6 Defect Density 9 Bug Count 15 7

Principles for metrics • Related to our goals • Leading vs lagging indicators • Process metrics vs outcome metrics • Use the right technology to display 8

Related to our goals 9

Use the right technology to collect & display 10

Provides actionable insights 11

Goal-Questions-Metric • GQM • Victor Basili • Align on a set of goals • Ask questions about those goals • Design & collect metrics to answer the questions 12

Why • Setting goals, in alignment with the wider organization, gains acceptance • Focus on what’s most important to your stakeholders • Provide “line of sight” from your metrics to your goals • Build comprehensive view of your goals. 13

Example - Context • Software as a service application • > 500K active users • Paying monthly subscription 14

Example • Goal: Deliver better quality to our customers • Questions: • How many defects do our customers report? • How are we trending on customer reported defects? • How quickly to we fix the defects? • What are the top causes of these defects? • Why aren’t we catching these bugs before release? 15

Example –Delivered Quality 70% Reasons for Escape RCA Pareto 60% 50% 45% 40% 40% 35% 30% 30% 20% 25% 20% 10% 15% 0% 10% 5% 0% 16

Characteristics to think about • Process Metrics vs Outcomes • LOC / Review hour vs Defects found per review • Leading Indicators vs Lagging Indicators • Code coverage vs delivered quality • Median vs Average (Better yet: percentile) • Median page load vs Average page load • % fixes within SLA vs Average Age • 2012 average income in San Mateo County 17

Principles in using metrics • Direct measures instead of derived • “quality score” • Apdex • Actionable • Total crashes vs crash code pareto • Live data is best data • No powerpoint … 18

Fallacies of Metrics - Gamification • Goal: Improve Testing Efficiency • Metric: Testing Efficiency: (fixed bugs / total submitted) 19

Fallacies of Metrics – Confirmation Bias • Incoming bug rate improved dramatically – our quality must be outstanding! Image Credit: By User:KAMiKAZOW (Transferred from en.wikipedia to Commons.) [Public domain], via Wikimedia Commons 20

Fallacies of Metrics – Survivor Bias Image Credit: WyrdLight.com [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons 21

Fallacies of Metrics – Survivor Bias Open bugs By Severity 180 160 140 120 100 80 60 40 20 0 Critical Major Minor 22

Measurement Bias Phase Detection By Release 40% 35% 35% 33% 33% 30% 29% 30% 28% 28% 28% 24% 24% 25% 22% 21% 20% 20% 20% 18% 15% 15% 13% 12% 10% 9% 10% 8% 8% 8% 5% 5% 5% 3% 3% 3% 3% 2% 0% Release 1 Release 2 Release 3 Release 4 Release 5 Rqmts Design Code Int Sys Customer 23

Vanity Metrics • Don’t measure things that matter • Easily manipulated • But, make us feel good 24

Keeping the gains • Process Wrapper • Monitor & regulate • Automatic trigger • Wide distribution 25

• Questions? • JohnRuberto@gmail.com • @johnruberto • http://linkedin.com/in/ruberto 26

Photo Credits Nadia Comm: Ben Sutherland https://www.flickr.com/photos/bensutherland/ Bull: By Hollingsworth John and Karen, U.S. Fish and Wildlife Service [Public domain], via Wikimedia Commons Cow: By Keith Weller/USDA (www.ars.usda.gov: Image Number K5176-3) [Public domain], via Wikimedia Commons 27