BUILDING A CUSTOMER QUALITY DASHBOARD John Ruberto VP of Quality - - PowerPoint PPT Presentation

▶
building a customer quality dashboard
SMART_READER_LITE
LIVE PREVIEW

BUILDING A CUSTOMER QUALITY DASHBOARD John Ruberto VP of Quality - - PowerPoint PPT Presentation

Aug 25, 2022 318 likes •611 views

BUILDING A CUSTOMER QUALITY DASHBOARD John Ruberto VP of Quality Engineering Clover, a First Data Company 2 3 First, A story 4 9.86 5 6 What is 9.86? Metric First Negative Link in Google Source Lines of Code 3 Cyclomatic

slide-1
SLIDE 1
slide-2
SLIDE 2 2

BUILDING A CUSTOMER QUALITY DASHBOARD

John Ruberto VP of Quality Engineering – Clover, a First Data Company

slide-3
SLIDE 3 3
slide-4
SLIDE 4 4

First, A story

slide-5
SLIDE 5 5

9.86

slide-6
SLIDE 6 6
slide-7
SLIDE 7 7

What is 9.86?

Metric First Negative Link in Google Source Lines of Code 3 Cyclomatic Complexity 19 Function Points 11 Code Coverage 6 Defect Removal Efficiency 6 Defect Density 9 Bug Count 15
slide-8
SLIDE 8 8

Principles for metrics

  • Related to our goals
  • Leading vs lagging indicators
  • Process metrics vs outcome metrics
  • Use the right technology to display
slide-9
SLIDE 9 9

Related to our goals

slide-10
SLIDE 10 10

Use the right technology to collect & display

slide-11
SLIDE 11 11

Provides actionable insights

slide-12
SLIDE 12 12

Goal-Questions-Metric

  • GQM
  • Victor Basili
  • Align on a set of goals
  • Ask questions about those goals
  • Design & collect metrics to answer the questions
slide-13
SLIDE 13 13

Why

  • Setting goals, in alignment with the wider organization, gains acceptance
  • Focus on what’s most important to your stakeholders
  • Provide “line of sight” from your metrics to your goals
  • Build comprehensive view of your goals.
slide-14
SLIDE 14 14

Example - Context

  • Software as a service application
  • > 500K active users
  • Paying monthly subscription
slide-15
SLIDE 15 15

Example

  • Goal: Deliver better quality to our customers
  • Questions:
  • How many defects do our customers report?
  • How are we trending on customer reported defects?
  • How quickly to we fix the defects?
  • What are the top causes of these defects?
  • Why aren’t we catching these bugs before release?
slide-16
SLIDE 16 16

Example –Delivered Quality

0% 10% 20% 30% 40% 50% 60% 70% RCA Pareto 0% 5% 10% 15% 20% 25% 30% 35% 40% 45% Reasons for Escape
slide-17
SLIDE 17 17

Characteristics to think about

  • Process Metrics vs Outcomes
  • LOC / Review hour vs Defects found per review
  • Leading Indicators vs Lagging Indicators
  • Code coverage vs delivered quality
  • Median vs Average (Better yet: percentile)
  • Median page load vs Average page load
  • % fixes within SLA vs Average Age
  • 2012 average income in San Mateo County
slide-18
SLIDE 18 18

Principles in using metrics

  • Direct measures instead of derived
  • “quality score”
  • Apdex
  • Actionable
  • Total crashes vs crash code pareto
  • Live data is best data
  • No powerpoint…
slide-19
SLIDE 19 19

Fallacies of Metrics - Gamification

  • Goal: Improve Testing Efficiency
  • Metric: Testing Efficiency: (fixed bugs / total submitted)
slide-20
SLIDE 20 20

Fallacies of Metrics – Confirmation Bias

  • Incoming bug rate improved dramatically – our quality must be outstanding!
Image Credit: By User:KAMiKAZOW (Transferred from en.wikipedia to Commons.) [Public domain], via Wikimedia Commons
slide-21
SLIDE 21 21

Fallacies of Metrics – Survivor Bias

Image Credit: WyrdLight.com [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons
slide-22
SLIDE 22 22

Fallacies of Metrics – Survivor Bias

20 40 60 80 100 120 140 160 180 Critical Major Minor Open bugs By Severity
slide-23
SLIDE 23 23

Measurement Bias

2% 3% 5% 8% 10% 3% 3% 8% 9% 12% 5% 13% 20% 30% 35% 29% 28% 24% 24% 22% 33% 33% 28% 21% 18% 28% 20% 15% 8% 3% 0% 5% 10% 15% 20% 25% 30% 35% 40% Release 1 Release 2 Release 3 Release 4 Release 5 Phase Detection By Release Rqmts Design Code Int Sys Customer
slide-24
SLIDE 24 24

Vanity Metrics

  • Don’t measure things that matter
  • Easily manipulated
  • But, make us feel good
slide-25
SLIDE 25 25

Keeping the gains

  • Process Wrapper
  • Monitor & regulate
  • Automatic trigger
  • Wide distribution
slide-26
SLIDE 26 26
  • Questions?
  • JohnRuberto@gmail.com
  • @johnruberto
  • http://linkedin.com/in/ruberto
slide-27
SLIDE 27 27

Photo Credits

Nadia Comm: Ben Sutherland https://www.flickr.com/photos/bensutherland/ Bull: By Hollingsworth John and Karen, U.S. Fish and Wildlife Service [Public domain], via Wikimedia Commons Cow: By Keith Weller/USDA (www.ars.usda.gov: Image Number K5176-3) [Public domain], via Wikimedia Commons