ACTIONABLE CONTINUOUS DELIVERY METRICS Suzanne Prince, Head of - - PowerPoint PPT Presentation

▶
actionable continuous delivery metrics
SMART_READER_LITE
LIVE PREVIEW

ACTIONABLE CONTINUOUS DELIVERY METRICS Suzanne Prince, Head of - - PowerPoint PPT Presentation

Jul 27, 2023 91 likes •791 views

ACTIONABLE CONTINUOUS DELIVERY METRICS Suzanne Prince, Head of Product, ThoughtWorks Products Head of Product for ThoughtWorks Products 13+ years experience with agile, continuous integration and continuous delivery practices Tweeting

slide-1
SLIDE 1

ACTIONABLE CONTINUOUS DELIVERY METRICS

Suzanne Prince, Head of Product, ThoughtWorks Products

slide-2
SLIDE 2

Head of Product for ThoughtWorks Products 13+ years experience with agile, continuous integration and continuous delivery practices Tweeting @pm_suzie

slide-3
SLIDE 3
  • ver 20 years

4000 people 40 offices 14 countries

slide-4
SLIDE 4
slide-5
SLIDE 5

THIS TALK

slide-6
SLIDE 6

THIS TALK

  • What is continuous delivery (CD)
slide-7
SLIDE 7

THIS TALK

  • What is continuous delivery (CD)
  • Explain why you should measure your continuous delivery process
slide-8
SLIDE 8

THIS TALK

  • What is continuous delivery (CD)
  • Explain why you should measure your continuous delivery process
  • Share what continuous delivery metrics you should measure
slide-9
SLIDE 9

THIS TALK

  • What is continuous delivery (CD)
  • Explain why you should measure your continuous delivery process
  • Share what continuous delivery metrics you should measure
  • Review some scenarios to explain what certain metrics reveal about your

continuous delivery process

slide-10
SLIDE 10

THIS TALK

  • What is continuous delivery (CD)
  • Explain why you should measure your continuous delivery process
  • Share what continuous delivery metrics you should measure
  • Review some scenarios to explain what certain metrics reveal about your

continuous delivery process

  • Questions
slide-11
SLIDE 11

CONTINUOUS DELIVERY

slide-12
SLIDE 12

“Continuous Delivery is the ability to get changes of all types —including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.”

  • Jez Humble, continuousdelivery.com
slide-13
SLIDE 13

BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL

slide-14
SLIDE 14

BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL

slide-15
SLIDE 15

BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL

slide-16
SLIDE 16

BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL

slide-17
SLIDE 17

BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL

slide-18
SLIDE 18

BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL BUILD & UNIT TESTS SMOKE TESTS INTEGRATION TESTS REGRESSION TEST UAT PROD MATERIAL

slide-19
SLIDE 19
slide-20
SLIDE 20
slide-21
SLIDE 21
slide-22
SLIDE 22

WHY MEASURE

slide-23
SLIDE 23

FEEDBACK AND IMPROVEMENT PLAN DO CHECK ACT

slide-24
SLIDE 24

PREDICTABILITY

6.5 13 19.5 26 Story 1 2 3 4 5 6 7 8 9 10

Cycle time (days) Trend

slide-25
SLIDE 25

BENCHMARKING

Modified from Forsgren PhD, Nicole. Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations

High performers Median performers Low performers Deployment frequency On demand (multiple deploys per day) 1/week - 1/month 1/week - 1/month Lead time for changes <1hr 1 week - 1 month 1 week - 1 month Change failure rate <15% <15% 31-45% MTTR <1hr <1 day 1 day - 1 week

slide-26
SLIDE 26

WHAT TO MEASURE

slide-27
SLIDE 27

WHAT TO MEASURE

  • Throughput
  • Deployment frequency
  • Cycle time
  • Lead time
  • Mean time between failures
  • Mean time to recover (MTTR)
  • Failure rate
  • Defect fix times
  • Escaped defects
  • Total regression test time
  • Number of branches in version

control

  • Production outages during

deployment


slide-28
SLIDE 28

WHAT TO MEASURE

  • Throughput
  • Deployment frequency
  • Cycle time
  • Lead time
  • Mean time between failures
  • Mean time to recover (MTTR)
  • Failure rate
  • Defect fix times
  • Escaped defects
  • Total regression test time
  • Number of branches in version

control

  • Production outages during

deployment


slide-29
SLIDE 29

THROUGHPUT How often does code reach a certain point in the CD pipeline? E.g. How often do you deploy?

slide-30
SLIDE 30

DAYS THROUGHPUT THROUGHPUT: 25%

slide-31
SLIDE 31

CYCLE TIME How long does it take to go from one point to the to another point in the CD pipeline? E.g. How long does it take to go from code commit to code successfully running in production?

slide-32
SLIDE 32

CYCLE TIME DAYS THROUGHPUT: 25% CYCLE TIME: 3 DAYS

slide-33
SLIDE 33

FAILURE RATE What percentage of changes results a failure? E.g. What percentage of changes break builds? What percentage of deployments result in a service outage?

slide-34
SLIDE 34

THROUGHPUT: 25% CYCLE TIME: 3 DAYS FAILURE RATE: 75% DAYS FAILURE RATE

slide-35
SLIDE 35

MEAN TIME TO RECOVER (MTTR) How long does it generally take to fix a failure? E.g. How long does it take to fix a broken build? How long does it take to restore service during a deployment failure?

slide-36
SLIDE 36

MTTR DAYS MTTR THROUGHPUT: 25% CYCLE TIME: 3 DAYS FAILURE RATE: 75% MTTR: 2 DAYS

slide-37
SLIDE 37

WHAT TO MEASURE

  • Throughput
  • Deployment frequency
  • Cycle time
  • Lead time
  • Mean time between failures
  • Mean time to recover (MTTR)
  • Failure rate
  • Defect fix times
  • Escaped defects
  • Total regression test time
  • Number of branches in version

control

  • Production outages during

deployment


slide-38
SLIDE 38

CAUTION! Beware of:

  • vanity metrics
  • unclear metrics
  • invisible metrics
  • comparing across teams
  • the Hawthorne Effect or observer effect
  • gathering “all the data” and not using it
slide-39
SLIDE 39

VANITY METRICS

https://res.infoq.com/articles/Continuous-Delivery-Maturity-Model/en/resources/fig1large.jpg

slide-40
SLIDE 40

ACT ON YOUR METRICS

slide-41
SLIDE 41
slide-42
SLIDE 42

LOW THROUGHPUT

slide-43
SLIDE 43
slide-44
SLIDE 44
slide-45
SLIDE 45
slide-46
SLIDE 46
slide-47
SLIDE 47

LOW THROUGHPUT Causes

  • Slow builds
  • Builds that fail often
  • Long lived feature branches

How to resolve

  • Review cycle time and failure rates
  • Consider using feature toggles and short-lived branches
slide-48
SLIDE 48

SLOW CYCLE TIME

slide-49
SLIDE 49
slide-50
SLIDE 50
slide-51
SLIDE 51
slide-52
SLIDE 52
slide-53
SLIDE 53
slide-54
SLIDE 54

SLOW CYCLE TIME Causes:

  • slow individual builds
  • delays due to manual approvals

How to resolve:

  • speed up slow steps by rewriting or parallelizing
  • automate or simplify manual processes
slide-55
SLIDE 55

HIGH FAILURE RATE

slide-56
SLIDE 56
slide-57
SLIDE 57
slide-58
SLIDE 58
slide-59
SLIDE 59

HIGH FAILURE RATE Causes

  • genuine failures
  • flaky tests
  • tests too slow or difficult to run locally before check-in

How to resolve

  • fail fast
  • make it easier to run tests locally
slide-60
SLIDE 60

HIGH MTTR

slide-61
SLIDE 61
slide-62
SLIDE 62
slide-63
SLIDE 63

HIGH MTTR Causes

  • no-one cares
  • issues are hard to resolve
  • combination with high failure rate and/or slow cycle time

How to resolve

  • revert of failing commits
  • stop the line
slide-64
SLIDE 64

FEEDBACK CYCLES PLAN DO CHECK ACT

slide-65
SLIDE 65

RECAP

slide-66
SLIDE 66

RECAP

  • Metrics are important to set goals, improve and predict
  • Start with throughput, cycle time, failure rate and MTTR
  • Be thoughtful about what you measure
  • Look for connections between metrics
  • Understand your context
  • Review, change and improve your process
  • Consider using tools to help capture and visualize data

slide-67
SLIDE 67

ADDITIONAL RESOURCES

  • Download GoCD - https://www.gocd.org/
  • GoCD Analytics plugin - https://www.gocd.org/analytics/
  • More events and talks - https://www.gocd.org/events/
  • 4 important metrics for continuous delivery - https://www.gocd.org/

2018/01/31/continuous-delivery-metrics/

  • Why measure your CD process https://www.gocd.org/2018/10/30/measure-

continuous-delivery-process/

slide-68
SLIDE 68

QUESTIONS ?