DEVOPS METRICS ANDREW SHAFER - PATRICK DEBOIS - ISRAEL GAT Thursday - - PowerPoint PPT Presentation

▶
devops metrics
SMART_READER_LITE
LIVE PREVIEW

DEVOPS METRICS ANDREW SHAFER - PATRICK DEBOIS - ISRAEL GAT Thursday - - PowerPoint PPT Presentation

Apr 11, 2024 141 likes •1.12k views

DEVOPS METRICS ANDREW SHAFER - PATRICK DEBOIS - ISRAEL GAT Thursday 16 June 2011 Patrick Debois Jedi @patrickdebois http://jedi.be/blog http://www.vectronicsappleworld.com/collection/articlepics/joysticks/snap23.jpg Thursday 16 June 2011

slide-1
SLIDE 1

DEVOPS METRICS

ANDREW SHAFER - PATRICK DEBOIS - ISRAEL GAT

Thursday 16 June 2011

slide-2
SLIDE 2

http://www.vectronicsappleworld.com/collection/articlepics/joysticks/snap23.jpg

Patrick Debois Jedi @patrickdebois http://jedi.be/blog

Thursday 16 June 2011

slide-3
SLIDE 3

Israel Gat Cutter Consortium @agileexec http://theagileexecutive.com

Thursday 16 June 2011

slide-4
SLIDE 4

Andrew Shafer - Rabble Rouser - Cloudscaling @littleidea - http://stochasticresonance.wordpress.com

Thursday 16 June 2011

slide-5
SLIDE 5

http://www.flickr.com/photos/quinnanya/3588989219

Thursday 16 June 2011

slide-6
SLIDE 6

http://www.flickr.com/photos/zen/3233023066/

Thursday 16 June 2011

slide-7
SLIDE 7

To Compare

http://www.flickr.com/photos/thebusybrain/2492945625/

Thursday 16 June 2011

slide-8
SLIDE 8

http://www.flickr.com/photos/topshampatti/219953012/

Thursday 16 June 2011

slide-9
SLIDE 9

To Understand

Thursday 16 June 2011

slide-10
SLIDE 10

To Protect

Thursday 16 June 2011

slide-11
SLIDE 11

To Improve

http://www.flickr.com/photos/vizzzual-dot-com/2655969483/

Thursday 16 June 2011

slide-12
SLIDE 12

Describing a Metric

http://www.flickr.com/photos/chandramarsono/4324373384/

Thursday 16 June 2011

slide-13
SLIDE 13

Describing a Metric

  • basis of measurement

http://www.flickr.com/photos/chandramarsono/4324373384/

Thursday 16 June 2011

slide-14
SLIDE 14

Describing a Metric

  • basis of measurement
  • assumptions

http://www.flickr.com/photos/chandramarsono/4324373384/

Thursday 16 June 2011

slide-15
SLIDE 15

Describing a Metric

  • basis of measurement
  • assumptions
  • level and usage

http://www.flickr.com/photos/chandramarsono/4324373384/

Thursday 16 June 2011

slide-16
SLIDE 16

Describing a Metric

  • basis of measurement
  • assumptions
  • level and usage
  • desired trend

http://www.flickr.com/photos/chandramarsono/4324373384/

Thursday 16 June 2011

slide-17
SLIDE 17

Describing a Metric

  • basis of measurement
  • assumptions
  • level and usage
  • desired trend
  • when to use

http://www.flickr.com/photos/chandramarsono/4324373384/

Thursday 16 June 2011

slide-18
SLIDE 18

Describing a Metric

  • basis of measurement
  • assumptions
  • level and usage
  • desired trend
  • when to use
  • when to stop

http://www.flickr.com/photos/chandramarsono/4324373384/

Thursday 16 June 2011

slide-19
SLIDE 19

Describing a Metric

  • basis of measurement
  • assumptions
  • level and usage
  • desired trend
  • when to use
  • when to stop
  • how to game

http://www.flickr.com/photos/chandramarsono/4324373384/

Thursday 16 June 2011

slide-20
SLIDE 20

Describing a Metric

  • basis of measurement
  • assumptions
  • level and usage
  • desired trend
  • when to use
  • when to stop
  • how to game
  • warnings

http://www.flickr.com/photos/chandramarsono/4324373384/

Thursday 16 June 2011

slide-21
SLIDE 21

Beware of Vanity Metrics

UNACTIONABLE INACCESSIBLE UNAUDITABLE

http://www.flickr.com/photos/centralasian/5829549813/

Thursday 16 June 2011

slide-22
SLIDE 22

Red is good , Green is worthless

  • http://www.reliableplant.com/Read/21305/lean-metrics-red-is-good,-green-is-worthless

Thursday 16 June 2011

slide-23
SLIDE 23

Thursday 16 June 2011

slide-24
SLIDE 24

“In truth, a good case could be made that if your knowledge is meagre and unsatisfactory, the last thing in the world you should do is make measurements; the chance is negligible that you will measure the right things accidentally.”

http://stochasticresonance.wordpress.com/2008/08/03/miles-per-gallon/

George Miller

Thursday 16 June 2011

slide-25
SLIDE 25

What we can learn from monitoring

http://www.experimentalistsanonymous.com/stuff/paralleluniverse/images/2edmoneyshot.jpg Thursday 16 June 2011

slide-26
SLIDE 26

Level 0 - Individual Resource

Thursday 16 June 2011

slide-27
SLIDE 27

Availability

Network, Memory, CPU Processes, Port,SSH Login

Thursday 16 June 2011

slide-28
SLIDE 28

Availability

Network, Memory, CPU Processes, Port,SSH Login

Thursday 16 June 2011

slide-29
SLIDE 29

Capacity

Network, Memory, CPU Processes, Port,SSH Login

Thursday 16 June 2011

slide-30
SLIDE 30

Capacity

Network, Memory, CPU Processes, Port,SSH Login

Thursday 16 June 2011

slide-31
SLIDE 31

Health

http://www.geocities.jp/nikonikocalendar/index_en.html

Thresholds

http://www.smiley-faces.org/wallpaper/smiley-face-wallpaper-001.jpg Thursday 16 June 2011

slide-32
SLIDE 32

Health

http://www.geocities.jp/nikonikocalendar/index_en.html

Thresholds

http://www.smiley-faces.org/wallpaper/smiley-face-wallpaper-001.jpg Thursday 16 June 2011

slide-33
SLIDE 33

http://www.slideshare.net/krivitsky/agile-metrics-presentation-654259

Measurement Metric Diagnostic

Pete Behrens, Trail Ridge Consulting

Does it contribute to the bottom line?

Metric versus Diagnostic

Yes No

Thursday 16 June 2011

slide-34
SLIDE 34

Level 1 - Services Level

Thursday 16 June 2011

slide-35
SLIDE 35

Performance

SSH, HTTP , NTP

Thursday 16 June 2011

slide-36
SLIDE 36

Performance

SSH, HTTP , NTP # Servers/Admin

Thursday 16 June 2011

slide-37
SLIDE 37

Performance

# Servers/Admin # Line of Code/Developer SSH, HTTP , NTP

Thursday 16 June 2011

slide-38
SLIDE 38

“Gameable”

http://kithicor.org/wp/wp-content/uploads/2010/01/gamerEvolution.jpg Thursday 16 June 2011

slide-39
SLIDE 39

Level 2 - Team Level

Thursday 16 June 2011

slide-40
SLIDE 40

Aggregation

WEB 1 LB-WEB WEB 2 WEB 3

Thursday 16 June 2011

slide-41
SLIDE 41

Aggregation

DEV DEV Project Team DEV DEV

Thursday 16 June 2011

slide-42
SLIDE 42

Burn up chart

Thursday 16 June 2011

slide-43
SLIDE 43

Team Utilization

Thursday 16 June 2011

slide-44
SLIDE 44

“Velocity also has a direction component”

http://www.randomstew.com/wp-content/uploads/mooneye.jpg Thursday 16 June 2011

slide-45
SLIDE 45

What about efficiency?

http://www.flickr.com/photos/jdiggans/92981578/

Thursday 16 June 2011

slide-46
SLIDE 46

http://sievertschreiber.files.wordpress.com/2009/12/good-code-is-measured-in-wtf-per-minute.jpg Thursday 16 June 2011

slide-47
SLIDE 47

Incentives & Behaviors

http://www.flickr.com/photos/turtlemadness/2439050717/

Thursday 16 June 2011

slide-48
SLIDE 48

http://www.xgn.nl/images/upload/20081020225036.jpg http://www.kawaiikakkoiisugoi.com/wp-content/uploads/2011/03/pacman-screen.jpg

Operations Development

Level 3 - Integration Level

Thursday 16 June 2011

slide-49
SLIDE 49

Integration

DB WEB APP

Thursday 16 June 2011

slide-50
SLIDE 50

Collaboration

# of deploys / day OPS DEV

http://www.sage.org/lists/sage-members-archive/2001/msg01208.html http://www.datacenterknowledge.com/archives/2009/12/30/how-many-servers-can-one-admin-manage/ http://fc04.deviantart.net/fs70/f/2010/108/2/3/Pac_Man_Widescreen_by_molotov_arts.jpg

QA

Thursday 16 June 2011

slide-51
SLIDE 51

Counting Collaboration

  • http://www.nature.com/news/2010/101013/full/news.2010.538.html

More interaction doesn’t mean a better party

Thursday 16 June 2011

slide-52
SLIDE 52

Measuring Collaboration?

http:///www.healthylarimer.org/documents/PlayNice.ppt

“Play Nice Study”

Thursday 16 June 2011

slide-53
SLIDE 53

Levels Collaboration

Cooperation Coordination Coalition Collaboration

Thursday 16 June 2011

slide-54
SLIDE 54

Cooperation

provide information somewhat defined roles formal communication decisions made independently

Thursday 16 June 2011

slide-55
SLIDE 55

Coordination

  • share information & resources
  • defined roles
  • frequent communication
  • some shared decision making

Thursday 16 June 2011

slide-56
SLIDE 56

Coalition

  • share ideas
  • share resources
  • frequent & prioritized

communication

  • all members have a vote on

decisions

Thursday 16 June 2011

slide-57
SLIDE 57

Collaboration

  • members belong to one system
  • frequent communication

characterized by mutual trust

  • consensus on all decisions

Thursday 16 June 2011

slide-58
SLIDE 58

http://www.au.af.mil/au/awc/awcgate/navy/onr_metrics_on_collab.pdf

Knowledge sharing

Thursday 16 June 2011

slide-59
SLIDE 59

Team Resilience

Truck Factor 10th Floor Test

http://www.kitchensoap.com/

Resilience should be a 4th management objective, alongside Better/Faster/Cheaper

John Allspaw

The number of people on your team who have to be hit with a truck before the project is in serious trouble

to take a random machine in your infrastructure , drop it from the 10th floor, and be able to restore your infrastructure to working within 5-10 minute

Thursday 16 June 2011

slide-60
SLIDE 60

Black Hole Theory

http://www.collaborate.com/blog/2010/01/28/musing-metrics-collaboration

Absorbs light, to see the effects you need to look at nearby

  • bjects

Thursday 16 June 2011

slide-61
SLIDE 61

http://communityandbusiness.groupsite.com/group/Collaboration20/post/measuring-the-value-of-collaboation

  • Saving time or Money

(tangible)

  • Increase Quality

(tangible... but less so)

  • Innovating and/or

providing decision support (tangible but less than quality)

  • Easing access to and

interactions with subject- matter experts (intangible)

Quest for proof

Thursday 16 June 2011

slide-62
SLIDE 62

http://socialconsulting.wordpress.com/2009/02/24/measuring-collaboration/

Quality Improvement Tune existing servers Add more servers Process Improvement Easier More Difficult!!

More vs Better Widgets

Thursday 16 June 2011

slide-63
SLIDE 63

Collaboration Challenge

“Maximize Team Utility”

Thursday 16 June 2011

slide-64
SLIDE 64

Level 4 - Business Level

Thursday 16 June 2011

slide-65
SLIDE 65

AHA KA-CHING!

Thursday 16 June 2011

slide-66
SLIDE 66

http://www.flickr.com/photos/hanekomu/4481674104/

Return on Investment

Thursday 16 June 2011

slide-67
SLIDE 67

Inventory (WIP)

Thursday 16 June 2011

slide-68
SLIDE 68

Time to Market

http://stefanroock.wordpress.com/2010/03/02/kanban-definition-of-lead-time-and-cycle-time/ Thursday 16 June 2011

slide-69
SLIDE 69

http://museumdosgames.blogspot.com/2010/04/game-over.html

Mean Time Between Failure

Thursday 16 June 2011

slide-70
SLIDE 70

Mean Time Between Resolution

Thursday 16 June 2011

slide-71
SLIDE 71

Technical Debt

Thursday 16 June 2011

slide-72
SLIDE 72

Leaks

Thursday 16 June 2011

slide-73
SLIDE 73

Debt Metaphor

not all debt is created equal

http://www.flickr.com/photos/shoot-art/4536509717/

Thursday 16 June 2011

slide-74
SLIDE 74

Customer Responsiveness

Jim Highsmith

Thursday 16 June 2011

slide-75
SLIDE 75

Types of Tech debt

http://forums.construx.com/blogs/stevemcc/archive/2007/11/01/technical-debt-2.aspx

“oops” “yeah I know” “we’ll fix it the next minor revision” “we’ll fix it the next major revision” unintentional intentional short term (tactical) long term (strategical)

Thursday 16 June 2011

slide-76
SLIDE 76

Gartner

  • http://www.castsoftware.com/resources/materials/analyst/Gartner-How_to_Monetize_Application_Technical_Debt.pdf

Thursday 16 June 2011

slide-77
SLIDE 77

cious Cycle of Technical Debt

(More) Relentless Pressure Take Technical Debt Fail to Pay Debt Back Neglect Maintenance Technical Debt Accrues Diminished Dev Velocity

Vicious Cycle of Technical Debt

Israel Gat

http://theagileexecutive.com/2010/09/20/how-to-break-the-vicious-cycle-of-technical-debt/ Thursday 16 June 2011

slide-78
SLIDE 78

Symptoms - Measurable

  • Poor Customer Responsiveness
  • Long Delivery Times
  • Late deliveries
  • Lots of defects (Toxic Code)
  • Rising Development costs
  • Frustrated & Poor performing Teams

http://www.theagileengineer.com/public/Home/Home_files/TechnicalDebt_published.pdf Thursday 16 June 2011

slide-79
SLIDE 79

Secret Sauce

http://radar.oreilly.com/archives/2007/10/Ops-ROI-jesserobbins.html Thursday 16 June 2011

slide-80
SLIDE 80

The Team Knows

http://kingsinsight.com/2010/07/31/ask-the-team-about-technical-debt-the-creeping-doom-graph/ Thursday 16 June 2011

slide-81
SLIDE 81

http://farm4.static.flickr.com/3167/2769773425_16c235dc28.jpg

Failure Demand Value Demand % As a Team

Thursday 16 June 2011

slide-82
SLIDE 82

Level 5 - End User Level

Thursday 16 June 2011

slide-83
SLIDE 83

End User Perspective

  • END to END monitoring
  • User Experience (browser testing)
  • External parties (Email, ISP

, Plugins)

Thursday 16 June 2011

slide-84
SLIDE 84

Participatory Design

http://www.informaworld.com/smpp/content~db=all~content=a790786652 Thursday 16 June 2011

slide-85
SLIDE 85

“Crowd” - Monitoring

Sudden Change in User Activity Pattern Detection of problem? A/B Testing

Thursday 16 June 2011

slide-86
SLIDE 86

http://www.informaworld.com/smpp/content~db=all~content=a790786652

Spend more time creatively , early in the process Automation => Traditional Engineering

Thursday 16 June 2011

slide-87
SLIDE 87

Design Strategies

http://www.psych.lancs.ac.uk/people/lindenball.html http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 p 4

To Design is Human To Design does not require super powers Design used to be a collective or shared ability

Thursday 16 June 2011

slide-88
SLIDE 88

Open Innovation

http://en.wikipedia.org/wiki/Open_innovation http://www.clickadvisor.com/downloads/Bughin_Next_Step_in_Open_Innovation.pdf

Wikipedia Shared Infrastructures Shared Config Management Open Source

Thursday 16 June 2011

slide-89
SLIDE 89

Novice vs Expert

  • depth first
  • vertical oriented
  • can not deal with

uncertainty

http://www.psych.lancs.ac.uk/people/lindenball.html http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 p 147

Linden Ball

  • breath first , depth dives
  • horizontal oriented
  • uncertainty = options

Thursday 16 June 2011

slide-90
SLIDE 90

Continuous Evolution

“Problem space evolves with the solution”

http://www.psych.lancs.ac.uk/people/lindenball.html http://www.amazon.com/Research-Design-Thinking-Nigel-Cross/dp/9062757960 Thursday 16 June 2011

slide-91
SLIDE 91

Cultural Taxonomy

Thursday 16 June 2011

slide-92
SLIDE 92

Cultural Taxonomy

Thursday 16 June 2011

slide-93
SLIDE 93

tl;dr

  • avoid vanity metrics
  • insight > data
  • don`t hate the player
  • hate the game
  • business value good
  • beware of technical debt

Thursday 16 June 2011

slide-94
SLIDE 94

Questions?

Thursday 16 June 2011

slide-95
SLIDE 95

Thursday 16 June 2011

slide-96
SLIDE 96

Thank you!

Thursday 16 June 2011