Using Monitoring and Metrics to learn in Development Patrick Debois - - PowerPoint PPT Presentation

▶
using monitoring and metrics to learn in development
SMART_READER_LITE
LIVE PREVIEW

Using Monitoring and Metrics to learn in Development Patrick Debois - - PowerPoint PPT Presentation

Oct 16, 2022 109 likes •778 views

Using Monitoring and Metrics to learn in Development Patrick Debois Atlassian http://jedi.be/blog @patrickdebois Monday, May 21, 12 HOSTEDOPS https://my.atlassian.com/ondemand/ http://www.cutter.com http://itrevolution.com Vagrant &

slide-1
SLIDE 1

Using Monitoring and Metrics to learn in Development

Patrick Debois Atlassian

http://jedi.be/blog @patrickdebois

Monday, May 21, 12

slide-2
SLIDE 2

HOSTEDOPS

Vagrant & Veewee

http://devopsdays.org http://itrevolution.com https://my.atlassian.com/ondemand/ http://www.cutter.com

Monday, May 21, 12

slide-3
SLIDE 3

Your Code

Monday, May 21, 12

slide-4
SLIDE 4

Monday, May 21, 12

slide-5
SLIDE 5

“Test Driven Development”

  • CI Coward

Monday, May 21, 12

slide-6
SLIDE 6

Testing in Controlled Environment

Monday, May 21, 12

slide-7
SLIDE 7

Your Code in Test

Monday, May 21, 12

slide-8
SLIDE 8

“Until code runs in production , your code is just inventory”

  • Agile Anonymous

Monday, May 21, 12

slide-9
SLIDE 9

OPS DEV

CLASH

Continuous Delivery

Monday, May 21, 12

slide-10
SLIDE 10

Monday, May 21, 12

slide-11
SLIDE 11

Shit Happens in production

Monday, May 21, 12

slide-12
SLIDE 12

“You go to production with the code you have, not the code you wish you had”

  • Devops Rumsfeld

Monday, May 21, 12

slide-13
SLIDE 13

OPS DEV

Three Phases of Creativity (Disney) Dreamer Realist Critic

Monday, May 21, 12

slide-14
SLIDE 14

OPS DEV

Area 4: Embed Operations knowledge into Project Area 2: Extend operations feedback to project Area 1: Extend delivery to production Area 3: Embed Project knowledge into Operations

Devops Areas

http://jedi.be/blog/2012/05/12/codifying-devops-area-practices/

Monday, May 21, 12

slide-15
SLIDE 15

OPS DEV

Culture Automate Measure Share

http://www.opscode.com/blog/2010/07/16/what-devops-means-to-me/

Monday, May 21, 12

slide-16
SLIDE 16

OPS DEV

Faster and better feedback Smaller and more frequent changes

Monday, May 21, 12

slide-17
SLIDE 17

OPS DEV

Area 1: Extend delivery to production “think Jez Humble” Area 1

Monday, May 21, 12

slide-18
SLIDE 18

http://www.slideshare.net/Urbancode/continuous-delivery-maturity-model

Continuous Integration Maturity Model

Monday, May 21, 12

slide-19
SLIDE 19

Infrastructure Code Repository DB WEB APP DB WEB APP DB WEB APP DEV TEST PROD Reuse “Infra code” across Environments

Monday, May 21, 12

slide-20
SLIDE 20

DB WEB APP HOST VM JEOS http://vagrantup.com/

Simple CLI

UP PROVISION HALT DESTROY

Vagrantfile Config Basebox Puppet/Chef VM Management Reuse “workflow” across Environments

Monday, May 21, 12

slide-21
SLIDE 21

INFRA

DB WEB APP

DEV TEST PROD Integrate with Continuous Integration

APP

DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Infrastructure Code Repository Application Code Repository

Monday, May 21, 12

slide-22
SLIDE 22

INFRA

DB WEB APP

DEV TEST PROD Continuous Integration to Continuous Delivery

APP

DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Infrastructure Code Repository Application Code Repository

Faster/Delivery Faster/Feedback

Monday, May 21, 12

slide-23
SLIDE 23

OPS DEV

Area 2: Extend operations feedback to project think “John Allspaw” Area2

Monday, May 21, 12

slide-24
SLIDE 24

Monitoring Layers

OS Middleware App OS Middleware App Clustered Service External Services User/Business Network/Security Hardware Hardware Network Storage Services

Monday, May 21, 12

slide-25
SLIDE 25

Monitoring Layers

Behavioral (larger coverage) Diagnostics

Monday, May 21, 12

slide-26
SLIDE 26

“What part made it fail” ~unit test

CPU Memory Network

Monday, May 21, 12

slide-27
SLIDE 27

“Is it working?” ~behavioral test

Monday, May 21, 12

slide-28
SLIDE 28

Monitoring Driven Development

Create a monitor check before implementing a feature

Monday, May 21, 12

slide-29
SLIDE 29

Information Radiator

Monday, May 21, 12

slide-30
SLIDE 30

http://jedi.be/blog/2012/01/03/monitoring-wonderland-survey-introduction/ http://jedi.be/blog/2012/01/04/monitoring-wonderland-moving-up-the-stack-application-user-metrics/ http://jedi.be/blog/2012/01/04/monitoring-wonderland-visualization/ http://jedi.be/blog/2012/01/03/monitoring-wonderland-nagios-the-mighty-beast/ http://jedi.be/blog/2012/01/03/monitoring-wonderland-metrics-api-gateways/

Monitoring Wonderland

Monday, May 21, 12

slide-31
SLIDE 31

“traditional In house monitoring”

Monday, May 21, 12

slide-32
SLIDE 32

Monday, May 21, 12

slide-33
SLIDE 33

“traditional In house metrics”

http://collectd.org/ http://munin-monitoring.org/

Monday, May 21, 12

slide-34
SLIDE 34

Ganglia

http://ganglia.info/

Monday, May 21, 12

slide-35
SLIDE 35

https://github.com/ monitoringsucks

Monday, May 21, 12

slide-36
SLIDE 36

“1980” style UI

Monday, May 21, 12

slide-37
SLIDE 37

No API

Monday, May 21, 12

slide-38
SLIDE 38

Hard to scale

Monday, May 21, 12

slide-39
SLIDE 39

Static configurations

Monday, May 21, 12

slide-40
SLIDE 40

Hard to expose “data” UI vs API

Monday, May 21, 12

slide-41
SLIDE 41

Not self-servicing

Monday, May 21, 12

slide-42
SLIDE 42

“Monitoring as a service”

Monday, May 21, 12

slide-43
SLIDE 43

https://github.com/sensu/sensu

Sonian Cloud Monitoring “Sensu”

Monday, May 21, 12

slide-44
SLIDE 44

http://code.google.com/p/rocksteady/

Monday, May 21, 12

slide-45
SLIDE 45

http://labs.omniti.com/labs/reconnoiter

Reconnoiter's goal is to better the world of monitoring by marrying fault detection and trending together. Through ease of configuration and ongoing maintenance, Reconnoiter encourages monitoring important technical metrics along side critical business metrics to improve awareness and ultimately accountability.

Monday, May 21, 12

slide-46
SLIDE 46

http://opentsdb.net/

Scalable writes

Monday, May 21, 12

slide-47
SLIDE 47

http://graphite.wikidot.com/

Selfservicing Graphs

Monday, May 21, 12

slide-48
SLIDE 48

“App Metrics”

Monday, May 21, 12

slide-49
SLIDE 49

“End-User Metrics”

Monday, May 21, 12

slide-50
SLIDE 50

http://www.appdynamics.com/

Monday, May 21, 12

slide-51
SLIDE 51

https://github.com/lookfirst/jmxtrans

Monday, May 21, 12

slide-52
SLIDE 52

https://github.com/etsy/statsd APP STATSD counting timing sampling GRAPHITE

Monday, May 21, 12

slide-53
SLIDE 53

Metrics Java Library

http://metrics.codahale.com/ https://github.com/tritonrc/metricsd

Monday, May 21, 12

slide-54
SLIDE 54

Business Level Metrics

Usage patterns - Google Analytics

Monday, May 21, 12

slide-55
SLIDE 55

A/B Testing in Prod

Metrics / Information

Monday, May 21, 12

slide-56
SLIDE 56

Meta-Ops Metrics

Know context

Monday, May 21, 12

slide-57
SLIDE 57

https://github.com/rashidkpc/Kibana http://graylog2.org/ http://logstash.net/

Logs as feedback

Monday, May 21, 12

slide-58
SLIDE 58

Unix Pipe Analogy

Monday, May 21, 12

slide-59
SLIDE 59

INFRA

DB WEB APP

DEV TEST PROD Integrate with Continuous Integration

APP

DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Infrastructure Code Repository Application Code Repository

Monday, May 21, 12

slide-60
SLIDE 60

INFRA

DB WEB APP

DEV TEST PROD Reuse Tools across dev & ops

APP

DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Infrastructure Code Repository Application Code Repository

Monitoring/ Metrics

Monday, May 21, 12

slide-61
SLIDE 61

INFRA

DB WEB APP

DEV TEST PROD

APP

DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Infrastructure Code Repository Application Code Repository

Monitoring/ Metrics Monitoring/ Metrics Monitoring/ Metrics

Integrate with Continuous Monitoring

Monday, May 21, 12

slide-62
SLIDE 62

INFRA

DB WEB APP

DEV TEST PROD Reuse Workflow

APP

DB WEB APP DB WEB APP DB WEB APP DB WEB APP DB WEB APP

Debugging

Monitoring/ Metrics Monitoring/ Metrics Monitoring/ Metrics

Post-Mortem

Monday, May 21, 12

slide-63
SLIDE 63

Metrics Driven Engineering

http://www.slideshare.net/mikebrittain/metricsdriven- engineering

Monday, May 21, 12

slide-64
SLIDE 64

Monday, May 21, 12

slide-65
SLIDE 65

“The future is metrics are here they are just not evenly distributed yet”

  • Morpheus Monitoring

Monday, May 21, 12

slide-66
SLIDE 66

https://github.com/monitoringsucks http://jedi.be/blog

Monday, May 21, 12