remediation patterns Jez Humble GOTO Copenhagen 2011 #gotocph - - PowerPoint PPT Presentation

▶

Jun 05, 2023 196 likes •434 views

remediation patterns Jez Humble GOTO Copenhagen 2011 #gotocph jez@thoughtworks.com @jezhumble #continuousdelivery Wednesday, May 11, 2011 remediation ITIL: Recovery to a known state after a failed Change or Release. Recovery:

SLIDE 1

remediation patterns

Jez Humble GOTO Copenhagen 2011 #gotocph jez@thoughtworks.com @jezhumble #continuousdelivery

Wednesday, May 11, 2011

SLIDE 2

ITIL: “Recovery to a known state after a failed Change or Release.” Recovery: “Returning a Configuration Item or an IT Service to a working state.” Jez: “Fixing shit when it breaks”

remediation

Wednesday, May 11, 2011

SLIDE 3

prevention patterns for low-risk release patterns for incremental delivery

strategies for remediation

Wednesday, May 11, 2011

SLIDE 4

1oz of prevention

Diagram invented by Brian Marick

Wednesday, May 11, 2011

SLIDE 5

deployment pipeline

Delivery team Version control Build & unit tests Automated acceptance tests User acceptance tests Release Check in Feedback Trigger Check in Feedback Trigger Trigger Check in Trigger Trigger Approval Approval Feedback Feedback Feedback Feedback

Wednesday, May 11, 2011

SLIDE 6

testing on production environments creating maintainable acceptance tests testing cross-functional requirements

the hard bits

Wednesday, May 11, 2011

SLIDE 7

automate provisioning and deployment ensure devs, testers and ops collaborate throughout

reducing release risk

Wednesday, May 11, 2011

SLIDE 8

canary releasing

Diagram by Martin Fowler

Wednesday, May 11, 2011

SLIDE 9

canary releasing

Diagram by Martin Fowler

Wednesday, May 11, 2011

SLIDE 10

reduce risk of release multivariant testing performance testing

canary releasing

Wednesday, May 11, 2011

SLIDE 11

immune system

what if someone replaced your “buy” button with spacer.gif?

T cells http://www.flickr.com/photos/gehealthcare/3326186490/

Wednesday, May 11, 2011

SLIDE 12

Business metrics - revenue, # orders, # users Ops metrics - changes, incidents, TTD, TTR, TBF Technical metrics - TPS, response time, hits

monitoring

http://www.flickr.com/photos/wwarby/3296379139/

Wednesday, May 11, 2011

SLIDE 13

root cause analysis collaboration data

the hard bits

Wednesday, May 11, 2011

SLIDE 14

incremental delivery

John Allspaw: “Ops Metametrics” http://slidesha.re/dsSZIr

Wednesday, May 11, 2011

SLIDE 15

incremental deployments develop on mainline feature toggles and branch by abstraction dark launching

incremental delivery

Wednesday, May 11, 2011

SLIDE 16

feature toggles

[featureToggles] wobblyFoobars: true flightyForkHandles: false

Config File

<toggle name=wobblyFoobars> ... various UI elements </toggle>

some.jsp

forkHandle = (featureConfig.isOn(‘flightlyForkHandles)) ? new FlightyForkHander(aCandle) : new ForkHandler(aCandle)

ther.java

Stolen from Martin Fowler http://martinfowler.com/bliki/FeatureToggle.html

Wednesday, May 11, 2011

SLIDE 17

branch by abstraction

Component A Component B

Seam

Component A

Wednesday, May 11, 2011

SLIDE 18

branch by abstraction

Component A Component A Component B’ Abstraction layer Component B

Wednesday, May 11, 2011

SLIDE 19

incremental deployment

STATIC CONTENT /static/1.1 /static/1.0 DEPENDENT SERVICE 1.0 1.1 Abstraction layer Abstraction layer APPLICATION

Database

Router / Load balancer Interwebs

Wednesday, May 11, 2011

SLIDE 20

dark launching

Diagram by Martin Fowler

Wednesday, May 11, 2011

SLIDE 21

dark launching

Diagram by Martin Fowler

Wednesday, May 11, 2011

SLIDE 22

How long would it take you to release a change to a single line of code? Ops metrics - changes, incidents, TTD, TTR, TBF If your data center blew up, how long would you take to restore service?

measuring effectiveness

Wednesday, May 11, 2011

SLIDE 23

thank you!

Jez Humble jez@thoughtworks.com @jezhumble #continuousdelivery

Wednesday, May 11, 2011