remediation patterns Jez Humble QCon London 2011 - - PowerPoint PPT Presentation

▶

Oct 11, 2022 148 likes •390 views

remediation patterns Jez Humble QCon London 2011 jez@thoughtworks.com @jezhumble #continuousdelivery Thursday, March 10, 2011 1 remediation ITIL: Recovery to a known state after a failed Change or Release. Recovery: Returning a

SLIDE 1

remediation patterns

Jez Humble QCon London 2011 jez@thoughtworks.com @jezhumble #continuousdelivery

1 Thursday, March 10, 2011

SLIDE 2

ITIL: “Recovery to a known state after a failed Change or Release.” Recovery: “Returning a Configuration Item or an IT Service to a working state.” Jez: “Fixing shit when it breaks”

remediation

2 Thursday, March 10, 2011

SLIDE 3

prevention patterns for low-risk release patterns for incremental delivery

strategies for remediation

3 Thursday, March 10, 2011

SLIDE 4

1oz of prevention

4 Thursday, March 10, 2011

SLIDE 5

deployment pipeline

5 Thursday, March 10, 2011

SLIDE 6

testing on production environments creating maintainable acceptance tests testing cross-functional requirements

the hard bits

6 Thursday, March 10, 2011

SLIDE 7

automate provisioning and deployment ensure devs, testers and ops collaborate throughout

reducing release risk

7 Thursday, March 10, 2011

SLIDE 8

canary releasing

8 Thursday, March 10, 2011

SLIDE 9

canary releasing

9 Thursday, March 10, 2011

SLIDE 10

reduce risk of release multivariant testing performance testing

canary releasing

10 Thursday, March 10, 2011

SLIDE 11

immune system

what if someone replaced your “buy” button with spacer.gif?

T cells http://www.flickr.com/photos/gehealthcare/3326186490/

11 Thursday, March 10, 2011

SLIDE 12

Business metrics - revenue, # orders, # users Ops metrics - changes, incidents, TTD, TTR, TBF Technical metrics - TPS, response time, hits

monitoring

http://www.flickr.com/photos/wwarby/3296379139/

12 Thursday, March 10, 2011

SLIDE 13

root cause analysis collboration data

the hard bits

13 Thursday, March 10, 2011

SLIDE 14

incremental delivery

John Allspaw: “Ops Metametrics” http://slidesha.re/dsSZIr

14 Thursday, March 10, 2011

SLIDE 15

incremental deployments develop on mainline feature toggles and branch by abstraction dark launching

incremental delivery

15 Thursday, March 10, 2011

SLIDE 16

feature toggles

[featureToggles] wobblyFoobars: true flightyForkHandles: false

Config File

<toggle name=wobblyFoobars> ... various UI elements </toggle>

some.jsp

forkHandle = (featureConfig.isOn(‘flightlyForkHandles)) ? new FlightyForkHander(aCandle) : new ForkHandler(aCandle)

ther.java

Stolen from Martin Fowler <link to his bliki entry>

16 Thursday, March 10, 2011

SLIDE 17

branch by abstraction

Component A Component B

Seam

Component A

17 Thursday, March 10, 2011

SLIDE 18

branch by abstraction

Component A Component A Component B’ Abstraction layer Component B

18 Thursday, March 10, 2011

SLIDE 19

incremental deployment

STATIC CONTENT /static/1.1 /static/1.0 DEPENDENT SERVICE 1.0 1.1 Abstraction layer Abstraction layer APPLICATION

Database

Router / Load balancer Interwebs

19 Thursday, March 10, 2011

SLIDE 20

dark launching

20 Thursday, March 10, 2011

SLIDE 21

dark launching

21 Thursday, March 10, 2011

SLIDE 22

How long would it take you to release a change to a single line of code? Ops metrics - changes, incidents, TTD, TTR, TBF If your data center blew up, how long would you take to restore service?

measuring effectiveness

22 Thursday, March 10, 2011

SLIDE 23

questions

Jez Humble jez@thoughtworks.com @jezhumble #continuousdelivery

23 Thursday, March 10, 2011