remediation patterns Jez Humble GOTO Copenhagen 2011 #gotocph - PowerPoint PPT Presentation

remediation patterns Jez Humble GOTO Copenhagen 2011 #gotocph jez@thoughtworks.com @jezhumble #continuousdelivery Wednesday, May 11, 2011

remediation ITIL: “Recovery to a known state after a failed Change or Release.” Recovery: “Returning a Configuration Item or an IT Service to a working state.” Jez: “Fixing shit when it breaks” Wednesday, May 11, 2011

strategies for remediation prevention patterns for low-risk release patterns for incremental delivery Wednesday, May 11, 2011

1oz of prevention Diagram invented by Brian Marick Wednesday, May 11, 2011

deployment pipeline Delivery team Version control Release Build & unit Automated User acceptance tests acceptance tests tests Check in Trigger Feedback Check in Trigger Feedback Trigger Feedback Check in Trigger Feedback Trigger Feedback Approval Approval Feedback Wednesday, May 11, 2011

the hard bits testing on production environments creating maintainable acceptance tests testing cross-functional requirements Wednesday, May 11, 2011

reducing release risk automate provisioning and deployment ensure devs, testers and ops collaborate throughout Wednesday, May 11, 2011

canary releasing Diagram by Martin Fowler Wednesday, May 11, 2011

canary releasing Diagram by Martin Fowler Wednesday, May 11, 2011

canary releasing reduce risk of release multivariant testing performance testing Wednesday, May 11, 2011

immune system what if someone replaced your “buy” button with spacer.gif? T cells http://www.flickr.com/photos/gehealthcare/3326186490/ Wednesday, May 11, 2011

monitoring Business metrics - revenue, # orders, # users Ops metrics - changes, incidents, TTD, TTR, TBF Technical metrics - TPS, response time, hits http://www.flickr.com/photos/wwarby/3296379139/ Wednesday, May 11, 2011

the hard bits root cause analysis collaboration data Wednesday, May 11, 2011

incremental delivery John Allspaw: “Ops Metametrics” http://slidesha.re/dsSZIr Wednesday, May 11, 2011

incremental delivery develop on mainline feature toggles and branch by abstraction incremental deployments dark launching Wednesday, May 11, 2011

feature toggles Config File [featureToggles] some.jsp wobblyFoobars: true flightyForkHandles: false <toggle name=wobblyFoobars> ... various UI elements </toggle> other.java forkHandle = (featureConfig.isOn(‘flightlyForkHandles)) ? new FlightyForkHander(aCandle) : new ForkHandler(aCandle) Stolen from Martin Fowler http://martinfowler.com/bliki/FeatureToggle.html Wednesday, May 11, 2011

branch by abstraction Component A Component A Seam Component B Wednesday, May 11, 2011

branch by abstraction Component A Component A Abstraction layer Component B Component B’ Wednesday, May 11, 2011

incremental deployment DEPENDENT SERVICE 1.0 1.1 Database Abstraction layer Abstraction layer STATIC CONTENT /static/1.1 APPLICATION /static/1.0 Router / Load balancer Interwebs Wednesday, May 11, 2011

dark launching Diagram by Martin Fowler Wednesday, May 11, 2011

dark launching Diagram by Martin Fowler Wednesday, May 11, 2011

measuring effectiveness How long would it take you to release a change to a single line of code? Ops metrics - changes, incidents, TTD, TTR, TBF If your data center blew up, how long would you take to restore service? Wednesday, May 11, 2011

thank you! Jez Humble jez@thoughtworks.com @jezhumble #continuousdelivery Wednesday, May 11, 2011