preparing for the unexpected
play

Preparing for the Unexpected Samuel Parkinson - PowerPoint PPT Presentation

Dec 09, 2022 •137 likes •1.96k views

Preparing for the Unexpected Samuel Parkinson samuel.parkinson@ft.com #qconlondon #prepfortheunexpected Photo by Hush Naidoo on Unsplash #qconlondon #prepfortheunexpected #qconlondon #prepfortheunexpected Lets start with a story

  1. Preparing for the Unexpected Samuel Parkinson samuel.parkinson@ft.com #qconlondon #prepfortheunexpected Photo by Hush Naidoo on Unsplash #qconlondon #prepfortheunexpected

  2. #qconlondon #prepfortheunexpected

  3. Let’s start with a story #qconlondon #prepfortheunexpected

  4. What’s the worst thing that could happen? #qconlondon #prepfortheunexpected

  5. #qconlondon #prepfortheunexpected

  6. #qconlondon #prepfortheunexpected

  7. #qconlondon #prepfortheunexpected

  8. #qconlondon #prepfortheunexpected

  9. #qconlondon #prepfortheunexpected

  10. ************* #qconlondon #prepfortheunexpected

  11. #qconlondon #prepfortheunexpected

  12. The FT.com zone was missing #qconlondon #prepfortheunexpected

  13. #qconlondon #prepfortheunexpected

  14. FT.com has over 5,100 subdomains 😭 #qconlondon #prepfortheunexpected

  15. This impacted the whole company #qconlondon #prepfortheunexpected

  16. #qconlondon #prepfortheunexpected

  17. #qconlondon #prepfortheunexpected

  18. 😲 #qconlondon #prepfortheunexpected

  19. We have never prepared for such an incident #qconlondon #prepfortheunexpected

  20. It’s a classic data loss situation #qconlondon #prepfortheunexpected

  21. #qconlondon #prepfortheunexpected

  22. Our provider had a partial backup #qconlondon #prepfortheunexpected

  23. But critical records we used for DNS load balancing were missing 👼 #qconlondon #prepfortheunexpected

  24. About 10 people worked to resolve the incident #qconlondon #prepfortheunexpected

  25. And over 30 people were online to follow along #qconlondon #prepfortheunexpected

  26. Most were not called, but still volunteered their time #qconlondon #prepfortheunexpected

  27. #qconlondon #prepfortheunexpected

  28. 4h 30m The first hour was a total outage. #qconlondon #prepfortheunexpected

  29. Lack of panic in the moment #qconlondon #prepfortheunexpected

  30. It was a slick operation and we recovered #qconlondon #prepfortheunexpected

  31. It took restoring from a backup and manual entry to get there #qconlondon #prepfortheunexpected

  32. We were focused on recovery, not what happened #qconlondon #prepfortheunexpected

  33. People were joining the incident to learn #qconlondon #prepfortheunexpected

  34. #qconlondon #prepfortheunexpected

  35. #qconlondon #prepfortheunexpected

  36. This is where we are today #qconlondon #prepfortheunexpected

  37. #qconlondon #prepfortheunexpected

  38. #qconlondon #prepfortheunexpected

  39. #qconlondon #prepfortheunexpected

  40. Photo by Victor Garcia on Unsplash #qconlondon #prepfortheunexpected

  41. Photo by Markus Spiske on Unsplash #qconlondon #prepfortheunexpected

  42. 0. How do we do on-call? 1. Our incident management challenges 2. Making out-of-hours sustainable 3. The results and takeaways #qconlondon #prepfortheunexpected

  43. 0. How do we do on-call? 1. Our incident management challenges 2. Making out-of-hours sustainable 3. The results and takeaways #qconlondon #prepfortheunexpected

  44. Internal FT Core Products Enterprise Services Customer Products Operations & Reliability FT Group Products #qconlondon #prepfortheunexpected

  45. Customer We are Products #qconlondon #prepfortheunexpected

  46. 45 engineers and counting 📉 #qconlondon #prepfortheunexpected

  47. And we own about 180 systems #qconlondon #prepfortheunexpected

  48. #qconlondon #prepfortheunexpected

  49. Split into 9 teams #qconlondon #prepfortheunexpected

  50. #qconlondon #prepfortheunexpected

  51. #qconlondon #prepfortheunexpected

  52. Operations monitor our entire estate 24/7 #qconlondon #prepfortheunexpected

  53. #qconlondon #prepfortheunexpected

  54. Our systems are a drop in the pond #qconlondon #prepfortheunexpected

  55. You build it, you run it #qconlondon #prepfortheunexpected

  56. Supporting our systems out-of-hours #qconlondon #prepfortheunexpected

  57. This is our approach to DevOps #qconlondon #prepfortheunexpected

  58. Our engineers wear many hats Photo by Joshua Coleman on Unsplash #qconlondon #prepfortheunexpected

  59. We’re putting on our incident management hat #qconlondon #prepfortheunexpected

  60. How do we do support out-of-hours? #qconlondon #prepfortheunexpected

  61. Our engineers volunteer to be part of the out-of-hours team #qconlondon #prepfortheunexpected

  62. We don’t have shifts #qconlondon #prepfortheunexpected

  63. We don’t have shifts #qconlondon #prepfortheunexpected

  64. Which means, we could all be unavailable #qconlondon #prepfortheunexpected

  65. What do we care about? #qconlondon #prepfortheunexpected

  66. We’re talking about our business capabilities #qconlondon #prepfortheunexpected

  67. We’re talking about our business capabilities #qconlondon #prepfortheunexpected

  68. What is an incident at the FT? #qconlondon #prepfortheunexpected

  69. Customer Products has two really important business capabilities #qconlondon #prepfortheunexpected

  70. 1. Our users can always read the news #qconlondon #prepfortheunexpected

  71. 2. Journalists must be able to publish the news #qconlondon #prepfortheunexpected

  72. If either of these go wrong we declare an incident #qconlondon #prepfortheunexpected

  73. #qconlondon #prepfortheunexpected

  74. #qconlondon #prepfortheunexpected

  75. 0. How do we do on-call? 1. Our incident management challenges 2. Making out-of-hours sustainable 3. The results and takeaways #qconlondon #prepfortheunexpected

  76. What were our challenges? #qconlondon #prepfortheunexpected

  77. We were not immediately productive on call → #qconlondon #prepfortheunexpected

  78. We were not immediately productive on call We had an engineering mindset in an operations situation #qconlondon #prepfortheunexpected

  79. We were not immediately productive on call Because we don’t have any SRE or DevOps specialists #qconlondon #prepfortheunexpected

  80. “ ” I always start with the impact and the comms, they kinda jump in at the Tech. #qconlondon #prepfortheunexpected

  81. We were not immediately productive on call Our incident management process wasn’t second nature #qconlondon #prepfortheunexpected

  82. We had very few incidents in the first half of the year #qconlondon #prepfortheunexpected

  83. We had very few incidents in the first half of the year #qconlondon #prepfortheunexpected

  84. And we were down to 5 people on the out-of-hours support team #qconlondon #prepfortheunexpected

  85. So we needed to make out-of-hours team sustainable #qconlondon #prepfortheunexpected

  86. 0. How do we do on-call? 1. Our incident management challenges 2. Making out-of-hours sustainable 3. The results and takeaways #qconlondon #prepfortheunexpected

  87. We surveyed engineers about helping out during an incident #qconlondon #prepfortheunexpected

  88. There were many people on the fence #qconlondon #prepfortheunexpected

  89. There were many people 7 people on the fence 3 people 6 people #qconlondon #prepfortheunexpected

  90. And they told us why #qconlondon #prepfortheunexpected

  91. “ ” I will need much more confidence in systems and domains knowledge. #qconlondon #prepfortheunexpected

  92. “ ” If I were to have a better understanding of how it works and what I would need to do, I would very likely join. #qconlondon #prepfortheunexpected

  93. We set out to convince people to join our out-of-hours team #qconlondon #prepfortheunexpected

  94. We built and ran incident workshops #qconlondon #prepfortheunexpected

  95. So our engineers are better prepared to take on incidents #qconlondon #prepfortheunexpected

  96. And we wrote a generic runbook for our microservices #qconlondon #prepfortheunexpected

  97. So engineers knew what they can do, and apply it to our ~180 systems #qconlondon #prepfortheunexpected

  98. We set out in the last 6 months of 2019 to address the situation #qconlondon #prepfortheunexpected

  99. Building your incident workshop → #qconlondon #prepfortheunexpected

  100. Building your incident workshop Don’t Panic! #qconlondon #prepfortheunexpected

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Dialogue Managing the Unexpected Managing the Unexpected the Role of the Regulatory Body

Dialogue Managing the Unexpected Managing the Unexpected the Role of the Regulatory Body

Swiss Federal Nuclear Safety Inspectorate ENSI Dialogue Managing the Unexpected Managing the Unexpected the Role of the Regulatory Body Cornelia Ryser, ENSI, Switzerland IAEA Technical Meeting Vienna, 25 29 June 2012 Assumptions

543 views • 4 slides
Pitfalls in PID: Expect the unexpected Faculty Disclosure X No, nothing to disclose Yes, please

Pitfalls in PID: Expect the unexpected Faculty Disclosure X No, nothing to disclose Yes, please

Pitfalls in PID: Expect the unexpected Faculty Disclosure X No, nothing to disclose Yes, please specify: PITFALLS IN PID: EXPECT THE UNEXPECTED First daughter of non-consanguineous parents, no relevant family history. 2 months old

258 views • 10 slides
Managing the Unexpected: Interactions between, Human Excellence and Organization Safety Culture

Managing the Unexpected: Interactions between, Human Excellence and Organization Safety Culture

Managing the Unexpected: Interactions between, Human Excellence and Organization Safety Culture Dr. Abdul Nasir Afghan (IBA) Mr. Shahbaz Ali Nasir Bhatti (PNRA) IAEA TM on Managing the Unexpected From the Perspective of the interaction

250 views • 21 slides
Testing for the Unexpected: An Automated T ti f th U t d A A t t d Method of Injecting

Testing for the Unexpected: An Automated T ti f th U t d A A t t d Method of Injecting

Testing for the Unexpected: An Automated T ti f th U t d A A t t d Method of Injecting Faults for Engine Management Development M t D l t Scott James Applied Dynamics International Sh Shaun Fuller Pickering Interfaces F ll

353 views • 18 slides
Scaling for the Expected and Unexpected Brian Moon Senior Web Engineer dealnews.com

Scaling for the Expected and Unexpected Brian Moon Senior Web Engineer dealnews.com

Scaling for the Expected and Unexpected Brian Moon Senior Web Engineer dealnews.com http://brian.moonspot.net/ twitter.com/brianlmoon Tuesday, June 23, 2009 Tuesday, June 23, 2009 Unexpected Likely one page.... at first Likely coming

420 views • 13 slides
Computing the unexpected and Unpredicted and Deceptive and Interesting Reflections

Computing the unexpected and Unpredicted and Deceptive and Interesting Reflections

Computing the unexpected and Unpredicted and Deceptive and Interesting Reflections Fifty year prediction about home computers Computing the unexpected and Unpredicted and Deceptive and Interesting Reflections

501 views • 10 slides
Preparing for Cascadia 9.0 Preparing for Cascadia 9.0 Individual, Household, and Community

Preparing for Cascadia 9.0 Preparing for Cascadia 9.0 Individual, Household, and Community

Preparing for Cascadia 9.0 Preparing for Cascadia 9.0 Individual, Household, and Community Preparedness Bill Hasan Hillsdale Neighborhood Emergency Team members hillsdaleornet@gmail.com March 10, 2019 | Portland Bureau of Emergency

352 views • 31 slides
How to Make a Formal Presentation Contents Preparing Content ( Written ) Theory

How to Make a Formal Presentation Contents Preparing Content ( Written ) Theory

How to Make a Formal Presentation Contents Preparing Content ( Written ) Theory Practice Preparing Content ( Visual ) Preparing Yourself Body Language Preparing for your Presentation Video Preparing Written

233 views • 8 slides
Flash Reliability in Produc4on: The Expected and the Unexpected

Flash Reliability in Produc4on: The Expected and the Unexpected

Flash Reliability in Produc4on: The Expected and the Unexpected Bianca Schroeder Raghav Lagisetty, University of Toronto Arif Merchant Google Inc. Why flash? More and more

210 views • 20 slides
Preparing IRB Submissions for Human Subjects Research Tips for Preparing IRB Protocols IRB

Preparing IRB Submissions for Human Subjects Research Tips for Preparing IRB Protocols IRB

Preparing IRB Submissions for Human Subjects Research Tips for Preparing IRB Protocols IRB Educational Session-Psychological Sciences September 15, 2017 Post Approval Monitor-Joan Levine, MPH, CIP Research Compliance Services Objectives You

386 views • 25 slides
Preparing Preparing for a for a Succe Successful ssful HAZOP/LO HAZOP/LOPA PA (Making or

Preparing Preparing for a for a Succe Successful ssful HAZOP/LO HAZOP/LOPA PA (Making or

Preparing Preparing for a for a Succe Successful ssful HAZOP/LO HAZOP/LOPA PA (Making or Breaking Quality & Efficiency) (GCPS-2018 Paper 513595) Steven T. Maher, PE CSP / Carine M. Black / John M. Johnson Risk Management

397 views • 29 slides
Creating Chaos: Engineering for the Unexpected Presented by:

Creating Chaos: Engineering for the Unexpected Presented by:

DW6 Microservices & Cloud Wednesday, November 7th, 2018 1:30 PM Creating Chaos: Engineering for the Unexpected Presented by:

509 views • 14 slides
Preparing For The Rising Tide Designing with Water Preparing For The Rising Tide Published

Preparing For The Rising Tide Designing with Water Preparing For The Rising Tide Published

Preparing For The Rising Tide Designing with Water Preparing For The Rising Tide Published February 2013 Maps Bostons vulnerability to coastal flooding Encourages property owners in today and tomorrows flood zone to know

283 views • 26 slides
Investigation of Sudden Investigation of Sudden and Unexpected Infant and Unexpected Infant

Investigation of Sudden Investigation of Sudden and Unexpected Infant and Unexpected Infant

Investigation of Sudden Investigation of Sudden and Unexpected Infant and Unexpected Infant Deaths Deaths Melanie Newbould Melanie Newbould Paediatric and Perinatal Paediatric and Perinatal Pathologist Pathologist RMCH RMCH Every year

963 views • 62 slides
Preparing for New School Year This session u What is the Preparing for the new school year

Preparing for New School Year This session u What is the Preparing for the new school year

5/12/16 Preparing for New School Year This session u What is the Preparing for the new school year process u Key term review u How to use the Prepare for Next School year administrative tool u Options for moving students 1 5/12/16 What

406 views • 11 slides
10 tips for preparing a presentation

10 tips for preparing a presentation

10 tips for preparing a presentation 1. Purpose What is the reason for the presentation you are preparing ? This sounds obvious but stay focused on this as you build the

298 views • 3 slides
Multiclass Boosting with Repartitioning Ling Li Learning Systems Group, Caltech ICML 2006

Multiclass Boosting with Repartitioning Ling Li Learning Systems Group, Caltech ICML 2006

Introduction Multiclass Boosting Repartitioning Experiments Summary Multiclass Boosting with Repartitioning Ling Li Learning Systems Group, Caltech ICML 2006 Introduction Multiclass Boosting Repartitioning Experiments Summary Binary

545 views • 23 slides
Adversarial Music: Real world audio adversary against wake-word detection systems Juncheng B. Li

Adversarial Music: Real world audio adversary against wake-word detection systems Juncheng B. Li

NeurIPS 2019, Vancouver, Canada Adversarial Music: Real world audio adversary against wake-word detection systems Juncheng B. Li , Shuhui Qu , Xinjian Li , Joseph Szurley , Zico Kolter , , Florian Metze Carnegie

356 views • 12 slides
Opening the pod bay doors building intelligent agents that can interpret, generate and learn

Opening the pod bay doors building intelligent agents that can interpret, generate and learn

Opening the pod bay doors building intelligent agents that can interpret, generate and learn from natural language Jacob Andreas, MIT / Microsoft web.mit.edu/jda/www / @jacobandreas Following natural language instructions

1.61k views • 122 slides
Kalman Filter Kalman Filter = special case of a Bayes filter with dynamics model and n

Kalman Filter Kalman Filter = special case of a Bayes filter with dynamics model and n

EKF, UKF Pieter Abbeel UC Berkeley EECS Many slides adapted from Thrun, Burgard and Fox, Probabilistic Robotics Kalman Filter Kalman Filter = special case of a Bayes filter with dynamics model and n sensory model being linear Gaussian: 2

535 views • 17 slides
Product Development Dilemma Product Development Dilemma

Product Development Dilemma Product Development Dilemma

A Revolutionary Solution for A Revolutionary Solution for Unified RF System and Circuit Unified RF System and Circuit Design Design EDP-2002 Conference Monterey, California April 21-23 James Spoto President and CEO AWR Product

172 views • 6 slides
Unique equilibrium states for geodesic flows in nonpositive curvature Todd Fisher Department of

Unique equilibrium states for geodesic flows in nonpositive curvature Todd Fisher Department of

Unique equilibrium states for geodesic flows in nonpositive curvature Todd Fisher Department of Mathematics Brigham Young University Fractal Geometry, Hyperbolic Dynamics and Thermodynamical Formalism Joint work with K. Burns, V. Climenhaga,

597 views • 40 slides
Probabilistic reasoning with graphical security models Barbara Kordy Clermont-Ferrand, January

Probabilistic reasoning with graphical security models Barbara Kordy Clermont-Ferrand, January

Probabilistic reasoning with graphical security models Barbara Kordy Clermont-Ferrand, January 7, 2016 Digital Confidence seminar Joint work Prof. Dr. Marc Pouly Lucerne University of Applied Sciences and Arts Dr. Patrick Schweitzer

903 views • 54 slides
Consistent Multitask Learning with Nonlinear Output Constraints Carlo Ciliberto Department of

Consistent Multitask Learning with Nonlinear Output Constraints Carlo Ciliberto Department of

Consistent Multitask Learning with Nonlinear Output Constraints Carlo Ciliberto Department of Computer Science, UCL joint work w/ Alessandro Rudi, Lorenzo Rosasco and Massi Pontil Multitask Learning (MTL) MTL Mantra : leverage the similarities

329 views • 22 slides

More recommend