pitfalls in measuring slos
play

Pitfalls in Measuring SLOs Danyel Fisher @fisherdanyel An Outage - PowerPoint PPT Presentation

Sep 21, 2022 •356 likes •1.18k views

Pitfalls in Measuring SLOs Danyel Fisher @fisherdanyel An Outage Danyel Fisher @fisherdanyel Danyel Fisher @fisherdanyel Danyel Fisher @fisherdanyel What do you do when things break? How bad was this break? Danyel Fisher @fisherdanyel

  1. Pitfalls in Measuring SLOs Danyel Fisher @fisherdanyel

  2. An Outage Danyel Fisher @fisherdanyel

  3. Danyel Fisher @fisherdanyel

  4. Danyel Fisher @fisherdanyel

  5. What do you do when things break? How bad was this break? Danyel Fisher @fisherdanyel

  6. Danyel Fisher @fisherdanyel

  7. We need to improve Build new features! quality! Danyel Fisher @fisherdanyel

  8. Management How broken is “too broken”? Engineering What does “good enough” mean? Clients and Users Combatting alert fatigue Danyel Fisher @fisherdanyel

  9. A telemetry system produces events that correspond to real world use We can describe some of these events as eligible We can describe some of them as good Danyel Fisher @fisherdanyel

  10. Given an event , is it eligible? Is it good? Eligible: “Had an http status code” Good: “... that was a 200, and was served under 500 ms ” Danyel Fisher @fisherdanyel

  11. Danyel Fisher @fisherdanyel

  12. Danyel Fisher @fisherdanyel

  13. Minimum Quality ratio over a Number of bad events allowed. period of time Danyel Fisher @fisherdanyel

  14. Deploy faster Room for experimentation Opportunity to tighten SLO Danyel Fisher @fisherdanyel

  15. We always store incoming user 99.99% ~4.3 minutes data Default dashboards usually load 99.9% 45 minutes in < 1s 99% 7.3 hours Queries often return in < 10 s Danyel Fisher @fisherdanyel

  16. User Data Throughput We blew through three months’ budget in those 12 minutes. Danyel Fisher @fisherdanyel

  17. We dropped customer data Danyel Fisher @fisherdanyel

  18. We dropped customer data We rolled it back (manually) We communicated to customers We halted deploys Danyel Fisher @fisherdanyel

  19. We checked in code that didn’t build . We had experimental CI build wiring. Our scripts deployed empty binaries . There was no health check and rollback. Danyel Fisher @fisherdanyel

  20. We stopped writing new features We prioritized stability We mitigated risks Danyel Fisher @fisherdanyel

  21. SLOs allowed us to characterize what went wrong, how badly it went wrong, and how to prioritize repair Danyel Fisher @fisherdanyel

  22. Learning from SLOs Danyel Fisher @fisherdanyel

  23. Final point A one-line description of it Danyel Fisher @fisherdanyel

  24. ● Design Thinking Expressing and Viewing ● Burndown Alerts and Responding ● ● Learning from our Experiences ● Success Stories Danyel Fisher @fisherdanyel

  25. Design Thinking and Task Analysis Understand user goals and needs Learn from informants and experts Collaborate with internal team Collect feedback and ideas externally Danyel Fisher @fisherdanyel

  26. Displays and Views Danyel Fisher @fisherdanyel

  27. See where the burndown was happening, explain why, and remediate Danyel Fisher @fisherdanyel

  28. Expressing SLOs Event based Time based “How many events had a duration < “How many 5 minute periods, had a 500 ms” P95(duration) < 500 ms” Danyel Fisher @fisherdanyel

  29. How do we express SLOs? Good events Bad events How often Time range Danyel Fisher @fisherdanyel

  30. How do we express SLOs? Good events Bad events How often Time range Danyel Fisher @fisherdanyel

  31. How do we express SLOs? Eligible: $name is “run_trigger_detailed” Good: $app.error does not exist Good events Bad events How often Time range Danyel Fisher @fisherdanyel

  32. How do we express SLOs? Good events Bad events How often Time range Danyel Fisher @fisherdanyel

  33. How do we express SLOs? Good events Bad events How often Time range Danyel Fisher @fisherdanyel

  34. Status of an SLO Danyel Fisher @fisherdanyel

  35. How have we done? Danyel Fisher @fisherdanyel

  36. Danyel Fisher @fisherdanyel

  37. Where did it go? Danyel Fisher @fisherdanyel

  38. When did the errors happen? Danyel Fisher @fisherdanyel

  39. When did the errors happen? Danyel Fisher @fisherdanyel

  40. What went wrong? High dimensional data High cardinality data Danyel Fisher @fisherdanyel

  41. Why did it happen? Danyel Fisher @fisherdanyel

  42. Why did it happen? Danyel Fisher @fisherdanyel

  43. Why did it happen? Danyel Fisher @fisherdanyel

  44. See where the burndown was happening, explain why, and remediate Danyel Fisher @fisherdanyel

  45. User Feedback “The Bubble Up in the SLO page is really powerful at highlighting what is contributing the most to missing our SLIs, it has definitely confirmed our assumptions.” Danyel Fisher @fisherdanyel

  46. User Feedback “Your customers have to be happy... we have to have an understanding of the customer experience. … To the millisecond we knew what our percentage was of success versus failure .” -Josh Hull, Site Reliability Engineering Lead, Clover Health Danyel Fisher @fisherdanyel

  47. User Feedback “The historical SLO chart also confirms a fix for a performance issue we did greatly contributed to the SLO compliance by showing a nice upward trend line. :)” Danyel Fisher @fisherdanyel

  48. User Feedback “I’d love to drive alerts off our SLOs. Right now we don’t have anything to draw us in and have some alerts on the average error rate but they’re a little spiky to be useful. It would be great to get a better sense of when the budget is going and define alerts that way.” Danyel Fisher @fisherdanyel

  49. Burndown Alerts Danyel Fisher @fisherdanyel

  50. How is my system doing? Am I over budget? When will my alarm fail? Danyel Fisher @fisherdanyel

  51. When will I fail? User goal: get alerts to exhaustion time Human-digestible units 24 hours: “I’ll take a look in the morning” 4 hours: “All hands on deck!” Danyel Fisher @fisherdanyel

  52. Danyel Fisher @fisherdanyel

  53. How is my system doing? Am I over budget? When will my alarm fail? Danyel Fisher @fisherdanyel

  54. Implementing Burn Alerts Run a 30 day query at a 5 minute resolution every minute Danyel Fisher @fisherdanyel

  55. Danyel Fisher @fisherdanyel

  56. Caching is Fun! Danyel Fisher @fisherdanyel

  57. Fun with Caching Vital to cache results … but not incomplete results … … at what resolution of cache? Danyel Fisher @fisherdanyel

  58. Flappy Alerts “It’ll expire at 3:55” (We added a 10%ish buffer) “Wait, make that 4:05” “Nope, 3:55 again!” Danyel Fisher @fisherdanyel

  59. Recovering from Bankruptcy A failure a month ago brought us to -169% and still hasn’t aged out? That means we don’t get alerts anymore Customer workaround: delete and re-create the SLO, thus blowing the cache Danyel Fisher @fisherdanyel

  60. Learning from Experience Danyel Fisher @fisherdanyel

  61. Volume is important Tolerate at least dozens of bad events per day Danyel Fisher @fisherdanyel

  62. Faults

  63. SLOs for Customer Service Remember that user having a bad day? ADD IMAGE Danyel Fisher @fisherdanyel

  64. Blackouts are easy … but brownouts are much more interesting Danyel Fisher @fisherdanyel

  65. Danyel Fisher @fisherdanyel

  66. Timeline 1:29 am SLO alerts. “Maybe it’s just a blip” 1.5% brownout for 20 minutes Danyel Fisher @fisherdanyel

  67. Timeline 1:29 am SLO alerts. “Maybe it’s just a blip” 4:21 am Minor incident. “It might be an AWS problem” 6:25 am SLO alerts again. “Could it be ALB compat?” 9:55 am “Why is our system uptime dropping to zero?” It’s out of memory We aren’t alerting on that crash Danyel Fisher @fisherdanyel

  68. Timeline 1:29 am SLO alerts. “Maybe it’s just a blip” 1.5% brownout for 20 minutes 4:21 am Minor incident. “It might be an AWS problem” Danyel Fisher @fisherdanyel

  69. Timeline 1:29 am SLO alerts. “Maybe it’s just a blip” 1.5% brownout for 20 minutes 4:21 am Minor incident. “It might be an AWS problem” 6:25 am SLO alerts again. “Could it be ALB compat?” Danyel Fisher @fisherdanyel

  70. Timeline 1:29 am SLO alerts. “Maybe it’s just a blip” 1.5% brownout for 20 minutes 4:21 am Minor incident. “It might be an AWS problem” 6:25 am SLO alerts again. “Could it be ALB compat?” 9:55 am “Why is our system uptime dropping to zero?” It’s out of memory We aren’t alerting on that crash Danyel Fisher @fisherdanyel

  71. Timeline 1:29 am SLO alerts. “Maybe it’s just a blip” 4:21 am Minor incident. “It might be an AWS problem” 6:25 am SLO alerts again. “Could it be ALB compat?” 9:55 am “Why is our system uptime dropping to zero?” It’s out of memory We aren’t alerting on that crash Danyel Fisher @fisherdanyel

  72. Timeline 1:29 am SLO alerts. “Maybe it’s just a blip” 1.5% brownout for 20 minutes 4:21 am Minor incident. “It might be an AWS problem” 6:25 am SLO alerts again. “Could it be ALB compat?” 9:55 am “Why is our system uptime dropping to zero?” It’s out of memory We aren’t alerting on that crash 10:32 am Fixed Danyel Fisher @fisherdanyel

  73. Danyel Fisher @fisherdanyel

  74. Danyel Fisher @fisherdanyel

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Student Learning Outcomes Course Objectives Course SLOs Program SLOs

Student Learning Outcomes Course Objectives Course SLOs Program SLOs

Student Learning Outcomes Course Objectives Course SLOs Program SLOs Introductions Long Beach City College Curriculum/ SLO Team: Kenna Hillman; Chair, Committee on Curriculum &amp; Instruction Haley Nguyen; Coordinator of

300 views • 27 slides
The Art of SLOs In the midst of chaos , there is also opportunity reliability Sun Tzu, The Art

The Art of SLOs In the midst of chaos , there is also opportunity reliability Sun Tzu, The Art

The Art of SLOs In the midst of chaos , there is also opportunity reliability Sun Tzu, The Art of War https://cre.page.link/art-of-slos-slides Welcome! Don't be shy say hello to your neighbours https://cre.page.link/art-of-slos-slides

1.19k views • 98 slides
Measuring of homelessness in Belgium: pitfalls and challenges Prof. dr. Koen Hermans Belgian

Measuring of homelessness in Belgium: pitfalls and challenges Prof. dr. Koen Hermans Belgian

Measuring of homelessness in Belgium: pitfalls and challenges Prof. dr. Koen Hermans Belgian homelessness policies Federal level : Social assistance Health care (including mental health care) Regional level : Housing

76 views • 7 slides
Exploring Student Learning Objectives (SLOs) &amp; Student Outcome Objectives (SOOs) American

Exploring Student Learning Objectives (SLOs) &amp; Student Outcome Objectives (SOOs) American

Exploring Student Learning Objectives (SLOs) &amp; Student Outcome Objectives (SOOs) American Institutes for Research in Partnership With the Oklahoma State Department of Education Session Overview This session will provide an overview of SLOs/SOOs

641 views • 36 slides
What are the Pitfalls for Bangladesh? Presented by Towfiqul Islam Khan Research Fellow, CPD 10

What are the Pitfalls for Bangladesh? Presented by Towfiqul Islam Khan Research Fellow, CPD 10

Public Dialogue on Bangladeshs Graduation from the LDC Group Pitfalls and Promises Session Three: Graduating in a Brave New World Pursuing a Graduation Strategy within the Global and Regional Environment What are the Pitfalls for

650 views • 24 slides
Pitfalls in Using a case based approach, we will Arrhythmias review pitfalls in management of:

Pitfalls in Using a case based approach, we will Arrhythmias review pitfalls in management of:

Goals Pitfalls in Using a case based approach, we will Arrhythmias review pitfalls in management of: Tachydysrhythmias Jeffrey Tabas, M.D. Narrow Wide Professor of Emergency Medicine Bradydysrhythmias Director of Outcomes and

438 views • 19 slides
SLOs: Assessment &amp; Alignment of Outcomes San Bernardino Valley College January 10, 2013

SLOs: Assessment &amp; Alignment of Outcomes San Bernardino Valley College January 10, 2013

SLOs: Assessment &amp; Alignment of Outcomes San Bernardino Valley College January 10, 2013 David W. Marshall, PhD California State University-San Bernardino Overview Two Cultures: A Contrast in Emphasis Defining Student Learning

750 views • 45 slides
Cake: : Enabling Hig igh-level SLOs on Shared Sto torage Syste tems Andrew Wang, Shivaram

Cake: : Enabling Hig igh-level SLOs on Shared Sto torage Syste tems Andrew Wang, Shivaram

Cake: : Enabling Hig igh-level SLOs on Shared Sto torage Syste tems Andrew Wang, Shivaram Venkataraman, Sara Alspaugh, Randy Katz, Ion Stoica University of California, Berkeley SO SOCC CC 20 2012 12 Content Introduction Problem

841 views • 30 slides
SLO Assessment: Training the Trainers Marcy Alancraig Learning Outcomes Assessment Coordinator

SLO Assessment: Training the Trainers Marcy Alancraig Learning Outcomes Assessment Coordinator

SLO Assessment: Training the Trainers Marcy Alancraig Learning Outcomes Assessment Coordinator Cabrillo College Writing SLOs SLOs: The Big Picture Requires HIGHER LEVEL thinking skills Synthesizes many discreet skills Requires

401 views • 24 slides
SNC-Meister: Admitting More Tenants With Tail Latency SLOs Timothy Zhu Daniel S. Berger Mor

SNC-Meister: Admitting More Tenants With Tail Latency SLOs Timothy Zhu Daniel S. Berger Mor

SNC-Meister: Admitting More Tenants With Tail Latency SLOs Timothy Zhu Daniel S. Berger Mor Harchol-Balter Carnegie Mellon University University of Kaiserslautern Carnegie Mellon University Presented By: Zane Ma &amp; Shuo Feng

566 views • 32 slides
ITU on Measuring Speech Quality Measuring Perceived Quality Typically done by using standards

ITU on Measuring Speech Quality Measuring Perceived Quality Typically done by using standards

Outline Measuring Perceived Quality of Introduction Speech and Video in Multimedia Measuring Perceived Quality Conferencing Applications What is Multimedia Quality? UCL Approach to Measuring Quality Anna Watson and M. Angela

328 views • 4 slides
Pitfalls in PID: Expect the unexpected Faculty Disclosure X No, nothing to disclose Yes, please

Pitfalls in PID: Expect the unexpected Faculty Disclosure X No, nothing to disclose Yes, please

Pitfalls in PID: Expect the unexpected Faculty Disclosure X No, nothing to disclose Yes, please specify: PITFALLS IN PID: EXPECT THE UNEXPECTED First daughter of non-consanguineous parents, no relevant family history. 2 months old

258 views • 10 slides
Describing is good: measuring is better. A new means of measuring the effectiveness of networks

Describing is good: measuring is better. A new means of measuring the effectiveness of networks

Describing is good: measuring is better. A new means of measuring the effectiveness of networks and simultaneously strengthening their function 9 th September 2015 Claire Grealy (Urbis) Gail Winkworth &amp; Michael White Known dimensions of

422 views • 16 slides
Long-term SLOs for reclaimed cloud computing resources Carvalho et al. (2014) Christopher

Long-term SLOs for reclaimed cloud computing resources Carvalho et al. (2014) Christopher

Long-term SLOs for reclaimed cloud computing resources Carvalho et al. (2014) Christopher Little Motivation Available Capacity CPU Cores Reservation Allocation g Usage Memory Motivation Available Capacity Reservation Allocation CPU

785 views • 15 slides
How to prevent cryptographic pitfalls by design February 3, 2019 How to prevent cryptographic

How to prevent cryptographic pitfalls by design February 3, 2019 How to prevent cryptographic

Department of Informatics Security and Privacy Maximilian Blochberger How to prevent cryptographic pitfalls by design February 3, 2019 How to prevent cryptographic pitfalls by design Goal Raise awareness of cryptographic misuse Disclaimer

910 views • 29 slides
Pitfalls in Arrhythmias Jeffrey Tabas, M.D. Professor of Emergency Medicine Office of CME UCSF

Pitfalls in Arrhythmias Jeffrey Tabas, M.D. Professor of Emergency Medicine Office of CME UCSF

Pitfalls in Arrhythmias Jeffrey Tabas, M.D. Professor of Emergency Medicine Office of CME UCSF School of Medicine Goals Using a case based approach, we will review pitfalls in management of: Tachydysrhythmias Narrow Wide

340 views • 12 slides
Robotic Navigation Unit Team 42 Robotic Navigation Unit Dr. Crassidis Faculty Mentor

Robotic Navigation Unit Team 42 Robotic Navigation Unit Dr. Crassidis Faculty Mentor

Robotic Navigation Unit Team 42 Robotic Navigation Unit Dr. Crassidis Faculty Mentor Antonija Soldo Team Leader/EE Aneta Rozwadowska - EE Andrew Fishbaugh - ME Ian Kunsch - ME Michael Oswald - ME Stephen

392 views • 16 slides
Springing into Summer How to support student learning during the summer May 16th, 2017, West

Springing into Summer How to support student learning during the summer May 16th, 2017, West

Springing into Summer How to support student learning during the summer May 16th, 2017, West School Presented by: Darren Bruce- Writing Specialist Gail Felberbaum- Writing Specialist Joanne Shulman- Reading Specialist Erin Gildea- Reading

369 views • 11 slides
2017: A Survival Guide Learning from Brexit and Trump to help you succeed in a changed world A

2017: A Survival Guide Learning from Brexit and Trump to help you succeed in a changed world A

Thursday 19 th January 2017 2017: A Survival Guide Learning from Brexit and Trump to help you succeed in a changed world A BritainThinks Breakfast Briefing britainthinks.com BritainThinks | Breakfast Briefing presentation (19 th January 2017)

625 views • 34 slides
A ____________ F_____ F__________ F_______ AFFF works by creating a _______ that stays

A ____________ F_____ F__________ F_______ AFFF works by creating a _______ that stays

Firefighting Foam Updates Study Guide ...... Name_______________ Date ________ 3M Foam contains a Persistent Bioaccumulative Toxic Pollutant (PBT) called __ __ __ __ and was removed from service for employee safety, health and potential damage

316 views • 6 slides
Policy Direction Workshop October 9, 2017 AGENDA Foundation Materials Regulatory &amp;

Policy Direction Workshop October 9, 2017 AGENDA Foundation Materials Regulatory &amp;

Policy Direction Workshop October 9, 2017 AGENDA Foundation Materials Regulatory &amp; Legislative Update Customer Survey Power Supply Options Customer Considerations (if time permits) Foundation Materials How The

816 views • 45 slides
Available in steel, 409 S.S., and aluminum body construction. Available in corrugated sides or

Available in steel, 409 S.S., and aluminum body construction. Available in corrugated sides or

Available in steel, 409 S.S., and aluminum body construction. Available in corrugated sides or sheet on post (smooth inside) as shown. Available in Roll-off configuration. Bolt on arms are now standard on all WaBo bucket units. Featuring 3

137 views • 12 slides
Who Are We? BENCOR In Florida, Special Pay plans in 63 Public School Systems Nearly 1

Who Are We? BENCOR In Florida, Special Pay plans in 63 Public School Systems Nearly 1

Who Are We? BENCOR In Florida, Special Pay plans in 63 Public School Systems Nearly 1 Billion in AUM* BENCOR Retirement Consultants Local Independent Advisors with a combined 50 years of experience working with Public School

451 views • 4 slides
Financial Update Faculty Associates, Inc. Board of Directors Meeting June 14, 2018 HIGH:

Financial Update Faculty Associates, Inc. Board of Directors Meeting June 14, 2018 HIGH:

The University of Florida College of Dentistry Financial Update Faculty Associates, Inc. Board of Directors Meeting June 14, 2018 HIGH: $17.82M 06.11.18 90-Day Reserve 6.70M LOW: ($1.07M) 01.07.13 Clinical cash balance as of 06.11.18:

555 views • 12 slides

More recommend