Crisis to Calm: A Story of Data Validation @ Netflix Lavanya - PowerPoint PPT Presentation

Mar 17, 2024 •232 likes •778 views

Crisis to Calm: A Story of Data Validation @ Netflix Lavanya Kanchanapalli Rollback data Increase capacity Netflix starts to work Root Cause Duplicate objects Apps failed Cascading failures Netflix goes down! Data

Crisis to Calm: A Story of Data Validation @ Netflix Lavanya Kanchanapalli
● Rollback data ● Increase capacity ● Netflix starts to work
Root Cause ● Duplicate objects ● Apps failed ● Cascading failures ● Netflix goes down!
Data Change Netflix Microservices Cloud App1 Appn App2 Change in Behavior
Metadata Architecture Traffic Netflix Netflix Source Services Netflix Services Netflix System Services Netflix Services Video Service Metadata Service Source System Amazon S3
● Single publisher ● Multiple consumers ● Hollow (hollow.how) ● Versioned data ● Fast propagation
Bad data happens!
Leaked Content
Disables Features
Deletes data
Data change = Code Push
1 2 2 3 3 Detection Staggering Staggering Rollback Rollback
1 2 3 3 Detection Staggering Rollback Rollback
1 2 3 Detection Staggering Rollback
Many Unknowns
Would it be too slow?
Would there be too many failures?
Would it cost too much?
1 2 3 Detection Staggering Rollback
1 2 2 3 3 Detection Staggering Staggering Rollback Rollback
Chapter 1: Circuit Breakers
Metadata Architecture Traffic Netflix Netflix Source Services Netflix Services Netflix System Services Netflix Services Video Service Metadata Service Source System Amazon S3
Circuit Breakers ● Integrity checks ● Duplicate detection ● Object counts ● Semantic checks
Know your data change
Knobs are key to sanity ● On/ off ● Threshold ● Exclusions
Business value is the key
Efficiency ● Change Isolation ● Sampling
Chapter 2: Canaries
Traditional Canaries Traffic Shadow Shadow Netflix Traffic Traffic Netflix Source Services Netflix Services Netflix System Services Netflix Services Video Service Metadata Service Baseline Canary Source (Old Code) (New Code) System Amazon S3
Data Canaries Traffic Netflix Netflix Data Netflix Service Canary Netflix Services Netflix Source Service Services Netflix System Services Netflix Services Video Service Metadata Service Source System Amazon S3
Netflix Data Canary Service ● Pick key use case(s) Netflix Data Netflix Service ● Pick data to test Canary Service ● Test with latest data
1 2 3 Detection Staggering Rollback Rollback
Amazon WS Global Infrastructure STAGGERED ROLLOUT
1 2 3 Detection Staggering Rollback
Keep calm & rollback ● Pin back ● Root cause ● Unpin
Rollback ● Visibility ● Traversing
Data Diff UI
Circuit Breaker UI
Pinning UI
Data validation is key to high availability ● Data change = Code push ● Circuit breakers & canaries ● Staggering and rollback
Thank You

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

CALM Sueann Tupy, Educator AGENDA Grounding Activity The Story of CALM Our Vision Program

SHAC Meeting March 20, 2019 CALM Sueann Tupy, Educator AGENDA Grounding Activity The Story of CALM Our Vision Program Goals Staff & Community Teacher Feedback Summary Grounding Activity What is CALM? C hildren and C ommunities A

396 views • 18 slides

Containers at Netflix - An Evolving Story Sangeeta Narayanan Engineering Manager @Netflix

Containers at Netflix - An Evolving Story Sangeeta Narayanan Engineering Manager @Netflix @sangeetan Going Global Recent Launches: Italy, Spain, Portugal Early 2016: Korea, Hong Kong, Taiwan, Singapore Growing BIGGER Winning Moments of

865 views • 61 slides

8 WAYS TO STAY CALM IN A CRISIS! www.bulletproofactor.com www.actonthis.tv #CORONAVIRUS THERE

8 WAYS TO STAY CALM IN A CRISIS! www.bulletproofactor.com www.actonthis.tv #CORONAVIRUS THERE IS A LOT OF FEAR AND UNCERTAINTY IN SOCIETY RIGHT NOW - AND RIGHTLY SO. THE SPREAD OF COVID-19 HAS TAKEN US ALL DOWN A PATH WE NEVER COULD

316 views • 19 slides

Occams Razor Sampling Bias : generate the data carefuly Data Snooping : handle the data

recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train Learning From Data D ( N K ) g 1 g 2 g N Lecture 14 ( x 1 , y 1 ) ( x 2 , y 2 ) ( x N , y N ) Three Learning Principles D

486 views • 15 slides

Netflix: Netflix: Petabyte Scale Petabyte Scale Analytics Infrastructure in Analytics

Netflix: Netflix: Petabyte Scale Petabyte Scale Analytics Infrastructure in Analytics Infrastructure in the Cloud the Cloud Daniel C. Weeks Tom Gianos Overview Data at Netflix Netflix Scale Platform Architecture Data

887 views • 60 slides

Netflix: Integrating Spark At Petabyte Scale Ashwin Shankar Cheolsoo Park Outline 1. Netflix

Netflix: Integrating Spark At Petabyte Scale Ashwin Shankar Cheolsoo Park Outline 1. Netflix big data platform 2. Spark @ Netflix 3. Multi-tenancy problems 4. Predicate pushdown 5. S3 file listing 6. S3 insert overwrite 7. Zeppelin, Ipython

526 views • 52 slides

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far

Data Mining II Model Validation Heiko Paulheim Why Model Validation? We have seen so far Various metrics (e.g., accuracy, F-measure, RMSE, ) Evaluation protocol setups Split Validation Cross Validation Special

1.04k views • 55 slides

How We Know Where You Are in House of Cards @zimmermatt Netflix Scale @zimmermatt Netflix

Netflixs Viewing Data Microservices How We Know Where You Are in House of Cards @zimmermatt Netflix Scale @zimmermatt Netflix Scale 62,000,000 @zimmermatt Netflix Scale 62,000,000 50+ @zimmermatt Netflix Scale 62,000,000 50+

1.07k views • 90 slides

1 QC STORY -32 QC STORY -32 QC STORY -32 QC Story-1 QC Story-1 QC Story-1 Awards and

QC STORY -32 QC STORY -32 QC STORY -32 Shop Floor view QC Story-1 QC Story-1 FACTORY OVERVIEW QC Story-1 On Behalf of Susira Industries Ltd., Chennai. We Welcome the Delegates for the QCI DL SHAH NATIONAL AWARDS ON ECONOMICS OF

976 views • 13 slides

Keep Calm Keep Calm and Use Parser and Use Parser Nov, 2015 Howard Huang, Huawei Julien

Keep Calm Keep Calm and Use Parser and Use Parser Nov, 2015 Howard Huang, Huawei Julien Zhang, ZTE How would OPNFV attract more open source software developers ? 2 I wrote the script based on the instructions in ETSI NFV documents, but

293 views • 15 slides

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection

Learning From Data Lecture 13 Validation and Model Selection The Validation Set Model Selection Cross Validation M. Magdon-Ismail CSCI 4100/6100 recap: Regularization Regularization combats the effects of noise by putting a leash on the

349 views • 31 slides

Calm Computing The Coming Age of Mark Weiser and John Seely Brown Calm Computing Whyfor, Calm

Calm Computing The Coming Age of Mark Weiser and John Seely Brown Calm Computing Whyfor, Calm Computing? Whyfor, Calm Computing? Computer Evolution Whyfor, Calm Computing? Computer Evolution Mainframes Whyfor, Calm Computing? Computer

1.08k views • 95 slides

Calm Technology Calm Technology Design for the Next Generation of Devices Amber Case |

Calm Technology Calm Technology Design for the Next Generation of Devices Amber Case | @caseorganic Research Fellow, Harvard Berkman Klein Center, @medialab QCon SF | 9 Nov 2016 calmtech.com caseorganic.com 50 billion devices 50 billion will

688 views • 47 slides

Update on aircraft validation efforts; T/q retrieval validation using ARM data Dave Tobin, Leslie

Update on aircraft validation efforts; T/q retrieval validation using ARM data Dave Tobin, Leslie Moy, Bob Knuteson, Hank Revercomb, Fred Best, Joe Taylor CIMSS / SSEC / UW-Madison AIRS Science Team Meeting Pasadena, CA 27-30 March 2007

425 views • 30 slides

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data

Learning From Data Lecture 14 Three Learning Principles Occams Razor Sampling Bias Data Snooping M. Magdon-Ismail CSCI 4100/6100 recap: Validation and Cross Validation Validation Cross Validation D ( N ) D 1 D 2 D N D train D

1.01k views • 58 slides

Data validation and exploration Data validation and exploration Abhijit Dasgupta Abhijit

Data validation and exploration Data validation and exploration Abhijit Dasgupta Abhijit Dasgupta Fall, 2019 Fall, 2019 1 BIOF339, Fall, 2019 Plan today Dynamic exploration of data Data validation Missing data evaluation 2 BIOF339,

570 views • 27 slides

EMA new premises 30 Churchill Place London E14 5EU Presented by: Agneta Brandt Head of

EMA new premises 30 Churchill Place London E14 5EU Presented by: Agneta Brandt Head of Infrastructure Services Department An agency of the European Union Background Leases for present building expires December 2014 The Agency

313 views • 13 slides

CORPORATE PRESENTATION INDEX 1. Introduction 2. Key Figures of the Montebalito Consolidated

January 2019 CORPORATE PRESENTATION INDEX 1. Introduction 2. Key Figures of the Montebalito Consolidated Group 1) Assets by Business Line 3. Geographic Distribution 4. Breakdown by Country: 1) Spain 2) Germany 3) France 4)

663 views • 47 slides

LAWYER SERVICES: NON RESIDENT BUYERS FUERTEVENTURA CANARY ISLANDS PLATINUM ASSOCIATES GUIDING

LAWYER SERVICES: NON RESIDENT BUYERS FUERTEVENTURA CANARY ISLANDS PLATINUM ASSOCIATES GUIDING YOU EVERY STEP OF THE WAY SUMMARY OF SERVICES ADMINISTRATION OF NIE AND REGISTRATION IN TAX AGENCY ADMINISTRATION OF RESIDENCE

381 views • 15 slides

El Paraguas productions for events, congresses and fairs. Company specialized in furniture

El Paraguas productions for events, congresses and fairs. Company specialized in furniture rental and ephemeral production for all kinds of events. Fairs, congresses, stands, retail, corner and scenographies for companies in the public and

260 views • 8 slides

Developing Kubernetes Services at Airbnb Scale @MELANIECEBULA What is kubernetes?

@MELANIECEBULA / MARCH 2019 / CON LONDON Developing Kubernetes Services at Airbnb Scale @MELANIECEBULA What is kubernetes? @MELANIECEBULA Who am I? A BRIEF HISTORY @MELANIECEBULA Why Microservices? 4000000 3000000 2000000 MONOLITH

1.74k views • 111 slides

Social Media: The Generation Gap Judge Christine Harapiak Provincial Court of Manitoba 1981

Social Media: The Generation Gap Judge Christine Harapiak Provincial Court of Manitoba 1981 August 1982 Talk to Strangers! Independence Integrity Diligence Equality Impartiality Poll Question Has anyone ever uploaded

707 views • 41 slides

Engineering Velocity: Continuous Delivery at Netflix Dianne Marsh SATURN 2014 en-gi-neer-ing +

Engineering Velocity: Continuous Delivery at Netflix Dianne Marsh SATURN 2014 en-gi-neer-ing + ve-loc-i-ty applying science and technology to designing and building speed into a system Availability vs. Rate of Change 6 5 Availablity

326 views • 29 slides

AGENDA ITEM G.3 FIVE-YEAR CATCH SHARE FOLLOW-ON ACTION FMP AND REGULATORY AMENDMENTS FINAL

Agenda Item G.3. Supplemental Staff Presentation 1 November 2018 AGENDA ITEM G.3 FIVE-YEAR CATCH SHARE FOLLOW-ON ACTION FMP AND REGULATORY AMENDMENTS FINAL ACTION November, 2018 PFMC 1 ACTION 1. Adopt final preferred alternatives

475 views • 35 slides