failure sketching a technique for automated root cause
play

Failure Sketching: A Technique for Automated Root Cause Diagnosis - PowerPoint PPT Presentation

Mar 28, 2023 •7 likes •937 views

Failure Sketching: A Technique for Automated Root Cause Diagnosis of In-Production Failures Baris Kasikci, Benjamin Schubert, Cristiano Pereira, Gilles Pokam, George Candea Debugging In-Production Software Failures Today 2 Debugging

  1. Failure Sketching: A Technique for Automated Root Cause Diagnosis of In-Production Failures Baris Kasikci, Benjamin Schubert, Cristiano Pereira, Gilles Pokam, George Candea

  2. Debugging In-Production Software Failures Today 2

  3. Debugging In-Production Software Failures Today 2

  4. Debugging In-Production Software Failures Today #0 0x00007f51abae820b in raise (sig=11) at ../nptl/ sysdeps/unix/sysv/linux/pt-raise.c:37 #1 0x000000000042d289 in ap_buffered_log_writer (r=0x7f51a40053d0, handle=0x20eeba0, strs=0x7f51a4003578, strl=0x7f51a40035e8, nelts=14, len=82) at mod_log_config.c:1368 #2 0x000000000042b10d in config_log_transaction (r=0x7f51a40053d0, cls=0x20b9d50, default_format=0x20ee370) at mod_log_config.c:930 #3 0x000000000042aad6 in multi_log_transaction (r=0x7f51a40053d0) at mod_log_config.c:950 #4 0x000000000046cb2d in ap_run_log_transaction (r=0x7f51a40053d0) at protocol.c:1563 #5 0x0000000000436e81 in ap_process_request (r=0x7f51a40053d0) at http_request.c:312 #6 0x000000000042e9da in ap_process_http_connection (c=0x7f519c000b68) at http_core.c:293 #7 0x0000000000465cdd in ap_run_process_connection (c=0x7f519c000b68) at connection.c:85 #8 0x00000000004661f5 in ap_process_connection (c=0x7f519c000b68, csd=0x7f519c000a20) at connection.c:211 #9 0x0000000000451ba0 in process_socket (p=0x7f519c0009b8, sock=0x7f519c000a20, my_child_num=0, my_thread_num=0, bucket_alloc=0x7f51a4001348) at worker.c:632 #10 0x0000000000451221 in worker_thread (thd=0x210fa90, dummy=0x7f51a40008c0) at worker.c:946 #11 0x00007f51ac87c555 in dummy_worker (opaque=0x210fa90) at thread.c:127 #12 0x00007f51abae0182 in start_thread (arg=0x7f51aa8ef700) at pthread_create.c:312 #13 0x00007f51ab80d47d in clone () at ../sysdeps/ unix/sysv/linux/x86_64/clone.S:111 2

  5. Debugging In-Production Software Failures Today Understand root cause #0 0x00007f51abae820b in raise (sig=11) at ../nptl/ sysdeps/unix/sysv/linux/pt-raise.c:37 #1 0x000000000042d289 in ap_buffered_log_writer (r=0x7f51a40053d0, handle=0x20eeba0, strs=0x7f51a4003578, strl=0x7f51a40035e8, nelts=14, len=82) at mod_log_config.c:1368 #2 0x000000000042b10d in config_log_transaction (r=0x7f51a40053d0, cls=0x20b9d50, default_format=0x20ee370) at mod_log_config.c:930 #3 0x000000000042aad6 in multi_log_transaction (r=0x7f51a40053d0) at mod_log_config.c:950 #4 0x000000000046cb2d in ap_run_log_transaction (r=0x7f51a40053d0) at protocol.c:1563 #5 0x0000000000436e81 in ap_process_request (r=0x7f51a40053d0) at http_request.c:312 #6 0x000000000042e9da in ap_process_http_connection (c=0x7f519c000b68) at http_core.c:293 #7 0x0000000000465cdd in ap_run_process_connection (c=0x7f519c000b68) at connection.c:85 #8 0x00000000004661f5 in ap_process_connection (c=0x7f519c000b68, csd=0x7f519c000a20) at connection.c:211 #9 0x0000000000451ba0 in process_socket (p=0x7f519c0009b8, sock=0x7f519c000a20, my_child_num=0, my_thread_num=0, bucket_alloc=0x7f51a4001348) at worker.c:632 #10 0x0000000000451221 in worker_thread (thd=0x210fa90, dummy=0x7f51a40008c0) at worker.c:946 #11 0x00007f51ac87c555 in dummy_worker (opaque=0x210fa90) at thread.c:127 #12 0x00007f51abae0182 in start_thread (arg=0x7f51aa8ef700) at pthread_create.c:312 #13 0x00007f51ab80d47d in clone () at ../sysdeps/ unix/sysv/linux/x86_64/clone.S:111 2

  6. Debugging In-Production Software Failures Today Understand root cause #0 0x00007f51abae820b in raise (sig=11) at ../nptl/ sysdeps/unix/sysv/linux/pt-raise.c:37 #1 0x000000000042d289 in ap_buffered_log_writer (r=0x7f51a40053d0, handle=0x20eeba0, strs=0x7f51a4003578, strl=0x7f51a40035e8, nelts=14, len=82) at mod_log_config.c:1368 #2 0x000000000042b10d in config_log_transaction (r=0x7f51a40053d0, cls=0x20b9d50, default_format=0x20ee370) at mod_log_config.c:930 #3 0x000000000042aad6 in multi_log_transaction (r=0x7f51a40053d0) at mod_log_config.c:950 #4 0x000000000046cb2d in ap_run_log_transaction (r=0x7f51a40053d0) at protocol.c:1563 #5 0x0000000000436e81 in ap_process_request (r=0x7f51a40053d0) at http_request.c:312 #6 0x000000000042e9da in ap_process_http_connection (c=0x7f519c000b68) at http_core.c:293 #7 0x0000000000465cdd in ap_run_process_connection (c=0x7f519c000b68) at connection.c:85 #8 0x00000000004661f5 in ap_process_connection (c=0x7f519c000b68, csd=0x7f519c000a20) at connection.c:211 #9 0x0000000000451ba0 in process_socket (p=0x7f519c0009b8, sock=0x7f519c000a20, my_child_num=0, my_thread_num=0, bucket_alloc=0x7f51a4001348) at worker.c:632 #10 0x0000000000451221 in worker_thread (thd=0x210fa90, dummy=0x7f51a40008c0) at worker.c:946 #11 0x00007f51ac87c555 in dummy_worker (opaque=0x210fa90) at thread.c:127 #12 0x00007f51abae0182 in start_thread (arg=0x7f51aa8ef700) at pthread_create.c:312 #13 0x00007f51ab80d47d in clone () at ../sysdeps/ unix/sysv/linux/x86_64/clone.S:111 Reproduce the failure 2

  7. Debugging In-Production Software Failures Today Understand root cause #0 0x00007f51abae820b in raise (sig=11) at ../nptl/ sysdeps/unix/sysv/linux/pt-raise.c:37 #1 0x000000000042d289 in ap_buffered_log_writer (r=0x7f51a40053d0, handle=0x20eeba0, strs=0x7f51a4003578, strl=0x7f51a40035e8, nelts=14, len=82) at mod_log_config.c:1368 #2 0x000000000042b10d in config_log_transaction (r=0x7f51a40053d0, cls=0x20b9d50, default_format=0x20ee370) at mod_log_config.c:930 #3 0x000000000042aad6 in multi_log_transaction (r=0x7f51a40053d0) at mod_log_config.c:950 #4 0x000000000046cb2d in ap_run_log_transaction (r=0x7f51a40053d0) at protocol.c:1563 #5 0x0000000000436e81 in ap_process_request (r=0x7f51a40053d0) at http_request.c:312 #6 0x000000000042e9da in ap_process_http_connection (c=0x7f519c000b68) at http_core.c:293 #7 0x0000000000465cdd in ap_run_process_connection (c=0x7f519c000b68) at connection.c:85 #8 0x00000000004661f5 in ap_process_connection (c=0x7f519c000b68, csd=0x7f519c000a20) at connection.c:211 #9 0x0000000000451ba0 in process_socket (p=0x7f519c0009b8, sock=0x7f519c000a20, my_child_num=0, my_thread_num=0, bucket_alloc=0x7f51a4001348) at worker.c:632 #10 0x0000000000451221 in worker_thread (thd=0x210fa90, dummy=0x7f51a40008c0) at worker.c:946 #11 0x00007f51ac87c555 in dummy_worker (opaque=0x210fa90) at thread.c:127 #12 0x00007f51abae0182 in start_thread (arg=0x7f51aa8ef700) at pthread_create.c:312 #13 0x00007f51ab80d47d in clone () at ../sysdeps/ unix/sysv/linux/x86_64/clone.S:111 Reproduce the failure 2

  8. Related Work • Collaborative approaches • WER [SOSP’09], CBI [PLDI’05], CCI [OOPSLA’10] • Identifying differences of failing and successful runs • Delta debugging [TSE’02], Symbiosis [PLDI’15] • Record & replay, checkpointing • ODR [SOSP’09], Triage [SOSP’07] • Hardware support • PBI [ASPLOS’13], LBRA/LCRA [ASPLOS’14] 3

  9. Related Work • Collaborative approaches • WER [SOSP’09], CBI [PLDI’05], CCI [OOPSLA’10] • Identifying differences of failing and successful runs • Delta debugging [TSE’02], Symbiosis [PLDI’15] • Record & replay, checkpointing • ODR [SOSP’09], Triage [SOSP’07] • Hardware support • PBI [ASPLOS’13], LBRA/LCRA [ASPLOS’14] 3

  10. Contributions 4

  11. Contributions Goal: automate the manual detective work of debugging 
 4

  12. Contributions Goal: automate the manual detective work of debugging 
 Failure sketching Complements in-house static analysis with in-production dynamic analysis Automatically and efficiently builds accurate failure sketches that show root causes of failures 4

  13. Failure Sketch Thread 1 Thread 2 Time 1 1 main() { 2 2 queue* f = init(size); 3 3 create_thread(cons, f); cons(queue* f) { 4 4 ... ... 5 5 6 6 free(f->mut); mutex_unlock(f->mut); 7 7 ... } 8 8 } Segfault 5

  14. Failure Sketch Thread 1 Thread 2 Time 1 1 main() { 2 2 queue* f = init(size); 3 3 create_thread(cons, f); cons(queue* f) { 4 4 ... ... 5 5 6 6 free(f->mut); mutex_unlock(f->mut); 7 7 ... } 8 8 } Segfault 5

  15. Failure Sketch Thread 1 Thread 2 Time 1 1 main() { 2 2 queue* f = init(size); 3 3 create_thread(cons, f); cons(queue* f) { 4 4 ... ... 5 5 6 6 free(f->mut); mutex_unlock(f->mut); 7 7 ... } 8 8 } Segfault 5

  16. Failure Sketch Thread 1 Thread 2 Time 1 1 main() { 2 2 queue* f = init(size); 3 3 create_thread(cons, f); cons(queue* f) { 4 4 ... ... 5 5 6 6 free(f->mut); mutex_unlock(f->mut); 7 7 ... } 8 8 } Segfault 5

  17. Failure Sketch Thread 1 Thread 2 Time 1 1 main() { 2 2 queue* f = init(size); 3 3 create_thread(cons, f); cons(queue* f) { 4 4 ... ... 5 5 6 6 free(f->mut); mutex_unlock(f->mut); 7 7 ... } 8 8 } Segfault 5

  18. Failure Sketch Thread 1 Thread 2 Time 1 1 main() { 2 2 queue* f = init(size); 3 3 create_thread(cons, f); cons(queue* f) { 4 4 ... Root ... 5 5 cause 6 6 free(f->mut); mutex_unlock(f->mut); 7 7 ... } 8 8 } Segfault 5

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Getting the most out of the ROOT tutorials Automated conversion from ROOT macros to Jupyter

Getting the most out of the ROOT tutorials Automated conversion from ROOT macros to Jupyter

Getting the most out of the ROOT tutorials Automated conversion from ROOT macros to Jupyter notebooks Pau Miquel, Summer 2016 Supervisors: Pere Mat, Danilo Piparo 1 / 24 Outline 1. Overview 2. ROOT Tutorials 3. Conversion a. Steps

746 views • 30 slides
EE3CL4: Sketching the Root Locus, Introduction to Linear Control Systems Steps 14 Steps 1

EE3CL4: Sketching the Root Locus, Introduction to Linear Control Systems Steps 14 Steps 1

EE 3CL4, 5 1 / 65 Tim Davidson Preliminary examples Principles EE3CL4: Sketching the Root Locus, Introduction to Linear Control Systems Steps 14 Steps 1 and 2 Section 5: Root Locus Procedure Review of Principles Review of Steps 1,

936 views • 58 slides
Plan of the Lecture Review: rules for sketching root loci; introduction to dynamic

Plan of the Lecture Review: rules for sketching root loci; introduction to dynamic

Plan of the Lecture Review: rules for sketching root loci; introduction to dynamic compensation Todays topic: lead and lag dynamic compensation Plan of the Lecture Review: rules for sketching root loci; introduction to dynamic

1.13k views • 97 slides
a culture of failure mathias meyer, @roidrage travis-ci.org / travis-ci.com failure risk 28

a culture of failure mathias meyer, @roidrage travis-ci.org / travis-ci.com failure risk 28

a culture of failure mathias meyer, @roidrage travis-ci.org / travis-ci.com failure risk 28 january 1986 sts-51-l 73 seconds there is no root cause "What you call root cause is simply the place where you stop looking any further"

610 views • 48 slides
Iterative Sketching Agile Arizona 2017 Agenda Who am I? The Power of Sketching When

Iterative Sketching Agile Arizona 2017 Agenda Who am I? The Power of Sketching When

Iterative Sketching Agile Arizona 2017 Agenda Who am I? The Power of Sketching When to Sketch? Who Sketches? Sketching Practice: Software Elements Lets Sketch! Wrap-up The Power of Sketching Create Shared

416 views • 9 slides
INC 342 Lecture 2: Root Locus (cont.) Dr. Benjamas Panomruttanarug Benjamas.pan@kmutt.ac.th

INC 342 Lecture 2: Root Locus (cont.) Dr. Benjamas Panomruttanarug Benjamas.pan@kmutt.ac.th

INC 342 Lecture 2: Root Locus (cont.) Dr. Benjamas Panomruttanarug Benjamas.pan@kmutt.ac.th Sketching Root Locus (review) 1. Number of branches 2. Symmetry 3. Real axis segment 4. Starting and ending points 5. Behavior at infinity BP INC 342

723 views • 12 slides
James 4:1-6 ...But all these causes for management failure have their root in one common

James 4:1-6 ...But all these causes for management failure have their root in one common

James 4:1-6 ...But all these causes for management failure have their root in one common cause: pride. In the simplest terms, pride is devastating. Im not talking about the pride one has in their work or their accomplishments. Im

484 views • 7 slides
1 Wri<ng and maintaining tests is tedious and error-prone.

1 Wri<ng and maintaining tests is tedious and error-prone.

{HEADSHOT} This lesson introduces a powerful automated tes<ng technique called dynamic symbolic execu<on. This technique is based on hybrid

1.18k views • 72 slides
Failure mode and effect analysis (FMEA) for Paediatric Intubation A technique to evaluate high

Failure mode and effect analysis (FMEA) for Paediatric Intubation A technique to evaluate high

Failure mode and effect analysis (FMEA) for Paediatric Intubation A technique to evaluate high risk procedures Third QPEM Conference 11-13 th of January 2019 Beverley Ludick Disclosure I do not have any relevant financial relationship with

300 views • 13 slides
Failure Sketches: A Be2er Way to Debug Baris Kasikci,

Failure Sketches: A Be2er Way to Debug Baris Kasikci,

Failure Sketches: A Be2er Way to Debug Baris Kasikci, Cris<ano Pereira, Gilles Pokam, Benjamin Schubert, Madan Musuvathi, George Candea Failure and Root

666 views • 37 slides
An Automated Social Graph De-anonymization Technique Kumar Sharad 1 George Danezis 2 1 2

An Automated Social Graph De-anonymization Technique Kumar Sharad 1 George Danezis 2 1 2

An Automated Social Graph De-anonymization Technique Kumar Sharad 1 George Danezis 2 1 2 November 3, 2014 Workshop on Privacy in the Electronic Society, Scottsdale, Arizona, USA This Talk 1 The Art of Data Anonymization 2 The D4D Challenge 3

527 views • 51 slides
Curve Sketching Michael Freeze MAT 151 UNC Wilmington Summer 2013 1 / 10 Section 5.4 :: Curve

Curve Sketching Michael Freeze MAT 151 UNC Wilmington Summer 2013 1 / 10 Section 5.4 :: Curve

Curve Sketching Michael Freeze MAT 151 UNC Wilmington Summer 2013 1 / 10 Section 5.4 :: Curve Sketching 2 / 10 Curve Sketching Make an accurate sketch of the graph the function. Identify any important features such as intercepts,

287 views • 10 slides
Evolving Technique Update Cup positioning to avoid Metalosis Bone preparation to avoid Failure of

Evolving Technique Update Cup positioning to avoid Metalosis Bone preparation to avoid Failure of

OS ET 2017 Bellagio Las Vegas Evolving Technique Update Cup positioning to avoid Metalosis Bone preparation to avoid Failure of Ingrowth Thomas Gross 8:52 AM 33 #1 Preventing Metalosis in Hip Resurfacing using RAIL guidelines and NS IOR

344 views • 33 slides
W O R L D B A N K , 1 8 T H A P R I L 2 0 1 7 THE FAILURE OF THE OLD ECONOMIC DEVELOPMENT

W O R L D B A N K , 1 8 T H A P R I L 2 0 1 7 THE FAILURE OF THE OLD ECONOMIC DEVELOPMENT

W O R L D B A N K , 1 8 T H A P R I L 2 0 1 7 THE FAILURE OF THE OLD ECONOMIC DEVELOPMENT MODELS 2 THE EMERGENCE OF GRASS-ROOT, ENTERPRISE LED MOVEMENT TO DEVELOPMENT 3 THE EXAMPLE OF THE ECONOMY OF COMMUNION 4 EOCS VOLUNTARY SHARING

137 views • 11 slides
Sulfur Recovery Unit Reaction Furnaces Waste Heat Boilers Ferrule Failure Root Cause Analysis

Sulfur Recovery Unit Reaction Furnaces Waste Heat Boilers Ferrule Failure Root Cause Analysis

Sulfur Recovery Unit Reaction Furnaces Waste Heat Boilers Ferrule Failure Root Cause Analysis Presented By: Muhammad Bashir Khan Manager, Technical Page 1 Abu Dhabi Gas Development Company Ltd. CONFIDENTIAL SRU and Ferrule Configuration

346 views • 30 slides
AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 2 ISSTA

AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 2 ISSTA

1 THOMAS WALSH 1 , PHIL MCMINN 1 , GREGORY KAPFHAMMER 2 1 UNIVERSITY OF SHEFFIELD, 2 ALLEGHENY COLLEGE AUTOMATED LAYOUT FAILURE DETECTION FOR RESPONSIVE WEB PAGES WITHOUT AN EXPLICIT ORACLE 2 ISSTA 2017 A LONG TIME AGO THOMAS WALSH

1k views • 72 slides
Meta-interpretive learning of data transformation programs Andrew Cropper, Alireza

Meta-interpretive learning of data transformation programs Andrew Cropper, Alireza

Meta-interpretive learning of data transformation programs Andrew Cropper, Alireza Tamaddoni-Nezhad, Stephen H. Muggleton Imperial College London Input P_011 67 year Output lung disease: n/a, Diagnosis: Unknown P_011 67 Unknown 80.78%

247 views • 12 slides
Automated Diagnosis of Software Configuration Errors Sai Zhang , Michael D. Ernst University of

Automated Diagnosis of Software Configuration Errors Sai Zhang , Michael D. Ernst University of

Automated Diagnosis of Software Configuration Errors Sai Zhang , Michael D. Ernst University of Washington

372 views • 35 slides
Cost of Debugging The huge prinMng presses for a major

Cost of Debugging The huge prinMng presses for a major

G 2 : A Graph Processing System for Diagnosing Distributed Systems Zhenyu GUO , Dong ZHOU, Haoxiang LIN, Mao YANG, Fan LONG, Chaoqiang DENG, Changshu

493 views • 28 slides
The MApUCE project: an interdisciplinary approach to integrate 1.

The MApUCE project: an interdisciplinary approach to integrate 1.

The MApUCE project: an interdisciplinary approach to integrate 1. Context energy ad urban climate into urban planning 2. Development of GENIUS

589 views • 48 slides
Model Selection, Evaluation, Diagnosis INFO-4604, Applied Machine Learning University of

Model Selection, Evaluation, Diagnosis INFO-4604, Applied Machine Learning University of

Model Selection, Evaluation, Diagnosis INFO-4604, Applied Machine Learning University of Colorado Boulder October 31 November 2, 2017 Prof. Michael Paul Today How do you estimate how well your classifier will perform? Pipeline for

977 views • 58 slides
Differential Diagnosis of ASD Hold shares/warrants in Chemocentryx, Retrotope, Jacaranda (a

Differential Diagnosis of ASD Hold shares/warrants in Chemocentryx, Retrotope, Jacaranda (a

3/3/2017 Disclosures Advisory Board to Invitae (genetic information company) Differential Diagnosis of ASD Hold shares/warrants in Chemocentryx, Retrotope, Jacaranda (a Neurologists perspective) Biosciences Elliott Sherr MD PhD

371 views • 6 slides
Inference and Representation David Sontag New York University Lecture 1, September 8, 2015

Inference and Representation David Sontag New York University Lecture 1, September 8, 2015

Inference and Representation David Sontag New York University Lecture 1, September 8, 2015 David Sontag (NYU) Inference and Representation Lecture 1, September 8, 2015 1 / 35 One of the most exciting advances in machine learning (AI, signal

890 views • 35 slides
Security and Data Privacy Instructor: Matei Zaharia cs245.stanford.edu Outline Security

Security and Data Privacy Instructor: Matei Zaharia cs245.stanford.edu Outline Security

Security and Data Privacy Instructor: Matei Zaharia cs245.stanford.edu Outline Security requirements Key concepts and tools Differential privacy Other security tools CS 245 2 Outline Security requirements Key concepts and tools

873 views • 62 slides

More recommend