USENIX / ACM / IFIP 10th International Middleware Conference How To Keep Your Head Above Water While Detecting Errors Ignacio Laguna, Fahad A. Arshad, David M. Grothe, Saurabh Bagchi Dependable Computing System Lab School of Electrical and Computer Engineering Purdue University Slide 1/27
Impact of Failures in Internet Services • Internet services are expected to be running 24/7 – System downtime can cost $1 million / hour (Source : Meta Group, 2002) • Service degradation is the most frequent problem – Service is slower than usual; almost unavailable – Can be difficult to detect and diagnose • Internet-based applications are very large and dynamic – Complexity increases as new components are added Slide 2/27
Complexity of Internet-based Applications Software Components Servlets, JavaBeans, EJBs • Each tier has multiple components • Components can be stateful Slide 3/27
P R E V I O U S W O R K The Monitor Detection System ( TDSC ’06 ) Monitor Observed System • Non-intrusive observe messages between components • Online faster detection than offline approaches • Black-box detection components treated as black boxes – No knowledge of components’ internals Slide 4/27
P R E V I O U S W O R K Stateful Rule-based Detection Monitor Finite State Machine Packet Normal- (event-based state Capturer Behavior Rules transition model) Detection Process Match rules If a rule does Deduce current A message based on the not satisfy, application is captured deduced state signal alarm state 1 2 3 4 Slide 5/27
P R E V I O U S W O R K The Breaking Point in High Rate of Messages Detection Latency 1200 900 (msec) 600 300 0 0 100 200 300 400 Breaking point Captured Packets / second (Incoming msg rate at Monitor) • After breaking point, latency increases sharply • True alarms rate decreases because packets are dropped • Breaking point expected in any stateful detection system Slide 6/27
P R E V I O U S W O R K Avoiding the Breaking Point: Random Sampling (SRDS ‘07) • Processing Load in Monitor δ = R × C R : Incoming message rate, C : Processing cost per message • Processing Load δ is reduced by reducing R – Only a portion of incoming messages is processed – n out of m messages are randomly sampled • Sampling is activated if R ≥ breaking point Slide 7/27
P R E V I O U S W O R K The Non-Determinism Caused by Sampling A portion of a Events in Monitor Finite State Machine (1) SV = { S0 } m3 … S3 (2) A message is dropped S1 m1 m4 (3) SV = { S1 , S2 } … S4 … S0 (4) A message is sampled m2 m5 S2 (5) The message is m5 … S5 (6) SV = { S5 } Definitions: State Vector (SV) The state(s) of the application from Monitor’s point of view (deduced state(s)) Non-Determinism Monitor is no longer aware of the exact state the application is in (because of dropped messages) Slide 8/27
C U R R E N T W O R K Remaining Agenda I. Addressing the problem of non-determinism A. Intelligent Sampling B. Hidden Markov Model (HMM) for state determination II. Experimental Test-bed III. Performance Results IV. Efficient Rule Matching and Triggering Slide 9/27
Challenges with Non-Determinism Decrease Detection Latency (Large SV increases rule Hidden Markov matching time) Intelligent Model Sampling SV reduction Non-Determinism Decrease False Increase True Alarms Alarms (Reduce effect of (Reduce incorrect incorrect messages) states in SV) Slide 10/27
Intelligent Sampling State Vector S2 Random S3 Finite State S1 Sampling Machine S5 S6 S4 Sampled Messages Incoming Messages State Vector Intelligent Finite State S1 S3 Sampling Machine S4 Sampled Messages Incoming Messages Message with desirable property Slide 11/27
What is a Desirable Property of a Message? Suppose, SV ={ S1 , S2 , S3 }, Sampling Rate = 1/3 Sampled A portion of a Discriminative SV Message Finite State Machine Size m1 … S1 m1 { S4 , S5 , S6 } 3 m2 S4 m2 { S7 , S8 } S7 2 m3 m2 m1 … … S2 S5 S9 m3 { S9 } 1 m3 S8 m2 m1 … S3 S6 Discriminative Size Number of times a message appears in transitions to different states in the FSM • Desirable property is a small discriminative size Slide 12/27
Benefits of Intelligent Sampling Random Sampling Intelligent Sampling • SV can grow into large size • SV is kept small – Detection latency reduction • Multiple incorrect states • Less incorrect states in SV – Increase of false alarms – False alarms reduction Slide 13/27
The Problem of Sampling an Incorrect Message • What if an incorrect message is sampled? – The message is incorrect in current states, e.g., a message from buggy component m3 … S3 S1 m1 m4 … S4 … S0 m2 S2 m5 … S5 • Suppose SV = { S1 , S2 } and m3 is changed to m5 ⇒ SV = { S5 } (incorrect SV !, it should be { S3 } ) Slide 14/27
Probabilistic State Vector Reduction: A Hidden Markov Model Approach • Hidden Markov Model (HMM) used to reduce SV – An HMM is an extended Markov Model where states are not observable – States are hidden as in the monitored application • Given an HMM, we can ask: The probability of the application being in any state, given a sequence of messages? Cost is O( N 3 L ) , N : number of states L : sequence length • The HMM is trained (offline) with application traces Slide 15/27
State Vector Reduction with the HMM • Monitor asks the HMM: { p 1 , p 2 ,…, p N } p i = P( S i | O ), S i : application state i , O: observation’s sequence p 1 , p 2 , Messages Sort by HMM …, (observations) Probabilities p N p 3 , α = Top k α ∩ SV = new SV p 5 , probabilities …, New SV is robust to p 1 incorrect messages Slide 16/27
Experimental Testbed: Java Duke’s Bank Application • Simulates multi-tier online banking system • User transactions: – Access account information, transfer money, etc. • Application stressed with different workloads – Incoming message rate at Monitor varies with user load Slide 17/27
Web Interaction : A Sequence of calls and returns Components: Java servlets, JavaBeans, EJBs Slide 18/27
Error Injection Types Error Type Description a response delay in a method call Response Delays a call to a component that is never executed Null calls exception thrown by execution that is never Unhandled Exceptions caught by the program change randomly the web interaction Incorrect Message structure Sequences • Errors are injected in components touched by web interactions – A web interaction is faulty if at least one of its components is faulty Slide 19/27
Performance Metrics Used in Experiments % of true detections out of web interactions where errors Accuracy (True Alarms) were injected % of true detections out of the total number of detections Precision (False Alarms) time elapsed between the error injection and its detection Detection Latency Web Interactions 1 2 3 4 5 X X X Error Injected Example: Detection X X X X (An alarm is signaled) Precision = 2/4 = 0.5 Accuracy = 2/3 = 0.67 Slide 20/27
Results: State Vector Reduction Random Sampling State Vector Size 1 30 20 0.8 10 0 10 20 30 40 50 60 70 80 90 100 0.6 CDF Discrete Time Intelligent Sampling State Vector Size 0.4 30 0.2 20 Random Sampling 10 Intelligent Sampling 0 0 0 2 4 6 8 10 20 30 40 50 60 70 80 90 100 Discrete Time Pruned State Vector Size • Peaks are not observed in intelligent sampling (IS) – IS capability of selecting messages with small discriminative size • SV of size 1 is more frequent in IS Slide 21/27
Results: Monitor vs. Pinpoint (Accuracy, Precision) • Pinpoint ( NSDI ‘04), traces paths though multiple components • Use of PCFG to detect abnormal paths 1 1 0.8 0.8 Accuracy Precision 0.6 0.6 0.4 0.4 Monitor Monitor 0.2 0.2 Pinpoint-PCFG Pinpoint-PCFG 0 0 4 8 12 16 20 24 4 8 12 16 20 24 Concurrent Users Concurrent Users • Monitor and Pinpoint expose similar levels of accuracy • Precision in Monitor (1.0) is higher than in Pinpoint (0.9) Slide 22/27
Results: Monitor vs. Pinpoint (Detection Latency) 200 200 Detection Latency (msec) Detection Latency (sec) Pinpoint-PCFG Monitor 150 150 100 100 50 50 0 0 4 8 12 16 20 24 4 8 12 16 20 24 Concurrent Users Concurrent Users • Detection latency in Monitor is in the order of milliseconds , while in Pinpoint is in seconds • The PCFG has a high space and time complexity Slide 23/27
Results: Memory Consumption (MB) Virtual Memory RAM Monitor 282.27 25.53 Pinpoint-PCFG 933.56 696.06 • Monitor doesn’t rely in large data structures • PCFG in Pinpoint has high space and time complexity – O( RL 2 ) and O( L 3 ) R : number of rules in the grammar L : size of a web interaction • Pinpoint thrashes due to high memory requirements Slide 24/27
Efficient Rule Matching . . . No Yes Yes System Expensive Match unstable? to match? . No . . Next rule • Selectively match computationally expensive rules – Expensive rules don’t have to be matched all the time • Rules are matched only if instability is present Slide 25/27
Recommend
More recommend
