Precision, Recall, and Sensitivity of Monitoring Partially - PowerPoint PPT Presentation

RunHme VerificaHon 2016 (RV’16) Precision, Recall, and Sensitivity of Monitoring Partially Synchronous Distributed Systems Sorrachai Yingchareonthawornchai 1 , Duong Nguyen 1 , Vidhya Tekken Valapil 1 , Sandeep Kulkarni 1 , and Murat Demirbas 2 1 Michigan State University, MI, USA 2 University at Buffalo, NY, USA

Introduction • Run-Hme verificaHon is focused on ensuring/verifying properHes such as • Always p • p leads to q • p UNTIL q • … • One key problem in all these instances • Detect if a given predicate p is true • Trivial if p is a local predicate • Not so trivial if p is a predicate involving mulHple processes

Consistent Snapshot • AssumpHon about monitors • At runHme: evaluate predicates in consistent snapshot • If true, report hWp://slideplayer.com/slide/9232333/

Requirements for Monitoring Distributed Systems • Wait-free • Low overhead Hmestamp

Requirements for Monitoring Distributed Systems • Wait-free • Low overhead Hmestamp ASYNC SYNC Par*ally SYNC Clocks No wall clock No clock dri\ Bounded clock dri\ Wait-Free Low overhead *mestamp

Monitoring with Partial Synchrony • Pros: • Wait-free • Low overhead Hmestamp • Cons: • You do not know uncertainty window • Need to understand: Precision, Recall, SensiHvity

Outline • IntroducHon • MoHvaHng Example • System Model • Precision, Recall and SensiHvity of Asynchronous Monitors • Precision, Recall, and SensiHvity of ParHally Synchronous Monitors • Quasi-Synchronous Monitors • Conclusion

Inherent Uncertainty in Evaluating Distributed Predicates (1) • Is predicate x > 0 ∧ y > 0 true x > 0 • No message m y > 0

Inherent Uncertainty in Evaluating Distributed Predicates (2) • Is predicate x > 0 ∧ y > 0 true x > 0 • It could be • We focus on such `possible’ modality y > 0

Inherent Uncertainty in Evaluating Distributed Predicates (2) • Is predicate x > 0 ∧ y > 0 true x > 0 • It could be • We focus on such `possible’ modality y > 0 x > 0 y > 0

Inherent Uncertainty in Evaluating Distributed Predicates (3) [10..20] on P1 • Is predicate x > 0 ∧ y > 0 true x > 0 • It depends upon assumpHons about clock synchronizaHon “Implicit Hmeout mechanism” • Clocks of P1 and P2 differ by at most 5 y > 0 [30..40] on P2

Inherent Uncertainty in Evaluating Distributed Predicates (4) • Is predicate x > 0 ∧ y > 0 true x > 0 • Asynchronous: (Possibly) True y > 0

Inherent Uncertainty in Evaluating Distributed Predicates (4) • Is predicate x > 0 ∧ y > 0 true x > 0 • Asynchronous: (Possibly) True • Synchronous: False y > 0

Inherent Uncertainty in Evaluating Distributed Predicates (4) • Is predicate x > 0 ∧ y > 0 true x > 0 • Asynchronous: (Possibly) True • Synchronous: False • ParHally Synchronous: depends on uncertainty bound y > 0

Inherent Uncertainty in Evaluating Distributed Predicates (4) • Is predicate x > 0 ∧ y > 0 true x > 0 • Asynchronous: (Possibly) True • Synchronous: False • ParHally Synchronous: depends on synchronizaHon bound y > 0 • If Hght, False SynchronizaHon Bound

Inherent Uncertainty in Evaluating Distributed Predicates (4) • Is predicate x > 0 ∧ y > 0 true x > 0 • Asynchronous: (Possibly) True • Synchronous: False • ParHally Synchronous: depends on synchronizaHon bound y > 0 • If Hght, False • If loose, True SynchronizaHon Bound

Inherent Uncertainty in Evaluating Distributed Predicates (5) • Is predicate x > 0 ∧ y > 0 true x > 0 • Depends upon clock synchronizaHon assumpHons made in the applicaHon • To work correctly, monitor must also make same assumpHons as y > 0 those made in the applicaHon • What if these assumpHons are ad-hoc • It would be impossible to match it perfectly

Our focus: analyze the errors caused by uncertainty of the underlying distributed system and the impedance mismatch between the monitor and the applica+on

Impedance Mismatch • Is predicate x > 0 ∧ y > 0 true x > 0 • (Monitor, App) à (async, sync) • False posiHve • (Monitor, App) à (sync, async) • False negaHve y > 0 • IntuiHon • ApplicaHon is the ground truth • Monitor’s job is to match with ApplicaHon

3 Problems • (1) Asynchronous Monitors • (2) ParHally Synchronous Monitors • Monitor assumes asynchronous • Monitor assumes ε 1 system • But, applicaHon relies on ε 2 • But, applicaHon is parHally synchronized • Characterize Precision/Recall? • Characterize Precision/Recall? How sensi*ve to the varia*on of uncertainty bound ? • (3) Quasi-Synchronous Monitors • Approximate synchronous monitors

Outline • IntroducHon • MoHvaHng Example • System Model • Precision, Recall and SensiHvity of Asynchronous Monitors • Precision, Recall, and SensiHvity of ParHally Synchronous Monitors • Quasi-Synchronous Monitors • Conclusion

System Model • A set of n processes that communicate via messages • ε app = clock synchronizaHon assumpHon made by the applicaHon • ε mon = clock synchronizaHon assumpHon made by the monitor • δ min = minimum message delay • δ max = maximum message delay (always ∞ in this discussion) • α = Probability of sending a message to others (random recipient )

System Model (2) • Focus on detecHon of conjuncHon of local predicates • Predicate being detected is ∧ pr j • pr j local predicate on process j • Other predicates can be expressed in terms of disjuncHon of such conjuncHve predicates • Parameter • β : Probability that pr j is true • L : length of interval for which pr j is true • We consider cases where β of different processes are independent/dependent • Monitors are responsible for ensuring that they combine local snapshots appropriately (e.g., to avoid inconsistencies)

Outline • IntroducHon • MoHvaHng Example • System Model • Precision, Recall and SensiHvity of Asynchronous Monitors • Precision, Recall, and SensiHvity of ParHally Synchronous Monitors • Quasi-Synchronous Monitors • Conclusion

Precision and Sensitivity of Asynchronous Monitors • Assume that ε mon = ∞ • Only false posiHves are possible

Precision and Sensitivity of Asynchronous Monitors • Assume that ε mon = ∞ • Only false posiHves are possible • Theorem 1: Precision is P = ((1 − (1 − β ) ✏ + ` − 1 ) n − 1 • Remark: Independent of communicaHon paWern

Precision

Precision, Sensitivity

Precision, Sensitivity

Precision, Sensitivity CriHcal points: ε p1 , ε p2

Sensitivity of Asynchronous Monitors 0 ε p1 ε p2 ∞

Sensitivity of Asynchronous Monitors Not effecHve, √ log 1 − β (3 n − 4 − 5 n 2 − 16 n + 12 not sensiHve ) 2( n − 1) 2 0 ε p1 ε p2 ∞

Sensitivity of Asynchronous Monitors √ EffecHve log 1 − β (3 n − 4 + 5 n 2 − 16 n + 12 ) Not SensiHve 2( n − 1) 2 0 ε p1 ε p2 ∞

Sensitivity of Asynchronous Monitors HypersensiHve RelaHvely, small EffecHve Not effecHve, range Not SensiHve not sensiHve ✏ p 2 − ✏ p 1 lim = 0 n →∞ ✏ p 1

Validation with Simulation n = 20, α =0.05, δ =10, β =0.03 1 n = 20, β =0.1 1.5 ε =50 0.8 ε =100 False positive rate False positive rate 1 ε =200 0.6 0.5 analytical model 0.4 simulation α =0.05 δ =10 0 α =0.1 δ =10 0.2 α =0.05 δ =100 Time -0.5 α =0.1 δ =100 0 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 ε 0 0 20 40 60 80 100 120 140

Validation with Simulation (2) • Effect of changing n , β and ε

Correlated events • Examples: PosiHve Majority CorrelaHon (PMAJ) • Process 1 has Beta probability • Process i’s event depends on 1 • Majority of [1, 2 …, i-1] with 50% Chance PMAJ distribution 0.8 • Beta probability with another 50% n=20 α =0.01 β =0.10 δ =100 False positive rate • In long run, 0.6 Analytical( β /2,n/4) • the distribuHon converges to Simulation • Uniform analyHcal model ! 0.4 0.2 • We have considered many other correlaHons with similar results ε 0 0 50 100 150 200 250 300

Outline • IntroducHon • MoHvaHng Example • System Model • Precision, Recall and SensiHvity of Asynchronous Monitors • Precision, Recall, and SensiHvity of ParHally Synchronous Monitors • Quasi-Synchronous Monitors • Conclusion

Precision, Recall, and Sensitivity of Partially Synchronous Monitors • What if ε app differs from ε mon • False posiHves can occur if ε app < ε mon • False negaHves can occur if ε mon < ε app • We use same analyHcal model and compute Precision and Recall and sensiHvity

Precision and Recall • Theorem 3: ImplicaHon: CondiHon for getng high precision and recall at the same Hme QuesHon: How sensiHve?

Precision and Recall Diagram Point-based precision (P) and recall (R) β =0.05, α =0.01, δ =10, n = 5 and 20 160 140 n=5 R=0.9 P=0.9 120 R=0.8 100 P=0.8 ε app 100 % 80 n=20 R=0.9 60 P=0.9 R=0.8 40 P=0.8 20 ε mon 0 0 20 40 60 80 100 120 140

Sensitivity