Precision, Recall, and Sensitivity of Monitoring Partially - - PowerPoint PPT Presentation

▶
precision recall and sensitivity of monitoring partially
SMART_READER_LITE
LIVE PREVIEW

Precision, Recall, and Sensitivity of Monitoring Partially - - PowerPoint PPT Presentation

Jan 24, 2024 452 likes •1.02k views

RunHme VerificaHon 2016 (RV16) Precision, Recall, and Sensitivity of Monitoring Partially Synchronous Distributed Systems Sorrachai Yingchareonthawornchai 1 , Duong Nguyen 1 , Vidhya Tekken Valapil 1 , Sandeep Kulkarni 1 , and Murat Demirbas 2

slide-1
SLIDE 1

Precision, Recall, and Sensitivity of Monitoring Partially Synchronous Distributed Systems

Sorrachai Yingchareonthawornchai1, Duong Nguyen1, Vidhya Tekken Valapil1, Sandeep Kulkarni1, and Murat Demirbas2

1Michigan State University, MI, USA 2University at Buffalo, NY, USA

RunHme VerificaHon 2016 (RV’16)

slide-2
SLIDE 2

Introduction

  • Run-Hme verificaHon is focused on ensuring/verifying properHes such

as

  • Always p
  • p leads to q
  • p UNTIL q
  • …
  • One key problem in all these instances
  • Detect if a given predicate p is true
  • Trivial if p is a local predicate
  • Not so trivial if p is a predicate involving mulHple processes
slide-3
SLIDE 3

Consistent Snapshot

  • AssumpHon about monitors
  • At runHme: evaluate predicates in consistent snapshot
  • If true, report

hWp://slideplayer.com/slide/9232333/

slide-4
SLIDE 4

Requirements for Monitoring Distributed Systems

  • Wait-free
  • Low overhead Hmestamp
slide-5
SLIDE 5

Requirements for Monitoring Distributed Systems

  • Wait-free
  • Low overhead Hmestamp

ASYNC SYNC Par*ally SYNC

Clocks No wall clock No clock dri\ Bounded clock dri\ Wait-Free Low overhead *mestamp

slide-6
SLIDE 6

Monitoring with Partial Synchrony

  • Pros:
  • Wait-free
  • Low overhead Hmestamp
  • Cons:
  • You do not know uncertainty window
  • Need to understand: Precision, Recall, SensiHvity
slide-7
SLIDE 7

Outline

  • IntroducHon
  • MoHvaHng Example
  • System Model
  • Precision, Recall and SensiHvity of Asynchronous Monitors
  • Precision, Recall, and SensiHvity of ParHally Synchronous Monitors
  • Quasi-Synchronous Monitors
  • Conclusion
slide-8
SLIDE 8

Inherent Uncertainty in Evaluating Distributed Predicates (1)

  • Is predicate x > 0 ∧ y > 0 true
  • No

x > 0 y > 0 message m

slide-9
SLIDE 9

Inherent Uncertainty in Evaluating Distributed Predicates (2)

  • Is predicate x > 0 ∧ y > 0 true
  • It could be
  • We focus on such `possible’ modality

x > 0 y > 0

slide-10
SLIDE 10

Inherent Uncertainty in Evaluating Distributed Predicates (2)

  • Is predicate x > 0 ∧ y > 0 true
  • It could be
  • We focus on such `possible’ modality

x > 0 y > 0 x > 0 y > 0

slide-11
SLIDE 11

Inherent Uncertainty in Evaluating Distributed Predicates (3)

  • Is predicate x > 0 ∧ y > 0 true
  • It depends upon assumpHons

about clock synchronizaHon

  • Clocks of P1 and P2 differ by at

most 5

x > 0 y > 0 “Implicit Hmeout mechanism” [10..20] on P1 [30..40] on P2

slide-12
SLIDE 12

Inherent Uncertainty in Evaluating Distributed Predicates (4)

  • Is predicate x > 0 ∧ y > 0 true
  • Asynchronous: (Possibly) True

x > 0 y > 0

slide-13
SLIDE 13

Inherent Uncertainty in Evaluating Distributed Predicates (4)

  • Is predicate x > 0 ∧ y > 0 true
  • Asynchronous: (Possibly) True
  • Synchronous: False

x > 0 y > 0

slide-14
SLIDE 14

Inherent Uncertainty in Evaluating Distributed Predicates (4)

  • Is predicate x > 0 ∧ y > 0 true
  • Asynchronous: (Possibly) True
  • Synchronous: False
  • ParHally Synchronous: depends on

uncertainty bound

x > 0 y > 0

slide-15
SLIDE 15

Inherent Uncertainty in Evaluating Distributed Predicates (4)

  • Is predicate x > 0 ∧ y > 0 true
  • Asynchronous: (Possibly) True
  • Synchronous: False
  • ParHally Synchronous: depends on

synchronizaHon bound

  • If Hght, False

x > 0 y > 0 SynchronizaHon Bound

slide-16
SLIDE 16

Inherent Uncertainty in Evaluating Distributed Predicates (4)

  • Is predicate x > 0 ∧ y > 0 true
  • Asynchronous: (Possibly) True
  • Synchronous: False
  • ParHally Synchronous: depends on

synchronizaHon bound

  • If Hght, False
  • If loose, True

x > 0 y > 0 SynchronizaHon Bound

slide-17
SLIDE 17

Inherent Uncertainty in Evaluating Distributed Predicates (5)

  • Is predicate x > 0 ∧ y > 0 true
  • Depends upon clock

synchronizaHon assumpHons made in the applicaHon

  • To work correctly, monitor must

also make same assumpHons as those made in the applicaHon

  • What if these assumpHons are

ad-hoc

  • It would be impossible to match it

perfectly

x > 0 y > 0

slide-18
SLIDE 18

Our focus: analyze the errors caused by uncertainty of the underlying distributed system and the impedance mismatch between the monitor and the applica+on

slide-19
SLIDE 19

Impedance Mismatch

  • Is predicate x > 0 ∧ y > 0 true
  • (Monitor, App) à (async, sync)
  • False posiHve
  • (Monitor, App) à (sync, async)
  • False negaHve
  • IntuiHon
  • ApplicaHon is the ground truth
  • Monitor’s job is to match with

ApplicaHon

x > 0 y > 0

slide-20
SLIDE 20

3 Problems

  • (1) Asynchronous Monitors
  • Monitor assumes asynchronous

system

  • But, applicaHon is parHally

synchronized

  • Characterize Precision/Recall?
  • (3) Quasi-Synchronous Monitors
  • Approximate synchronous monitors

How sensi*ve to the varia*on of uncertainty bound ?

  • (2) ParHally Synchronous Monitors
  • Monitor assumes ε1
  • But, applicaHon relies on ε2
  • Characterize Precision/Recall?
slide-21
SLIDE 21

Outline

  • IntroducHon
  • MoHvaHng Example
  • System Model
  • Precision, Recall and SensiHvity of Asynchronous Monitors
  • Precision, Recall, and SensiHvity of ParHally Synchronous Monitors
  • Quasi-Synchronous Monitors
  • Conclusion
slide-22
SLIDE 22

System Model

  • A set of n processes that communicate via messages
  • εapp = clock synchronizaHon assumpHon made by the applicaHon
  • εmon= clock synchronizaHon assumpHon made by the monitor
  • δmin = minimum message delay
  • δmax= maximum message delay (always ∞ in this discussion)
  • α = Probability of sending a message to others (random recipient)
slide-23
SLIDE 23

System Model (2)

  • Focus on detecHon of conjuncHon of local predicates
  • Predicate being detected is ∧ prj
  • prj local predicate on process j
  • Other predicates can be expressed in terms of disjuncHon of such conjuncHve predicates
  • Parameter
  • β : Probability that prj is true
  • L : length of interval for which prj is true
  • We consider cases where β of different processes are independent/dependent
  • Monitors are responsible for ensuring that they combine local snapshots appropriately

(e.g., to avoid inconsistencies)

slide-24
SLIDE 24

Outline

  • IntroducHon
  • MoHvaHng Example
  • System Model
  • Precision, Recall and SensiHvity of Asynchronous Monitors
  • Precision, Recall, and SensiHvity of ParHally Synchronous Monitors
  • Quasi-Synchronous Monitors
  • Conclusion
slide-25
SLIDE 25

Precision and Sensitivity of Asynchronous Monitors

  • Assume that εmon = ∞
  • Only false posiHves are possible
slide-26
SLIDE 26

Precision and Sensitivity of Asynchronous Monitors

  • Assume that εmon = ∞
  • Only false posiHves are possible
  • Theorem 1: Precision is
  • Remark: Independent of communicaHon paWern

P = ((1 − (1 − β)✏+`−1)n−1

slide-27
SLIDE 27

Precision

slide-28
SLIDE 28

Precision, Sensitivity

slide-29
SLIDE 29

Precision, Sensitivity

slide-30
SLIDE 30

CriHcal points: εp1, εp2

Precision, Sensitivity

slide-31
SLIDE 31

Sensitivity of Asynchronous Monitors

0 εp1 εp2 ∞

slide-32
SLIDE 32

Sensitivity of Asynchronous Monitors

0 εp1 εp2 ∞

Not effecHve, not sensiHve

log1−β(3n − 4 − √ 5n2 − 16n + 12 2(n − 1)2 )

slide-33
SLIDE 33

Sensitivity of Asynchronous Monitors

0 εp1 εp2 ∞

EffecHve Not SensiHve

log1−β(3n − 4 + √ 5n2 − 16n + 12 2(n − 1)2 )

slide-34
SLIDE 34

Sensitivity of Asynchronous Monitors

Not effecHve, not sensiHve HypersensiHve RelaHvely, small range EffecHve Not SensiHve

lim

n→∞

✏p2 − ✏p1 ✏p1 = 0

slide-35
SLIDE 35

Validation with Simulation

  • 0.5

0.5 1 1.5 10000 20000 30000 40000 50000 60000 70000 80000 90000 100000 False positive rate Time n = 20, α=0.05, δ=10, β=0.03 ε=50 ε=100 ε=200

0.2 0.4 0.6 0.8 1 20 40 60 80 100 120 140 False positive rate ε n = 20, β=0.1 analytical model simulation α=0.05 δ=10 α=0.1 δ=10 α=0.05 δ=100 α=0.1 δ=100

slide-36
SLIDE 36

Validation with Simulation (2)

  • Effect of changing n, β and ε
slide-37
SLIDE 37

Correlated events

  • Examples: PosiHve Majority

CorrelaHon (PMAJ)

  • Process 1 has Beta probability
  • Process i’s event depends on
  • Majority of [1, 2 …, i-1] with 50%

Chance

  • Beta probability with another 50%
  • In long run,
  • the distribuHon converges to
  • Uniform analyHcal model !
  • We have considered many other

correlaHons with similar results

0.2 0.4 0.6 0.8 1 50 100 150 200 250 300 False positive rate ε PMAJ distribution n=20 α=0.01 β=0.10 δ=100 Analytical(β/2,n/4) Simulation

slide-38
SLIDE 38

Outline

  • IntroducHon
  • MoHvaHng Example
  • System Model
  • Precision, Recall and SensiHvity of Asynchronous Monitors
  • Precision, Recall, and SensiHvity of ParHally Synchronous Monitors
  • Quasi-Synchronous Monitors
  • Conclusion
slide-39
SLIDE 39

Precision, Recall, and Sensitivity of Partially Synchronous Monitors

  • What if εapp differs from ε mon
  • False posiHves can occur if εapp < ε mon
  • False negaHves can occur if ε mon < εapp
  • We use same analyHcal model and compute Precision and Recall and

sensiHvity

slide-40
SLIDE 40

Precision and Recall

  • Theorem 3:

ImplicaHon: CondiHon for getng high precision and recall at the same Hme QuesHon: How sensiHve?

slide-41
SLIDE 41

Precision and Recall Diagram

20 40 60 80 100 120 140 160 20 40 60 80 100 120 140 εapp εmon Point-based precision (P) and recall (R) β=0.05, α=0.01, δ=10, n = 5 and 20 n=5 R=0.9 P=0.9 R=0.8 P=0.8 100 % n=20 R=0.9 P=0.9 R=0.8 P=0.8

slide-42
SLIDE 42

Sensitivity

slide-43
SLIDE 43

Take away: partially synchronous monitor

  • Use analyHcal model to check if hypersensiHvity is avoidable
  • If not, the monitor must decide which one is more important and

sacrifice the other one

  • Safety predicates vs. Liveness/progress predicates
slide-44
SLIDE 44

Outline

  • IntroducHon
  • MoHvaHng Example
  • System Model
  • Precision, Recall and SensiHvity of Asynchronous Monitors
  • Precision, Recall, and SensiHvity of ParHally Synchronous Monitors
  • Quasi-Synchronous Monitors
  • Conclusion
slide-45
SLIDE 45

Quasi Synchronous Model

  • If local predicates are true exactly

at the same Hme then the global predicate is true

  • Perfect clock synchronizaHon is

hard

  • What if we had a simulated clock that

had the same property?

  • You could use Lamport’s logical clocks,
  • r
  • Hybrid logical clocks that provide

similar property while staying close to physical clock

x > 0 y > 0

slide-46
SLIDE 46

Quasi Synchronous Model (2)

  • If monitor finds that local predicates are true at the same `simulated

clock’ value then it declares that the global predicate is true

  • Only suffers from false negaHves
  • Can be pracHcal if local predicates are true for an `interval’
  • Longer the beWer
slide-47
SLIDE 47

Quasi Synchronous Simulation

  • Impact of interval length

Recall

slide-48
SLIDE 48

Conclusion

  • There is an inherent uncertainty in distributed systems
  • If assumpHons made by the monitor differ from those made in the

applicaHon, monitor can make an error

  • We characterized Precision/Recall and SensiHvity of monitors

designed for

  • Asynchronous systems
  • ParHally synchronous systems
  • Quasi-synchronous systems
slide-49
SLIDE 49

Questions

slide-50
SLIDE 50

Appendix

slide-51
SLIDE 51

Requirements for monitoring distributed systems

  • Wait-free
  • Low overhead for Hmestamp
  • Able to idenHfy all consistent snapshots
  • Neither asynchronous nor synchronous systems saHsfy all three

requirements simultaneously

slide-52
SLIDE 52

Asynchronous System

Wait-free Low overhead for Hmestamp Able to idenHfy all consistent snapshots

No wall clocks Pros:

  • Wait-free
  • Able to idenHfy all consistent snapshots

Cons:

  • High overhead Hmestamp (Vector Clocks) [J. Fidge, 1988]
  • VC of size n is necessary [B. Charron-Bost, 1991]
slide-53
SLIDE 53

Synchronous System

Wait-free Low overhead for Hmestamp Able to idenHfy all consistent snapshots

Clock-dri\ = 0 For instance, Google’s Spanner [J. CorbeW et al., 2012] Pros:

  • Low overhead for Hmestamp
  • Strong consistency

Cons:

  • Wait-out synchronizaHon bound to ensure consistency
slide-54
SLIDE 54

ParHally Synchronous System

Wait-free Low overhead for Hmestamp Able to idenHfy all consistent snapshots

Clock-dri\ is bounded Loosely-synchronized clocks e.g., NTP protocol Pros:

  • Wait-free
  • Low overhead for Hmestamp (Hybrid Vector Clocks)
  • HVC size is small for most systems [Yingchareonthawornchai et al., 2015]
  • Able to idenHfy all consistent snapshots
slide-55
SLIDE 55

ParHally Synchronous System

Wait-free Low overhead for Hmestamp Able to idenHfy all consistent snapshots

Clock-dri\ is bounded Loosely-synchronized clocks e.g., NTP protocol Pros:

  • Wait-free
  • Low overhead for Hmestamp (Hybrid Vector Clocks)
  • HVC size is small for most systems [Yingchareonthawornchai et al., 2015]
  • Able to idenHfy all consistent snapshots

Cons:

  • You do not know uncertainty bound
  • Need to understand: Precision, Recall, SensiHvity