Leveraging Existing Instrumentation to Automatically Infer - - PowerPoint PPT Presentation
Leveraging Existing Instrumentation to Automatically Infer Invariant-Constrained Models Ivan Beschastnikh Yuriy Brun Sigurd Schneider Michael Sloan University of Washington Saarland University Michael D. Ernst 1 Synoptic: * ... Mining
Ivan Beschastnikh Yuriy Brun Sigurd Schneider Michael Sloan Michael D. Ernst
Saarland University University of Washington
1
...
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Ivan Beschastnikh Yuriy Brun Sigurd Schneider Michael Sloan Michael D. Ernst
Saarland University University of Washington
* http://en.wikipedia.org/wiki/File:DFAexample.svg *
2
3
(IN
4
7
Manually defined
Concise and exact
Trivial instrumentation A low-level view
7
8
Manually defined
Concise and exact
Trivial instrumentation A low-level view
8
9
Input
Output
Tool
Dallmeier et al. WODA 2006
Prior work:
Cook et al. TSE 1998 Lo et al. ASE 2010
9
10
Input
Output
propose commit tx-commit tx-abort abort commit abort
Does not require access to code Uses refinement instead of coarsening Finds a compact accurate model
Invariants
Mines and preserves temporal log invariants
10
11
11
12
12
13
13
14
Two phase commit protocol log
each replica replies with an abort or a commit.
i) TX commit if all replicas commit ii) TX abort otherwise Manager maintains a totally ordered log of events for all transactions in the system
src : 2, dst : 0, timestamp : 0, type : prepare src : 2, dst : 1, timestamp : 1, type : prepare src : 0, dst : 2, timestamp : 2, type : commit src : 1, dst : 2, timestamp : 3, type : commit src : 2, dst : 0, timestamp : 4, type : tx_commit src : 2, dst : 1, timestamp : 5, type : tx_commit src : 0, dst : 2, timestamp : 6, type : ack src : 1, dst : 2, timestamp : 7, type : ack src : 2, dst : 0, timestamp : 8, type : prepare src : 2, dst : 1, timestamp : 9, type : prepare src : 0, dst : 2, timestamp : 10, type : commit src : 1, dst : 2, timestamp : 11, type : commit src : 2, dst : 0, timestamp : 12, type : tx_commit src : 2, dst : 1, timestamp : 13, type : tx_commit src : 0, dst : 2, timestamp : 14, type : ack src : 1, dst : 2, timestamp : 15, type : ack src : 2, dst : 0, timestamp : 16, type : prepare src : 2, dst : 1, timestamp : 17, type : prepare src : 0, dst : 2, timestamp : 18, type : commit src : 1, dst : 2, timestamp : 19, type : commit src : 2, dst : 0, timestamp : 20, type : tx_commit src : 2, dst : 1, timestamp : 21, type : tx_commit src : 0, dst : 2, timestamp : 22, type : ack src : 1, dst : 2, timestamp : 23, type : ack src : 2, dst : 0, timestamp : 0, type : prepare src : 2, dst : 1, timestamp : 1, type : prepare src : 0, dst : 2, timestamp : 2, type : commit src : 1, dst : 2, timestamp : 3, type : commit src : 2, dst : 0, timestamp : 4, type : tx_commit src : 2, dst : 1, timestamp : 5, type : tx_commit src : 0, dst : 2, timestamp : 6, type : ack src : 1, dst : 2, timestamp : 7, type : ack src : 2, dst : 0, timestamp : 8, type : prepare src : 2, dst : 1, timestamp : 9, type : prepare src : 0, dst : 2, timestamp : 10, type : commit src : 1, dst : 2, timestamp : 11, type : commit src : 2, dst : 0, timestamp : 12, type : tx_commit src : 2, dst : 1, timestamp : 13, type : tx_commit src : 0, dst : 2, timestamp : 14, type : ack src : 1, dst : 2, timestamp : 15, type : ack src : 2, dst : 0, timestamp : 16, type : prepare src : 2, dst : 1, timestamp : 17, type : preparelog.txt
2PC System
14
15
log.txt
2PC System
15
16
1/5. Parse log into a trace graph 2/5. Construct the initial model 3/5. Mine temporal invariants
1/5. Parse log into a trace graph
propose propose abort commit tx-abort tx-abort propose propose commit abort tx-abort tx-abort propose propose commit commit tx-commit tx-commit
: initial event : terminal event : intermediate event
X Y Z
16
16
1/5. Parse log into a trace graph 2/5. Construct the initial model 3/5. Mine temporal invariants
propose
A compact model with
2/5. Construct the initial model
propose propose abort commit tx-abort tx-abort propose propose commit abort tx-abort tx-abort propose propose commit commit tx-commit tx-commit
16
16
1/5. Parse log into a trace graph 2/5. Construct the initial model 3/5. Mine temporal invariants
propose commit
A compact model with
2/5. Construct the initial model
propose propose abort commit tx-abort tx-abort propose propose commit abort tx-abort tx-abort propose propose commit commit tx-commit tx-commit
16
16
1/5. Parse log into a trace graph 2/5. Construct the initial model 3/5. Mine temporal invariants
propose abort tx-commit tx-abort commit
A compact model with
2/5. Construct the initial model
propose propose abort commit tx-abort tx-abort propose propose commit abort tx-abort tx-abort propose propose commit commit tx-commit tx-commit
16
16
1/5. Parse log into a trace graph 2/5. Construct the initial model 3/5. Mine temporal invariants
commit tx-commit abort propose tx-abort
A compact model with
2/5. Construct the initial model
propose propose abort commit tx-abort tx-abort propose propose commit abort tx-abort tx-abort propose propose commit commit tx-commit tx-commit
16
16
1/5. Parse log into a trace graph 2/5. Construct the initial model 3/5. Mine temporal invariants
A compact model with
commit tx-commit abort propose tx-abort
2/5. Construct the initial model
propose propose abort commit tx-abort tx-abort propose propose commit abort tx-abort tx-abort propose propose commit commit tx-commit tx-commit propose propose abort commit tx-abort tx-abort propose propose commit abort tx-abort tx-abort propose propose commit commit tx-commit tx-commit
16
16
1/5. Parse log into a trace graph 2/5. Construct the initial model 3/5. Mine temporal invariants
A compact model with
3/5. Mine temporal invariants
commit tx-commit abort propose tx-abort
abort tx-abort abort tx-commit abort tx-abort
propose propose abort commit tx-abort tx-abort propose propose commit abort tx-abort tx-abort propose propose commit commit tx-commit tx-commit
16
17
5/5. Coarsen model without unsatisfying any invariants 4/5. Refine the initial model until all invariants satisfied 4/5. Refine the initial model until all invariants satisfied Choose an invariant invalid in the model
abort tx-abort
True for log, false for initial model
17
17
5/5. Coarsen model without unsatisfying any invariants 4/5. Refine the initial model until all invariants satisfied 4/5. Refine the initial model until all invariants satisfied Choose an invariant invalid in the model
abort tx-abort
Refine a model node
commit tx-commit abort propose tx-abort
commit tx-commit abort propose tx-abort commit
Unsatisfied invariants exist
17
17
5/5. Coarsen model without unsatisfying any invariants 4/5. Refine the initial model until all invariants satisfied 5/5. Coarsen model without unsatisfying any invariants Choose an invariant invalid in the model
abort tx-abort
Refine a model node
commit tx-commit abort propose tx-abort
commit tx-commit abort propose tx-abort commit
Unsatisfied invariants exist All invariants satisfied
17
17
5/5. Coarsen model without unsatisfying any invariants 4/5. Refine the initial model until all invariants satisfied 5/5. Coarsen model without unsatisfying any invariants Choose an invariant invalid in the model
abort tx-abort
Refine a model node
commit tx-commit abort propose tx-abort
commit tx-commit abort propose tx-abort commit
Unsatisfied invariants exist All invariants satisfied
propose commit tx-commit tx-abort abort commit abort abort
Merge nodes that exhibit the same behaviors
propose commit tx-commit tx-abort abort commit abort
17
among events
18
Logs System
18
19
19
Invariant Example Type
x y
always followed by
liveness x y
always precedes
safety x y
never followed by
safety
that are true for all the logged executions
Dwyer et al. ICSE 1999
20
x y y x z x x y x x y y y z X X Y y X Y X Y y y y
Yang et al. PASTE 2004
20
Invariant Example Type
x y
always followed by
liveness x y
always precedes
safety x y
never followed by
safety
that are true for all the logged executions
Dwyer et al. ICSE 1999
20
x x y y y z X X Y y X Y X Y y y y X Y Y x z x X Y
Yang et al. PASTE 2004
20
Invariant Example Type
x y
always followed by
liveness x y
always precedes
safety x y
never followed by
safety
that are true for all the logged executions
Dwyer et al. ICSE 1999
20
X X Y y X Y X Y y y y X Y Y x z x X Y X X y y y z
Yang et al. PASTE 2004
20
21
abort, commit, tx-abort, tx-commit
abort, commit, tx-abort, tx-commit
18
propose propose abort commit tx-abort tx-abort propose propose commit commit tx-commit tx-commit
21
22
abort, commit, tx-abort, tx-commit
abort, commit, tx-abort, tx-commit
mined invariants:
18
a corresponding path
eliminate such path counter-examples
commit tx-commit abort propose tx-abort
One node per event type
22
Example invariant: abort tx-abort
23
commit tx-commit abort propose tx-abort
1.
Initial model
commit tx-commit abort propose tx-abort
2.
Invariant counter-example Identify partition to refine
commit tx-commit abort propose tx-abort commit
3.
commit commit
Eliminate counter-example
commit tx-commit abort propose tx-abort commit
4.
commit commit
Refined model
commit tx-commit abort propose tx-abort
5.
commit
23
...
24
c1 c3 B a1 a2 c2 2 3 1,4 a3 a4 5 c1 c3 B a2 c2 3 4 a3 a4 5 2 a1 1 c1 c3 B a2 c2 3 4 a3 5 a1 1 a4 2 c1 c3 B a1 a2 c2 3 1,4 a4 5 a3 2 a1 a3 a2 B C 1 2 3 4 a4 5 a1 a3 a2 B c1 c3 1 2 3 4 a4 5 c2 c1 c3 a1 a3 a2 a4 B c2 2 1 4 3,5 c1 c3 B a2 a4 c2 3,5 4 a1 1 a3 2 a1 B 3 1 C a3 a4 5 2 a2 4 c1 c3 B A c2 2 3,5 1,4 a1 a3 a2 a4 B C 1 2 3,5 4 a1 a2 a4 B C a3 3,5 2 1,4 a1 a2 B 1 2 3 4 a4 5 C a3 a2 a4 B C a1 a3 2 3,5 4 1 a1 a2 a4 B c1 c3 a3 c2 2 1,4 3,5 a1 a2 B C a4 a3 1,4 3 5 2 a1 a2 B C a3 a4 3 5 2 1,4 B C A 1,4 2 3,5Larger models: fewer behaviors Smaller models: more behaviors Initial model Trace graph
24
...
24
c1 c3 B a1 a2 c2 2 3 1,4 a3 a4 5 c1 c3 B a2 c2 3 4 a3 a4 5 2 a1 1 c1 c3 B a2 c2 3 4 a3 5 a1 1 a4 2 c1 c3 B a1 a2 c2 3 1,4 a4 5 a3 2 a1 a3 a2 B C 1 2 3 4 a4 5 a1 a3 a2 B c1 c3 1 2 3 4 a4 5 c2 c1 c3 a1 a3 a2 a4 B c2 2 1 4 3,5 c1 c3 B a2 a4 c2 3,5 4 a1 1 a3 2 a1 B 3 1 C a3 a4 5 2 a2 4 c1 c3 B A c2 2 3,5 1,4 a1 a3 a2 a4 B C 1 2 3,5 4 a1 a2 a4 B C a3 3,5 2 1,4 a1 a2 B 1 2 3 4 a4 5 C a3 a2 a4 B C a1 a3 2 3,5 4 1 a1 a2 a4 B c1 c3 a3 c2 2 1,4 3,5 a1 a2 B C a4 a3 1,4 3 5 2 a1 a2 B C a3 a4 3 5 2 1,4 B C A 1,4 2 3,5Larger models: fewer behaviors Smaller models: more behaviors Initial model Trace graph
24
...
24
c1 c3 B a1 a2 c2 2 3 1,4 a3 a4 5 c1 c3 B a2 c2 3 4 a3 a4 5 2 a1 1 c1 c3 B a2 c2 3 4 a3 5 a1 1 a4 2 c1 c3 B a1 a2 c2 3 1,4 a4 5 a3 2 a1 a3 a2 B C 1 2 3 4 a4 5 a1 a3 a2 B c1 c3 1 2 3 4 a4 5 c2 c1 c3 a1 a3 a2 a4 B c2 2 1 4 3,5 c1 c3 B a2 a4 c2 3,5 4 a1 1 a3 2 a1 B 3 1 C a3 a4 5 2 a2 4 c1 c3 B A c2 2 3,5 1,4 a1 a3 a2 a4 B C 1 2 3,5 4 a1 a2 a4 B C a3 3,5 2 1,4 a1 a2 B 1 2 3 4 a4 5 C a3 a2 a4 B C a1 a3 2 3,5 4 1 a1 a2 a4 B c1 c3 a3 c2 2 1,4 3,5 a1 a2 B C a4 a3 1,4 3 5 2 a1 a2 B C a3 a4 3 5 2 1,4 B C A 1,4 2 3,5Some invariants falsified
Larger models: fewer behaviors Smaller models: more behaviors Initial model Trace graph
24
...
24
c1 c3 B a1 a2 c2 2 3 1,4 a3 a4 5 c1 c3 B a2 c2 3 4 a3 a4 5 2 a1 1 c1 c3 B a2 c2 3 4 a3 5 a1 1 a4 2 c1 c3 B a1 a2 c2 3 1,4 a4 5 a3 2 a1 a3 a2 B C 1 2 3 4 a4 5 a1 a3 a2 B c1 c3 1 2 3 4 a4 5 c2 c1 c3 a1 a3 a2 a4 B c2 2 1 4 3,5 c1 c3 B a2 a4 c2 3,5 4 a1 1 a3 2 a1 B 3 1 C a3 a4 5 2 a2 4 c1 c3 B A c2 2 3,5 1,4 a1 a3 a2 a4 B C 1 2 3,5 4 a1 a2 a4 B C a3 3,5 2 1,4 a1 a2 B 1 2 3 4 a4 5 C a3 a2 a4 B C a1 a3 2 3,5 4 1 a1 a2 a4 B c1 c3 a3 c2 2 1,4 3,5 a1 a2 B C a4 a3 1,4 3 5 2 a1 a2 B C a3 a4 3 5 2 1,4 B C A 1,4 2 3,5All invariants satisfied Some invariants falsified
Larger models: fewer behaviors Smaller models: more behaviors Initial model Trace graph
24
...
24
c1 c3 B a1 a2 c2 2 3 1,4 a3 a4 5 c1 c3 B a2 c2 3 4 a3 a4 5 2 a1 1 c1 c3 B a2 c2 3 4 a3 5 a1 1 a4 2 c1 c3 B a1 a2 c2 3 1,4 a4 5 a3 2 a1 a3 a2 B C 1 2 3 4 a4 5 a1 a3 a2 B c1 c3 1 2 3 4 a4 5 c2 c1 c3 a1 a3 a2 a4 B c2 2 1 4 3,5 c1 c3 B a2 a4 c2 3,5 4 a1 1 a3 2 a1 B 3 1 C a3 a4 5 2 a2 4 c1 c3 B A c2 2 3,5 1,4 a1 a3 a2 a4 B C 1 2 3,5 4 a1 a2 a4 B C a3 3,5 2 1,4 a1 a2 B 1 2 3 4 a4 5 C a3 a2 a4 B C a1 a3 2 3,5 4 1 a1 a2 a4 B c1 c3 a3 c2 2 1,4 3,5 a1 a2 B C a4 a3 1,4 3 5 2 a1 a2 B C a3 a4 3 5 2 1,4 B C A 1,4 2 3,5All invariants satisfied Some invariants falsified
Larger models: fewer behaviors Smaller models: more behaviors
Find the smallest model satisfying all invariants
Synoptic’s goal:
Initial model Trace graph
24
...
24
c1 c3 B a1 a2 c2 2 3 1,4 a3 a4 5 c1 c3 B a2 c2 3 4 a3 a4 5 2 a1 1 c1 c3 B a2 c2 3 4 a3 5 a1 1 a4 2 c1 c3 B a1 a2 c2 3 1,4 a4 5 a3 2 a1 a3 a2 B C 1 2 3 4 a4 5 a1 a3 a2 B c1 c3 1 2 3 4 a4 5 c2 c1 c3 a1 a3 a2 a4 B c2 2 1 4 3,5 c1 c3 B a2 a4 c2 3,5 4 a1 1 a3 2 a1 B 3 1 C a3 a4 5 2 a2 4 c1 c3 B A c2 2 3,5 1,4 a1 a3 a2 a4 B C 1 2 3,5 4 a1 a2 a4 B C a3 3,5 2 1,4 a1 a2 B 1 2 3 4 a4 5 C a3 a2 a4 B C a1 a3 2 3,5 4 1 a1 a2 a4 B c1 c3 a3 c2 2 1,4 3,5 a1 a2 B C a4 a3 1,4 3 5 2 a1 a2 B C a3 a4 3 5 2 1,4 B C A 1,4 2 3,5All invariants satisfied Some invariants falsified
Larger models: fewer behaviors Smaller models: more behaviors
Coarsening Refinement
Find the smallest model satisfying all invariants
Synoptic’s goal:
Initial model Trace graph
24
25
c1 c3 B a1 a2 c2 2 3 1,4 a3 a4 5 c1 c3 B a2 c2 3 4 a3 a4 5 2 a1 1 c1 c3 B a2 c2 3 4 a3 5 a1 1 a4 2 c1 c3 B a1 a2 c2 3 1,4 a4 5 a3 2 a1 a3 a2 B C 1 2 3 4 a4 5 a1 a3 a2 B c1 c3 1 2 3 4 a4 5 c2 c1 c3 a1 a3 a2 a4 B c2 2 1 4 3,5 c1 c3 B a2 a4 c2 3,5 4 a1 1 a3 2 a1 B 3 1 C a3 a4 5 2 a2 4 c1 c3 B A c2 2 3,5 1,4 a1 a3 a2 a4 B C 1 2 3,5 4 a1 a2 a4 B C a3 3,5 2 1,4 a1 a2 B 1 2 3 4 a4 5 C a3 a2 a4 B C a1 a3 2 3,5 4 1 a1 a2 a4 B c1 c3 a3 c2 2 1,4 3,5 a1 a2 B C a4 a3 1,4 3 5 2 a1 a2 B C a3 a4 3 5 2 1,4 B C A 1,4 2 3,5All invariants satisfied Some invariants satisfied
Refine Refine Refine Coarsen
Synoptic finds a local
a global one
Select a refinement that satisfies an unsatisfied invariant Select a coarsening that maintains all the invariants
Initial model
25
25
c1 c3 B a1 a2 c2 2 3 1,4 a3 a4 5 c1 c3 B a2 c2 3 4 a3 a4 5 2 a1 1 c1 c3 B a2 c2 3 4 a3 5 a1 1 a4 2 c1 c3 B a1 a2 c2 3 1,4 a4 5 a3 2 a1 a3 a2 B C 1 2 3 4 a4 5 a1 a3 a2 B c1 c3 1 2 3 4 a4 5 c2 c1 c3 a1 a3 a2 a4 B c2 2 1 4 3,5 c1 c3 B a2 a4 c2 3,5 4 a1 1 a3 2 a1 B 3 1 C a3 a4 5 2 a2 4 c1 c3 B A c2 2 3,5 1,4 a1 a3 a2 a4 B C 1 2 3,5 4 a1 a2 a4 B C a3 3,5 2 1,4 a1 a2 B 1 2 3 4 a4 5 C a3 a2 a4 B C a1 a3 2 3,5 4 1 a1 a2 a4 B c1 c3 a3 c2 2 1,4 3,5 a1 a2 B C a4 a3 1,4 3 5 2 a1 a2 B C a3 a4 3 5 2 1,4 B C A 1,4 2 3,5All invariants satisfied Some invariants satisfied
Refine Refine Refine Coarsen
Synoptic finds a local
a global one
Select a refinement that satisfies an unsatisfied invariant Select a coarsening that maintains all the invariants
Initial model
Biermann et al. 1972 Lorenzoli et al. ICSE 2008 Paige et al. 1987
25
26
26
27
See the paper for more
27
28
destination to a source host
KATZ-BASSETT et al. NSDI 2010
28
do_measurements do_fail_callback get_next_hop reverse_hops_assume_ symmetric do_measurements reverse_hops_assume_ symmetric_peek reverse_hops_ tr_to_src do_measurements get_next_hop check_cache reverse_hops_assume_ symmetric_peek reverse_hops_ tr_to_src do_reach_callback do_fail_callback get_next_hop do_reach_callback reverse_hops_ts reverse_hops_rr reverse_hops_assume_ symmetric
0.82 0.14 1.00 0.24 1.00 1.00 0.92 0.33 0.44 0.14 0.95 1.00 1.00 0.20 0.78 0.67 0.15 0.71 0.11 0.13 0.85 0.50 0.12 0.43 0.81 0.01 0.01 0.04 0.02 0.02 0.48 0.01
2 bugs found:
should not be terminal
should not exist
29
Shaded
Synoptic was useful for:
29
granularity of logging statements and user input
be irrelevant
30
30
31
Lou et al. KDD 2010 Gabel et al. FSE 2008 Ernst et al. TSE 2001 Vaarandi et al. IFIP ICICS 2004 Zhu et al. OSR 2010
31
32
and complex logs, and can help find bugs
Synoptic: a tool that extracts models from logs
32