Ultimate Referee, Ultimate Automizer, and Incremental Verification - - PowerPoint PPT Presentation
Ultimate Referee, Ultimate Automizer, and Incremental Verification Matthias Heizmann University of Freiburg CPAchecker Workshop 2019 Outline Running Example and Floyd-Hoare Annotations Ultimate Referee A strict proof checker.
Matthias Heizmann
University of Freiburg
CPAchecker Workshop 2019
A strict proof checker.
The verification approach of Ultimate Automizer
ℓ0:
assume p != 0;
ℓ1:
while(n >= 0)
{ ℓ2:
assert p != 0; if(n == 0)
{ ℓ3:
p := 0;
} ℓ4:
n--;
}
pseudocode
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
control flow graph
Definition:
{ ϕ } st { ϕ′ } is valid Hoare triple iff if program is in state that satisfies ϕ and program executes st then program is in a state that satisfies ϕ′
Example: { p = 0 ∨ n = −1 } n>=0 { p = 0 } is a valid Hoare triple
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
control flow graph
Definition: A Floyd-Hoare annotation is a map- ping that assigns each location ℓi a for- mula ϕi such that there is an edge ℓi ϕi
ℓj
ϕj
st
{ ϕ } st { ϕ′ } is valid. Proposition: Given a program P, if there is a Floyd- Hoare annotation such that ◮ every initial location is labeled with true and ◮ every error location is labeled with false then P is correct.
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
true p = 0 ∨ n = −1 p = 0 true false n = 0 p = 0 ∨ n = 0
control flow graph
A strict proof checker.
The verification approach of Ultimate Automizer
Correctness Witnesses: Control-flow graph annotated by invariants ◮ not required to annoted every location ◮ invariants to not have to be inductive ◮ invariants do not have to be sufficient
Correctness Witnesses: Control-flow graph annotated by invariants ◮ not required to annoted every location ◮ invariants to not have to be inductive ◮ invariants do not have to be sufficient Shortcomings of Ultimate Automizer as Witness validator ◮ Different tools have different notions of a control-flow graph we cannot always match invariants to the intended location.
Obstacles ◮ procedure entry values
Obstacles ◮ procedure entry values ◮ valid memory
Obstacles ◮ procedure entry values ◮ valid memory ◮ programs with gotos
A strict proof checker.
The verification approach of Ultimate Automizer
Matthias Heizmann, Jochen Hoenicke, and Andreas Podelski. “Refine- ment of Trace Abstraction”. In: SAS. vol. 5673. Lecture Notes in Com- puter Science. Springer, 2009, pp. 69–85 Matthias Heizmann, Jochen Hoenicke, and Andreas Podelski. “Nested interpolants”. In: POPL. ACM, 2010, pp. 471–482 Matthias Heizmann, Jochen Hoenicke, and Andreas Podelski. “Software Model Checking for People Who Love Automata”. In: CAV. vol. 8044. Lecture Notes in Computer Science. Springer, 2013, pp. 36–52
◮ trace sequence of statements ◮ error trace labeling along path from initial location to error location ◮ infeasible trace trace π such that Hoare triple
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0 p != 0 n >= 0 p == 0
program P examples infeasible feasible error trace of P
p != 0 n >= 0 p == 0
? not error trace of P
n == 0 n-- n == 0 n >= 0 n-- n == 0
Show that every error trace is infeasible. Decompose infeasible error traces into sets such that there is a “simple” infeasibility proof for each set.
Show that every error trace is infeasible. Decompose infeasible error traces into sets such that there is a “simple” infeasibility proof for each set. ◮ Reason 1: If we assume that p is not 0 and do not modify p then p cannot be 0. ◮ Reason 2: If we assume that n is 0 and we decrement n then n cannot be non-negative.
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
program P
Implementation based on automata theory Set of statements: alphabet of formal language here: Σ = { p != 0 ,
n >= 0 , n == 0 , p := 0 , n != 0 , p == 0 , n-- , n < 0 }
◮ Set of traces: automaton over the alphabet
◮ Control flow graph: automaton over the alphabet
◮ Error location: accepting state of this automaton ◮ Error trace of program: word accepted by this automaton
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
program P
ℓ0:
assume p != 0;
ℓ1:
while(n >= 0)
{ ℓ2:
assert p != 0; if(n == 0)
{ ℓ3:
p := 0;
} ℓ4:
n--;
}
pseudocode
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
control flow graph
ℓ0:
assume p != 0;
ℓ1:
while(n >= 0)
{ ℓ2:
assert p != 0; if(n == 0)
{ ℓ3:
p := 0;
} ℓ4:
n--;
}
pseudocode
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0 p != 0 n >= 0 p == 0
control flow graph
p != 0 n >= 0 p == 0
1: assume p != 0; 2: assume n >= 0; 3: assert p != 0; pseudocode of A1
p != 0 n >= 0 p == 0
p != 0 n >= 0 p == 0
true p = 0 p = 0 false
◮ add transitions
{ p = 0 }
n--
{ p = 0 } is valid Hoare triple p != 0 n >= 0 p == 0
true p = 0 p = 0 false
◮ add transitions
{ p = 0 }
n--
{ p = 0 } is valid Hoare triple { p = 0 }
n != 0
{ p = 0 } is valid Hoare triple p != 0 n >= 0 p == 0
true p = 0 p = 0 false
n--
◮ add transitions
{ p = 0 }
n--
{ p = 0 } is valid Hoare triple { p = 0 }
n != 0
{ p = 0 } is valid Hoare triple { p = 0 }
n >= 0
{ p = 0 } is valid Hoare triple p != 0 n >= 0 p == 0
true p = 0 p = 0 false
n-- n >= 0
◮ add transitions
p != 0 n >= 0 p == 0
true p = 0 p = 0 false
◮ add transitions
p != 0 n >= 0 p == 0
true p = 0 p = 0 false
all all \{ p := 0 } all
◮ add transitions ◮ merge locations
q0
true
q1
p = 0
q2
false
Σ Σ p != 0 p == 0 Σ\{ p := 0 }
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
program P
q0
true
q1
p = 0
q2
false
Σ Σ p != 0 p == 0 Σ\{ p := 0 }
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
program P
q0
true
q1
p = 0
q2
false
Σ Σ p != 0 p == 0 Σ\{ p := 0 }
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
program P
q0
true
q1
p = 0
q2
false
Σ Σ p != 0 p == 0 Σ\{ p := 0 }
Consider P and A1 as au- tomata and consider con- struct set theoretic differ- ence L(P)\L(A1). P A1
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
program P
q0 q1 q2 Σ Σ p != 0 p == 0 Σ\{ p := 0 }
Consider P and A1 as au- tomata and consider con- struct set theoretic differ- ence L(P)\L(A1). P A1
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0 p != 0 n >= 0 n == 0 p := 0 n-- p == 0
program P
q0 q1 q2 Σ Σ p != 0 p == 0 Σ\{ p := 0 }
Consider P and A1 as au- tomata and consider con- struct set theoretic differ- ence L(P)\L(A1). P A1
p != 0 n >= 0 n == 0 p := 0 n-- n >= 0 p == 0
p != 0 n >= 0 n == 0 p := 0 n-- n >= 0 p == 0
p != 0 n >= 0 n == 0 p := 0 n-- n >= 0 p == 0
true true true n = 0 n = 0 n = −1 false false
◮ add transitions ◮ merge locations
q0
true
q1
p = 0
q2
p = 0
q3
false
Σ Σ n == 0 n-- n >= 0 Σ\{ n-- } Σ\{ n-- }
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
program P
q0 q1 q2 Σ Σ p != 0 p == 0 Σ\{ p := 0 }
q0 q1 q2 q3 Σ Σ n == 0 n-- n >= 0 Σ\{ n-- } Σ\{ n-- }
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
program P
q0 q1 q2 Σ Σ p != 0 p == 0 Σ\{ p := 0 }
q0 q1 q2 q3 Σ Σ n == 0 n-- n >= 0 Σ\{ n-- } Σ\{ n-- }
program P “P is correct” “P is incorrect” L(P) ⊆ L(A1) ∪ · · · ∪ L(An) is π feasible ? no pick new error trace π no construct infeasiblity proof for π construct generalized automaton Ai yes yes
program P “P is correct” “P is incorrect” L(P) ⊆ L(A1) ∪ · · · ∪ L(An) is π feasible ? no pick new error trace π no construct infeasiblity proof for π construct generalized automaton Ai yes yes
program P “P is correct” “P is incorrect” L(P) ⊆ L(A1) ∪ · · · ∪ L(An) is π feasible ? no pick new error trace π no construct infeasiblity proof for π construct generalized automaton Ai yes yes
program P “P is correct” “P is incorrect” L(P) ⊆ L(A1) ∪ · · · ∪ L(An) is π feasible ? no pick new error trace π no construct infeasiblity proof for π construct generalized automaton Ai yes yes
program P “P is correct” “P is incorrect” L(P) ⊆ L(A1) ∪ · · · ∪ L(An) is π feasible ? no pick new error trace π no construct infeasiblity proof for π construct generalized automaton Ai yes yes
A strict proof checker.
The verification approach of Ultimate Automizer
verify program P construct modified program P′ verify program P′ construct modified program P′′ verify program P′′ . . .
verify program P construct modified program P′ verify program P′ construct modified program P′′ verify program P′′ . . . Which information can we reuse while verifying via trace abstraction?
Bat-Chen Rothenberg Daniel Dietsch Matthias Heizmann
Bat-Chen Rothenberg, Daniel Dietsch, and Matthias Heizmann. “Incre- mental Verification Using Trace Abstraction”. In: SAS. vol. 11002. Lec- ture Notes in Computer Science. Springer, 2018, pp. 364–382
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0 ℓ′ p==0 n:=-2
program Pnew with Σnew = Σ ∪ { n:=-2 }
q0 q1 q2 Σ Σ p != 0 p == 0 Σ\{ p := 0 }
Floyd-Hoare automaton A1
q0 q1 q2 q3 Σ Σ n == 0 n-- n >= 0 Σ\{ n-- } Σ\{ n-- }
Floyd-Hoare automaton A2
Pnew ∩ A1 ∩ A2 = ∅ ?
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0 ℓ′ p==0 n:=-2
program Pnew with Σnew = Σ ∪ { n:=-2 }
q0 q1 q2 Σ Σ p != 0 p == 0 Σ\{ p := 0 }
Floyd-Hoare automaton A1
q0 q1 q2 q3 Σ Σ n == 0 n-- n >= 0 Σ\{ n-- } Σ\{ n-- }
Floyd-Hoare automaton A2
Pnew ∩ A1 ∩ A2 = ∅ ?
No! Counterexample to emptiness: π =
p==0 n:=-2 n ≥ 0 p==0
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0 ℓ′ p==0 n:=-2
program Pnew with Σnew = Σ ∪ { n:=-2 }
q0 q1 q2 Σ Σ p != 0 p == 0 Σ\{ p := 0 }
Floyd-Hoare automaton A1
q0 q1 q2 q3 Σ Σ n == 0 n-- n >= 0 Σ\{ n-- } Σ\{ n-- }
Floyd-Hoare automaton A2
r0 r1 r2
n:=-2 n>=0
Σnew Σnew\{ n-- } Σnew
Floyd-Hoare automaton A3
Pnew ∩ A1 ∩ A2 ∩ A3 = ∅ !
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p := 23 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0
program Pnew with Σnew = Σ ∪ { n:=-2 }
q0 q1 q2 Σ Σ p != 0 p == 0 Σ\{ p := 0 }
Floyd-Hoare automaton A1
q0 q1 q2 q3 Σ Σ n == 0 n-- n >= 0 Σ\{ n-- } Σ\{ n-- }
Floyd-Hoare automaton A2
Pnew ∩ A1 ∩ A2 = ∅ ?
No! Automata A1 and A2 are useless! Statement
p := 23 not in Σ.
Idea: extend Floyd-Hoare automata with new statements
q0
true
q1
p = 0
q2
false
p := 23 Σ Σ p != 0 p == 0 Σ\{ p := 0 }
Floyd-Hoare automaton A1
{ true } p := 23 { p = 0 } is valid Hoare triple
Idea: extend Floyd-Hoare automata with new statements
q0
true
q1
p = 0
q2
false
Σnew Σnew p := 23 p == 0 Σnew\{ p := 0 }
Floyd-Hoare automaton A1
{ true } p := 23 { p = 0 } is valid Hoare triple { true } p := 23 { true } is valid Hoare triple { false } p := 23 { false } is valid Hoare triple . . .
ℓ0 ℓ1 ℓ2 ℓ3 ℓ4 ℓ5 ℓerr p != 0 n >= 0 n == 0 p := 0 n != 0 p == 0 n-- n < 0 ℓ′ p==0 n:=-2
program Pnew with Σnew = Σ ∪ { n:=-2 }
q0 q1 q2 Σnew Σnew p := 23 p == 0 Σnew\{ p := 0 }
Floyd-Hoare automaton A1
q0 q1 q2 q3 Σnew Σnew n == 0 n-- n >= 0 Σnew\{ n-- } Σnew\{ n-- }
Floyd-Hoare automaton A2
Pnew ∩ A1 ∩ A2 = ∅ !
program P over Σnew Floyd-Hoare automata A1, · · · , Am over Σ P is correct Floyd-Hoare automata A1, · · · , An P is incorrect Floyd-Hoare automata A1, · · · , An extend A1, · · · , Am to Aext
1 , · · · , Aext m over Σnew
L(P ∩ A1 ∩ · · · ∩ An) = ∅ ? π is infeasible? no return trace π such that π ∈ L(P ∩ A1 ∩ · · · ∩ An) yes return Floyd-Hoare automaton An+1 such that π ∈ L(An+1) yes no ∀1 ≤ i ≤ m Ai := Aext
i
n := m
program P over Σnew Floyd-Hoare automata A1, · · · , Am over Σ P is correct Floyd-Hoare automata A1, · · · , An P is incorrect Floyd-Hoare automata A1, · · · , An extend A1, · · · , Am to Aext
1 , · · · , Aext m over Σnew
L(P ∩ A1 ∩ · · · ∩ An) = ∅ ? π is infeasible? ∃1 ≤ i ≤ m s.t. π ∈ L(Aext
i
)? no return trace π s.t. π ∈ L(P ∩ A1 ∩ · · · ∩ An) yes return Floyd-Hoare automaton An+1 such that π ∈ L(An+1) no yes An+1 = Aext
i
yes no n := 0
Saved costs: ◮ Analysis of (potentially spurious) counterexamples
checking feasibility, computation of interpolants
◮ Construction of Floyd-Hoare automata
checking Hoare triples
Saved costs: ◮ Analysis of (potentially spurious) counterexamples
checking feasibility, computation of interpolants
◮ Construction of Floyd-Hoare automata
checking Hoare triples
Additional costs: ◮ Larger automata
e.g., P ∩ A1 ∩ . . . ∩ An
◮ Extending automata
checking Hoare triples
◮ Reading and writing automata
I/O operations on hard drive, parsing automata
Implemented in the Ultimate Automizer software verifier ◮ http://ultimate.informatik.uni-freiburg.de/ ◮ Open source https://github.com/ultimate-pa/ultimate Automata written in the format of the Ultimate Automata Library
◮ Produced by the Linux Verification Center http://linuxtesting.org/ ◮ 4,193 verification tasks from 1,119 revisions of 62 device drivers ◮ Bechmark set used in related work
Dirk Beyer et al. “Precision reuse for efficient regression verification”. In: ESEC/SIGSOFT FSE. ACM, 2013, pp. 389–399 https://www.sosy-lab.org/research/cpa-reuse/regression-benchmarks
Default Eager Lazy Driver Spec Tasks Overall Overall Analysis Overall Analysis Speedup Overall Speedup Analysis [5] Speedup dvb-usb-rtl28xxu 08 1a 10 20.509 0.352 0.187 0.416 0.257 49.30 79.80 3.6 dvb-usb-rtl28xxu 39 7a 10 110.893 4.081 1.992 4.059 2.546 27.32 43.55 6.3 dvb-usb-rtl28xxu 32 7a 10 35.551 1.306 0.725 1.550 0.844 22.93 42.12 4.9 dvb-usb-az6007 08 1a 5 4.620 0.173 0.118 0.187 0.132 24.70 35.00 3.5 dvb-usb-az6007 39 7a 5 17.952 1.378 0.862 1.425 0.989 12.59 18.15 4.9 cx231xx-dvb 08 1a 13 3.330 0.303 0.206 0.323 0.228 10.30 14.60 1.8 panasonic-laptop 08 1a 16 3.466 0.337 0.222 0.384 0.257 9.02 13.48 2.4 spcp8x5 43 1a 13 5.531 0.632 0.437 0.618 0.432 8.94 12.80 1.6 panasonic-laptop 32 1 4 0.623 0.100 0.061 0.072 0.051 8.65 12.21 3.4 panasonic-laptop 39 7a 16 18.961 2.377 1.654 2.617 1.906 7.24 9.94 3.6 leds-bd2802 68 1 4 1.039 0.180 0.112 0.191 0.123 5.43 8.44 4.4 leds-bd2802 32 1 4 0.484 0.089 0.057 0.097 0.064 4.98 7.56 3.9 wm831x-dcdc 32 1 3 0.330 0.063 0.044 0.066 0.047 5.00 7.02 2.1 cx231xx-dvb 39 7a 13 17.536 3.389 2.425 3.464 2.517 5.06 6.96 3.2 ems usb 08 1a 21 2.334 0.502 0.327 0.543 0.362 4.29 6.44 2.9 . . . (for full results cf. http://batg.cswp.cs.technion.ac.il/publications/) ar7part 32 7a 6 0.071 0.067 0.056 0.074 0.063 0.95 1.12 1.3 metro-usb 08 1a 25 0.394 0.497 0.330 0.518 0.356 0.76 1.10 2.1 rtc-max6902 32 7a 9 0.133 0.124 0.106 0.147 0.126 0.90 1.05 1.1 i2c-algo-pca 43 1a 7 0.012 0.018 0.018 0.019 0.019 1.00 1.00 1.0 dvb-usb-vp7045 43 1a 2 0.001 0.002 0.002 0.027 0.027 1.00 1.00 2.6 cfag12864b 43 1a 2 0.036 0.039 0.036 0.040 0.037 0.90 0.97 1.0 rtc-max6902 43 1a 5 0.278 0.273 0.262 0.303 0.291 0.91 0.95 1.1 magellan 32 7a 2 0.015 0.018 0.016 0.018 0.016 0.83 0.93 0.93 vsxxxaa 43 1a 2 0.030 0.037 0.033 0.036 0.032 0.83 0.93 6.8 ar7part 43 1a 2 0.036 0.043 0.038 0.044 0.039 0.81 0.92 1.2 Sum (All) 2,660 3, 048.373 434.853 334.603 448.424 349.69 Mean (All) 15 16.749 2.389 1.838 2.464 1.921 6.798 8.717 4.3
◮ Measure semantical similarity of programs ◮ Floyd-Hoare automata with alpha renaming ◮ Database of Floyd-Hoare automata in the cloud ◮ Machine learning to determine most promising Floyd-Hoare automata from database
Dirk Beyer et al. “Correctness witnesses: exchanging verification results between verifiers”. In: SIGSOFT FSE. ACM, 2016, pp. 326–337. Dirk Beyer et al. “Precision reuse for efficient regression verification”. In: ESEC/SIGSOFT
Matthias Heizmann, Jochen Hoenicke, and Andreas Podelski. “Nested interpolants”. In:
Matthias Heizmann, Jochen Hoenicke, and Andreas Podelski. “Refinement of Trace Abstraction”. In: SAS. Vol. 5673. Lecture Notes in Computer Science. Springer, 2009,
Matthias Heizmann, Jochen Hoenicke, and Andreas Podelski. “Software Model Checking for People Who Love Automata”. In: CAV. Vol. 8044. Lecture Notes in Computer Science. Springer, 2013, pp. 36–52. Bat-Chen Rothenberg, Daniel Dietsch, and Matthias Heizmann. “Incremental Verification Using Trace Abstraction”. In: SAS. Vol. 11002. Lecture Notes in Computer Science. Springer, 2018, pp. 364–382.