Learning to Specify soundly Suresh Jagannathan Joint work with He - PowerPoint PPT Presentation

Learning to Specify … soundly Suresh Jagannathan Joint work with He Zhu, Stephen Magill, and Gustavo Petri

Goal + Verification Program Specifications Conditions Types Assertions Contracts Pre/Post Loop Invariants … Spec Spec Automated Manual How do we automatically discover useful specifications to facilitate verification?

Learning ... Feature extraction: P ➔ F Set of features: F H: Hypothesis space over F S: Sample space Learner (F) C: Concept class of program P: Data structures, Numeric domains, ...

Context and Challenges Data-Driven Precondition Inference with Learned Features • What is the language in which specifications are expressed? From Invariant Checking to Invariant Inference Using Randomized Search ★ Decidability Dependent Array Type Inference from Tests • How do we generate samples? Verification as Learning Geometric Concepts ★ Coverage ICE: A Robust Framework for Learning Invariants • How do we generalize from samples? From Tests to Proofs ★ Turn postulated invariants to true invariants A Data Driven Approach for Algebraic Loop ★ Soundness Invariants ⋆ • How do we infer inductive invariants? Using Dynamic Analysis to Generate Disjunctive Invariants ★ Necessary for automated verification Testing, Abstraction, Theorem Proving: Better Together! • How do we guarantee progress? The Daikon system for dynamic detection of likely invariants ★ Relate number of observations to quality of inference • How do we ensure convergence? Interpolants as Classifiers ? Learning Invariants using Decision Trees ★ Will we eventually learn a true invariant? and Implication Counterexamples • Quality of specifications (simplicity, minimality, ….) Learning Commutativity Specifications

A A Programmer’s Day ... Defining data structures ... type ‘a list = type ‘a tree = | Nil | Leaf | Cons ‘a * | Node ‘a * ‘a list ‘a tree * ‘a tree Writing functions ... // flat: ‘a list -> ‘a tree -> ‘a list let rec flat accu t = No assertions / match t with loop invariants | Leaf -> accu pre-conditions / | Node (x, l, r) -> post-conditions! flat (x::(flat accu r)) l // elements: ‘a tree -> ‘a list let elements t = flat [] t

A A Programmer’s Day ... Testing code ... // elements: ‘a tree -> ‘a list let elements t = flat [] t l = elements t x4 x2 x5 x1 x2 x3 x4 x5 l t x1 x3 Implicitly discovers: // specification: // elements: ‘a tree -> ‘a list // l = elements t ≡ // // in-order( t ) forward-order( l )

A Features of Data Structures ... // elements: ‘a tree -> ‘a list let elements t = flat [] t Hypothesis Domain over l = elements t data structure features: t Containment t 99K u t 99K 5 4 t : u & v Reachability t : u . v 2 5 t : u v x t : 4 . 1 1 3 t : 3 5 x l 99K u l : u → v l : 1 → 3 l : 3 → 5 l 5 1 2 3 4 l 99K 5

A From features to specifications ... // elements: ‘a tree -> ‘a list let elements t = flat [] t l = elements t Predict truth of output features using a Boolean combination of input features ... Classification t : u . v ∧ ⇒ = l : u → v t : u & v ∨ ⇐ = t : u input output x v features features ∧ ⇒ = t 99K u l 99K u ∨ ⇐ = t 99K v

A Specifications of Data Structures ... // specification: // in-order of t ≡ forward-order of l l :list = elements ( t :tree) t : v . u _ ⇒ l : u → v ) ⇐ ( ∀ u v, t : u & v _ t : u v x t ⇐ ⇒ l v u v u v u v u

A Feature Extraction ... t : u t x v val type ‘a tree = | Leaf l r | Node ‘a * ‘a tree * ‘a tree t : u . v t val Node t : u & v l r ‘a tree ‘a tree t : u ⇒ ( l 99K u ∧ r 99K v ) ∨ l : u v ∨ r : u v ⇐ v x x x ⇒ (( u = val ^ r 99K v ) _ r : u & v _ l : u & v ) t : u & v ⇐ v root node left subtree right subtree (( u = val ^ l 99K v ) _ l : u . v _ r : u . v t : u . v ( )

A Learner ... // elements: ‘a tree -> ‘a list let elements t = flat [] t t 4 l = elements t v l 2 5 1 2 3 4 5 u ( u , v ) 1 3 output features input features ( u , v ) t : u . v t : u & v t : u t : v . u t : v & u t : v t 99K u t 99K v l : u → v x v x u (1,2) 
 0 0 0 1 0 0 1 1 1 
 0 0 0 1 0 0 1 1 1 
 (1,2) (4,5) 0 1 0 0 0 0 1 1 1 
 pos (2,5) 
 0 0 1 0 0 0 1 1 1 
 (3,1) 
 0 0 0 0 0 1 1 1 0 
 (3,2) 
 0 0 0 0 1 0 1 1 0 
 neg (4,1) 
 1 0 0 0 0 0 1 1 0 
 . . . Sample space


 
 
 
 
 
 
 
 
 A Learner ... input features ( u , v ) t : u . v t : u & v t : u t : v . u t : v & u t : v t 99K u t 99K v x v x u (1,2) 
 0 
 0 
 1 
 0 
 pos samples 0 
 0 
 1 
 1 
 (4,5) 1 0 0 0 0 0 1 1 ϕ l : u → v (2,5) 0 1 0 0 0 0 1 1 neg samples (3,1) 
 0 
 0 
 0 
 1 
 0 
 0 
 1 
 1 
 (3,2) 
 0 
 0 
 0 
 0 
 0 
 1 
 1 
 1 
 ¬ l : u → v ¬ ϕ (4,1) 
 0 
 0 
 0 
 0 
 1 
 0 
 1 
 1 
 ⇒ l : u → v ϕ ⇐ t 4 v 2 5 u v u l 1 2 3 4 5 1 3

A Learner ... Truth Table ( u , v ) t : u . v t : u & v t : u t : v . u t : v & u t : v l : u → v t 99K u t 99K v x v x u (1,2) 
 0 
 0 
 1 
 0 
 1 
 0 
 0 
 1 
 1 
 (4,5) 1 0 0 0 1 0 0 1 1 pos (2,5) 
 0 
 1 
 0 
 0 
 1 
 0 
 0 
 1 
 1 
 (3,1) 
 0 
 0 
 0 
 1 
 0 
 0 
 0 
 1 
 1 
 neg (3,2) 
 0 
 0 
 0 
 0 
 0 
 0 
 1 
 1 
 1 
 (4,1) 
 0 
 0 
 0 
 0 
 0 
 1 
 0 
 1 
 1 
 • Optimization task: • Constraint solvers

A Learner ... l :list = elements ( t :tree) t : u & v t : u v t : v . u l : u → v x 1 
 0 
 0 
 1 
 1 1 0 0 Truth Table 1 
 0 
 1 
 0 
 0 
 0 
 0 
 0 
 0 
 0 
 0 
 0 
 0 
 0 
 0 
 0 
 0 1 t : v . u _ A ( � � 8 u v , t : u v _ ) l : u ! v x @ t : u & v


 
 
 
 
 
 
 
 A Learner ... ⇐ ⇒ If and only if specifications are nice, but … input input input input input input input input feature1 feature2 feature3 feature4 feature5 feature6 feature7 feature8 0 
 0 
 1 
 0 
 0 
 0 
 1 
 1 
 pos samples 1 0 0 0 0 0 1 1 output feature 0 1 0 0 0 0 1 1 No classifier! 0 
 0 
 1 
 0 
 0 
 0 
 1 
 1 
 neg samples 0 
 0 
 0 
 0 
 0 
 1 
 1 
 1 
 ¬output feature 0 
 0 
 0 
 0 
 1 
 0 
 1 
 1 
 ? = ⇒ ⇐ =

A Binary Search Tree Insertion ... r = insert 3 t let rec insert x t = match t with r x =3 | Leaf -> Node (x, Leaf, Leaf) 4 4 t | Node (y, l, r) -> if x < y then Node (y, insert x l, r) 2 2 else if y < x then Node (y, l, insert x r) else t 3 input features input features output features Π 10 Π 8 Π 0 t : u . v ( u , v ) Π 0 u = x Π 1 Π 2 Π 3 Π 4 Π 5 Π 6 Π 7 Π 9 Π 10 Π 8 Π 9 Π 1 t : u & v r : u . v v = x (4,3) 0 0 0 0 0 0 1 0 0 1 1 pos . Π 2 neg t : u (2,3) 0 0 0 0 0 0 1 0 0 1 0 . x v . Π 3 t : v . u Problem: Π 4 t : v & u Samples are not separable Π 5 t : v x u with existing features Π 6 t 99K u Π 7 t 99K v

A Binary Search Tree Insertion ... r x =3 4 4 t output features input features 2 2 Π 10 Π 8 Π 0 t : u . v u = x Π 9 Π 1 t : u & v v = x r : u . v 3 r = insert 3 t . Π 2 t : u . x v input features . Π 3 t : v . u Π 4 Π 0 Π 1 Π 2 Π 3 Π 4 Π 5 Π 6 Π 7 Π 9 Π 10 Π 8 t : v & u Π 5 t : v (4,3) 
 0 
 0 
 0 
 0 
 0 
 0 
 1 0 
 0 
 1 
 1 
 x u Π 6 (4,2) 1 0 0 0 0 0 1 1 0 0 1 t 99K u (2,3) 0 0 0 0 0 0 1 0 0 1 0 Π 7 t 99K v 0 0 1 0 0 0 1 1 0 0 0 (2,4) 
 8 u v , t : u . v ) r : u . v ✓ ◆ ( t 99K u ^ v = x ) _ 8 u v , r : u . v ) t : u . v