An Active Learning Approach to the Falsification of Black Box Cyber-Physical Systems Simone Silvetti , Alberto Policriti, Luca Bortolussi silvetti.simone@spes.uniud.it silvetti@esteco.com 13th International Conference on integrated Formal Methods S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 1 / 35
Outline Overview 1 Model Based Development Signal Temporal Logic Search-Based Testing Domain Estimation Problem 2 Algorithm Idea Test Case & Results 3 Challenges & Further studies 4 S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 2 / 35
Overview Model Based Development Overview Model Based Development Methodology based on a computational model of a real target system used at the early stage of the design phase used at the end to verify the compliance of the real system Motivations reducing the time of prototyping reducing the cost of development S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 3 / 35
Overview Model Based Development Models Software: Block Diagram Systems Computational Models LabView Hybrid Systems CPS Automata Statistical Models Simulink Problem Too Much Complexity ⇒ no stan- dard Model checking techniques. Solution Black Box Assumption and Search-based approach . S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 4 / 35
Overview Model Based Development Simulink Model S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 5 / 35
Overview Model Based Development Simulink Model - Inputs S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 6 / 35
Overview Model Based Development Simulink Model - Outputs S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 7 / 35
Overview Model Based Development Simulink Model - Continuous Dynamics S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 8 / 35
Overview Model Based Development Simulink Model - Finite State Machine S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 9 / 35
Overview Model Based Development Black Box Assumption Inputs & Outputs The Inputs are Piece Wise Constant (PWC) Functions, the Outputs are PWC functions (Gear) or Continuous Functions. S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 10 / 35
Overview Model Based Development Black Box Assumption Black Box Assumption less information an more general approach (interesting by an industrial point of view) S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 11 / 35
Overview Signal Temporal Logic The requirements: Signal Temporal Logic (STL) Signal temporal logic is: a linear continuous time temporal logic. X ) ≥ 0 ] where g : R n → R is a continuous the atomic predicates are of the form µ ( � X ):=[ g ( � function. the syntax is φ := ⊥ | ⊤ | µ | ¬ φ | φ ∨ φ | φ U [ T 1 , T 2 ] φ, (1) Example φ 1 := F [ 0 , 50 ] | X 1 − X 2 | > 10 The Booleans semantics: if a given path 1 satisfies or not a given STL formula. The Quantitative semantics: How much a 2 given path satisfies or not a given STL formula. S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 12 / 35
Overview Search-Based Testing Search-Based Testing Falsification Goal: Find the input functions (1) which violate the requirements (4) Problems 1 Falsify with a low number of simulations ⇒ Active Learning 2 Functional Input Space(!!) ⇒ Adaptive Space Parameterization S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 13 / 35
Overview Search-Based Testing Fixed Parameterization n adaptive control points ⇒ n variable to optimize S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 14 / 35
Overview Search-Based Testing Fixed Parameterization n fixed control points ⇒ n variable to optimize S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 15 / 35
Overview Search-Based Testing Adaptive Parameterization n adaptive control points ⇒ 2n variable to optimize S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 16 / 35
Domain Estimation Problem Domain Estimation Problem Domain Estimation Problem Consider a function ρ : Θ → R and an interval I ⊆ R . We define the domain estimation problem as the task of identifying the set: B = { θ ∈ Θ | f ( θ ) ∈ I } ⊆ Θ (2) In practice, if B � = ∅ , we will limit us to identify a subset B ⊆ B of size n. Falsification ∼ Domain estimation problems B = { θ ∈ Θ | ρ ( θ ) ∈ ( −∞ , 0 ) } ⊆ Θ Gaussian Processes S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 17 / 35
Domain Estimation Problem Gaussian Processes Definition A random variable f ( θ ) , θ ∈ Θ is a GP f ∼ GP ( m , k ) ⇐ ⇒ ( f ( θ 1 ) , f ( θ 2 ) , . . . , f ( θ n )) ∼ N ( m , K ) where m = ( m ( θ 1 ; h 1 ) , m ( θ 2 ; h 1 ) , . . . , m ( θ n ; h 1 )) and K ij = k ( f ( θ i ) , f ( θ j ); h 2 ) Prediction { f ( θ 1 ) , . . . , f ( θ n ) , f ( θ ′ ) } ∼ N ( m ′ , K ′ ) E ( f ( θ ′ )) = ( k ( θ ′ , θ 1 ) , . . . , k ( θ ′ , θ N )) K − 1 r N var ( f ( θ ′ )) = k ( θ ′ , θ ′ ) − K ( θ, r ) K − 1 K ( θ, r ) T N S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 18 / 35
Domain Estimation Problem Algorithm Idea Domain Estimation Problem Domain Estimation Problem Train Set: K ( ρ ) = { ( θ i , ρ ( θ i ) ) } i ≤ n (the partial knowledge) Gaussian Process: ρ K ( θ ) ∼ GP ( m K ( θ ) , σ K ( θ )) (the partial model) � 0 − m K ( θ ) � P ( ρ K ( θ ) < 0 ) = CDF σ K ( θ ) Simple Idea Iteratively explore the area which is more probable to falsify the system by sampling from P ( ρ K ( θ ) < 0 ) . S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 19 / 35
Domain Estimation Problem Algorithm Idea Algorithm - I S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 20 / 35
Domain Estimation Problem Algorithm Idea Algorithm - II S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 21 / 35
Domain Estimation Problem Algorithm Idea Aglorithm - III S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 22 / 35
Domain Estimation Problem Algorithm Idea Algorithm - IV S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 23 / 35
Domain Estimation Problem Algorithm Idea Algorithm - V S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 24 / 35
Domain Estimation Problem Algorithm Idea Algorithm - VI S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 25 / 35
Domain Estimation Problem Algorithm Idea Algorithm - VII S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 26 / 35
Domain Estimation Problem Algorithm Idea Algorithm - VIII S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 27 / 35
Domain Estimation Problem Algorithm Idea Algorithm - IX S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 28 / 35
Domain Estimation Problem Algorithm Idea Algorithm - X S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 29 / 35
Domain Estimation Problem Algorithm Idea Algorithm - XI S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 30 / 35
Domain Estimation Problem Algorithm Idea Probabilistic Approximation Semantics Definition ( L 0 and L ) L 0 : [ ⊂ STL]: atomic propositions + φ 1 U T φ 2 , F T φ, G T φ , that cannot be equivalently written as Boolean combinations of simpler formulas; F T ( φ 1 ∨ φ 2 ) ≡ F T φ 1 ∨ F T φ 2 �∈ L 0 L : the boolean connective closure of L 0 . Definition (Probabilistic Approximation Semantics of L ) The probabilistic approximation function γ : L × Path M × [ 0 , ∞ ) → [ 0 , 1 ] is defined by: γ ( φ, θ, t ) = P ( f K ( φ ) ( θ ) > 0 ) γ ( ¬ ψ, θ, t ) = 1 − γ ( ψ, θ, t ) γ ( ψ 1 ∧ ψ 2 , θ, t ) = γ ( ψ 1 , θ, t ) ∗ γ ( ψ 2 , θ, t ) γ ( ψ 1 ∨ ψ 2 , θ, t ) = γ ( ψ 1 , θ, t ) + γ ( ψ 2 , θ, t ) − γ ( ψ 1 ∧ ψ 2 , θ, t ) S. Silvetti , A. Policriti, L. Bortolussi iFM 2017 September 20, 2017 31 / 35
Recommend
More recommend
