Testing and Monitoring of Cyber-Physical Systems Georgios Fainekos - - PowerPoint PPT Presentation

▶
testing and monitoring of
SMART_READER_LITE
LIVE PREVIEW

Testing and Monitoring of Cyber-Physical Systems Georgios Fainekos - - PowerPoint PPT Presentation

Dec 30, 2022 259 likes •658 views

Dagstuhl: December 2016 1 Formal Specification Debugging for Testing and Monitoring of Cyber-Physical Systems Georgios Fainekos Joint work with Adel Dokhanchi and Bardh Hoxha School of Computing, Informatics and Decision System Engineering

slide-1
SLIDE 1

1

Lab Lab CPS PS

Dagstuhl: December 2016

School of Computing, Informatics and Decision System Engineering Arizona State University  fainekos at asu.edu

Georgios Fainekos

Formal Specification Debugging for Testing and Monitoring of Cyber-Physical Systems

Joint work with Adel Dokhanchi and Bardh Hoxha

slide-2
SLIDE 2

2

Lab Lab CPS PS

Dagstuhl: December 2016

Problem: automotive software recalls on the rise

No downshifting from 5th to 4th gear Rough idling or stalling due to complicated adaptive ECU Electric motor to rotate in the direction opposite to that selected by the transmission Cruise control does not disengage unless turning off the ignition Samples of recent recalls from different OEMs:

slide-3
SLIDE 3

3

Lab Lab CPS PS

Dagstuhl: December 2016

Challenge: Why there are so many bugs?

Due to the complex interactions between the physical system and software:

  • Analytical tools and theory cannot provide correct-by-design guarantees
  • The verification problem is undecidable / complete methods do not scale
  • Humans cannot predict the conditions that lead to bad behaviors

Correct Software  Correct System Behavior

Insulin Infusion Pump wearable continuous glucose monitor

Controller

Time Blood Glucose

slide-4
SLIDE 4

4

Lab Lab CPS PS

Dagstuhl: December 2016

Vision: a complete theory for MBD for CPS

Autocode Generation (with multi-core in mind) Formal Specifications Model Design System Deployment Informal Requirements

Transparent from the user perspective: 1. Automated synthesis 2. T esting and verification support with guarantees

Awards: 1017074, 1116136, 1319560, 1350420, 1446730

Any opinions, findings, and conclusions

  • r recommendations expressed in this

material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

System Calibration Hardware In the Loop (HIL) Processor In the Loop (PIL)

slide-5
SLIDE 5

5

Lab Lab CPS PS

Dagstuhl: December 2016

S-Taliro support in the V-process

Autocode Generation (with multi-core in mind) S-Taliro support 1 2 Formal Specifications Model Design System Deployment Informal Requirements 4 3 1 2 5 System Calibration Hardware In the Loop (HIL) Processor In the Loop (PIL)

  • 1. Testing formal specifications and specification mining [TECS 2013, ICTSS 2012, …]
  • 2. Conformance testing: models, HIL/PIL or tuned/calibrated model [MEMOCODE 2014]
  • 3. Testing formal specifications on the HIL/PIL calibrated system [TECS 2013, …]
  • 4. Runtime monitoring of formal requirements [RV 2014]
  • 5. Specification visualization [IROS 2015] & Debugging [MEMOCODE 2015]
slide-6
SLIDE 6

6

Lab Lab CPS PS

Dagstuhl: December 2016

Previous Work: ViSpec

  • ViSpec helps transforming pre-specified templates in NL
  • No need for MITL background

Hoxha, Bach, Abbas, Dokhanchi, Kobayashi, Fainekos, Towards Formal Specification Visualization for Testing and Monitoring of Cyber-Physical Systems, DIFTS 2014 Hoxha, Mavridis and Fainekos, VISPEC: A graphical tool for easy elicitation of MTL requirements, IROS 2015

slide-7
SLIDE 7

7

Lab Lab CPS PS

Dagstuhl: December 2016

ViSpec – Specification Classes

Safety: □𝐽𝜚 Reachability: ◇𝐽𝜚 Stabilization: ◇𝐽□𝐽𝜚 Recurrence: □𝐽◇𝐽𝜚 Implication: 𝜚 → 𝜔 Reactive Response: □𝐽(𝜚 → 𝑁𝐽𝜔) Conjunction: 𝜚 ∧ 𝜔 Non-strict Sequencing: 𝑂𝐽(𝜚 ∧ 𝑁𝐽𝜔)

𝑁, 𝑂 ∈ □, ◇

slide-8
SLIDE 8

8

Lab Lab CPS PS

Dagstuhl: December 2016

Motivating Example: On-Line Survey

We asked: “At some time in the first 30 seconds, the vehicle speed (v) will go

  • ver 100 and stay above 100 for 20 seconds”

Response: 𝜒 = ◇[0,30]((𝑤 > 100) ⇒ □[0,20](𝑤 > 100))

∃t ∈ [0,30]( 𝑤 𝑢 > 100 ⇒ ∀𝑢′ ∈ [t,t+20](𝑤(𝑢) > 100))

𝜒 is a tautology

  • 𝑤 > 100 =⊥ at any time in [0,30]

(𝑤 > 100) ⇒ □[0,20](𝑤 > 100) = ⊤

  • 𝑤 > 100 = ⊤ for all the time in [0,30]

□[0,20] 𝑤 > 100 = ⊤between [0,10] (𝑤 > 100) ⇒ □[0,20](𝑤 > 100) = ⊤ between [0,10]

  • B. Hoxha, N. Mavridis and G. Fainekos, VISPEC: A graphical tool for easy elicitation of MTL requirements, IROS 2015
slide-9
SLIDE 9

9

Lab Lab CPS PS

Dagstuhl: December 2016

Somewhere in Switzerland …

Not a software bug! From the Tesla Model X Owner’s manual: Invalid requirements: At the presence of stationary vehicles the Traffic-Aware Cruise Control may or may not break! At the absence of vehicles the Traffic-Aware Cruise Control may or may not break!

slide-10
SLIDE 10

10

Lab Lab CPS PS

Dagstuhl: December 2016

Problem Formulation

Problem 1 (System Independent MITL Analysis): Given an MITL formula ϕ, find whether ϕ has any of the following logical issues:

  • Validity: the specification is unsatisfiable or a tautology.
  • Redundancy: the formula has redundant conjuncts.
  • Vacuity: some subformulas do not contribute to the satisfiability of the

formula.

Problem 2 (System Dependent Vacuity Checking): Given an MITL formula ϕ, and signal µ, check whether µ satisfies the antecedent failure mutation of ϕ.

  • A. Dokhanchi, B. Hoxha, and G. Fainekos, Metric interval temporal logic specification elicitation and
  • debugging. MEMOCODE 2015, Austin, TX, USA
slide-11
SLIDE 11

11

Lab Lab CPS PS

Dagstuhl: December 2016

Overview

  • Motivation
  • Preliminaries
  • System Independent MITL Analysis
  • System Dependent Vacuity Checking
  • Experiments
  • Conclusion & Future Research
slide-12
SLIDE 12

12

Lab Lab CPS PS

Dagstuhl: December 2016

□𝑏- always a ◇[1,3]a - eventually a 𝑏 𝑉 𝑐- a until b 𝑏 𝑉[1,1.5] 𝑐 -a until b

a a a a a a * * a * * * a a b * * a 0.4 0.7 1.1 1.2 1.7 time 𝜚 ∷= ⊤ | 𝑞 ¬𝜚 𝜚1 ∨ 𝜚2 □𝐽𝜚 ◇𝐽𝜚 | 𝜚1𝑉𝐽𝜚2

Metric Interval Temporal Logic: Semantic Intuition

now

slide-13
SLIDE 13

13

Lab Lab CPS PS

Dagstuhl: December 2016

Subset of MITL

  • Bounded-MITL(◇, □) with only Always & Eventually operator
  • Negation Normal Form
  • Syntax:
  • No Until, Release, Next operator is used

𝜚 ∷= ⊤ ⊥ 𝑞 ¬𝑞 𝜚1 ∨ 𝜚2 | 𝜚1 ∧ 𝜚2 □𝐽𝜚 ◇𝐽𝜚

slide-14
SLIDE 14

14

Lab Lab CPS PS

Dagstuhl: December 2016

Signal Temporal Logic

Time t

a

Boolean abstraction

a

Time t

1.1 𝑦 𝑢 ∈ R Specification example: ◇[1.1,3.2](𝑦(𝑢) ≥ 𝑦0) x0 Real-Value Signal Boolean Signal

Notice example is MITL if we replace the predicate with a proposition: 𝑏 ≡ (𝑦(𝑢) ≥ 𝑦0)

3.2 1.1 3.2

slide-15
SLIDE 15

15

Lab Lab CPS PS

Dagstuhl: December 2016

Overview

  • Motivation
  • Preliminaries
  • System Independent MITL Analysis
  • System Dependent Vacuity Checking
  • Experiments
  • Conclusion & Future Research
slide-16
SLIDE 16

16

Lab Lab CPS PS

Dagstuhl: December 2016

Transforming STL to MITL

Example: 𝑡𝑞𝑓𝑓𝑒 > 100 ⇒ 𝑡𝑞𝑓𝑓𝑒 > 80

time

a

t

Speed b

100 80

time

a≡ 𝑇𝑞𝑓𝑓𝑒 > 100

time

Boolean abstraction

b≡ 𝑇𝑞𝑓𝑓𝑒 > 80

time

100 ≥ 𝑇𝑞𝑓𝑓𝑒 > 80

c

Question: 𝑏 ⟹ 𝑐 ? 𝑏 ⇒ 𝑏 ∨ 𝑑

slide-17
SLIDE 17

17

Lab Lab CPS PS

Dagstuhl: December 2016

Debugging MITL Specification

Specification Elicitation Framework 3-Levels of Specification Debugging

MITL Passed

slide-18
SLIDE 18

18

Lab Lab CPS PS

Dagstuhl: December 2016

Validity Issues Detection

Checking whether 𝜒 is unsatisfiable or a tautology A valid specification is one where 𝜒 and ¬𝜒 are satisfiable We asked: “At some time in the first 30 seconds, the vehicle speed (v) will go

  • ver 100 and stay above 100 for 20 seconds”

Response: 𝜒 = ◇[0,30]( (𝑤 > 100) ⇒ □[0,20](𝑤 > 100) ) 𝜒 is a tautology

slide-19
SLIDE 19

19

Lab Lab CPS PS

Dagstuhl: December 2016

Redundancy Issues Detection

Conjunctive formula: Φ =ٿ𝑘=1

𝑙

𝜒𝑘 Removing conjunct: ٿ𝑘=1

𝑗−1 𝜒𝑘 ∧ٿ𝑘=𝑗+1 𝑙

𝜒𝑘 ≡ Φ\𝜒𝑗 If ∃ 𝜒𝑗 Φ\𝜒𝑗 ⊨ 𝜒𝑗 Then 𝜒𝑗 is redundant Example 𝜒2 = 𝑞 ∧ □[0,10]𝑞 □[0,10]𝑞 ⊨ 𝑞 Algorithm 1: Checks Φ\𝜒𝑗 ⊨ 𝜒𝑗 for each conjunct Creates a list of redundant conjuncts

  • H. Chockler and O. Strichman, Before and after vacuity.
  • Form. Methods Syst. Des., 34(1):37–58, Feb. 2009.
slide-20
SLIDE 20

20

Lab Lab CPS PS

Dagstuhl: December 2016

Redundancy Example

User response to “At some point in time in the first 30 seconds, vehicle speed will go over 100 and stay above for 20 seconds.” 𝜒 = ◇[0,30](𝑡𝑞𝑓𝑓𝑒 > 100) ∧ ◇[0,20](𝑡𝑞𝑓𝑓𝑒 > 100) ◇[0,30](𝑤 > 100) is redundant since ◇[0,20](𝑤 > 100) ⊨ ◇[0,30](𝑤 > 100)

slide-21
SLIDE 21

21

Lab Lab CPS PS

Dagstuhl: December 2016

Vacuity Issues Detection

If sub-formula 𝜔 ∈ 𝜒 does not affect the satisfiability of 𝜒, then 𝜒 is vacuous Remove 𝜔 Vacuous specifications are equivalent to their mutant

  • H. Chockler and O. Strichman, Before and after vacuity.
  • Form. Methods Syst. Des., 34(1):37–58, Feb. 2009.
slide-22
SLIDE 22

22

Lab Lab CPS PS

Dagstuhl: December 2016

Mutation of MITL for Vacuity Checking

Mutation with assigning ⊥ to literal occurrence

𝜒 = (¬𝑞 ∧ 𝑟) ∨ ◇[0,10]𝑞 ∨ □[0,10]𝑟 𝜒[¬𝑞 ←⊥] = (⊥∧ 𝑟) ∨ ◇[0,10]𝑞 ∨ □[0,10]𝑟

4 literal occurrence => 4 mutation Algorithm 2: Checks Φ ⊨ 𝜒𝑗 𝑚 ←⊥ for each mutation Creates a list of mutated sub-formulas

slide-23
SLIDE 23

23

Lab Lab CPS PS

Dagstuhl: December 2016

Vacuity Example

CPS example

𝜒𝑇𝑈𝑀=◇[0,10]((𝑡𝑞𝑓𝑓𝑒 > 100) ∧ ◇[0,10](𝑡𝑞𝑓𝑓𝑒 > 80)) 𝜒=◇[0,10](𝑏 ∧ ◇[0,10] 𝑐) is not vacuous

However… 𝜒′=◇[0,10](𝑏 ∧ ◇[0,10](𝑏 ∨ 𝑑)) is vacuous

Where 𝑏:(𝑡𝑞𝑓𝑓𝑒 > 100) and 𝑑:(100 ≥ 𝑡𝑞𝑓𝑓𝑒 > 80)

𝜒′ ⊨ ◇[0,10](𝑏 ∧ ◇[0,10](𝑏 ∨⊥))

slide-24
SLIDE 24

24

Lab Lab CPS PS

Dagstuhl: December 2016

Overview

  • Motivation
  • Preliminaries
  • System Independent MITL Analysis
  • System Dependent Vacuity Checking
  • Experiments
  • Conclusion & Future Research
slide-25
SLIDE 25

25

Lab Lab CPS PS

Dagstuhl: December 2016

Vacuous Signals

  • The MITL specification

𝜒 = □ [0,5]( (𝑠𝑓𝑟𝑣𝑓𝑡𝑢) ⇒ ◇[0,10](𝑏𝑑𝑙𝑜𝑝𝑥𝑚𝑓𝑒𝑕𝑓) )

  • 𝜒 is passed the MITL Specification Debugging Framework
  • Any signal µ that does not satisfy 𝑠𝑓𝑟𝑣𝑓𝑡𝑢 at any point in time

will vacuously satisfy 𝜒.

  • Signals that do not satisfy the antecedent (precondition) of the

subformula are called vacuous signals.

  • Vacuous Signals satisfy antecedent failure mutation of 𝜒
slide-26
SLIDE 26

26

Lab Lab CPS PS

Dagstuhl: December 2016

Antecedent Failure Mutation

  • For each implication (ϕ ⇒ ψ),

(ϕ) is the precondition (antecedent) of the implication.

  • Antecedent Failure Mutation is the assertion that the

precondition (ϕ) never happens.

  • Example:

Antecedent Failure of ϕ is ¬ϕ

  • Signals that satisfy ¬ϕ are vacuous signals
slide-27
SLIDE 27

27

Lab Lab CPS PS

Dagstuhl: December 2016

Vacuity Detection in Testing

slide-28
SLIDE 28

28

Lab Lab CPS PS

Dagstuhl: December 2016

Overview

  • Motivation
  • Preliminaries
  • System Independent MITL Analysis
  • System Dependent Vacuity Checking
  • Experiments
  • Conclusion & Future Research
slide-29
SLIDE 29

29

Lab Lab CPS PS

Dagstuhl: December 2016

Implementation and Experiments

We used MITL satisfiability solver for this method: 𝜒 ⊨ 𝜔 iff (𝜒 ⟹ 𝜔) ≡ 𝑈 iff (¬𝜒 ∨ 𝜔) ≡ 𝑈 iff (𝜒 ∧ ¬𝜔) ≡⊥ We detected all the issues with MITL SAT solver

  • M. Bersani and M. Rossi and P. San Pietro, A tool for

deciding the satisfiability of continuous-time metric temporal logic. Acta Informatica, pages 1–36, 2015.

QTL Solver Zot Z3 MITL SAT CLTLoc SAT SMT Validity Redundancy Vacuity

slide-30
SLIDE 30

30

Lab Lab CPS PS

Dagstuhl: December 2016

ViSpec – Usability Study

Each user received ten tasks:

  • To formalize a NL specification in automotive industry through ViSpec

Group I: Non-expert users No experience in working with requirements. 20 subjects from the student community at ASU Group 2: Expert users Experienced in working with requirements (not necessarily formal requirements) 10 subjects from the industry in the Phoenix area

  • B. Hoxha, N. Mavridis and G. Fainekos, VISPEC: A graphical tool

for easy elicitation of MTL requirements, IROS 2015

slide-31
SLIDE 31

31

Lab Lab CPS PS

Dagstuhl: December 2016

Specification Checks

Seven of ten tasks have no detected issue Example task with erroneous specifications: Stabilization “At some point in time in the first 30 seconds, vehicle speed will go over 100 and stay above for 20 seconds.” Correct answer: ◇[0,30] □[0,20]( p1 ) p1 : speed>100 Incorrect Answers:

Specification Detected Error ◇[0,30]( p1 ) ∧ ◇[0,20] ( p1 ) ◇[0,30]( p1 ) is redundant ◇[0,30]( p1 ⇒ □[0,20] ( p1 ) ) Tautology

slide-32
SLIDE 32

32

Lab Lab CPS PS

Dagstuhl: December 2016

Error in Oscillation Task

“At every point in time in the first 40 seconds, vehicle speed will go

  • ver 100 in the next 10 seconds.”

Correct answer: □ [0,40] ◇[0,10]( p1 ) p1 : speed>100 Incorrect answer (with Redundancy Error): □ [0,40] (p1 ) ∧ □ [0,40] ◇[0,20]( p1 ) Issue: □ [0,40] ◇[0,20]( p1 ) is redundant

slide-33
SLIDE 33

33

Lab Lab CPS PS

Dagstuhl: December 2016

Error in Long Sequence Task

“If, at some point in time in the first 40 seconds, vehicle speed goes

  • ver 80 then from that point on, if within the next 20 seconds

the engine speed goes over 4000, then, for the next 30 seconds, the vehicle speed should be over 100.” ◇[0,40] ( (speed>80) ⇒ ◇[0,20](rpm>4000 ⇒ □[0,30]speed>100) ) ↓(STL2MITL) ◇[0,40] ((p1 ∨ p3) ⇒ ◇[0,20](p2 ⇒ □[0,30]p1) ) p1: speed>100 p2 : rpm>4000 p3: 100≥speed>80

Specification Detected Error ◇[0,40](((p1 ∨ p3) ⇒ ◇[0,20] p2) ∧ □[0,30]p1 ) Vacuous formula 𝜒 ⊨ 𝜒 𝑞3 ←⊥ ◇[0,40](p1 ∨ p3) ∧ ◇[0,40] p2 ∧ ◇[0,40]□[0,30]p1 ◇[0,40](p1 ∨ p3) is redundant

Incorrect Answers:

slide-34
SLIDE 34

34

Lab Lab CPS PS

Dagstuhl: December 2016

Runtime Overhead

slide-35
SLIDE 35

35

Lab Lab CPS PS

Dagstuhl: December 2016

Antecedent Failure Detection

  • B. Hoxha, H. Abbas and Georgios Fainekos, Benchmarks for

Temporal Logic Requirements for Automotive Systems, ARCH 2014

slide-36
SLIDE 36

36

Lab Lab CPS PS

Dagstuhl: December 2016

Overview

  • Motivation
  • Preliminaries
  • System Independent MITL Analysis
  • System Dependent Vacuity Checking
  • Experiments
  • Conclusion & Future Research
slide-37
SLIDE 37

37

Lab Lab CPS PS

Dagstuhl: December 2016

Conclusions

  • We developed a debugging framework for MITL
  • We extended the existing LTL vacuity detection algorithms to

MITL

  • We implemented the antecedent failure detection algorithm

that can find signal vacuity

  • Our tool can improve the users ability to create correct MITL

specifications

slide-38
SLIDE 38

38

Lab Lab CPS PS

Dagstuhl: December 2016

Future Research

  • Integrate MITL analysis into ViSpec
  • Finding the Coverage of specification with respect to falsifying

signals

  • Improving the stochastic search algorithms for falsification of

the requirement in CPS with signal vacuity detection.

slide-39
SLIDE 39

39

Lab Lab CPS PS

Dagstuhl: December 2016

Acknowledgements

On-Line Survey: http://goo.gl/forms/YW0reiDtgi Sponsors

NSF awards: IIP-1454143 CNS-1116136 CNS-1350420 CNS-1319560 Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.