Machine Learning Components Shakiba Yaghoubi, Georgios Fainekos CPS - - PowerPoint PPT Presentation

Gray-box Adversarial Testing for Control Systems with Machine Learning Components

Shakiba Yaghoubi, Georgios Fainekos

1

@

CPS V&V I&F Workshop – Dec. 11, 2019

@

Accidents happen - IFCS

2

[1] Tomayko, The story of Self-Repairing Flight Control Systems, Dryden Historical Study No 1, 2003 [2] NASA Facts, FS-2002-09-076-DFRC

@

Control Systems with ML Components

3

Neural Networks

* Nice NN pictures from Mathworks!

Feed-Forward Neural Network Recurrent Neural Network

@

New up and coming verification methods

• Dutta, S.; Jha, S.; Sankaranarayanan, S. & Tiwari, A., Learning and

Verification of Feedback Control Systems using Feedforward Neural Networks, ADHS 2018

• Xiang, W.; Lopez, D. M.; Musau, P. & Johnson, T. T., Reachable set

estimation and verification for neural network models of nonlinear dynamic systems, Safe, Autonomous and Intelligent Vehicles, 2019

• Ivanov, R.; Weimer, J.; Alur, R.; Pappas, G. J. & Lee, I., Verisig:

verifying safety properties of hybrid systems with neural network controllers, HSCC 2019

• Sun, X.; Khedr, H. & Shoukry, Y., Formal verification of neural network

controlled autonomous systems, HSCC 2019

• Way more work when considering the NN in isolation (not in the loop)

…

4

@

Why falsification?

• We need to stay as close as possible to Black-Box (BB) testing ...

 In practice, models may have BB components

• We would like to validate complex space-time requirements (with some

quantitative interpretation)

• Recurrent NN
• Use falsification on counter-example based / adversarial training

(more on this later) Our assumptions:

1.

Smooth system dynamics (working now on a relaxation)

2.

Smooth activation functions (for now this is necessary)

3.

Gray box testing: Linearizations at specific operating points are available (analytical or numerical)

5

@

Metric Temporal Logic* (MTL)

• Syntax:
• Semantics:

6

𝜚 ∷= ⊤ |𝑞 ¬𝜚 𝜚1 ∨ 𝜚2 □𝐽𝜚 ◇𝐽𝜚|◯𝜚| 𝜚1𝑉𝐽𝜚2

□[0,∞)𝑏 - Always a ◇[1,3] 𝑏 - Eventually a ◯[0.1,0.8]𝑏 - Next a 𝑏 𝑉[𝟐,𝟐.𝟔] 𝑐 - a until b

a a a a a a * * a * * * a a b * * a 0 0.4 0.7 1.1 1.2 time now * a * * * *

* R. Koymans "Specifying real-time properties with metric temporal logic" Real-Time Systems, 2(4):255–299, 1990

@

Signal Temporal Logic*

7

Time t a 1.1 𝑦 𝑢 ∈ R

Specification example: ◇[1.1,3.2](𝑦(𝑢) ≥ 𝑨)

z Real-Value Signal 3.2 Notice example is MITL if we replace the predicate with a proposition: 𝑏 ≡ (𝑦(𝑢) ≥ 𝑦0) 𝑦2 𝑢 ∈ R 𝑦1 𝑢 ∈ R z

Specification example: ◇ 1.1,3.2 (𝑦1

2 𝑢 + 𝑦2 2 𝑢 ≤ 𝑨2) Notice example is MITL if we replace the predicate with a proposition: 𝑏 ≡ (𝑦1

2 𝑢 + 𝑦2 2 𝑢 ≤ 𝑨2)

𝑦 𝑢 = 𝑦1(𝑢) 𝑦2(𝑢) ∈ R2

* Maler, O. & Nickovic, D., Monitoring Temporal Properties of Continuous Signals, FORMATS-FTRTFT 2004

@

Signal Temporal Logic & Robustness

8

𝑦 𝑢 Time t a 1.1 z 3.2 𝑦2 𝑢 𝑦1 𝑢 z 𝑦2 𝑦1 𝑢 z Time t a 1.1 z 3.2 𝑦 𝑢

@

Semantics

𝑆⊤ 𝑡, 𝜐 , 𝑗 = +∞ 𝑆𝑞((𝑡, 𝜐), 𝑗) = Dist𝑒(𝑡 𝑗 , 𝑦 𝑦 ⊨ 𝑞}) 𝑆𝜒1∨𝜒2 𝑡, 𝜐 , 𝑗 = 𝑆𝜒1 𝑡, 𝜐 , 𝑗 ⊔ 𝑆𝜒2 𝑡, 𝜐 , 𝑗 𝑆◯I𝜒 𝑡, 𝜐 , 𝑗 = ቊ 𝜐 𝑗 + 1 ∈ 𝜐 𝑗 + 𝐽 ∞ ⊓ 𝑆𝜒 𝑡, 𝜐 , 𝑗 𝑗𝑔 𝜐 > 𝑗 −∞ 𝑝𝑢ℎ𝑓𝑠𝑥𝑗𝑡𝑓 𝑆𝜒1𝑉𝐽𝜒2 𝑡, 𝜐 , 𝑗 =⊔𝑘∈𝜐−1(𝜐 𝑗 +𝐽) 𝑆𝜒2 𝑡, 𝜐 , 𝑘 ⊓⊓𝑙=𝑗

𝑘−1 𝑆𝜒1

𝑡, 𝜐 , 𝑙

9

𝑏 ⊓ 𝑐 = inf{𝑏, 𝑐} 𝑏 ⊔ 𝑐 = sup{𝑏, 𝑐} Dist𝑒 𝑦, 𝐵 = ቊ −inf 𝑒 𝑦, 𝑧 𝑧 ∈ 𝐵 𝑗𝑔 𝑦 ∉ 𝐵 inf 𝑒 𝑦, 𝑧 𝑧 ∈ 𝑌\𝐵 𝑗𝑔 𝑦 ∈ 𝐵

Complexity based on dynamic programming: O(|φ| |τ| c), where c = max 0j|τ|, IT(φ) |[j, max J(j, I)]|

@

Temporal Logic falsification as robustness minimization 10

Spec : ◇p ε s X0

• We need to solve an optimization problem:
• Challenges:
• Non-linear system dynamics
• Unknown input signals
• Unknown system parameters
• Non-differentiable cost function
• not known in closed form
• needs to be computed

min RΦ(y) yY is the set of all observable trajectories of the system 𝑧 𝑧 ⊨ 𝑞}

@

How do these “robustness” functions look like? 11

• 1

1

• 1
• 0.5

0.5 1

• 1
• 0.5

0.5 1 1.5 2 2.5 3 3.5 x1 x2 Robustenss

@

Primary Problem of Interest: 12

Consider the dynamical system including an NN component ሶ 𝑦𝑞 = 𝑔

𝑞 𝑦𝑞, 𝑣, 𝑂𝑂 𝑢, 𝑦𝑞 . , 𝑣 .

Add the possible states of the NN to the closed loop system (𝑦 = 𝑦𝑞

𝑈, 𝑦𝑂𝑂 𝑈 𝑈)

ሶ 𝑦 = 𝑔 𝑦, 𝑣 Find the initial condition 𝑦0 ∈ 𝑌0, and the time varying adversarial input 𝑣 . ∈ 𝑉 0,𝑈 of the system that minimize the robustness function corresponding to a specification of interest. ሶ 𝑦 = 𝐺(𝑦, 𝑣) 𝑦0

∗

𝑌0

t 𝑣∗(𝑢) 𝑦1 𝑦2

𝑠∗

Steam Condenser with RNN controller

@

• smooth and differentiable
• using gradients we can find directions of

improvement to guide the search in the large dimensional search space.

Cost function

13

The primary robustness function is complicated, non-smooth, and non-convex. Instead we minimize the following cost function which locally approximates the robustness function. p2 p1

𝑦𝑞(0)

𝑠

∗

𝑦(𝑢∗)

Critical time Closest point in the unsafe set

t

𝑣(𝑢)

𝑣

□p1  □ p2

min

𝑦𝑞 0 ,𝑣 𝐾𝑗 = 1

2 𝑦 𝑢∗

𝑗 − 𝑠 ∗ 𝑗 𝑈 𝑦 𝑢∗ 𝑗 − 𝑠 ∗ 𝑗

𝑡. 𝑢 ሶ 𝑦 = 𝑔 𝑦, 𝑣 𝑦𝑞 0 ∈ 𝑌0, 𝑣 ∈ 𝑉

𝑣(𝑢) + 𝜀𝑣(𝑢) 𝑦𝑞(0) + 𝜀𝑦𝑞(0)

@ ሶ 𝜇 = − 𝜖𝐼 𝜖𝑦

𝑈

= − 𝜖𝑔 𝜖𝑦 ቚ

𝑦𝑗,𝑣𝑗 𝑈

𝜇 𝜇 𝑢∗

𝑗 =

𝑒𝜚𝑗 𝑦𝑗 𝑢∗

𝑗

𝑒𝑦

𝑈

= 𝑦𝑗 𝑢∗

𝑗 − 𝑠 ∗ 𝑗

Decent direction Calculation

14

• Using the method of the Lagrange multipliers, the problem can be reduced to

the problem of minimizing the cost function

• Forming the Hamiltonian as 𝐼 𝑦, 𝑣 = 𝜇𝑈𝑔(𝑦, 𝑣), co-states and descent directions can be

calculated as Co-states Local optimal perturbations

Extractable from Simulink using command “Linearize”

• r numerical linearization

ҧ 𝐾𝑗 = 1 2 𝑦 𝑢∗

𝑗 − 𝑠 ∗ 𝑗 𝑈 𝑦 𝑢∗ 𝑗 − 𝑠 ∗ 𝑗 + න 𝑢∗

𝑗

𝜇𝑈 𝑔 𝑦, 𝑣 − 𝑒𝑦 𝑒𝑢 𝑒𝑢 𝜀𝑦𝑗(0) = −𝜇(0) 𝜀𝑥𝑗 𝑢 = − 𝜖𝐼 𝜖𝑥 = − 𝜖𝑔 𝜖𝑥 ቚ

𝑦𝑗,𝑣𝑗 𝑈

𝜇(𝑢)

@

Approach

15

t u(t)

𝑦0 = 𝑦1 𝑦2 : 𝑦𝑜

t x(t)

Linearizations: 𝜖𝑔 𝜖𝑦 , 𝜖𝑔 𝜖𝑣

• 1. Compute

specification robustness 𝝇.

• 2. Compute optimal

control based changes in 𝑣(𝑢) , 𝑦0

𝜒

t 𝜀u(t) 𝜀𝑦0, 𝜍

Compute new 𝑣(𝑢), 𝑦0

@

Example (Nonlinear system with FNN controller)

• Specification:

u 𝑢 ∈[-0.1,0.1]

16

• Falsifying trajectory with robustness −7.7 × 10−7.

ሶ 𝑦1 = −0.5 𝑦1 − 2𝑓−0.5𝑢 sin 3𝑢 + sin 𝑦2 ሶ 𝑦2 = −𝑦2 + 𝑦1

2(cos 𝑦2 + 𝑣 𝑢

+ 𝐺𝑂𝑂(𝑦1, 𝑦2) 𝑦1 0 = −0.2, 𝑦2(0) = 5 □((𝑦1 𝑢 < 0 ∧◇ 0,𝜗 𝑦1 𝑢 > 0) → ◇ 0,7 □ (𝑦1 𝑢 < 0.1))

@

• Model of a steam condenser with 5

continuous states based on energy balance and cooling water mass balance under an RNN controller with 6 discrete states

• Specification:

□ 30,35 𝑞(𝑢) ∈ [87,87.5] u 𝑢 ∈[3.99,4.01] Initial robustness: 0.20633 Final robustness: 0.00030222

Example (Steam Condenser with RNN Controller) 17

* Yi Cao, Dynamic Modelling of a Steam Condenser.

@

Experimental results 18

• We used Uniform Random Sampling (UR) and Simulated Annealing (SA)

implementations of S-Taliro unaided and aided by the optimal local search (UR+GD and SA+GD, respectively).

• Total runs: 40 times
• Maximum Number of simulations in each run: 600
• The UR and SA implementations include 12 control points and we let the switch times

vary.

• The improvement in the results from left to right in Table is evident and it motivates

the use of the proposed local search.

@

Experimental results

19

SA minimizer with 18 CP: Not falsifying SA+GD minimizer: Falsifying

In fact, from the ARCH 2019 Falsification competition: …

@

Scalability to the size of NN

20

The approach was tested on systems with:

• FNNs with 20 to 100 layers.
• Small RNNs:
• The approach works on simpler architectures of RNN, since layers of

RNN with delays increase the size of the state space

• 100 layers of RNN with 10 neurons on each layer which each

include 5 delay blocks will add 100 × 10 × 5 = 5000 states to the system.

• But … training RNNs with complex dynamics is anyway very hard

and RNNs used in systems usually have simple architectures

@

But wait … there is more! No need for NN: Steam Condenser with PID 21

• Model of a steam condenser with 5

continuous states based on energy balance and cooling water mass balance under a PID controller.

• Specification:

□ 30,35 𝑞(𝑢) ∈ [87.25,87.75] u 𝑢 ∈[3.99,4.01] Final robustness: -0.0040075

@

Experiments (Steam Condenser with PID)

22

SA+GD # of falsifications: 40/50 Avg min robustness: -0.00032895 Avg Execution time: 57.1898 Avg # of simulations: 58.84 Min robustness = -0.0196 SA # of falsifications: 1/50 Avg min robustness: 0.040579 Avg Execution time: 78.9605 Avg # of simulations: 200 Min robustness = -0.00021831

@

Summary

23

• Local optimal control improves falsification
• Our method locally improves the search for falsifying behaviors by

computing descent directions for the TL robustness in the search space

• f the falsification problem.
• Arbitrary (not parameterized) input signals
• We searched the infinite dimensional search space of the input using

results from optimal control.

• Can handle RNN or FNN with smooth activation functions
• Using an example of a steam condenser system controlled with a

Recurrent Neural Network, we demonstrated that our framework locally decreases the TL robustness.

@

counter-example guided adversarial training*

Bonus talk (2 talks in 1)

*Shakiba Yaghoubi & Georgios Fainekos, Worst-case Satisfaction of STL Specifications Using Feedforward Neural Network Controllers: A Lagrange Multipliers Approach, EMSOFT 2019

24

@

Vehicle Navigation Using the Lagrange Multiplier Update 25

Goal 1 Goal 2

ሶ 𝑦 = ሶ 𝑞𝑦 ሶ 𝑞𝑧 ሶ 𝜄 = 𝑤 cos(𝜄) 𝑤 𝑡𝑗𝑜(𝜄) 𝑤 𝑢𝑏𝑜(𝛿) Inputs 𝑤 ∈ [0,5], 𝛿 ∈ [−

𝜌 4 , 𝜌 4]

Car’s initial angle can vary in 𝜄0 ∈ [−

3𝜌 4 , − 5𝜌 8 ]

STL specification: 𝜒 = ◇ 1,40 𝐻𝑝𝑏𝑚1 ∧ ◇𝐻𝑝𝑏𝑚2 ∧ □ 1,40 ¬𝑉𝑜𝑡𝑏𝑔𝑓

@

Problem Formulation

26

Problem 1

max

𝑋

min

𝑦0∈𝑌0 𝜍𝜒 x 𝑦0, 𝑋

s.t. ൞ 𝑧 𝑢 = 𝑕(𝑦𝑢) x 𝑦0, 𝑋 = 𝑦0, 𝑦1, … , 𝑦𝑂 𝑦𝑢+1 = 𝑔(𝑦𝑢, 𝑂𝑂 𝑧𝑢, 𝑋 ) Maximize the worst-case behavior in X0 by tuning the NN (W)

@

Training NN on 𝑌0

𝑡 - Using Lagrange Multipliers

Consider the following objective function for some 𝑦0 ∈ 𝑌0

s

max

𝑋

𝐾𝑦0 𝑋 = 𝜍𝜒 𝑦0, 𝑦1, … , 𝑦𝑂 s.t. 𝑦𝑢+1 = 𝑔(𝑦𝑢, 𝑂𝑂 𝑕(𝑦(𝑢), 𝑋)

27

Problem 3

• max

𝑋

min

𝑦0∈𝑌0

s 𝜍𝜒 x 𝑦0, 𝑋

• s.t. ൞

𝑧 𝑢 = 𝑕(𝑦𝑢) x 𝑦0, 𝑋 = 𝑦0, 𝑦1, … , 𝑦𝑂 𝑦𝑢+1 = 𝑔(𝑦𝑢, 𝑂𝑂 𝑧𝑢, 𝑋 )

Dynamical constraints are added to the objective function using co-states max

𝑋

ҧ 𝐾𝑦0 𝑋 = 𝜍𝜒 𝑦0, 𝑦1, … , 𝑦𝑂 + ෍

𝑢=0 𝑂−1

𝜇𝑢+1

⊤

( 𝑔(𝑦𝑢, 𝑂𝑂(𝑕(𝑦𝑢), 𝑋))) − 𝑦𝑢+1)

ҧ 𝜍𝜒 : Smooth Approximation of Robustness (see Pant, Abbas, & Mangharam. Smooth

• perator: Control using the smooth robustness of temporal logic. CCTA 2017)

@

Quadrotor Mission (comparison)

28

@

• We cannot guarantee global optimality.
• A state feedback controller may not be enough. Some

memory may be required.

• Smooth robustness semantics are introduced for discrete

time systems.

Limitations

29

𝐻2 𝐻1

𝜒 = ◇ (𝐻1 ∧ ◇𝐻2)

@

30

Summary

• An approach for training NN controllers that satisfy system properties

given in STL was provided.

• The loss function for training is inspired by the robustness of the STL

formula defined over temporal sequences of the closed-loop system response.

• A formulation for gradient based training which does not result in

numerical issues when the length of the temporal sequences are large.

• The NNs were retrained using adversarial samples.

@

Summary

• An approach for training NN controllers that satisfy system properties

given in STL was provided.

• The loss function for training is inspired by the robustness of the STL

formula defined over temporal sequences of the closed-loop system response.

• A formulation for gradient based training which does not result in

numerical issues when the length of the temporal sequences are long.

• The NNs were retrained using adversarial samples.

31

@

Challenge?

From requirements for perception systems to guaranteed descent for closed-loop systems

32

BB requirements-driven testing seems to work!

Tuncali et al. Requirements-driven Test Generation for Autonomous Vehicles with Machine Learning Components, (To Appear in) IEEE TIV

@

Formal requirements for perception systems

33

Car in adjacent lane (Red Box) becomes undetected for 3 frames (Yellow Boxes) “At every time step, for all the objects (id) in the frame, if the object class is car with probability > 0.7, then in the next 5 frames the object (id) should still be detected and classified as a car with probability > 0.6”

𝜚2=□ 𝑦. ∀𝑗𝑒@𝑦, (𝐷 𝑦, 𝑗𝑒 = 𝐷𝑏𝑠 ∧ 𝑄 𝑦, 𝑗𝑒 > 0.7) → □(𝑧. 𝑦 ≤ 𝑧 ∧ 𝑧 ≤ 𝑦 + 5 → 𝐷 𝑧, 𝑗𝑒 = 𝐷𝑏𝑠 ∧ 𝑄 𝑧, 𝑗𝑒 > 0.6 )

@

What is the right level of abstraction?

34

Vehicle Control Perception System & Sensor Fusion Sensors Camera Controller Decision Making Low-level Control LIDAR Clustering etc Object Detection & Classification

@

Download our tools!

35

S-TaLiRo

Requirements driven test generation & monitoring

□( (g=5  ω<x) → ◇[0,τ] g=4) □( idle → ω>1100 RPM) □( (g1  “other” → ωem>0) □( turnoff → ◇[0,τ] cc=off)

https://sites.google.com/a/asu.edu/s-taliro/

Questions?!?

Any opinions, findings, and conclusions

• r recommendations expressed in this

material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

CNS 1350420, CNS 1932068

?