[PPT] - Safe Learning: A Challenge Talk CPS V&V I&F Workshop 2019 PowerPoint Presentation

SLIDE 1

Motivation Safety Bounds Safe Behaviors Future

Safe Learning: A Challenge Talk

CPS V&V I&F Workshop 2019

Kristin Yvonne Rozier

Iowa State University

December 11, 2019

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 2

Motivation Safety Bounds Safe Behaviors Future Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 3

Motivation Safety Bounds Safe Behaviors Future

What is Safe Learning?

Learning in an environment that is physically, emotionally, and socially safe 1

1study.com Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 4

Motivation Safety Bounds Safe Behaviors Future

– Safe Learning and Safe Acting –

What is learning? adding a behavior to an automated system in response to some observed pattern of operation

can be performed by a person or a machine can take many forms (automated, semi-automated)

“safe learning:” learned behavior is a safe action

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 5

Motivation Safety Bounds Safe Behaviors Future

– Safe Learning and Safe Acting –

What is learning? adding a behavior to an automated system in response to some observed pattern of operation

can be performed by a person or a machine can take many forms (automated, semi-automated)

“safe learning:” learned behavior is a safe action What is safe acting? performing an action that:

does not harm humans may prevent harm resulting from no action

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 6

Motivation Safety Bounds Safe Behaviors Future

How do we know an action is safe?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 7

Motivation Safety Bounds Safe Behaviors Future

How do we know an action is safe?

Need a proof!

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 8

Motivation Safety Bounds Safe Behaviors Future

How do we know an action is safe?

Need a proof!

Proof that the action is within a safety region?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 9

Motivation Safety Bounds Safe Behaviors Future

How do we know an action is safe?

Need a proof!

Proof that the action is within a safety region? Proof that harmful actions aren’t within the behavior space?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 10

Motivation Safety Bounds Safe Behaviors Future

How do we know an action is safe?

Need a proof!

Proof that the action is within a safety region? Proof that harmful actions aren’t within the behavior space? . . .

We need a specification of what is safe!

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 11

Motivation Safety Bounds Safe Behaviors Future

How do we know an action is safe?

Need a proof!

Proof that the action is within a safety region? Proof that harmful actions aren’t within the behavior space? . . .

Need a specification!

What are the safety requirements? What are the assumed safety bounds? How do we identify a violation?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 12

Motivation Safety Bounds Safe Behaviors Future

How do we know an action is safe?

Need a proof!

Proof that the action is within a safety region? Proof that harmful actions aren’t within the behavior space? . . .

Need a specification!

What are the safety requirements? What are the assumed safety bounds? How do we identify a violation?

Need a way of checking the implementation follows the proof, generated from the specification

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 13

Motivation Safety Bounds Safe Behaviors Future

Down a Level: What is Safe Learning?

What are the inputs and outputs?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 14

Motivation Safety Bounds Safe Behaviors Future

Safe Learning In Six Definitions

1 Learning within safety bounds Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 15

Motivation Safety Bounds Safe Behaviors Future

Safe Learning In Six Definitions

1 Learning within safety bounds 2 Learning safe behaviors Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 16

Motivation Safety Bounds Safe Behaviors Future

Safe Learning In Six Definitions

1 Learning within safety bounds 2 Learning safe behaviors → learning safety requirements Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 17

Motivation Safety Bounds Safe Behaviors Future

Safe Learning In Six Definitions

1 Learning within safety bounds 2 Learning safe behaviors → learning safety requirements →

safe behavior genesis

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 18

Motivation Safety Bounds Safe Behaviors Future

Safe Learning In Six Definitions

1 Learning within safety bounds 2 Learning safe behaviors → learning safety requirements →

safe behavior genesis

3 Refining behaviors to be more safe/conservative Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 19

Motivation Safety Bounds Safe Behaviors Future

Safe Learning In Six Definitions

1 Learning within safety bounds 2 Learning safe behaviors → learning safety requirements →

safe behavior genesis

3 Refining behaviors to be more safe/conservative 4 Learning that generates verification artifacts Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 20

Motivation Safety Bounds Safe Behaviors Future

Safe Learning In Six Definitions

1 Learning within safety bounds 2 Learning safe behaviors → learning safety requirements →

safe behavior genesis

3 Refining behaviors to be more safe/conservative 4 Learning that generates verification artifacts

Learning that passes verification tests

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 21

Motivation Safety Bounds Safe Behaviors Future

Safe Learning In Six Definitions

1 Learning within safety bounds 2 Learning safe behaviors → learning safety requirements →

safe behavior genesis

3 Refining behaviors to be more safe/conservative 4 Learning that generates verification artifacts

Learning that passes verification tests

5 Learning of proofs Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 22

Motivation Safety Bounds Safe Behaviors Future

2

2Pedro A. Ortega, Vishal Maini, and the DeepMind safety team. “Building safe artificial intelligence: specification, robustness, and assurance.” https://medium.com/@deepmindsafetyresearch/building-safe-artificial-intelligence-52f5f75058f1 Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 23

Motivation Safety Bounds Safe Behaviors Future

Specification: The Biggest Bottleneck in Formal Methods and Autonomy3

You are here Specifications Completeness Correctness Coverage Quality

Where are we now?

Continuously re-assess . . .

Where will we get specifications from? How should we measure specification quality? How do we best use specifications? How should we organize specifications?

3For expansions on these ideas, see: K.Y.Rozier. “Specification: The Biggest Bottleneck in Formal Methods and Autonomy.” VSTTE, 2016. Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 24

Motivation Safety Bounds Safe Behaviors Future

Specification: The Biggest Bottleneck in Formal Methods and Autonomy3

You are here Specifications Completeness Correctness Coverage Quality

Where are we now?

Continuously re-assess . . .

Where will we get specifications from? How should we measure specification quality? How do we best use specifications? How should we organize specifications? ... in the context of learning, autonomously acting systems?

3For expansions on these ideas, see: K.Y.Rozier. “Specification: The Biggest Bottleneck in Formal Methods and Autonomy.” VSTTE, 2016. Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 25

Motivation Safety Bounds Safe Behaviors Future

Learning in a Safety Region

Physics Design−time requirement Measurement Constraint

Learning

Logically follows: + 3 sides

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 26

Motivation Safety Bounds Safe Behaviors Future

Safety Bounds

Can use logical deduction (e.g., bound by SAT/SMT) Can use a priori known bounds (e.g., bounded learning) Can we use design-time requirements? Can we use technological limits?

what we can measure computational complexity what we can verify

Physics Design−time requirement Measurement Constraint

Learning

Logically follows: + 3 sides

Bottleneck: Where do we get these bounds from?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 27

Motivation Safety Bounds Safe Behaviors Future

Safety Bound Extraction from Learning

Post Learning: What Safety Bounds Were Learned? Rule extraction for Deep Neural Networks4 ML feature selection ML feature extraction5

4T. Hailesilassie. “Rule Extraction Algorithm for Deep Neural Networks: A Review.” IJCSIS, Vol. 14, No. 7, 2016
5S. Khalid, T. Khalil, S. Nasreen. “A Survey Of Feature Selection And Feature Extraction Techniques In Machine

Learning.” Science and Information Conference, 2014 Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 28

Motivation Safety Bounds Safe Behaviors Future

An Observation. . .

These bounds look a lot like sanity checks . . .

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 29

Motivation Safety Bounds Safe Behaviors Future

Dynamic Sanity Checking: Some Challenges

Dynamic Sanity Checks: change with different mission modes accommodate re-planning respond to unexpected environmental conditions allow human interaction

how to explain the purpose behind findings to humans how to create and monitor additional sanity checks per human request how to allow humans to refine definition of safety

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 30

Motivation Safety Bounds Safe Behaviors Future

Challenge: What Do The Bounds Look Like?

To be useful, bounds must obey patterns. . . What are the patterns? Measurable Precise Domain-specific (in the system domain, level of abstraction, units of the action being bounded) Translatable: English ⇐ ⇒ System-level (Semi-) Automatable What else?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 31

Motivation Safety Bounds Safe Behaviors Future

Challenge: What Do The Bounds Look Like?

To be useful, bounds must obey patterns. . . What are the patterns? Measurable Precise Domain-specific (in the system domain, level of abstraction, units of the action being bounded) Translatable: English ⇐ ⇒ System-level (Semi-) Automatable What else?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 32

Motivation Safety Bounds Safe Behaviors Future

Learning a Behavior

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 33

Motivation Safety Bounds Safe Behaviors Future

Learning a Behavior or Runtime Verification?

6

6Grigore Rosu and Klaus Havelund, 2001, https://www.runtimeverification.com/presentations/ Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 34

Motivation Safety Bounds Safe Behaviors Future

Learning a Behavior or Runtime Verification? Simulation?

6 7

6Grigore Rosu and Klaus Havelund, 2001, https://www.runtimeverification.com/presentations/ 7Kristin Yvonne Rozier. “From Simulation to Runtime Verification and Back: Connecting Single-Run Verification Techniques.” In Spring Simulation Conference (SpringSim19) 2019. Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 35

Motivation Safety Bounds Safe Behaviors Future

Purpose

The purpose of simulation is insight 8 whereas the purpose of RV is fault detection 9.

8Leemis, L. M., and S. K. Park. 2006. Discrete-event simulation: A first course. Pearson Prentice Hall Upper Saddle River, NJ. 9Leucker, M., and C. Schallhart. 2009. A brief account of runtime verification. The Journal of Logic and Algebraic Programming vol. 78 (5), pp. 293303. Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 36

Motivation Safety Bounds Safe Behaviors Future

Purpose

The purpose of simulation and learning? is insight 10 whereas the purpose of RV is fault detection 11.

10Leemis, L. M., and S. K. Park. 2006. Discrete-event simulation: A first course. Pearson Prentice Hall Upper Saddle River, NJ. 11Leucker, M., and C. Schallhart. 2009. A brief account of runtime verification. The Journal of Logic and Algebraic Programming vol. 78 (5), pp. 293303. Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 37

Motivation Safety Bounds Safe Behaviors Future

The Specification Bottleneck

Specification is the biggest bottleneck to RV.12 Can learning provide RV requirements?

12Rozier, K. Y. 2016, July. Specification: The Biggest Bottleneck in Formal Methods and Autonomy. In Proceedings of 8th Working Conference on Verified Software: Theories, Tools, and Experiments (VSTTE 2016), Volume 9971 of LNCS, pp.

119. Toronto, ON, Canada, Springer-Verlag.

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 38

Motivation Safety Bounds Safe Behaviors Future

Learning → RV

State Variables Σ = Specification Model M Model Computational Single Traces Generate Output Statistics Aggregate

f Safe Trace

Execution Characterization Engine RV Requirements Specification ϕ = Formal

...

σ ?

Figure : Possible workflow for connecting the outputs of leaning simulation runs to the inputs for runtime verification: if we can formalize and automate the translation of simulation output statistics to supply the requirements from which we create runtime monitors, we can mitigate the biggest bottleneck in RV.14

13Kristin Yvonne Rozier. “From Simulation to Runtime Verification and Back: Connecting Single-Run Verification Techniques.” In Spring Simulation Conference (SpringSim19) 2019. 14Kristin Yvonne Rozier. “From Simulation to Runtime Verification and Back: Connecting Single-Run Verification Techniques.” In Spring Simulation Conference (SpringSim19) 2019. Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 39

Motivation Safety Bounds Safe Behaviors Future

RV → Learning

}

State Variables Σ = Model Computational Output Statistics Aggregate Specification Model M Single Traces Generate Engine RV ϕ = Formal Requirements Specification

? ...

σ Learned Safe Behoviors Execution Set

Figure : Possible workflow for filtering individual runs using RV. But how do we know what checks to run?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 40

Motivation Safety Bounds Safe Behaviors Future

Safe Learning: Verification Artifacts and Proofs

How can learning algorithms generate verification inputs?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 41

Motivation Safety Bounds Safe Behaviors Future

Safe Learning: Verification Artifacts and Proofs

How can learning algorithms generate verification inputs? Can any learning algorithms generate verification artifacts?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 42

Motivation Safety Bounds Safe Behaviors Future

Safe Learning: Verification Artifacts and Proofs

How can learning algorithms generate verification inputs? Can any learning algorithms generate verification artifacts? Can they generate explainability artifacts?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 43

Motivation Safety Bounds Safe Behaviors Future

Safe Learning: Verification Artifacts and Proofs

How can learning algorithms generate verification inputs? Can any learning algorithms generate verification artifacts? Can they generate explainability artifacts? Can we even start to generate proofs?

Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges

SLIDE 44

Motivation Safety Bounds Safe Behaviors Future

Safe Learning Defined?

Doing one of more of the following, with or without automation or help from humans, driven by a specification that is checkable, with the provable result of minimizing harm to humans (through action or inaction):

1 Learning within safety bounds 2 Learning safe behaviors → learning safety requirements →

safe behavior genesis

3 Refining behaviors to be more safe/conservative 4 Learning that generates verification artifacts

Learning that passes verification tests

5 Learning of proofs Laboratory for

Temporal Logic

Kristin Yvonne Rozier Safe Learning Challenges