Correctness and Control for Human Cyber-Physical Systems Dorsa - - PowerPoint PPT Presentation

Correctness and Control for Human Cyber-Physical Systems

Dorsa Sadigh

Advisors: Sanjit A. Seshia, S. Shankar Sastry

University of California, Berkeley Department of Electrical Engineering and Computer Sciences

Non-Zero-Sum-Games and Control, Schloss Dagstuhl

February 3, 2015

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Motivation: Safety Critical Human CPS

Giving guarantees about the performance of control algorithms for safety critical Human CPS is a challenging problem.

2 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

The Human CPS Problem

Performing a task and satisfaction

• f properties.

autonomous control human control

Shared or Swtiched Control Setting Interface Layer

3 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

The Human CPS Problem

Performing a task and satisfaction

• f properties.

autonomous control human control

Shared or Swtiched Control Setting Interface Layer

human system envrionment specifjcation

3 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

The Human CPS Problem

Performing a task and satisfaction

• f properties.

autonomous control human control

Shared or Swtiched Control Setting Interface Layer

human system envrionment specifjcation

One challenge is detecting why and when the robot is going to fail and transfer control to the human in such scenarios.

3 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Contributions

• Formalization of human-in-the-loop control systems.1

4 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Contributions

• Formalization of human-in-the-loop control systems.1
• Proposing a new take on specification, algorithm and output of

controller synthesis from Temporal Logic specifications for human-in-the-loop systems.

4 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Contributions

• Formalization of human-in-the-loop control systems.1
• Proposing a new take on specification, algorithm and output of

controller synthesis from Temporal Logic specifications for human-in-the-loop systems.

• Extracting specifications from high level guidelines.
• Mining and Monitoring assumptions about the environment.
• Advisory controller that decides whether the human or

autonomous system should be in control.

4 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Contributions

• Formalization of human-in-the-loop control systems.1
• Proposing a new take on specification, algorithm and output of

controller synthesis from Temporal Logic specifications for human-in-the-loop systems.

• Extracting specifications from high level guidelines.
• Mining and Monitoring assumptions about the environment.
• Advisory controller that decides whether the human or

autonomous system should be in control.

human system envrionment specifjcation system envrionment specifjcation

• 1. Synthesis for Human-in-the-Loop Control Systems. W. Li, D. Sadigh, S. Sastry, S. Seshia. TACAS 2014.

4 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Unrealizable Left Turn Example

Temporal Spec Synthesis Tool Realizable Unrealizable Autonomous Control

:(

Synthesizing a controller for a vehicle making an unprotected left turn at a traffic light is unrealizable. Human driver decides how to change the objective or violate the specification.

5 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

National Highway Transportation and Safety Administration

Level 0 Level 1 Level 2 Level 3 Level 4 No Automation

Driver is in complete control

Function Specific Automation

precharged brakes

Combined Function Automation

Cruise Control + Lane Keeping

Limited Self Driving Automation Full Self Driving Automation

6 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

National Highway Transportation and Safety Administration

Level 0 Level 1 Level 2 Level 3 Level 4 No Automation

Driver is in complete control

Function Specific Automation

precharged brakes

Combined Function Automation

Cruise Control + Lane Keeping

Limited Self Driving Automation Full Self Driving Automation

7 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Level 3: Limited Self Driving Automation

“Vehicles at this level of automation enable the driver to cede full control

• f all safety-critical functions under certain traffic or environmental

conditions and in those conditions to rely heavily on the vehicle to monitor for changes in those conditions requiring transition back to driver control. The driver is expected to be available for occasional control, but with sufficiently comfortable transition time. The Google car is an example of limited self-driving automation.”1

• 1. National Highway Traffic Safety Administration. Preliminary statement of policy concerning automated vehicles,

May 2013.

8 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Criteria for Human in the Loop Controllers

• Monitoring

⋆ Determine if the human intervention is needed based on past and current information.

9 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Criteria for Human in the Loop Controllers

• Monitoring

⋆ Determine if the human intervention is needed based on past and current information.

• Minimally Intervening

⋆ Invoke human operator only if necessary.

9 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Criteria for Human in the Loop Controllers

• Monitoring

⋆ Determine if the human intervention is needed based on past and current information.

• Minimally Intervening

⋆ Invoke human operator only if necessary.

• Prescient

⋆ Allow sufficient response time for the human operator.

9 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Criteria for Human in the Loop Controllers

• Monitoring

⋆ Determine if the human intervention is needed based on past and current information.

• Minimally Intervening

⋆ Invoke human operator only if necessary.

• Prescient

⋆ Allow sufficient response time for the human operator.

• Conditionally Correct

⋆ Controller should operate correctly until the point of human intervention.

9 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Problem Formulation

Temporal Spec Autonomous Control Response time Intervention Cost Function Human Control

?

Given:

• a high level specification,
• driver’s response time,
• a cost function penalizing

human’s intervention Synthesize a fully autonomous controller satisfying the specification Or a Human in the Loop Controller (composition of auto-controller, human operator, advisory controller) that is: ⋆ Monitoring ⋆ Prescient ⋆ Minimally Intervening ⋆ Conditionally correct

10 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Synthesis from GR(1)

Specification: ψenv = GF¬x Environment Fairness ψsys = G(¬x → ¬y) System Transition ∧ GFy System Fairness

Environment System

x y

I = {x} set of inputs O = {y} set of outputs GR(1) Specifications: ψ := ψenv → ψsys ψ{env,sys} = ψi ∧ ψt ∧ ψf

11 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Game Graph: Satisfiable Run

Specification: ψenv = GF¬x ψsys = G(¬x → ¬y) ∧ GFy Env. Sys. x = 0 y = 0 x = 1 y = 1 x = 0 y = 0 x = 1 y = 1 . . . . . .

12 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Game Graph: Satisfiable Run

xy ¯ x ¯ y x ¯ y ¯ x ¯ y x ¯ y xy ¯ x ¯ y xy x ¯ y ¯ x ¯ y Specification: ψenv = GF¬x ψsys = G(¬x → ¬y) ∧ GFy Each state of the game graph G is marked by inputs and outputs. Env. Sys. x = 0 y = 0 x = 1 y = 1 x = 0 y = 0 x = 1 y = 1 . . . . . .

12 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Game Graph: Unsatisfiable Run

Specification: ψenv = GF¬x ψsys = G(¬x → ¬y) ∧ GFy System transition is satisfied, but system fairness is violated. The specification is unrealizable. Env. Sys. x = 0 y = 0 x = 0 y = 0 x = 0 y = 0 x = 0 y = 0 . . . . . .

13 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Game Graph: Unsatisfiable Run

xy ¯ x ¯ y x ¯ y ¯ x ¯ y x ¯ y xy ¯ x ¯ y xy x ¯ y ¯ x ¯ y Specification: ψenv = GF¬x ψsys = G(¬x → ¬y) ∧ GFy System transition is satisfied, but system fairness is violated. The specification is unrealizable. Env. Sys. x = 0 y = 0 x = 0 y = 0 x = 0 y = 0 x = 0 y = 0 . . . . . .

13 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Generating Counterstrategy Graph

Temporal Spec Synthesis Tool Realizable Unrealizable Autonomous Control Counterstrategy Graph

14 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Generating Counterstrategy Graph

Temporal Spec Synthesis Tool Realizable Unrealizable Autonomous Control Counterstrategy Graph

xy, ¯ x ¯ x ¯ y, ¯ x x ¯ y, ¯ x ¯ x ¯ y, ¯ x ¯ y ¯ y ¯ y ¯ y In the counterstrategy graph C, the states S = (Q, I) are marked by game graph’s states and the next input value. Transitions occur on receiving output variables from O.

14 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Why do we care about counterstrategy graphs?

Temporal Spec Synthesis Tool Realizable Unrealizable Autonomous Control Counterstrategy Graph Mine Assumptions

1 2,3 1-R. K¨

• nighofer, G. Hofferek, and R. Bloem. Debugging formal specifications using simple counterstrategies.

FMCAD 2009. 2-K. Chatterjee, T. A. Henzinger, and B. Jobstmann. Environment assumptions for synthesis. CONCUR 2008. 3-W. Li, L. Dworkin, and S. Seshia. Mining assumptions for synthesis. MEMOCODE 2011.

15 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Satisfaction of NHTSA Guideline

Temporal Spec Synthesis Tool Realizable Unrealizable Autonomous Control Counterstrategy Graph Mine Assumptions Response time Intervention Cost Function Prescient Minimally Intervening Conditionally Correct

16 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Counterstrategy Graph: Condensing Failure Prone Nodes

Nodes that violate safety conditions:

17 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Counterstrategy Graph: Condensing Failure Prone Nodes

Nodes that violate safety conditions: Nodes that violate liveness conditions:

17 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Counterstrategy Graph: Condensing Failure Prone Nodes

Nodes that violate safety conditions: Nodes that violate liveness conditions: Contracting all strongly connected components to a single node to convert Gc to a DAG.

17 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Prescient Condition

strongly connected components

source sink

... ... ... ...

T-1

Assuming T is human’s response time, we allow enough time to the human to take back control. Remove T − 1 nodes to failure prone nodes in the counter strategy graph.

18 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Weighted Counterstrategy Graph

strongly connected components

source sink

... ... ...

T-1

1 1 1

L (v,w)< 1

v w

• Num. of env. actions.
• Shortest path to failure

prone nodes.

• State of the human

driver.

19 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Minimally Intervening

min

• e:=(v,w)∈E

L(v, w) The minimization guarantees a minimal set of edges (assumptions) to be monitored and minimally intervening Human in the Loop controller.

source sink

...

L (v,w)

v w

20 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Mining Assumptions from the Edges

The edges that are found by min-cut algorithm correspond to environment specifications that need to be disallowed. G((x ∧ y) → ¬X(x′)) xy, x′ y ′

21 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Mining Assumptions from the Edges

The edges that are found by min-cut algorithm correspond to environment specifications that need to be disallowed. G((x ∧ y) → ¬X(x′)) xy, x′ y ′ For all the edges we mine assumptions of the following format: Φ =

• i

(G(ai → ¬Xbi))

21 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Mining Assumptions from the Edges

The edges that are found by min-cut algorithm correspond to environment specifications that need to be disallowed. G((x ∧ y) → ¬X(x′)) xy, x′ y ′ For all the edges we mine assumptions of the following format: Φ =

• i

(G(ai → ¬Xbi)) Here Φ is the set of all mined assumptions from the cut: Realizable: ψnew = (φ ∧ ψenv) → ψsys

21 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Conditionally Correct

A monitor outputs auto = false, if any of the assumptions are violated.

• G ((x ∧ y) → ¬X ¯

x)

• ∧
• G ((¯

x ∧ ¯ y) → ¬X ¯ x)

• ∧
• G ((x ∧ ¯

y) → ¬X ¯ x)

• xy, ¯

x ¯ x ¯ y, ¯ x x ¯ y, ¯ x ¯ x ¯ y, ¯ x ¯ y ¯ y ¯ y ¯ y

22 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Advisory Controller

Temporal Spec Synthesis Tool Realizable Unrealizable Autonomous Control Counterstrategy Graph Mine Assumptions Assumption Monitoring ψ

new

:= (φ Λ ψ

env)—>

ψ

sys

φ

23 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Advisory Controller

Temporal Spec Synthesis Tool Realizable Unrealizable Autonomous Control Counterstrategy Graph Mine Assumptions Assumption Monitoring Human Driving ψ

new

:= (φ Λ ψ

env)—>

ψ

sys

φ auto = false

24 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Our Human CPS Problem

Performing a task and satisfaction

• f properties.

Shared or Swtiched Control Setting Interface Layer

f a i l

One challenge is detecting why and when the robot is going to fail and transfer control to the human in such scenarios.

25 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Our future Human CPS Problem

Performing a task and satisfaction

• f properties.

Shared or Swtiched Control Setting Interface Layer

fail

One challenge is detecting why and when human is going to fail and transfer control to the robot in such scenarios.

26 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Future Directions

More expressive human models e.g. probabilistic human behvaior models,

• nline reaction time modeling, etc.

Detecting failure in other control algorithms by using the idea of counterstrategy graphs. Considering uncertain environments that both human and robot has to interact with.

27 / 27

Motivation NHTSA Guideline Human-in-the-Loop Control Control Advisor Future Directions

Future Directions

More expressive human models e.g. probabilistic human behvaior models,

• nline reaction time modeling, etc.

Detecting failure in other control algorithms by using the idea of counterstrategy graphs. Considering uncertain environments that both human and robot has to interact with.

https://www.eecs.berkeley.edu/∼dsadigh/

27 / 27