Automatic Test Generation for Autonomous Vehicular Systems Cumhur - - PowerPoint PPT Presentation

1

Lab CPS PS

Dagstuhl 2016: Robustness in CPS School of Computing, Informatics and Decision System Engineering Arizona State University  fainekos at asu edu  http://www.public.asu.edu/~gfaineko

Cumhur Erkan Tuncali, Theodore P. Pavlic and Georgios Fainekos

Automatic Test Generation for Autonomous Vehicular Systems

Based on work to be presented: International Conference on Intelligent Transportation Systems (ITSC) 2016

2

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Autonomous Driving

Toyota Google Volvo Mercedes

3

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Autonomous Driving

Toyota Google Volvo Mercedes Current (winning) strategy:

• 1. Use professional drivers to train deep NN
• 2. Collect as much data as possible so that you do not need humans for

supervision

• 3. Deploy vehicles and continue collecting knowledge

Main reasoning for the approach: You cannot possibly predict and program the vehicles for every possible situation so you have to learn and adjust.

4

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Autonomous Driving

Some obvious questions:

• 1. Do learning methods work for every vehicle (dynamics) or each car

model will need different training process? What about new car models?

• 2. If NN/ML is the winning approach, then how can we tell (after

deployment) if there concerning “corner” cases?

• 3. How do we achieve further levels of automation for more efficient

transportation systems, e.g., platooning, intersection navigation, vehicle coordination, etc?

• 4. How do we check if cooperating vehicular systems are robust to

disturbances?

5

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

How do we measure robustness?

Euclidean distance or other norms do not work Lane dependent measures also do not work

Time-to-Collision (TTC*) : Time required to collision with current heading and velocity TTC:  TTC: small number

*J. C. Hayward, “Near-miss determination through use of a scale of danger,” Highway Research Record, no. 384, 1972.

6

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

What if we are searching for non-robust behaviors?

Our claim: We need to detect and robustify “boundary” situations.

7

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Robustness Metric

Goal is to find boundaries between safe and unsafe behavior! Minimizing the robustness function should guide the search towards the boundary Time-to-Collision (TTC)*: Time required to collision with current motion

*J. C. Hayward, “Near-miss determination through use of a scale of danger,” Highway Research Record, no. 384, 1972.

Robustness function Simulation trajectory Collision speed (relative) Minimum collision severity Minimum TTC in trajectory Maximum possible collision speed

8

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Robustness Metric

An example robustness function. Maximum possible collision speed = 100

Large TTC: Small collision “risk” Collision at a small relative velocity Collision at a maximum relative velocity

9

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Problem Formulation

Given:

• a set of Vehicles Under Test (VUT),
• a set of dummy actors (objects physically interacting with VUT in the

environment, e.g., human-driven vehicles, pedestrians etc.),

• surroundings (e.g., road network, weather and road conditions etc.),

m and the boundaries for initial conditions, states and functions defining motion (of dummy actors or VUT), compute the initial conditions and inputs/trajectories which lead to an operation on the boundaries between safe and unsafe behavior.

Image from: http://www.techspot.com/news /64373-platoons-self-driving- trucks-made-their-way- across.html

10

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Framework Architecture

S-TaLiRo

• A MATLAB toolbox for systematic testing of hybrid systems.
• Robustness metric: How far a system trajectory is from falsifying formal

system requirements (negative robustness means a requirement is falsified)

• Optimization methods (e.g., simulated annealing, ant colony optimization)

11

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Case Study

Simulation Configuration:

• Two vehicles under test
• One dummy vehicle
• Two-lane straight road

Simulation Engine:

• Simulates VUT using a vehicle dynamic model
• Simulates dummy vehicles using a kinematic model
• Implemented in MATLAB (Can be changed to another platform)

Initial Conditions:

• Both VUT on the right lane separated by a distance
• The dummy vehicle is on the left lane next to one of the VUT

12

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Case Study

Vehicle Configuration:

• Critical points define the vehicle (corners, sensor positions etc.)
• Sensor locations, orientation and ranges are defined
• VUT controlled by a Model Predictive Controller
• Dummy vehicles are controlled by a PID controller

A side sensor with 5m range and 10° sensing angle A front sensor with 10m range and 10° sensing angle Blind area !

13

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Case Study

Experiment Results: Seeking a trajectory for the dummy vehicle which causes a behavior at the boundary between collision and no-collision operations. (A very slow speed collision or a very near miss) (The trajectory for the dummy vehicle) (A front collision right after avoiding a side collision)

14

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Where are we going with this?

High level defined with π-calculus expressions Low level defined with hybrid automata

Campbell et al Modeling Concurrency and Reconfiguration in Vehicular Systems: A π-calculus Approach, IEEE CASE, 2016

Mercedes-Benz More complicated sensor models

15

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Where are we going with this?

Testing sensing and perception algorithms as part of the system.

* Campbell et al Traffic Light Status Detection Using Movement Patterns of Vehicles, IEEE Intelligent Transportation Systems Conference, 2016

Typical Example: Traffic Light Status Detection Using Movement Patterns of Vehicles*

16

Lab CPS PS

Dagstuhl 2016: Robustness in CPS

Awards: CNS-1446730

Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Thank you! Questions?

Eventually we will be able to trust

• ur lives on CPS!