Impact of Traffic Lights on Trajectory Forecasting of Human-driven - - PowerPoint PPT Presentation

Impact of Traffic Lights on Trajectory Forecasting of Human-driven Vehicles Near Signalized Intersections

12th Workshop on Planning, Perception, Navigation for Intelligent Vehicle @ IROS2020

Geunseob (GS) Oh, Huei Peng University of Michigan

Video Reference: Waymo

Video Reference: Waymo

Host Vehicle (Present) Front Vehicle (Present) 𝐈𝐩𝐭𝐮 𝐖𝐟𝐢𝐣𝐝𝐦𝐟 (𝐆𝐯𝐮𝐯𝐬𝐟) Rear/Side Vehicle (Present)

We tackle this challenging vehicle forecasting problem near traffic lights (TLs) Goal: Trajectory forecasts for the host vehicle, 𝑌0:𝑈

𝐼𝑊.

Core elements of prediction near TL include:

𝐔𝐬𝐛𝐠𝐠𝐣𝐝 𝐦𝐣𝐡𝐢𝐮

Traffic Flow

Has received much less attention, despite the importance.

• 1. History of the host vehicle (HV), 𝑌−𝜐:0

𝐼𝑊

• 2. Interactions with other vehicles, 𝑌−𝜐:0

𝐺𝑊 , 𝑌−𝜐:0 𝑆𝑊 , 𝑌−𝜐:0 𝑇𝑊

• 3. Rule imposed by traffic light (TL), 𝑌𝑢

𝑈𝑀

Well addressed in literatures

Video Reference: Waymo

Host Vehicle (Present) Front Vehicle (Present) 𝐈𝐩𝐭𝐮 𝐖𝐟𝐢𝐣𝐝𝐦𝐟 (𝐆𝐯𝐮𝐯𝐬𝐟) Rear/Side Vehicle (Present)

We tackle this challenging vehicle forecasting problem near traffic lights (TLs) Goal: Trajectory forecasts for the host vehicle Our contribution:

• 1. Identification of the impacts of traffic lights on prediction; qualitative and quantitative
• 2. A novel prediction approach that is mindful of the impacts which utilizes vehicle-to-

infrastructure (V2I) communications.

𝐔𝐬𝐛𝐠𝐠𝐣𝐝 𝐦𝐣𝐡𝐢𝐮

Traffic Flow

Video Reference: Waymo

𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥

𝑢−𝜐

time

𝑢0 𝑢𝑈

𝐼𝑗𝑡𝑢𝑝𝑠𝑧 Possible predictions from methods that do not consider 𝑌𝑢

𝑈𝑀

Given the phase (Red) at t=0, 𝑌−𝜐:0

𝐼𝑊 , 𝑌−𝜐:0 𝐺𝑊

Existing methods would predict HV to stay put

𝐅𝐲𝐛𝐧𝐪𝐦𝐟 𝟐

How does traffic light impact the prediction?

𝑄ℎ𝑏𝑡𝑓

Video Reference: Waymo

𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥

𝑢−𝜐

time

𝑢0 𝑢𝑈

𝐼𝑗𝑡𝑢𝑝𝑠𝑧 Possible predictions from existing methods Given the phase (Red) at t=0, 𝑌−𝜐:0

𝐼𝑊 , 𝑌−𝜐:0 𝐺𝑊

Existing methods would predict HV to stay put

𝐅𝐲𝐛𝐧𝐪𝐦𝐟 𝟐

How does traffic light impact the prediction?

𝑄ℎ𝑏𝑡𝑓

𝑢−𝜐

time

𝑢0 𝑢𝑈

Ground-truth Actually, the phase changed to Green shortly after. The ground-truth trajectory started accelerating.

𝐔𝐢𝐟 𝐮𝐬𝐯𝐮𝐢 𝐣𝐭 …

𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝐼𝑗𝑡𝑢𝑝𝑠𝑧

Video Reference: Waymo

Given the phase (Yellow) at t=0, 𝑌−𝜐:0

𝐼𝑊 , 𝑌−𝜐:0 𝐺𝑊

Existing methods would predict HV to keep the speed

𝑢−𝜐

time

𝑢0 𝑢𝑈

𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝐼𝑗𝑡𝑢𝑝𝑠𝑧

𝐅𝐲𝐛𝐧𝐪𝐦𝐟 𝟑

How does traffic light impact the prediction?

Possible predictions from methods that do not consider 𝑌𝑢

𝑈𝑀

Video Reference: Waymo

Given the phase (Yellow) at t=0, 𝑌−𝜐:0

𝐼𝑊 , 𝑌−𝜐:0 𝐺𝑊

Existing methods would predict HV to keep the speed

𝑢−𝜐

time

𝑢0 𝑢𝑈

𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝐼𝑗𝑡𝑢𝑝𝑠𝑧

𝐅𝐲𝐛𝐧𝐪𝐦𝐟 𝟑

How does traffic light impact the prediction? 𝑢−𝜐

time

𝑢0 𝑢𝑈

Actually, the phase changed to Red shortly, The ground-truth trajectory started decelerating. 𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝐼𝑗𝑡𝑢𝑝𝑠𝑧

𝐔𝐢𝐟 𝐮𝐬𝐯𝐮𝐢 𝐣𝐭 …

Video Reference: Waymo

We propose a solution to the problem we identified

Future phase and timing can be shared through V2I Idea: Utilizing vehicle communications to infrastructures (V2I),

• btain the future profiles of TL states ahead of time

Image Reference: USDOT

Video Reference: Waymo

A sneak peek of the results

When we leverages the future profiles of TL (𝑌0:5𝑡

𝑈𝑀 ),

the predictions are so much better!

𝑢−𝜐 𝑢0 𝑢𝑈 𝑢−𝜐 𝑢0 𝑢𝑈

𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝐼𝑗𝑡𝑢𝑝𝑠𝑧 𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝐼𝑗𝑡𝑢𝑝𝑠𝑧

Blues and Reds (Fig. 4) are trajectories forecasted from our methods Pink: methods that do not leverage 𝑌0:5𝑡

𝑈𝑀

Video Reference: Waymo

A mapping function 𝑔 from states to actions

Prediction model - setup

𝑔 𝑌𝑢−𝜐:𝑢 = 𝑏𝑢

𝐼𝑊

We simplify the problem:

longitudinal prediction with the presence of a preceding vehicle Dataset limitation: rear/side vehicles were not modeled.

𝑌𝑢: state of the host vehicle + environment at time t 𝑏𝑢

𝐼𝑊: action of the host vehicle (acceleration)

HV Front Vehicle (FV)

A data-driven approach

Video Reference: Waymo

HV Front Vehicle (FV)

In detail, a state is defined as: 𝑌𝑢 ≔ 𝑌𝑢

𝐼𝑊, 𝑌𝑢 𝐺𝑊, 𝑌𝑢 𝑈𝑀, 𝑈𝑃𝐸𝑢

Host vehicle state (𝑌𝐼𝑊): Longitudinal position (i.e., distance to the intersection) & speed Context (𝐷 ≔ [𝑌𝐺𝑊, 𝑌𝑈𝑀, 𝑈𝑃𝐸]): 𝑌𝐺𝑊 ≔ [𝐺𝑊

𝑢, 𝑠𝑢, ሶ

𝑠𝑢] FV state: captures interactions with the front vehicle (binary flag for presence of FV, relative pos, speed) 𝑌𝑈𝑀 ≔ [𝑄𝑢, 𝑈𝑢] TL state: captures interactions with traffic light (phase (G,Y,R) and timing (time elapsed since the phase change)) 𝑈𝑃𝐸 Time of the day (0-24): macro-scopic traffic characteristics Output: Action taken by HV (longitudinal acceleration) 𝑔 𝑌𝑢−𝜐:𝑢 = 𝑏𝑢

𝐼𝑊

Prediction model - setup

Video Reference: Waymo

Dataset

Host vehicle (GPS, kinematics, time information) Traffic light (TL state profile) Front Camera (post-processed information on FV)

Image Reference: Google Map, UMTRI

We used real-world driving records & traffic light states from SPMD: Naturalistic Driving Records of 3,000 vehicles over 2 years

SPMD is a dataset established by USDOT & UMTRI

A signalized intersection (Plymouth-Huron Pkwy, Ann Arbor) was used for a study The study includes 50 cars passed through the intersection Total 502,253 sample trips made during 03/2015 – 05/2017 (27 months)

Video Reference: Waymo

MDN captures competing policies And allows probabilistic interpretation

Modeling Intuition

Prediction model

Deterministic Policy (𝑔

𝑒 ) Learning: RNN

LSTM 𝑔

𝑒 𝑌𝑢−𝜐:𝑢 𝐼𝑊 , 𝐷𝑢 = 𝑏𝑢 𝐼𝑊

𝑏𝑢

𝐼𝑊

𝐷𝑢 . . . 𝑌𝑢−𝜐

𝐼𝑊

𝑌𝑢

𝐼𝑊

+

MLP

(a)

Probabilistic Policy (𝑔

𝑞 ) Learning: RNN-MDN

𝑔

𝑞 𝑌𝑢−𝜐:𝑢 𝐼𝑊 , 𝐷𝑢 = 𝑎𝑢

LSTM 𝐷𝑢 . . . 𝑌𝑢−𝜐

𝐼𝑊

𝑌𝑢

𝐼𝑊

+

M D N 𝑞(𝑏𝑢

𝐼𝑊|𝑌𝑢−𝜐:𝑢 𝐼𝑊 , 𝐷𝑢; 𝑎𝑢)

𝑎𝑢

(b)

RNN(LSTM) models temporal dependencies

Video Reference: Waymo

Prediction framework

Autoregressive prediction using the learned policies to obtain the roll-outs

Video Reference: Waymo

Qualitative evaluation

Scenario YR 𝑢0 Scenario G Scenario GYR 𝑢𝑗𝑛𝑓 𝑢𝑗𝑛𝑓 𝑢𝑗𝑛𝑓 𝑢𝑈 𝑢0 𝑢0 𝑢𝑈 𝑢𝑈 𝑄𝑠𝑓𝑒𝑗𝑑𝑢𝑗𝑝𝑜 𝑥𝑗𝑜𝑒𝑝𝑥 𝑔

𝑒

𝑔

𝑒 𝑂𝑝𝐺𝑊

𝑔

𝑒 𝑂𝑝𝑈𝑀

[𝑌𝐼𝑊, 𝑌𝐺𝑊, 𝑌𝑈𝑀, 𝑈𝑃𝐸] → 𝑏𝑈𝑀 [𝑌𝐼𝑊, , 𝑌𝑈𝑀, 𝑈𝑃𝐸] → 𝑏𝑈𝑀 [𝑌𝐼𝑊, 𝑌𝐺𝑊, , 𝑈𝑃𝐸] → 𝑏𝑈𝑀 The impact of TL: 𝑔

𝑒 vs 𝑔 𝑒 𝑂𝑝𝑈𝑀

𝐼𝑊(𝑄𝑠𝑓𝑡𝑓𝑜𝑢) 𝐺𝑊(𝑄𝑠𝑓𝑡𝑓𝑜𝑢) 𝑰𝑾(𝑮𝒗𝒖𝒗𝒔𝒇)

The deterministic policies Scenarios Problem

Video Reference: Waymo

Qualitative evaluation with 3 sample episodes: TL vs NoTL

The results demonstrate how the utilization of the future 𝒀𝑼𝑴 can improve the predictions Our models (blue and red) produce more accurate predictions than the model (pink) that doesn’t utilize future 𝒀𝑼𝑴 Given ground-truth (Black) trajectories, the trajectory forecasts from the following 3 models are compared: All (𝑔

𝑒 , Blue) : a model which uses both 𝒀𝑮𝑩 and future 𝒀𝑼𝑴

No FV (𝑔

𝑒 𝑂𝑝𝐺𝑊, Red) : a model which doesn’t use 𝒀𝑮𝑩

No TL (𝑔

𝑒 𝑂𝑝𝐺𝑊, Pink) : a model which doesn’t use future 𝒀𝑼𝑴

Benchmarking purpose Our approach

Video Reference: Waymo

Quantitative evaluation with 3111 test samples & ablation study

Evaluation metrics: Models for the ablation study:

𝑔

𝑒

𝑔

𝑒 𝑂𝑝𝐺𝑊

𝑔

𝑒 𝑂𝑝𝑈𝑀

𝑔

𝑒 𝑂𝑝𝐺𝑊𝑈𝑀

[𝑌𝐼𝑊, 𝑌𝐺𝑊, 𝑌𝑈𝑀, 𝑈𝑃𝐸] → 𝑏𝑈𝑀 [𝑌𝐼𝑊, , 𝑌𝑈𝑀, 𝑈𝑃𝐸] → 𝑏𝑈𝑀 [𝑌𝐼𝑊, 𝑌𝐺𝑊, , 𝑈𝑃𝐸] → 𝑏𝑈𝑀 [𝑌𝐼𝑊, , 𝑈𝑃𝐸] → 𝑏𝑈𝑀 The impact of TL: 𝑔

𝑒 vs 𝑔 𝑒 𝑂𝑝𝑈𝑀

Scenarios: G, R, GY, YR, RG, GYR

Video Reference: Waymo

Quantitative evaluation with 3111 test samples & ablation study

Red: All X:=[𝑌𝐼𝑊, 𝑌𝐺𝑊, 𝑌𝑈𝑀, 𝑈𝑃𝐸] Green: No 𝑌𝐺𝑊 Blue: No 𝑌𝑈𝑀 Cyan: No 𝑌𝐺𝑊, 𝑌𝑈𝑀 Accuracy: No 𝒀𝑮𝑾 > All X >>> No 𝑌𝑈𝑀 > No 𝑌𝐺𝑊, 𝑌𝑈𝑀

Exclusion of 𝑌𝐺𝑊 increases the prediction accuracy, due to the uncertainty in 𝑌0:𝑈

𝐺𝑊

Note, 𝑌0:𝑈

𝐺𝑊 has also to be forecasted (unlike 𝑌0:𝑈 𝑈𝑀 which are available through V2I)

N: 688 N: 1909 N: 68 N: 81 N: 362 N: 32 Sample Size N ADN: Lower the better The access to 𝑌0:𝑈

𝑈𝑀 improves the quality of forecasts significantly,

resulting 1.5 - 30 times smaller error (No 𝑌𝐺𝑊 vs No 𝑌𝑈𝑀) depending on the scenario.

T=15s for GYR T=5s for others

Video Reference: Waymo

We identified the scenarios where the existing forecasting methods could perform poor and proposed a novel solution to this problem that leverages the future TL states. (1) Identification of a new problem where the existing forecasting methods might suffer (2) Demonstration of how the access to future TL states improve the predictions (3) Longitudinal trajectory forecasting algorithms which solve the problem

Contribution Conclusion

Due to the dataset availability, interactions with rear & side cars were not considered and no perception data (e.g., lidar, radar) was used. Nevertheless, we believe that the proposed solution makes a step forward towards more accurate modeling and trajectory forecasting of human-driven vehicles.

Video Reference: Waymo

Thank You