Long-Term Goal: Monitoring Driver Behavior /++01.23( - - PowerPoint PPT Presentation

▶
long term goal monitoring driver behavior
SMART_READER_LITE
LIVE PREVIEW

Long-Term Goal: Monitoring Driver Behavior /++01.23( - - PowerPoint PPT Presentation

Jul 21, 2023 516 likes •801 views

MSP - CRSS A NALYSIS OF D RIVER B EHAVIORS D URING C OMMON T ASKS U SING F RONTAL V IDEO C AMERA AND CAN-B US I NFORMATION J INESH J AIN C ARLOS B USSO July 14th, 2011 busso@utdallas.edu MSP - CRSS Problem Statement 100-car Naturalistic

slide-1
SLIDE 1

busso@utdallas.edu

MSP - CRSS

ANALYSIS OF DRIVER BEHAVIORS DURING COMMON TASKS USING FRONTAL VIDEO CAMERA AND CAN-BUS INFORMATION

JINESH JAIN CARLOS BUSSO

July 14th, 2011

slide-2
SLIDE 2

busso@utdallas.edu

MSP - CRSS

Problem Statement

  • 100-car Naturalistic Study: Over 78% of crashes

involved driver inattention

  • It is estimated that drivers engage in potentially

distracting secondary tasks about 30% of their time

[Ranney, 2008]

  • In-vehicle technologies, cell phones and navigation

systems are estimated to increase exponentially[Broy, 2006]

  • Detecting driver distraction early can have huge

advantages and reduce damage to lives and property

2

slide-3
SLIDE 3

busso@utdallas.edu

MSP - CRSS

Definition of Distraction

  • Report by Australian Road Safety Board
  • Highlights:
  • Voluntary or Involuntary diversion from primary driving

task

  • Not related to impairment due to alcohol, fatigue and

drugs

  • While performing secondary task focusing on a different
  • bject, event or person
  • Reduces situational awareness, decision making abilities

3

slide-4
SLIDE 4

busso@utdallas.edu

MSP - CRSS

Multimodal Information

  • Controller Area Network (CAN) Bus information
  • Steering wheel,

Vehicle speed, Brake, Gas [Kutila et al. 2007], [Liang et

  • al. 2007], [Ersal et al. 2010]
  • Video camera
  • Head pose, eyelid movement, lane tracking [Su et al. 2006], [Azman

et al. 2010]

  • Audio information from microphones [Sathyanarayana et al. 2010]
  • Invasive sensors to monitor physiological signals
  • EEG, ECG, pulse, respiration, head and leg movement [Putze

et al. 2010], [Sathyanarayana et al. 2008] 4

slide-5
SLIDE 5

busso@utdallas.edu

MSP - CRSS

Long-Term Goal: Monitoring Driver Behavior

5

!"#$%"&$#'( )&$*+&( ,+-.*$"&(

/++01.23( !$2&"4-"#+5( /&"#%.6(7.8+&.( 79:;,<5( =".0(7.8+&.(

Focus on this study is to identify relevant multimodal features

slide-6
SLIDE 6

busso@utdallas.edu

MSP - CRSS

Our Goal

  • Identify salient multimodal features to detect

driver distraction

  • Monitor driving behaviors while performing various

secondary tasks

  • Use real-world data
  • Use non-invasive sensors

6

slide-7
SLIDE 7

busso@utdallas.edu

MSP - CRSS

UTDrive

  • Highly sensorized driving research

platform.

  • Emphasis on understanding the driver

behavior during secondary tasks

  • cell-phone use, dialog systems, radio

tuning, navigation system.

  • Developing driver behavior models to

design human-centric active safety systems.

7

slide-8
SLIDE 8

busso@utdallas.edu

MSP - CRSS

UTDrive

  • Front facing camera
  • PBC-700
  • 320 x 240 at 30fps
  • 4 - channel Microphone array
  • 25kHz
  • CAN Bus for Steering wheel,

Vehicle speed, Brake, Gas

  • Road facing camera
  • 320 x 240 at 15fps

8

slide-9
SLIDE 9

busso@utdallas.edu

MSP - CRSS

UTDrive

9

  • Data Acquisition Unit - Dewetron
  • Data Extraction Software - Dewesoft
slide-10
SLIDE 10

busso@utdallas.edu

MSP - CRSS

Protocol

10

  • 2 runs of driving per subject
  • First run – with 7 tasks
  • Operating a Radio
  • Operating Navigation System

(GPS)

  • Operating and following
  • Cell phone
  • Operating and talking
  • Describing Pictures
  • Conversation with a Passenger
  • Second run – neutral driving

(without tasks)

8 drivers (updated version has 20 subjects) Good Day light, dry weather conditions to reduce environmental factors

slide-11
SLIDE 11

busso@utdallas.edu

MSP - CRSS

Modalities

  • CAN-Bus Information
  • Steering wheel angle (Jitter),

Vehicle Speed, Brake Value, Gas pedal pressures

  • Frontal Facing video Information:
  • Head pose (yaw and pitch), eye closure
  • Extracted with AFECT

11

slide-12
SLIDE 12

busso@utdallas.edu

MSP - CRSS

AFECT

Courtesy: Machine Perception Laboratory, University of California, San Diego

12

slide-13
SLIDE 13

busso@utdallas.edu

MSP - CRSS

Analysis of Driver Behavior

  • What features can be used to distinguish between

normal and task driving conditions?

  • Approach:
  • Contrasting features from task and normal conditions

(for each route segment)

  • Procedure:
  • Hypothesis testing (matched pairs)
  • Discriminant analysis (task versus normal conditions)

13

slide-14
SLIDE 14

busso@utdallas.edu

MSP - CRSS

Hypothesis Testing

  • Approach
  • Extract the mean and standard deviation of features
  • ver 5 sec windows
  • For each task and for each subject, evaluate the

different between normal and task conditions

  • Matched pairs Hypothesis Testing across speakers

14

slide-15
SLIDE 15

busso@utdallas.edu

MSP - CRSS

15

Hypothesis Testing

  • Matched pairs Hypothesis Testing (p = 0.05)
slide-16
SLIDE 16

busso@utdallas.edu

MSP - CRSS

16

Hypothesis Testing

  • The mean of head - yaw is an important feature
slide-17
SLIDE 17

busso@utdallas.edu

MSP - CRSS

Hypothesis Testing

  • Error plot for the mean of head - yaw

17

−30 −20 −10 10 20 Radio GPS Operating GPS Following Phone Operating Phone Talking Pictures Conversation Neutral Task −30 −20 −10 10 20 Radio GPS Operating GPS Following Phone Operating Phone Talking Pictures Conversation Neutral Task

slide-18
SLIDE 18

busso@utdallas.edu

MSP - CRSS

Hypothesis Testing

  • Histogram head yaw mean for Conversation

18

−30 −20 −10 10 20 R a d i
  • G
P S O p e r a t i n g G P S F
  • l
l
  • w
i n g P h
  • n
e O p e r a t i n g P h
  • n
e T a l k i n g P i c t u r e s C
  • n
v e r s a t i
  • n
Neutral Task

−50 −40 −30 −20 −10 10 20 30 40 50 0.2 0.4 0.6 Head −Yaw[

  • ]

Neutral Mean −50 −40 −30 −20 −10 10 20 30 40 50 0.2 0.4 Head −Yaw[

  • ]

Task Mean

slide-19
SLIDE 19

busso@utdallas.edu

MSP - CRSS

19

Hypothesis Testing

  • Some tasks produce higher deviation in the

features from normal conditions

slide-20
SLIDE 20

busso@utdallas.edu

MSP - CRSS

20

Hypothesis Testing

  • Other tasks produce small or no deviation in the

features from normal conditions

slide-21
SLIDE 21

busso@utdallas.edu

MSP - CRSS

Hypothesis Testing

  • Percentage of eye closure in task and normal conditions
  • Defined as percentage of frames in which the eyelids are

lowered below a given threshold

21

20 40 60 80 100 Radio GPS − Operating GPS − Following Phone − Operating Phone − Talking Pictures Conversation Neutral Task

slide-22
SLIDE 22

busso@utdallas.edu

MSP - CRSS

  • Binary classification per task: “Leave-one-out” cross

validation

  • Average classification Accuracy: k-NN classifier
  • Forward feature selection - Increase in performance

22

Binary Classification (task vs. normal conditions)

!"#$%& '()*+,-& .,-"%/& !"#$%& '())*& '()+*& '(+,'&

  • ./&0&1234"567&

'(+8+& '()+)& '(+,*&

  • ./&0&9%::%;$67&

'(*8)& '(*8+& '(*<=& .>%63&0&1234"567& '(?@'& '(?@'& '(),<& .>%63&0&A":B$67& '(*<*& '(=?'& '(=+,& .$CDE43F& '(+,)& '(+'*& '(+,)& G%6H34F"5%6& '(*<8& '(?,+& '(?@8& 0$1/&123%--&41-5-& 67898& 6789:& 678;<&

slide-23
SLIDE 23

busso@utdallas.edu

MSP - CRSS

Number of time that features were selected for binary classification tasks (out of 7)

23

Analysis of Driver Behavior

slide-24
SLIDE 24

busso@utdallas.edu

MSP - CRSS

  • 8 - class problem with k-NN
  • Normal and 7 tasks
  • “Leave-one-out” cross validation
  • Best accuracy = 40.7% at k = 10 compared to

baseline = 12.5%

24

Multiclass Classification

Secondary tasks

  • Radio
  • GPS - Operating
  • GPS - Following
  • Phone - Operating
  • Phone - Talking
  • Pictures
  • Conversation
slide-25
SLIDE 25

busso@utdallas.edu

MSP - CRSS

Conclusion and Discussion

  • Real-driving data while performing common secondary tasks
  • Multimodal features can discriminate between task and

normal conditions

  • Frontal camera 76.7%
  • CAN-Bus 76.5%
  • Fusion 78.9%
  • Highest accuracies
  • Radio, GPS Operating, Phone Operating and Pictures
  • Lowest accuracies
  • GPS - Following, Phone - Talking and Conversation

25

slide-26
SLIDE 26

busso@utdallas.edu

MSP - CRSS

Future Direction

  • Regression models to predict driver distraction.
  • We are collecting more data.
  • We now have 20 subjects.
  • We are studying other modalities.
  • Microphones, other CAN-bus signals.
  • Looking at the driver emotional state.
  • Study cognitive distractions.

26

slide-27
SLIDE 27

busso@utdallas.edu

MSP - CRSS

Discussion & Questions

27

THANK YOU!

Jinesh J. Jain and Carlos Busso

Multimodal Signal Processing (MSP) Laboratory

Erik Jonsson School of Engineering & Computer Science University of Texas at Dallas Richardson, Texas 75083, U.S.A.

Motivation Multimodal features Analysis of Features Discussion

  • Over 78% of crashes involved driver

inattention [Neale et al., 2005].

  • Drivers engage in potentially

distracting secondary tasks 30% the car is moving [Ranney, 2008].

  • Relevant problem since in-vehicle

technologies are estimated to increase.

  • Detection of distracted drivers is

crucial for the prevention of accidents.

Hypothesis Testing Future Directions

  • Multimodal features can discriminate

between task and normal conditions.

  • Frontal camera, 76.7%; CAN-Bus,
76.5%; and Fusion (78.9%).
  • Highest accuracies: Radio, GPS
Operating, Phone Operating and Pictures.
  • Lowest accuracies: GPS - Following,
Phone - Talking and Conversation.
  • CAN-Bus data is particularly useful for
Phone - Operating and Conversation.
  • CAN-Bus Information:
  • Jitter of steering wheel angle.
  • Vehicle speed.
  • Brake and gas pedal pressures
  • Frontal Facing video (AFECT [Barlett et al., 2008]):
  • Head pose (yaw and pitch).
  • Eye closure.
  • Features: mean & std of 5sec windows

Our Goal

  • Identify salient multimodal features to

detect inattentive drivers.

  • Use data from real driving conditions.
  • Use various noninvasive sensors.
  • Study common secondary tasks.

Driver Distraction

  • Diversion from primary driving task.
  • Not related to alcohol, fatigue and drugs.

Database

UTDrive

  • Frontal camera
  • Microphone array
  • CAN Bus
  • Road camera

Data Collection

  • 8 subjects.
  • First run - with 7 tasks.
  • Second run - normal driving (reference).
  • Secondary tasks:
  • Radio
  • GPS - Operating
  • GPS - Following
  • Phone - Operating
  • Phone - Talking
  • Pictures
  • Conversation
(5.6 mile track)
  • Normal versus tasks conditions.
  • Matched-pairs t-test (p-value = 0.05).
  • Head pose, blink and speed are salient.
  • Some tasks do not affect these features.
  • Phone – Talking, GPS – Following.

Error plots Discriminant analysis

Head – Yaw Vehicle Speed
  • Driver patterns change during

secondary tasks.

  • Drivers shift attention from the road.
  • Drivers reduce the car speed when
engaged in secondary tasks.
  • Characteristic of the route is an

important variable.

  • Task versus normal binary classification.
  • Forward feature selection.
  • K- Nearest Neighbor algorithm.
  • “Leave-one-out” cross validation.
  • Frequency that the features were selected.
  • 7 binary classifiers.
  • Regression models to predict driver

distraction.

  • We are collecting more data.
  • We now have 20 subjects.
  • We are studying other modalities.
  • Microphones, other CAN-bus signals.
  • Looking at the driver emotional state.
  • Study cognitive distractions.

ANALYSIS OF DRIVER BEHAVIORS DURING COMMON TASKS USING FRONTAL VIDEO CAMERA AND CAN-BUS INFORMATION

TH2: Poster 8 Today 13:30-15:30 Room: JHM