Driving Anomaly Detection with Conditional GAN using Physiological - PowerPoint PPT Presentation

Driving Anomaly Detection with Conditional GAN using Physiological Data & CAN-Bus Data Yuning Qiu (UT Dallas) Teruhisa Misu (Honda Research Inst.) Carlos Busso (UT Dallas)

Motivation Goal: Categorization of driving data for future driver assistance systems 1. Known-knowns 3. Unknown-knowns Beyond-control situations: Easy and usual situations: w g a n L i (Situations that we give-up n , n r a o e i (Situations that we know and t l a e l n u i g h e c R a M : to handle) n / o e i l t know how to handle) u u R o l S : n o i t u - Flying vehicle o l S - Freeway driving - Reckless ped. behaviors 2. known-unknowns 4. Unknown-unknowns Unexpected Very difficult situations: situations: (Situations that we know are difficult, r e v o - e k a T : but not know how to handle it) n ????? o ? i ? t u ? ? l o : S n o i t u - complex intersection l o S 5. Driver errors First target à Detection of “anomaly” situations 2-5 2

Motivation § Examples include: Driver errors: On-road § Lack of awareness of objects, pedestrians, or vehicles pedestrian Anomaly surrounding situations: § Hazard actions from other vehicles § Unexpected changes that leads to hazard scenarios § Our approach: ( ßà Supervised approach, Look-outside approach) Hazard § Unsupervised methods without pre-set patterns action § Patterns of abnormal driving scenarios are difficult to determine § To discover unknown abnormal scenarios § Detect anomalies from driver reaction ß Driver should have reacted to anomalies (Driving is “interaction”!) 3

Multimodal Signals – From our Previous Work § Relationship between driving maneuvers and drivers’ physiological signals [Qiu et al., 2019] § Heart rate (HR), Breath rate (BR), and electrodermal activity (EDA) can be used to discriminate different driving maneuvers § Physiological signals respond to other events (e.g., driver stress, surprise) § Physiological signals are useful for driving maneuver classification when combined with features extracted from CAN-bus data [Li et al., 2016] In this work, we tackle anomaly detection using • CAN-bus data • Physiological signals (HR, BR, EDA) • (NO images/videos) 4

1. Motivation 2. Driving Anomaly Dataset (DAD dataset) 3. Proposed Model 4. Experimental Evaluation 5. Conclusions

Driving Anomaly Dataset (DAD) § 250 hours of naturalistic driving recordings § Single driver § 48 hours used in this study § Collected in a city in Asia § Driving events are manually annotated based on forward- facing in-vehicle camera 6

<latexit sha1_base64="nWC41+LpVrxKQ9eyc6ToFGu8PSE=">AEKXicbVNb9NAEHUdPkr4SuHIZUQE4pIorpColAMtQIkEKFt2qImqtbribPKetfaXaeKTP4OF/4KF5BAwJU/wq7z3bKSpfF7O2/ejMdhypk2jcbvDb905eq165s3yjdv3b5zt7J170jLTFHsUMmlOgmJRs4EdgwzHE9ShSQJOR6Hw5bj0eoNJPi0IxT7CUkFqzPKDEWOtvyn3dDjJnIDQkzTtQkT/OgvkOTCXyCNHfBpNwdOPkyPIY9IaQpUjVAtztnXknCa5FiIxSwRx1t74RxhbG6WtKtGYibsJb7BswmRJN2GfxYB4XMCeCItABETHO6XWspaTW54QPl4Kd2roCQqhszmBFYAm9QxXj0veBYUnGM3J+4GRaRNe4ogRgyuNtkim0fEstiVb0prSLqUJh4r07ViBu6JNaGNkGcWIpdpEDTECStSK0p4xKGbViq5sMzDCAaN83qdD0hWdfSsyk29lpsYsNPXxXtSUJBGEjI6pW5pFzYWga3ytwkfkXJ7P9fZd+rlUw1XnINBaxlgWM1xa76KIFvtyVqk26o3iwOUgmAVb3baZ5Xv3UjSLHdU24/4mnQSE0vJ8o4Y3bZ7IRTQockxlMbCpKg7uXFpk/gkUi6EtlH2GgQFczcpJoPU5CezMhZqAvcg78H3eamf5OL2cizexnodNC/YyDkeB+G4iYspvMxzYgVDHr1e2kItTteNkOIbjY8uXgaLseNOrBh6fV3RezcWx6D7yH3hMv8J5u95r+1POp/9r/6P/yfpS+lb6VfpT/Tq/7GLOe+t3ZKf/8BW/pa6A=</latexit> <latexit sha1_base64="nWC41+LpVrxKQ9eyc6ToFGu8PSE=">AEKXicbVNb9NAEHUdPkr4SuHIZUQE4pIorpColAMtQIkEKFt2qImqtbribPKetfaXaeKTP4OF/4KF5BAwJU/wq7z3bKSpfF7O2/ejMdhypk2jcbvDb905eq165s3yjdv3b5zt7J170jLTFHsUMmlOgmJRs4EdgwzHE9ShSQJOR6Hw5bj0eoNJPi0IxT7CUkFqzPKDEWOtvyn3dDjJnIDQkzTtQkT/OgvkOTCXyCNHfBpNwdOPkyPIY9IaQpUjVAtztnXknCa5FiIxSwRx1t74RxhbG6WtKtGYibsJb7BswmRJN2GfxYB4XMCeCItABETHO6XWspaTW54QPl4Kd2roCQqhszmBFYAm9QxXj0veBYUnGM3J+4GRaRNe4ogRgyuNtkim0fEstiVb0prSLqUJh4r07ViBu6JNaGNkGcWIpdpEDTECStSK0p4xKGbViq5sMzDCAaN83qdD0hWdfSsyk29lpsYsNPXxXtSUJBGEjI6pW5pFzYWga3ytwkfkXJ7P9fZd+rlUw1XnINBaxlgWM1xa76KIFvtyVqk26o3iwOUgmAVb3baZ5Xv3UjSLHdU24/4mnQSE0vJ8o4Y3bZ7IRTQockxlMbCpKg7uXFpk/gkUi6EtlH2GgQFczcpJoPU5CezMhZqAvcg78H3eamf5OL2cizexnodNC/YyDkeB+G4iYspvMxzYgVDHr1e2kItTteNkOIbjY8uXgaLseNOrBh6fV3RezcWx6D7yH3hMv8J5u95r+1POp/9r/6P/yfpS+lb6VfpT/Tq/7GLOe+t3ZKf/8BW/pa6A=</latexit> <latexit sha1_base64="nWC41+LpVrxKQ9eyc6ToFGu8PSE=">AEKXicbVNb9NAEHUdPkr4SuHIZUQE4pIorpColAMtQIkEKFt2qImqtbribPKetfaXaeKTP4OF/4KF5BAwJU/wq7z3bKSpfF7O2/ejMdhypk2jcbvDb905eq165s3yjdv3b5zt7J170jLTFHsUMmlOgmJRs4EdgwzHE9ShSQJOR6Hw5bj0eoNJPi0IxT7CUkFqzPKDEWOtvyn3dDjJnIDQkzTtQkT/OgvkOTCXyCNHfBpNwdOPkyPIY9IaQpUjVAtztnXknCa5FiIxSwRx1t74RxhbG6WtKtGYibsJb7BswmRJN2GfxYB4XMCeCItABETHO6XWspaTW54QPl4Kd2roCQqhszmBFYAm9QxXj0veBYUnGM3J+4GRaRNe4ogRgyuNtkim0fEstiVb0prSLqUJh4r07ViBu6JNaGNkGcWIpdpEDTECStSK0p4xKGbViq5sMzDCAaN83qdD0hWdfSsyk29lpsYsNPXxXtSUJBGEjI6pW5pFzYWga3ytwkfkXJ7P9fZd+rlUw1XnINBaxlgWM1xa76KIFvtyVqk26o3iwOUgmAVb3baZ5Xv3UjSLHdU24/4mnQSE0vJ8o4Y3bZ7IRTQockxlMbCpKg7uXFpk/gkUi6EtlH2GgQFczcpJoPU5CezMhZqAvcg78H3eamf5OL2cizexnodNC/YyDkeB+G4iYspvMxzYgVDHr1e2kItTteNkOIbjY8uXgaLseNOrBh6fV3RezcWx6D7yH3hMv8J5u95r+1POp/9r/6P/yfpS+lb6VfpT/Tq/7GLOe+t3ZKf/8BW/pa6A=</latexit> <latexit sha1_base64="nWC41+LpVrxKQ9eyc6ToFGu8PSE=">AEKXicbVNb9NAEHUdPkr4SuHIZUQE4pIorpColAMtQIkEKFt2qImqtbribPKetfaXaeKTP4OF/4KF5BAwJU/wq7z3bKSpfF7O2/ejMdhypk2jcbvDb905eq165s3yjdv3b5zt7J170jLTFHsUMmlOgmJRs4EdgwzHE9ShSQJOR6Hw5bj0eoNJPi0IxT7CUkFqzPKDEWOtvyn3dDjJnIDQkzTtQkT/OgvkOTCXyCNHfBpNwdOPkyPIY9IaQpUjVAtztnXknCa5FiIxSwRx1t74RxhbG6WtKtGYibsJb7BswmRJN2GfxYB4XMCeCItABETHO6XWspaTW54QPl4Kd2roCQqhszmBFYAm9QxXj0veBYUnGM3J+4GRaRNe4ogRgyuNtkim0fEstiVb0prSLqUJh4r07ViBu6JNaGNkGcWIpdpEDTECStSK0p4xKGbViq5sMzDCAaN83qdD0hWdfSsyk29lpsYsNPXxXtSUJBGEjI6pW5pFzYWga3ytwkfkXJ7P9fZd+rlUw1XnINBaxlgWM1xa76KIFvtyVqk26o3iwOUgmAVb3baZ5Xv3UjSLHdU24/4mnQSE0vJ8o4Y3bZ7IRTQockxlMbCpKg7uXFpk/gkUi6EtlH2GgQFczcpJoPU5CezMhZqAvcg78H3eamf5OL2cizexnodNC/YyDkeB+G4iYspvMxzYgVDHr1e2kItTteNkOIbjY8uXgaLseNOrBh6fV3RezcWx6D7yH3hMv8J5u95r+1POp/9r/6P/yfpS+lb6VfpT/Tq/7GLOe+t3ZKf/8BW/pa6A=</latexit> Driving Anomaly Dataset (DAD) § Annotations [Ramanishka et al. 2018] Annotations § A four-layer representation Goal-driven Intersection passing; Left turn; Right turn; Left lance Action change; Right lance change; Crosswalk passing; U- § Goal-driven actions turn; Left lane branch; Right lane branch; Merge Stimulus- Stop; Deviate § Stimulus-driven actions driven Action § Cause/Attention Cause Sign; Congestion; Tra ffi c light; Pedestrian; Parked car § Traffic rule violations Attention Crossing vehicle; Crossing pedestrian; Red light; Cut-in; Sign; On-road bicyclist; Parked vehicle; Merging vehicle; Yellow light; Road work; Pedestrian § Signal data collected near ego lane § Drivers’ physiological data § Vehicle controller area network (CAN)-bus data § Speed § Steer angle § Heart Rate (HR) § Yaw § Pedal pressure § Breath Rate (BR) § Steer speed § Pedal angle § Skin conductance (EDA) 7

1. Motivation 2. Driving Anomaly Dataset (DAD dataset) 3. Proposed Model 4. Experimental Evaluation 5. Conclusions

Big Picture Idea Previous frames Future frames § Can we forecast the “signals” in No Yes Predictable upcoming recordings based on previous frames? No anomalies Anomalies Next step: How to define “predictability” * Videos are just for reference 9 à We actually generate predicted signal and compare with real signal

Generation and Comparison using GAN § Generative Adversarial Network § Conditional GAN (cGAN) (GAN) § Input: Condition & Random noise § Generate plausible data from random § Generate data constrained by condition noise [Real data sequence] [Real data sequence] [Real [Real Discriminator or or Discriminator Fake] [Condition] Fake] [Plausible data Generator sequence] Generator [Noise] [Plausible data [Noise] sequence] 10

���������������������������� ������� ������� Proposed Model for Anomaly Detection § Condition of CGAN § Training Process § Real data from previous 6-seconds § Random noise: ( Output: Score ) § Random noise, totally unrelated to real data [1, 1] Fully § Real signals: connected [1, 180] § Real data from the next 6-seconds [1, 6] layers prediction prediction [1, 60] Fully connected [1, 18] layers [1, 34] � 1 � 51] [1, 180] ( Input: Features extracted from Raw Data ) � 1 � 360] ( Input: Previous window & Random Noise ) 11

Proposed Model Structure § Testing Process § Analysis Frames: § Real data from next 6-seconds Fully § Predicted Signals: Unsupervised § Prediction from our conditional GAN model § Feature Extraction: § Time domain (CAN-Bus & Physiological): § Mean, Standard Deviation (Std), Max, Min § Frequency domain: Real Plausible § Energy covering the following 5 bands: [0- 0.04 Hz], [0.04-0.15 Hz], [0.15-0.5 Hz], [0.5- 4 Hz], [4-20 Hz] prediction 12

1. Motivation 2. Driving Anomaly Dataset (DAD dataset) 3. Proposed Model 4. Experimental Evaluation 5. Conclusions