Learning to Rank Authenticity from Facial Activity Descriptors Otto - PowerPoint PPT Presentation

Learning to Rank Authenticity from Facial Activity Descriptors Otto von Guericke University, Magdeburg - Germany Frerk Saxen, Philipp Werner, Ayoub Al-Hamadi

The Task – Real or Fake? Dataset statistics • Training set – 40 Subjects – 6 Emotions – 480 Videos • Validation set – 5 Subjects – 6 Emotions – 60 Videos • Test set – 5 Subjects – 6 Emotions – 60 videos • One real and one fake video per emotion and subject 2

The Task – Real or Fake? We changed the rules: • Which one is more real than the other? 3

Our Approach Videos (same subject and emotion) real AU Intensity Activity Estimation Descriptor Authen- Rank SVM ticity Ensemble Ranking AU Intensity Activity Estimation Descriptor fake 1. Face Detection 2. Face Recognition 3. Landmark detection 4. Action Unit Intensity Estimation 5. Facial Activity Descriptor 6. Rank SVM Ensemble 4

Our Approach Videos (same subject and emotion) real AU Intensity Activity Estimation Descriptor Authen- Rank SVM ticity Ensemble Ranking AU Intensity Activity Estimation Descriptor fake 1. Face Detection – CNN resnet with dlib [1] 1. 2. Face Recognition – CNN resnet with dlib [2] 3. 3. Landmark detection – Ensemble of regression trees with dlib [3] 4. Action Unit Intensity Estimation 2. 5. Facial Activity Descriptor 6. Rank SVM Ensemble [1] D. King. Easily create high quality object detectors with deep learning, 2016. http://blog.dlib.net/2016/10/easily-create-high-qualityobject.html. [2] D. King. High quality face recognition with deep metric learning, 2017. http://blog.dlib.net/2017/02/high-quality-face-recognition-withdeep.html. [3] D. King. Real-time face pose estimation, 2014. http://blog.dlib.net/2014/08/real-timeface-pose-estimation.html. 5

Our Approach Videos (same subject and emotion) real AU Intensity Activity Estimation Descriptor Authen- Rank SVM ticity Ensemble Ranking AU Intensity Activity Estimation Descriptor fake Source: [5] Source: [5] 1. Face Detection 2. Face Recognition 3. Landmark detection 4. Action Unit Intensity Estimation – Using previous work [4] Action Unit 12: Action Unit 4: – Affine face transform Lip corner puller Brow lowerer – LPB + shape features – Ensemble of SVR (Support Vector List of used Action Units: Regression) 1: Inner Brow Raiser 5. Facial Activity Descriptor 2: Outer Brow Raiser 6. Rank SVM Ensemble 4: Brow lowerer 6: Cheek Raiser 9: Nose Wrinkler [4] P. Werner, F. Saxen, and A. Al-Hamadi. Handling data imbalance in automatic facial action intensity estimation. In X. Xie, M.W. Jones, and G. K. L. Tam, editors, Proceedings of the British 12: Lip corner puller Machine Vision Conference (BMVC), pages 124.1 – 124.12. BMVA Press, 2015 25: Lips part [5] Bryn Farnsworth. Facial Action Coding System (FACS) – A Visual Guidebook. Online. 2017-10-12. 6 https://imotions.com/blog/facial-action-coding-system/

Our Approach Videos (same subject and emotion) real AU Intensity Activity Estimation Descriptor Authen- Rank SVM ticity Ensemble Ranking AU Intensity Activity Estimation Descriptor fake 1. Face Detection 2. Face Recognition 3. Landmark detection 4. Action Unit Intensity Estimation 5. Facial Activity Descriptor – Based on previous work [5] with some additional statistics – Noise reduction with Butterworth filter – Filtered signal + derivative – Signal statistics (max, mean , … ) 6. Rank SVM Ensemble [5] P. Werner, A. Al-Hamadi, K. Limbrecht-Ecklundt, S.Walter, S. Gruss, and H. Traue. Automatic pain assessment with facial activity descriptors. IEEE Transactions on Affective Computing, PP(99):1 – 1, 2016 7

Our Approach Videos (same subject and emotion) real AU Intensity Activity Estimation Descriptor Authen- Rank SVM Training : ticity Ensemble Ranking AU Intensity Activity 𝑔𝑓𝑏𝑢𝑣𝑠𝑓 → 𝑚𝑏𝑐𝑓𝑚 Estimation Descriptor fake 𝑦 𝑆 − 𝑦 𝐺 → 1 1. Face Detection 𝑦 𝐺 − 𝑦 𝑆 → 0 2. Face Recognition 3. Landmark detection Test : 4. Action Unit Intensity Estimation 𝑜𝐹𝑜𝑡𝑓𝑛𝑐𝑚𝑓𝑡 5. Facial Activity Descriptor 𝑂 0 = 𝑞𝑠𝑓𝑒 𝑗 (𝑦 0 − 𝑦 1 ) == 1 6. Rank SVM Ensemble 𝑗 – Learn the difference between two facial 𝑜𝐹𝑜𝑡𝑓𝑛𝑐𝑚𝑓𝑡 activity descriptors with libSVM [6] 𝑂 1 = 𝑞𝑠𝑓𝑒 𝑗 (𝑦 1 − 𝑦 0 ) == 1 – Create an Ensemble of 75 Rank SVMs – 𝑗 Aggregate by counting the votes. 𝑧 0 𝑛𝑝𝑠𝑓 𝑏𝑣𝑢ℎ𝑓𝑜𝑢𝑗𝑑 ← 𝑂 0 > 𝑂 1 𝑧 1 𝑛𝑝𝑠𝑓 𝑏𝑣𝑢ℎ𝑓𝑜𝑢𝑗𝑑 ← 𝑂 1 > 𝑂 0 [6] Chih-Chung Chang and Chih-Jen Lin, LIBSVM : a library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1--27:27, 2011. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm 8

Results • Validation set performance (5 subjects, 60 videos) • Human performance conducted with 22 subjects 9

Results • Test set performance (5 subjects, 60 videos) 10

Final Remarks • Professional organization team with quick and friendly response. Thank you! • Challenge introduced us to a new research field. • Validation and test set provided only 60 samples. • Data acquisition is questionable: – Some humans reported that all videos look fake – Fake vs. Fake? 11

Final Remarks 12

Contact • Frerk Saxen Tel.: +49 391 67 51671 E-Mail: frerk.saxen@ovgu.de • Philipp Werner Tel.: +49 391 67 51491 E-Mail: frerk.saxen@ovgu.de • Prof. Ayoub Al-Hamadi Tel.: +49 391 67 58709 E-Mail: ayoub.al-hamadi@ovgu.de Address: Otto-von-Guericke Universität Institut für Informations- und Kommunikationstechnik (IIKT) Universitätsplatz 2 30106 Magdeburg 13