Face Alignment in the Wild Mirrorability and Sensitivity Heng Yang yanghengnudt@gmail.com Rainbow Group, Computer Laboratory
Outline • Introduction • Face alignment mirrorability • Face alignment sensitivity • Conclusion and discussion
Introduction During 1964 and 1965, Bledsoe, along with Helen Chan and Charles Bisson, worked on using the computer to recognize human faces Man-Machine System Users are asked to localize a set of facial landmarks of the photograph “This recognition problem is made difficult by the great variability in head rotation and tilt, lighting intensity and angle, facial expression, aging, etc.” —Woody Bledsoe, 1966 Woodrow Wilson "Woody" Bledsoe Automatic Facial Landmarks Localization (Nov.12, 1921 – Oct. 4, 1995)
Introduction Detection Alignment Recognition Name: Kate Emotion: Happy Gender: Female Attractiveness: 5 Race: Caucasian Age: ~30 Occupation: ? … .
Introduction • Alignment is an essential step to the success of face recognition. [Yaniv Taigman et al. DeepFace: Closing the Gap to Human-Level Performance in Face Verification,CVPR2014] [Tal Hassner et al. Effective Face Frontalization in Unconstrained Images, CVPR2015]
Introduction • Conceptually it is very similar to many other problems 9 8 7 5 8 4 7 4 5 6 1011 6 1 2 6 7 5 3 2 3 7 6 5 8 4 9 10 3 2 3 4 12 5 3 4 6 1 8 14 1 13 9 10 13 14 12 2 11 1 1 2 7 8 View2 View4 View3 View1 View5 [Yang and Patras TIP’14] [Yang and Ramanan cvpr’11] [Yang, Zhang and Robinson Arxiv1509.04954] [Liu and Belhumeur, ICCV’13]
Introduction Industry Academia Sensetime, Face++, Linkface, • AAM [T. Cootes et al. 98] • CLM [T. Cootes et al. 06] Orbeus, CloudWalk ， BUPTAPI, 腾讯优图 ， 汉 王人 脸 通，一登 … FaceMark, eyedea ， sightcorp, ci2cv, Kairos, betaface, Luxand FaceSDK, Cognitec, ayonix, Omron ， Affectiva … https://sites.google.com/site/yanghengcv/face-alignment
Introduction • Cascaded face alignment framework PROS CONS # of landmarks has little impact on running ✔ It is an open system and usually requires speed, with a vector representation ✖ additional model to check the reliability It is easy to augment training samples ✔ It is very efficient in both training and testing ✔ ✖ It is very sensitive to initialization It is simple and more accurate ✔
Face Alignment Mirrorability • Mirrorability: the ability of a model/algorithm to preserve the mirror symmetry when applied on an image and its mirror image. NB. NOT USING SYMMETRY Why only mirror? It is easy to avoid training data bias. CONSTRAINS [Heng Yang and Ioannis Patras CVPR 15.]
Face Alignment Mirrorability • Mirror error: Big mirror error Small mirror error [Heng Yang and Ioannis Patras CVPR 15.]
Face Alignment Mirrorability • Property #1: Mirror error is calculated in a blind way—no ground truth is needed • Property #2 :Mirror error shows high correlation with the ground truth alignment error [X. P. Burgos-Artizzu, P. Perona and P. Dollár, "Robust face landmark estimation under occlusion ", ICCV2013.]
Face Alignment Mirrorability • Mirror error is calculated in a blind way—no ground truth is needed • Mirror error shows high correlation with the ground truth alignment error Face alignment by random ferns Face alignment by random trees [X. P. Burgos-Artizzu, P. Perona and P. Dollár, "Robust face landmark estimation under occlusion ", ICCV2013.] [Kazemi, Vahid and Sullivan, Josephine: "One Millisecond Face Alignment with an Ensemble of Regression Trees",CVPR,2014.]
Face Alignment Mirrorability • Similar observation is found in human pose estimation [F. Wang and Y. Li. Beyond physical connections: Tree models in human pose estimation. CVPR, 2013]
Face Alignment Mirrorability • How does it happen in regression forests based method? x 1 x 2 x 1 -x 2 > t ?
Face Alignment Mirrorability • How does it happen in regression forests based method?
Face Alignment Mirrorability • How does it happen in Convolutional Neural Netork method? [Karel Lenc, Andrea Vedaldi, Understanding image representations by measuring their equivariance and equivalence ", CVPR 2015.]
Face Alignment Mirrorability • Application #1: difficult samples selection Compared to samples selected with ground truth. Compared to samples selected w/o ground truth. 1. The samples that we have selected are truly ‘difficult’ 2. Selected difficult samples show high consistency across different approaches.
Face Alignment Mirrorability “open” system Feedback (Mirror error?)
Face Alignment Mirrorability • Application #2: re-start cascaded face alignment [Kazemi, Vahid and Sullivan, Josephine: "One Millisecond Face Alignment with an Ensemble of Regression Trees",CVPR,2014.] [X. P. Burgos-Artizzu, P. Perona and P. Dollár, "Robust face landmark estimation under occlusion ", ICCV2013.]
Face Alignment Mirrorability • Application examples Warning Good Bad
Face Alignment Mirrorability • Mirrorability in other computer vision problems 3.854 46.26 37.03 110.1 104.1 11.11 http://vision.ucsd.edu/~pdollar/toolbox/doc/index.html https://how-old.net/ http://places.csail.mit.edu/demo.html
Face Alignment Sensitivity • Motivation #1: face detection is not always consistent in consecutive frames. • Motivation #2: face detection might vary from training to testing GT HeatHunter Viola&J ones IBUG HOG+SVM
Face Alignment Sensitivity • A new evaluation metric over a testing set:
Face Alignment Sensitivity • Empirical study #1: synthesised face center shifts Viola&J ones Shift -0.10 Shift -0.21 N.B.: I use the off-the-shelf models, thus it is not very meaningful to assess the relative merits.
Face Alignment Sensitivity • Empirical study #2: real face center shifts
Face Alignment Sensitivity • Empirical study #3: synthesized face scale changes
Conclusion and Discussion ü We have proposed a new concept, mirrorability and a corresponding metric, mirror error to measure it. ü We show an important property that mirror error is highly correlated to alignment error, which can be used as a feedback for cascaded face alignment. ü We show that most recent face alignment methods are sensitive to initialisation center shifts, (less sensitive w.r.t scale variation). Future work o How about face alignment in videos? ICCV2015 300W Videos Challenge: http://ibug.doc.ic.ac.uk/resources/300-VW/ o How accurate is it desired for a specific task in face analysis? o Is mirrorability or equivariance worthy more attentions in computer vision? o Further improvement in face alignment, data vs. model?
Conclusion and Discussion • Further improvement in face alignment, data vs. model? [Yang et al. BMVC2015] [Yang, Zhang and Robinson Arxiv1509.04954]
Reference • Heng Yang, Ioannis Patras, "Mirror, mirror on the wall, tell me, is the error small? ", CVPR 2015. • Heng Yang, Ioannis Patras, "Fine-tuning regression forests votes for object alignment in the wild", IEEE Trans. Image Processing, Dec., 2014. • Heng Yang et al. "Face alignment assisted by head pose estimation", BMVC 2015. • SDM : Xiong, X., & De la Torre, F. (2013, June). Supervised descent method and its applications to face alignment. CVPR, 2013. • GNDPM: Tzimiropoulos, G., & Pantic, M. (2014, June). Gauss-newton deformable part models for face alignment in-the-wild, CVPR, 2014. • CFAN: Zhang, J., Shan, S., Kan, M., & Chen, X. (2014). Coarse-to-fine auto-encoder networks (CFAN) for real-time face alignment. ECCV 2014, Springer. • IFA: Asthana, S. Zafeiriou, S. Cheng, M. Pantic, Incremental Face Alignment in the Wild, CVPR 2014. • CCNF: Baltru š aitis, Tadas and Robinson, Peter and Morency, Louis-Philippe: "Continuous Conditional Neural Fields for Structured Regression", ECCV, 2014. • TCDCN: Zhang, Zhanpeng and Luo, Ping and Loy, Chen Change and Tang, Xiaoou: "Facial Landmark Detection by Deep Multi-task Learning",ECCV,2014 • TREE:Kazemi, Vahid and Sullivan, Josephine: "One Millisecond Face Alignment with an Ensemble of Regression Trees",CVPR,2014. • LBF: Ren, Shaoqing and Cao, Xudong and Wei, Yichen and Sun, Jian: "Face Alignment at 3000 FPS via Regressing Local Binary Features", CVPR,2014. • GNDPM: Tzimiropoulos, Georgios and Pantic, Maja: "Gauss-newton deformable part models for face alignment in-the-wild", CVPR,2014. • IFA: Asthana, Akshay and Zafeiriou, Stefanos and Cheng, Shiyang and Pantic, Maja: "Incremental Face Alignment in the Wild", CVPR, 2014. • Sagonas, Christos and Tzimiropoulos, Georgios and Zafeiriou, Stefanos and Pantic, Maja: "300 Faces in-the-Wild Challenge: The first facial landmark localization Challenge", ICCV,2013. • Burgos-Artizzu, Xavier P and Perona, Pietro and Dollar, Piotr: "Robust face landmark estimation under occlusion", ICCV,2013. • Y. Yang and D. Ramanan. Articulated human detection with flexible mixtures of parts. T-PAMI, 2013. • F. Wang and Y. Li. Beyond physical connections: Tree models in human pose estimation. CVPR, 2013
Recommend
More recommend
