facial landmark tracking for mobile
play

Facial Landmark Tracking for Mobile Instructor - Simon Lucey - PowerPoint PPT Presentation

Facial Landmark Tracking for Mobile Instructor - Simon Lucey 16-623 - Designing Computer Vision Apps Thatcher Effect Thompson, P. (1980). "Margaret Thatcher: a new illusion". Perception. 9 (4) Thatcher Effect Thompson, P. (1980).


  1. Reminder: SDMs • Like IC-LK, SDM assumes a linear relationship between appearance and geometry: ∆ p = R [ I ( p ) − T ( 0 )] • Iteratively updates until convergence X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

  2. Reminder: SDMs • Like IC-LK, SDM assumes a linear relationship between appearance and geometry: ∆ p = R [ I ( p ) − T ( 0 )] • Iteratively updates until convergence X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

  3. Reminder: SDMs • Like IC-LK, SDM assumes a linear relationship between appearance and geometry: ∆ p = R [ I ( p ) − T ( 0 )] • Iteratively updates until convergence X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

  4. Reminder: SDMs • Like IC-LK, SDM assumes a linear relationship between appearance and geometry: ∆ p = R [ I ( p ) − T ( 0 )] • Iteratively updates until convergence X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

  5. Reminder: SDMs • Like IC-LK, SDM assumes a linear relationship between appearance and geometry: ∆ p = R [ I ( p ) − T ( 0 )] • Iteratively updates until convergence X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

  6. Reminder: SDMs • Like IC-LK, SDM assumes a linear relationship between appearance and geometry: ∆ p = R [ I ( p ) − T ( 0 )] • Iteratively updates until convergence X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

  7. Reminder: SDMs • Like IC-LK, SDM assumes a linear relationship between appearance and geometry: ∆ p = R [ I ( p ) − T ( 0 )] • Iteratively updates until convergence … X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

  8. Reminder: SDMs • Like IC-LK, SDM assumes a linear relationship between appearance and geometry: ∆ p = R [ I ( p ) − T ( 0 )] • Iteratively updates until convergence … R ( k ) … • However, is iteration specific. X, Xuehan, and F. De la Torre. "Supervised descent method and its applications to face alignment." CVPR 2013.

  9. SDMs vs. CLMs • SDMs are inherently more computationally efficient than CLMs. • Computational cost is associated only with image warping, feature extraction, and matrix multiplication (no convolution).

  10. IntraFace http://www.humansensing.cs.cmu.edu/intraface

  11. IntraFace http://www.humansensing.cs.cmu.edu/intraface

  12. � � ����� ���� � � � � ����� ���� � � � � � � ��� � � � � � � � � � � � � � � � ����� ���� � � � � � � ����� ���� � � � � � � ����� ���� � � SIFT Feature - SDMs � � � �� �

  13. � � ����� ���� � � � � ����� ���� � � � � � � ��� � � � � � � � � � � � � � ����� ���� � ����� ���� � � � � � � � � � � � � � ����� ���� � � SIFT Feature - SDMs � � � �� � 1. Compute image gradients 2. Pool into local histograms 3. Concatenate histograms 4. Normalize histograms

  14. � � ����� ���� � � � � ����� ���� � � � � � � ��� � � � � � � � � � � � � � ����� ���� � ����� ���� � � � � � � � � � � � � � ����� ���� � � SIFT Feature - SDMs � � � �� � Calculating SIFT for each landmark on a mobile device is in general too costly for real-time performance. 1. Compute image gradients 2. Pool into local histograms 3. Concatenate histograms 4. Normalize histograms

  15. SIFT Speed - Mac vs. iPhone 7 A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arix 2016.

  16. Reminder: Binary Features • Proposed by Calonder et al. ECCV 2010 (BRIEF). • Borrows idea that binary comparison is very fast on modern chipset architectures, { 1 : I ( x + ∆ 1 ) > I ( x + ∆ 2 ) ψ I ( x , ∆ 1 , ∆ 2 ) = 0 : otherwise • Combine features together compactly, 2 i − 1 ψ I ( x , ∆ ( i ) 1 , ∆ ( i ) X ψ I ( x ) = 2 ) i

  17. Why do Binary Features Work? • Success of binary features says something about perception itself. • Absolute values of pixels do not matter. • Makes sense as variable lighting source will effect the gain and bias of pixels, but the local ordering should remain relatively constant.

  18. Reminder: BRIEF Descriptor • Do not need to “touch” all pixels, can choose pairs { ∆ 1 , ∆ 2 } randomly and sparsely,

  19. Hamming distance in least squares 1 0 1 1 0 0 1 0 178 0 0 1 1 0 0 1 0 50 1 0 0 0 0 0 0 0 128 Hamming distance Squared distance vs. 1 16384 H. Alismail, B. Browning, S. Lucey. "Enhancing Direct Camera Tracking with Feature Descriptors" ACCV 2016.

  20. � � ����� ���� � � � � ����� ���� � � � � � � ��� � � � � � � � � � � � � � ����� ���� � ����� ���� � � � � � � � � � � � � � ����� ���� � � Local Binary Feature - SDMs � � � �� � S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

  21. � � ����� ���� � � � � ����� ���� � � � � � � ��� � � � � � � � � � � � � � ����� ���� � ����� ���� � � � � � � � � � � � � � ����� ���� � � Local Binary Feature - SDMs � � � �� � S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

  22. � � ����� ���� � � � � ����� ���� � � � � � � ��� � � � � � � � � � � � � � ����� ���� � ����� ���� � � � � � � � � � � � � � ����� ���� � � Local Binary Feature - SDMs � � � �� � φ 1 = ∈ [0 , 1] > S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

  23. � � ����� ���� � � � � ����� ���� � � � � � � ��� � � � � � � � � � � � � � ����� ���� � � � � � � � � � � � � � � ����� ���� � � ����� ���� Local Binary Feature - SDMs � � � �� � φ 1 = ∈ [0 , 1] > . . . φ F = ∈ [0 , 1] > S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

  24. � � � ���������� � ��� � � � ��� � � � ��� � � � � � ���������� ���������� ����������� � ���������� � ��� ���������� � ��� Local Binary Feature - SDMs

  25. � � � ���������� � ��� � � � ��� � � � ��� � � � � � ���������� ���������� ����������� � ���������� � ��� ���������� � ��� Local Binary Feature - SDMs φ 1 0 1

  26. ���������� ���������� ����������� � ���������� � ��� � ���������� � ��� � � � ��� � � � ��� � � � � � � � ���������� � ��� Local Binary Feature - SDMs φ 1 0 1 φ 2 1 0

  27. ���������� � ��� ���������� � ��� � ���������� � ��� � � � ��� � � � ��� � � � � � ���������� ���������� ����������� � � � Local Binary Feature - SDMs φ 1 0 1 φ 2 1 0 φ 3 1 0

  28. � � � ���������� � ��� � � � ��� � � � ��� � � � � � ���������� ���������� ����������� � ���������� � ��� ���������� � ��� Local Binary Feature - SDMs φ 1 0 1 φ 2 1 0 φ 3 1 0 [0 1 0 0]

  29. � � � � � � Local Binary Feature - SDMs (a) concatenating � ���������� � ��� … … ��� � � � ��� � � � ���������� � ��� (b) � ���������� � ��� ���������� ���������� ����������� � S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

  30. � � � � � � � � � � � � � � � � � � � � � � � � � � Local Binary Feature - SDMs Learning Feature Local Binary Learning Linear Estimated Shape � ��� Estimated Shape � � Ground Truth Shape � � Mapping � � Features Projection � � Ground Truth Shape � � ����� ���� ����� ���� � � � � � � � �� � ����� ���� ����� ���� Train Test ����� ���� Feature Mapping Linear Projection S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

  31. Local Binary Feature - SDMs 300-W (68 landmarks) 2 Common Challenging Method Fullset FPS Subset Subset 70 ESR[5] 7.58 5.28 17.00 120 - 21 SDM[32] 7.52 5.60 15.40 70 LBF 6.32 4.95 11.98 320 LBF fast 7.37 5.38 15.50 3100 atasets, respectively. The errors of ESR and SDM are from our S. Ren, et al. "Face alignment at 3000 fps via regressing local binary features." CVPR 2014.

  32. LBF-SDMs - Drawbacks • Offer unprecedented speed, making its application ideal for mobile face tracking. • In reality, however, performance is often noisy/jittery as the approach is only touching a sparse set of pixels.

  33. Binary Approximated SIFT • Recently, Fagg et al. proposed Binary Approximated SIFT for SDMs. A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

  34. Binary Approximated SIFT • Recently, Fagg et al. proposed Binary Approximated SIFT for SDMs. θ = atan2( r y I ( x ) , r x I ( x )) A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

  35. Binary Approximated SIFT • Recently, Fagg et al. proposed Binary Approximated SIFT for SDMs. r y I ( x ) θ = atan2( r y I ( x ) , r x I ( x )) r x I ( x ) A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

  36. Binary Approximated SIFT • Recently, Fagg et al. proposed Binary Approximated SIFT for SDMs. r y I ( x ) r y I ( x ) > 0 r x I ( x ) > 0 | r y I ( x ) | > | r x I ( x ) | r x I ( x ) 2 3 = 8 orientation bins A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

  37. Binary Approximated SIFT • Recently, Fagg et al. proposed Binary Approximated SIFT for SDMs. • The approach approximates binning through binary comparison. 8 orientations 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

  38. � � ����� ���� � � � � ����� ���� � � � � � � ��� � � � � � � � � � � � � � ����� ���� � ����� ���� � � � � � � � � � � � � � ����� ���� � � BA-SIFT Feature - SDMs � � � �� � 1 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0

  39. � � ����� ���� � � � � ����� ���� � � � � � � ��� � � � � � � � � � � � � � ����� ���� � ����� ���� � � � � � � � � � � ����� ���� � � � � � BA-SIFT Feature - SDMs � � � �� � 1 0 0 0 0 0 Calculating BASIFT extremely efficient. 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0

  40. Inspiration from the FFT A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

  41. Sparse Compositional SDM (a) Component 1 Structure (b) Component 2 Structure (c) Component 3 Structure (d) Composition A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

  42. Speed Comparison A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

  43. 90 FPS - iPhone 7 L L L A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arxiv 2016.

  44. 90 FPS - iPhone 7 A. Fagg, S. Sridharan, and S. Lucey, "Fast, Dense Feature SDM on an iPhone" Arix 2016.

  45. More Examples.....

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend


More recommend