Landmark Labelling for 3D Faces Clement Creusot, Nick Pears, Jim - - PowerPoint PPT Presentation

▶
landmark labelling for 3d faces
SMART_READER_LITE
LIVE PREVIEW

Landmark Labelling for 3D Faces Clement Creusot, Nick Pears, Jim - - PowerPoint PPT Presentation

Apr 26, 2023 141 likes •544 views

Landmark Labelling for 3D Faces Clement Creusot, Nick Pears, Jim Austin Clment Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 1 Table of Content Motivation Motivation Problem Problem Solution Solution Results

slide-1
SLIDE 1

Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 1

Landmark Labelling for 3D Faces

Clement Creusot, Nick Pears, Jim Austin

slide-2
SLIDE 2

Motivation Problem Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 2

Table of Content

■ Motivation ■ Problem ■ Solution ■ Results ■ Conclusion

slide-3
SLIDE 3

Motivation

  • Non-cooperative Recognition

at a distance

  • Modality
  • Difficult Cases
  • Review
  • Assumptions

Problem Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 3

Motivation

slide-4
SLIDE 4

Motivation

  • Non-cooperative Recognition

at a distance

  • Modality
  • Difficult Cases
  • Review
  • Assumptions

Problem Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 4

Non-cooperative Recognition at a distance

■ Application: ◆ Surveillance ◆ Human-Machine Interaction

slide-5
SLIDE 5

Motivation

  • Non-cooperative Recognition

at a distance

  • Modality
  • Difficult Cases
  • Review
  • Assumptions

Problem Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 4

Non-cooperative Recognition at a distance

From [Savran et al., 2008] From [Savran et al., 2008]

■ Application: ◆ Surveillance ◆ Human-Machine Interaction ■ Problems: ◆ Pose ◆ Occlusion ◆ Speed

slide-6
SLIDE 6

Motivation

  • Non-cooperative Recognition

at a distance

  • Modality
  • Difficult Cases
  • Review
  • Assumptions

Problem Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 5

Modality

■ Non-Cooperative ⊃ Anti-cooperative ■ Proved possible for big database

From [Proenca, 2008] From [Yan and Bowyer, 2007] From [Phillips et al., 2005] From [Havasi et al., 2007]

slide-7
SLIDE 7

Motivation

  • Non-cooperative Recognition

at a distance

  • Modality
  • Difficult Cases
  • Review
  • Assumptions

Problem Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 5

Modality

■ Non-Cooperative ⊃ Anti-cooperative ■ Proved possible for big database

From [Proenca, 2008] From [Yan and Bowyer, 2007] From [Phillips et al., 2005] From [Havasi et al., 2007]

■ 2D or 3D ?

From [Liu et al., 2007]

slide-8
SLIDE 8

Motivation

  • Non-cooperative Recognition

at a distance

  • Modality
  • Difficult Cases
  • Review
  • Assumptions

Problem Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 6

Difficult Cases

■ Recognition: ◆ Holistic method → Need for good Registration ◆ Feature based method → Need for good Feature

Localisation

slide-9
SLIDE 9

Motivation

  • Non-cooperative Recognition

at a distance

  • Modality
  • Difficult Cases
  • Review
  • Assumptions

Problem Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 6

Difficult Cases

■ Recognition: ◆ Holistic method → Need for good Registration ◆ Feature based method → Need for good Feature

Localisation

■ Will often fail at preprocessing ◆ Naive methods for feature detection ◆ Strong assumptions

slide-10
SLIDE 10

Motivation

  • Non-cooperative Recognition

at a distance

  • Modality
  • Difficult Cases
  • Review
  • Assumptions

Problem Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 6

Difficult Cases

■ Recognition: ◆ Holistic method → Need for good Registration ◆ Feature based method → Need for good Feature

Localisation

■ Will often fail at preprocessing ◆ Naive methods for feature detection ◆ Strong assumptions ■ Recquire better feature detection

slide-11
SLIDE 11

Motivation

  • Non-cooperative Recognition

at a distance

  • Modality
  • Difficult Cases
  • Review
  • Assumptions

Problem Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 7

Review

■ Almost all paper expect non occluded frontal face ■ A few that don’t: ◆ Some orientation change:

■ [Colbry et al., 2005]: Curvature + ICP + Relaxation ■ [Lu and Jain, 2006]: Directional Maximum ■ [Faltemier et al., 2008]: Rotated Profile Signature

[Colbry et al., 2005] [Lu and Jain, 2006] [Faltemier et al., 2008]

■ Almost all papers expect the nose will be present ■ Most papers recquire two well defined inner corners of the

eyes

slide-12
SLIDE 12

Motivation

  • Non-cooperative Recognition

at a distance

  • Modality
  • Difficult Cases
  • Review
  • Assumptions

Problem Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 8

Assumptions

■ The ones we needed to make: ◆ At least half of the face is visible ◆ There exist features repeatable across individual ■ The ones we did not make: ◆ All landmark are present and will match there descriptor ◆ Candidates for one landmark descriptor are rare ■ The ones we made (only in post-processing) ◆ The face is roughly convex ◆ Faces are not too flexible (= hand) ◆ Only 1 face per scene

slide-13
SLIDE 13

Motivation Problem

  • The landmark Detection

Problem

  • Input Generation

Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 9

Problem

slide-14
SLIDE 14

Motivation Problem

  • The landmark Detection

Problem

  • Input Generation

Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 10

The landmark Detection Problem

Input Mesh Landmarks

Landmarking

Positions + Labels

■ Landmark = Position + Label ■ Two Approaches: ◆ Select One Label + Find Corresponding Position ◆ Find All Positions + Find Corresponding Labels

slide-15
SLIDE 15

Motivation Problem

  • The landmark Detection

Problem

  • Input Generation

Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 10

The landmark Detection Problem

Input Mesh Landmarks Points

Repeatable Point Detection Labelling

■ Landmark = Position + Label ■ Two Approaches: ◆ Select One Label + Find Corresponding Position ◆ Find All Positions + Find Corresponding Labels

slide-16
SLIDE 16

Motivation Problem

  • The landmark Detection

Problem

  • Input Generation

Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 10

The landmark Detection Problem

Input Mesh Landmarks Points

Labelling

■ Landmark = Position + Label ■ Two Approaches: ◆ Select One Label + Find Corresponding Position ◆ Find All Positions + Find Corresponding Labels

slide-17
SLIDE 17

Motivation Problem

  • The landmark Detection

Problem

  • Input Generation

Solution Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 11

Input Generation

Mesh Automatic Points Hand-Placed Points Input Points

+

slide-18
SLIDE 18

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 12

Solution

slide-19
SLIDE 19

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 13

Our Strategy

Input Position Graph Matching Post-Processing Results

  • Multi-attribute seeding
  • Relaxation by elimination
  • Threshold on scores
  • Unit-Quaternion clustering
slide-20
SLIDE 20

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 14

Graph Generation

Mesh Neighbourhood Scalars & Vectors Input Points

H,K,SI,Vol,LC

Node Attributes Edge Attributes

Eucl.Dist., Geod.Dist., Ratio, ∆H, . . .

Graphs

■ Graph Properties: ◆ Complete Graph (for now) ◆ 5 attributes per Node ◆ 7 attributes per Edge

slide-21
SLIDE 21

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 15

Graph Matching

■ Structure ◆ list of candidates ◆ Associated scores

slide-22
SLIDE 22

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 15

Graph Matching

■ Structure ◆ list of candidates ◆ Associated scores ■ Objective: ◆ Reduce correspondence Nb

slide-23
SLIDE 23

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 15

Graph Matching

Query 1.1 120.0 . . . 5.6 ×0.2 ×0.5 . . . ×0.3 Model

Σ

0.5×0.2 0.65×0.5 . . . 0.9×0.3 = Score N properties

■ Structure ◆ list of candidates ◆ Associated scores ■ Objective: ◆ Reduce correspondence Nb ■ Seeding ◆ Partial scores

LDA

→ Score

slide-24
SLIDE 24

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 15

Graph Matching

■ Structure ◆ list of candidates ◆ Associated scores ■ Objective: ◆ Reduce correspondence Nb ■ Seeding ◆ Partial scores

LDA

→ Score

■ Relaxation on hyperedges (= [Christmas et al., 1995])

slide-25
SLIDE 25

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 16

Elimination

QUERY

slide-26
SLIDE 26

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 16

Elimination

QUERY MODEL

slide-27
SLIDE 27

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 16

Elimination

QUERY MODEL

slide-28
SLIDE 28

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 16

Elimination

QUERY MODEL

slide-29
SLIDE 29

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 16

Elimination

QUERY MODEL

slide-30
SLIDE 30

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 16

Elimination

14.0 151.0 222.0 500 728.0 1000 Initial Seeding GM Final Nb of Node Correspondances

June

14.0 58.0 500 539.0 812.0 1000 Initial Seeding GM Final Nb of Node Correspondances

Sept.

slide-31
SLIDE 31

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 17

Post-Processing

Transformation Matrix 4x4:      R′

  • t

1      → ˙ q Unit Quaternion

  • t

Translation s Scale

slide-32
SLIDE 32

Motivation Problem Solution

  • Our Strategy
  • Graph Generation
  • Graph Matching
  • Elimination
  • Post-Processing

Results Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 17

Post-Processing

■ Clustering ■ Mean Transformation ■ Final Correspondence

slide-33
SLIDE 33

Motivation Problem Solution Results

  • Databases
  • Results

Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 18

Results

slide-34
SLIDE 34

Motivation Problem Solution Results

  • Databases
  • Results

Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 19

Databases

■ FRGC v2 ◆ 4950 faces from 557 people ◆ 200 in train set ◆ 4750 in test set (3108 Neutral, 1642 Expression) ◆ cropped ■ Bosphorus ◆ 4666 faces from 105 people ◆ Occlusion, Expression, Rotation ◆ 99 in train set (20 for profile)

slide-35
SLIDE 35

Motivation Problem Solution Results

  • Databases
  • Results

Conclusion Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 20

Results

FRGC v2 - Neutral FRGC v2 - All Bosphorus - Occluded Bosphorus - Rotation 45

■ For now: ◆ 6.3% bad final registration ■ If automatic landmarks only: ◆ 10.4% bad final registration ■ The system doesn’t collapse

when dealing with occlusion

  • r pose variation
slide-36
SLIDE 36

Motivation Problem Solution Results Conclusion

  • Conclusion
  • Bibliography

Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 21

Conclusion

slide-37
SLIDE 37

Motivation Problem Solution Results Conclusion

  • Conclusion
  • Bibliography

Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 22

Conclusion

■ Good ◆ Very few assumptions on the input data ◆ Graphs are very versatile ■ Bad ◆ Non optimised (preliminary results) ◆ Naive post-processing ■ Future Work ◆ Try different graph topologies ◆ Improve robustness to missing points ◆ Deal with non-cropped faces ◆ Try higher order hyperedges

slide-38
SLIDE 38

Motivation Problem Solution Results Conclusion

  • Conclusion
  • Bibliography

Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 22

Conclusion

■ Good ◆ Very few assumptions on the input data ◆ Graphs are very versatile ■ Bad ◆ Non optimised (preliminary results) ◆ Naive post-processing ■ Future Work ◆ Try different graph topologies ◆ Improve robustness to missing points ◆ Deal with non-cropped faces ◆ Try higher order hyperedges

Thank you !

slide-39
SLIDE 39

Motivation Problem Solution Results Conclusion

  • Conclusion
  • Bibliography

Clément Creusot, October 25th, 2010 3D Object Retrieval, Florence - p. 23

Bibliography

[Christmas et al., 1995] Christmas, W. J., Kittler, J., and Petrou, M. (1995). Structural matching in computer vision using probabilistic relaxation. IEEE Trans. Pattern Anal. Mach. Intell., 17(8):749–764. [Colbry et al., 2005] Colbry, D., Stockman, G., and Jain, A. (2005). Detection of anchor points for 3d face veri.cation. In Computer Vision and Pattern Recognition - Workshops, 2005. CVPR Workshops. IEEE Computer Society Conference on, pages 118 –118. [Faltemier et al., 2008] Faltemier, T., Bowyer, K., and Flynn, P . (2008). A region ensemble for 3-d face recognition. Information Forensics and Security, IEEE Transactions on, 3(1):62–73. [Havasi et al., 2007] Havasi, L., Szlavik, Z., and Sziranyi, T. (2007). Detection of gait characteristics for scene registration in video surveillance system. Image Processing, IEEE Transactions on, 16(2):503 –510. [Liu et al., 2007] Liu, C.-C., Dai, D.-Q., and Yan, H. (2007). Local discriminant wavelet packet coordinates for face recognition.

  • J. Mach. Learn. Res., 8:1165–1195.

[Lu and Jain, 2006] Lu, X. and Jain, A. (2006). Automatic feature extraction for multiview 3d face recognition. In Automatic Face and Gesture Recognition, 2006. FGR 2006. 7th International Conference on, pages 585 –590. [Phillips et al., 2005] Phillips, P ., Flynn, P ., Scruggs, T., Bowyer, K., Chang, J., Hoffman, K., Marques, J., Min, J., and Worek,

  • W. (2005). Overview of the face recognition grand challenge. Computer Vision and Pattern Recognition, 2005. CVPR
  • 2005. IEEE Computer Society Conference on, 1:947–954.

[Proenca, 2008] Proenca, H. (2008). Iris recognition: A method to segment visible wavelength iris images acquired

  • n-the-move and at-a-distance. Advances in Visual Computing, pages 731–742.

[Savran et al., 2008] Savran, A., Alyüz, N., Dibeklio˘ glu, H., Çeliktutan, O., Gökberk, B., Sankur, B., and Akarun, L. (2008). Bosphorus database for 3d face analysis. In Biometrics and Identity Management: First European Workshop, BIOID 2008, Roskilde, Denmark, May 7-9, 2008., pages 47–56, Berlin, Heidelberg. Springer-Verlag. [Yan and Bowyer, 2007] Yan, P . and Bowyer, K. (2007). Biometric recognition using 3d ear shape. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 29(8):1297 –1308.