3D Landmark Model Discovery from a Registered Set of Organic Shapes - - PowerPoint PPT Presentation

▶

Oct 30, 2023 415 likes •706 views

3D Landmark Model Discovery from a Registered Set of Organic Shapes Clement Creusot, Nick Pears, Jim Austin Department of Computer science PCP, CVPRW, Providence (RI), June 2012 Plan What/Why How Results Conclusion References What/Why:

SLIDE 1

3D Landmark Model Discovery from a Registered Set of Organic Shapes

Clement Creusot, Nick Pears, Jim Austin Department of Computer science

PCP, CVPRW, Providence (RI), June 2012

SLIDE 2

What/Why How Results Conclusion References

Plan

What/Why: Generalities and Problems How: Proposed method Results Conclusion

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 2 / 20

SLIDE 3

What/Why How Results Conclusion References

Generalities

Where is Wally?

Waldo? Charlie? Walter? ウォーリー? 威利? . . . Scene s

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 3 / 20

SLIDE 4

What/Why How Results Conclusion References

Generalities

Where is Wally?

Waldo? Charlie? Walter? ウォーリー? 威利? . . . Scene Output Detector s

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 3 / 20

SLIDE 5

What/Why How Results Conclusion References

Generalities

Where is Wally?

Waldo? Charlie? Walter? ウォーリー? 威利? . . . Scene Output Detector Model s

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 3 / 20

SLIDE 6

What/Why How Results Conclusion References

Generalities

Where is Wally?

Waldo? Charlie? Walter? ウォーリー? 威利? . . . Scene Output Detector Model Model Discoverer s

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 3 / 20

SLIDE 7

What/Why How Results Conclusion References

Generalities

Where is Wally?

Waldo? Charlie? Walter? ウォーリー? 威利? . . . Scene Output Detector Model Model Discoverer

?

s

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 3 / 20

SLIDE 8

What/Why How Results Conclusion References

Model Discovery for 3D Face Landmarking

s Scene Output Detector Model

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 4 / 20

SLIDE 9

What/Why How Results Conclusion References

Model Discovery for 3D Face Landmarking

s Scene Output Landmarker Model

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 4 / 20

SLIDE 10

What/Why How Results Conclusion References

Model Discovery for 3D Face Landmarking

s Scene Output Landmarker Model Model Discoverer

?

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 4 / 20

SLIDE 11

What/Why How Results Conclusion References

Why? - Gap in Research

[Amberg et al., 2007] [Creusot et al., 2011] [Gupta et al., 2007] [Romero-Huertas and Pears, 2008] [Szeptycki et al., 2009] [Zhao et al., 2011]

Easy to label or explain to an operator Linked to 2D projections and plane symmetries Overall arbitrary

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 5 / 20

SLIDE 12

What/Why How Results Conclusion References

Nature of a model for a 3D-object class

Sparse “Descriptive” Featural/Local information (nodes) Structural/Global information (edges/hyperedges) Possible Local Features: Object Points Curves Surfaces Volumes

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 6 / 20

SLIDE 13

What/Why How Results Conclusion References

Nature of a model for a 3D-object class

Sparse “Descriptive” Featural/Local information (nodes) Structural/Global information (edges/hyperedges) Possible Local Features: Object Points Curves Surfaces Volumes

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 6 / 20

SLIDE 14

What/Why How Results Conclusion References

Organicly-shape objects

More possible point-models than geometric shapes Less intuition about what model is good

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 7 / 20

SLIDE 15

What/Why How Results Conclusion References

Example of 3D-objects point models

Articulated Models: Articulations Extremities Non-Articulated Models: ??? ???

[Shotton et al., 2011] [Bray et al., 2004] [Creusot et al., 2011] Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 8 / 20

SLIDE 16

What/Why How Results Conclusion References

Hypothesis

Scene Output Landmarker Model Model Discoverer

?

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 9 / 20

SLIDE 17

What/Why How Results Conclusion References

Hypothesis

“Probabilistic” response map available One point per model Scene Output Landmarker Model Model Discoverer

?

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 9 / 20

SLIDE 18

What/Why How Results Conclusion References

Our Approach

Use Detector and Neighborhood definition from [Creusot et al., 2011]

8 Local Descriptors Gaussian Distributions Linear Combination (LDA based)

Test as many models as there are vertices in the template mesh (∼ 2000) Define two cost functions for each model:

Saliency: Different from its neighborhood (good) Ubiquity: Ubiquitous over the face (bad)

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 10 / 20

SLIDE 19

What/Why How Results Conclusion References

Databases

FRGC (real) BFM (synthetic)

(Coarse Correspondence) (Fine Correspondence)

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 11 / 20

SLIDE 20

What/Why How Results Conclusion References

Saliency Score per Vertex

0.962 0.962 0.981 0.981 1.00 1.00

0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.4 0.6 0.8 1 Density Score 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.4 0.6 0.8 1 Density Score 0.05 0.1 0.15 0.2 0.25 0.2 0.4 0.6 0.8 1 Density Score 0.05 0.1 0.15 0.2 0.25 0.2 0.4 0.6 0.8 1 Density Score 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.2 0.4 0.6 0.8 1 Density Score 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.2 0.4 0.6 0.8 1 Density Score 0.05 0.1 0.15 0.2 0.25 0.3 0.2 0.4 0.6 0.8 1 Density Score 0.05 0.1 0.15 0.2 0.25 0.3 0.2 0.4 0.6 0.8 1 Density Score

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 12 / 20

SLIDE 21

What/Why How Results Conclusion References

Ubiquity Score per Vertex

15.8 15.8 586. 586. 1.16e+03 1.16e+03

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 13 / 20

SLIDE 22

What/Why How Results Conclusion References

Results

Manual Automatic Initial Symmetry

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 14 / 20

SLIDE 23

What/Why How Results Conclusion References

Saliency

Similar ID 8 3 1 2 5 6 7 U 658.37 215.52 287.70 67.452 252.50 393.68 324.40 Automatic ID 1 2 3 4 5 6 U 683.62 192.43 492.43 87.408 328.08 226.66 206.76 Manual

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 15 / 20

SLIDE 24

What/Why How Results Conclusion References

Saliency

Different 4 9 U 325.51 561.31 416.18 Automatic ID 7 8 9 U 701.64 954.16 444.54 Manual

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 16 / 20

SLIDE 25

What/Why How Results Conclusion References

Problems

Different answers depending on the registration method:

Fine registration on clean data (BFM) Coarse registration on unclean data (FRGC) Fine registration on unclean data (???) Needed

Optimization method → Depends on the detector used (and its parameters) How to include structural information in the model discovery? How to project a newly discovered model to unseen training data? (again a registration problem)

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 17 / 20

SLIDE 26

What/Why How Results Conclusion References

Conclusion

Good:

Optimize model for a detector Validate most human-chosen landmarks Give quantifiable measure of landmark quality

Bad:

Only non-articulated objects for now Requires a large set of finely-registered objects (Do you have one to share?)

Questions to you:

How do you learn a model structure in your application domain? Are there applications where you think this might help? Brain teaser: How do you extend the idea to multi-dimensional features (curves, area, volumes)?

Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 18 / 20

SLIDE 27

What/Why How Results Conclusion References

References I

Amberg, B., Romdhani, S., and Vetter, T. (2007). Optimal step nonrigid icp algorithms for surface registration. In IEEE Int Conf. CVPR. Bray, M., Koller-Meier, E., Mueller, P., Gool, L. V., and Schraudolph, N. N. (2004). 3d hand tracking by rapid stochastic gradient descent using a skinning model. In Chambers, A. and Hilton, A., editors, 1st European Conference on Visual Media Production (CVMP), pages 59–68. IEE. Creusot, C., Pears, N., and Austin, J. (2011). Automatic keypoint detection on 3d faces using a dictionary of local shapes. In 3DIMPVT, pages 204–211. Gupta, S., Markey, M. K., Aggarwal, J., and Bovik, A. C. (2007). Three dimensional face recognition based on geodesic and euclidean distances. In IS&T/SPIE Symp. on Electronic Imaging: Vision Geometry XV. Romero-Huertas, M. and Pears, N. (2008). 3d facial landmark localisation by matching simple descriptors. In IEEE Int. Conf. BTAS, pages 1–6. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. (2011). Real-time human pose recognition in parts from single depth images. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1297 –1304. Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 19 / 20

SLIDE 28

What/Why How Results Conclusion References

References II

Szeptycki, P., Ardabilian, M., and Chen, L. (2009). A coarse-to-fine curvature analysis-based rotation invariant 3D face landmarking. In IEEE Int. Conf. BTAS, pages 32–37. Zhao, X., Dellandr´ e anda, E., Chen, L., and Kakadiaris, I. A. (2011). Accurate landmarking of three-dimensional facial data in the presence of facial expressions and occlusions using a three-dimensional statistical facial feature model. IEEE Trans. Syst. Man, and Cybernetics, Part B: Cybernetics, 41(5):1417–1428. Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 20 / 20