Accurate Object Shape and Pose M. Zeeshan Zia 1 Michael Stark 2,3 - - PowerPoint PPT Presentation

▶
accurate object shape and pose
SMART_READER_LITE
LIVE PREVIEW

Accurate Object Shape and Pose M. Zeeshan Zia 1 Michael Stark 2,3 - - PowerPoint PPT Presentation

Oct 09, 2022 50 likes •413 views

Revisiting 3D Geometric Models for Accurate Object Shape and Pose M. Zeeshan Zia 1 Michael Stark 2,3 Bernt Schiele 3 Konrad Schindler 1 3 Max-Planck-Institute for Informatics 1 Photogrammetry and Remote Sensing Laboratory 2 Artificial Intelligence

slide-1
SLIDE 1

Revisiting 3D Geometric Models for Accurate Object Shape and Pose

  • M. Zeeshan Zia1

Michael Stark2,3 Bernt Schiele3 Konrad Schindler1

1Photogrammetry and Remote Sensing Laboratory

Swiss Federal Institute of Technology (ETH), Zurich

3Max-Planck-Institute for Informatics

Saarbrücken, Germany

2Artificial Intelligence Lab

Stanford University, USA

slide-2
SLIDE 2

Current object models: coarse grained estimates

1

Zeeshan Zia

slide-3
SLIDE 3

Our goal: finer-grained models to aid scene-level reasoning

Zeeshan Zia

2

slide-4
SLIDE 4

Revival of 3D geometric representations

[Marr, Nishihara ’78] [Brooks ’81] [Pentland ’86] [Lowe ’87] [Koller, Daniilidis, Nagel ’93] [Sullivan, Worrall, Ferryman ’95] [Haag, Nagel ’99]

Zeeshan Zia

1970 1980 1990 2000 2010

3

slide-5
SLIDE 5

Revival of 3D geometric representations

[Marr, Nishihara ’78] [Brooks ’81] [Pentland ’86] [Lowe ’87] [Koller, Daniilidis, Nagel ’93] [Sullivan, Worrall, Ferryman ’95] [Haag, Nagel ’99] [Hoiem, Efros, Hebert ’08] [Ess, Leibe, Schindler, Van Gool ’09] [Wang, Gould, Koller ’10] [Hedau, Hoiem, Forsyth ’10] [Barinova, Lempitsky, Tretyak, Kohli ’10] [Gupta, Efros, Hebert ’10] [Wojek, Roth, Schindler, Schiele ’10]

Zeeshan Zia

1970 1980 1990 2000 2010

3

slide-6
SLIDE 6

Related work in viewpoint invariant detection

Multiple, viewpoint dependent representations (connected in different ways)

[Thomas et al., ’06] [Yan, Khan, Shah ’07] [Ozuysal, Lepetit, Fua ’09] [Nachimson, Basri ’09] [Su, Sun, Fei-Fei, Savarese ’09] [Gu, Ren ’10] [Stark, Goesele, Schiele ’10]1)

Explicit 3D geometry representation

[Liebelt, Schmid ’10]2) [Sun, Xu, Bradski, Savarese ’10] [Gupta, Efros, Hebert ’10] [Chen, Kim, Cipolla ‘10] [Gupta, Satkin, Efros, Hebert ’11]

Zeeshan Zia

1) 2)

4

slide-7
SLIDE 7

Overview

Zeeshan Zia

3D Active Shape Model Simplify

PCA

5

3D CAD Models

slide-8
SLIDE 8

Overview

Zeeshan Zia

3D Active Shape Model

Positive examples (per part)

Render Simplify

PCA

5

3D CAD Models

slide-9
SLIDE 9

Overview

Zeeshan Zia

3D Active Shape Model

Positive examples (per part) Negative examples (background)

AdaBoost

Render Simplify

PCA

5

3D CAD Models

slide-10
SLIDE 10

Overview

Zeeshan Zia

3D Active Shape Model

Negative examples (background)

AdaBoost Test image

Render Simplify

PCA

Detection maps

5

3D CAD Models

Positive examples (per part)

slide-11
SLIDE 11

Overview

Zeeshan Zia

3D Active Shape Model

Negative examples (background)

AdaBoost Test image Inference

Render Simplify

PCA

5

3D CAD Models

Positive examples (per part) Detection maps

slide-12
SLIDE 12

Representation: 3D geometry

Zeeshan Zia

6

  • Simplified 3D wireframes : fixed number of vertices
slide-13
SLIDE 13

Eigen-Cars

Learning: 3D geometry

Zeeshan Zia

  • Principal Components Analysis (PCA)

 Tightly constrained global geometry

7

slide-14
SLIDE 14

Representation: Local appearance

Zeeshan Zia

8

 Accurate foreground shape  Very cheap training data, dense sampling of viewpoints!

slide-15
SLIDE 15

Learning: Local appearance

Zeeshan Zia

9

  • Dense Shape Context features [Belongie, Malik. ’00]
  • AdaBoost classifiers (per part-viewpoint)
  • Annotated vertices are our ‘parts’.

+

  • …

…

Related work: [Andriluka, Roth, Schiele ’09]

slide-16
SLIDE 16

Inference

Zeeshan Zia

Test Image

10

slide-17
SLIDE 17

… …

Zeeshan Zia

Test Image

Inference

10

Detection maps

slide-18
SLIDE 18

… … … …

Zeeshan Zia

Test Image Detection maps Sample 3D wireframes, project, compute image likelihood

Inference

10

slide-19
SLIDE 19

Sample 3D cars, project, compute image likelihood Detection maps

… … … …

image evidence recognition hypothesis shape of wireframe camera focal length viewpoint parameters, azimuth and elevation image space translation and scaling local part scale part likelihood self-occlusion indicator Projection matrix

Zeeshan Zia

Inference

11

slide-20
SLIDE 20

Experimental evaluation – Test Dataset

  • Evaluations on 3D Object Classes dataset [Savarese et al., 2007]
  • Car class (8 azimuth angles, 2 elevation angles, 3 distances, varying

backgrounds) – 240 images, 5 cars

Zeeshan Zia

12

slide-21
SLIDE 21

Experimental evaluation - Training

  • 38 3D CAD models
  • 36 vertices as model points, 20 annotations per model (due to symmetry).
  • Separate local part shape detectors trained from:
  • 72 different azimuth angles,
  • 2 different elevation angles (7.5°, 15° from ground plane)

Zeeshan Zia

13

slide-22
SLIDE 22

Zeeshan Zia

20°

Experimental evaluation - Initialization

14

Two initializations :

  • Stark et al., 2010 (full system)
  • True initial value (tight bounding box, rough azimuth)
slide-23
SLIDE 23

Experimental evaluation - Inference

Zeeshan Zia

35° 35° 20°

14

slide-24
SLIDE 24

Example wireframe fits

Zeeshan Zia

Parts correctly localized Full system: 74.2% True initial value: 83.4%

15

slide-25
SLIDE 25

Fine-grained 3D geometry estimation

Zeeshan Zia

16

 Accurate estimation of closest 3D CAD model, camera parameters, and ground plane

slide-26
SLIDE 26

Ultra-wide baseline matching

Zeeshan Zia

17

Related work: [Bao, Savarese ’11]

  • UW-Baseline matching using only model fits (corresponding part locations)

 Impossible using interest point matching

slide-27
SLIDE 27

Ultra-wide baseline matching

Zeeshan Zia

18

Related work: [Bao, Savarese ’11]

  • UW-Baseline matching using only model fits (corresponding part locations)

 Impossible using interest point matching

slide-28
SLIDE 28

Ultra-wide baseline matching

Zeeshan Zia

19

Azimuth Difference

  • No. of

Image Pairs True initial value Full system SIFT Part detections

  • nly

45° 53 91% 55% 2% 27% 90° 35 91% 60% 0% 27% 135° 29 69% 52% 0% 10% 180° 17 59% 41% 0% 24%

  • Correct fit = Sampson error < Emax on ground truth correspondences

 3D Geometric model improves significantly over part detections only

slide-29
SLIDE 29

Multiview recognition

Zeeshan Zia

20

  • Rescored hypotheses

 Good 2D localization

slide-30
SLIDE 30

Continuous viewpoint estimation

Zeeshan Zia

Total Images True Positives % correct azimuth Average error azimuth Average error elevation Stark et al., 2010 48 46 67.4% 4.2° 4.0° Full system 48 45 73.3% 3.8° 3.6° True initial value* 48 48 89.6% 4.2° 3.6°

* Approximate pose initialization quantized to 45° steps

21

  • Comparison against ground truth pose, manually labeled.

 Full system improves 6% over Stark et al., 2010.

slide-31
SLIDE 31

Conclusion

 3D deformable object class model have potential for accurate geometric reasoning on scene level.

  • accurate object localization
  • geometric parts in 2D
  • 3D pose estimation

 Novel application examples

  • fine-grained object categorization
  • ultra-wide baseline matching
  • Future extensions
  • efficient multi-class methods for part likelihoods
  • analyze importance of geometric model vs. local appearance
  • occlusion invariance

Zeeshan Zia

22

slide-32
SLIDE 32

OLD SLIDES

slide-33
SLIDE 33

Eigen-Cars

Learning: 3D Geometry

mean wireframe direction of jth principal component standard deviation of jth principal component any wireframe weight of kth principal component residual (if r < m)

Zeeshan Zia

slide-34
SLIDE 34

Part localization

correct localization ~ localized within 4% of car length from ground truth

Zeeshan Zia

slide-35
SLIDE 35

Zeeshan Zia

Experimental evaluation - Inference

14

35° 35° 20°

slide-36
SLIDE 36

Zeeshan Zia

Experimental evaluation - Inference

14

35° 35° 20°