SLIDE 1
Using 3D data for image interpretation and geometric reasoning
Martial Hebert Abhinav Gupta David Fouhey, Adrien Matricon, Wajahat Hussain
and geometric reasoning Martial Hebert Abhinav Gupta David Fouhey, - - PowerPoint PPT Presentation
and geometric reasoning Martial Hebert Abhinav Gupta David Fouhey, - - PowerPoint PPT Presentation
Using 3D data for image interpretation and geometric reasoning Martial Hebert Abhinav Gupta David Fouhey, Adrien Matricon, Wajahat Hussain Sparse mid-level primitives can be used to transfer geometric information? Can this helps in
SLIDE 2
SLIDE 3
- Sparse mid-level primitives can be used to
transfer geometric information?
- Can this helps in detection and matching
tasks?
- Geometric reasoning can use this local
evidence to produce a consistent geometric interpretation?
SLIDE 4
Primitives
Visually Discriminative
Image
Geometrically Informative
Surface Normals
Saurabh Singh et al. Discriminative Mid-Level Patches
SLIDE 5
NYU v2 Dataset (Silberman et al., 2012)
SLIDE 6
Learning primitives
…
SLIDE 7
Representation
Instances Detector Canonical Form
SLIDE 8
SLIDE 9
SLIDE 10
SLIDE 11
Learning Primitives
Approach: iterative procedure
SLIDE 12
Inference
Sparse Transfer …
19s
SLIDE 13
Inference
Sparse Transfer …
SLIDE 14
Inference
Sparse Transfer
SLIDE 15
Inference
Dense Transfer
SLIDE 16
Sample Results – Qualitative
795 /654
SLIDE 17
Confidence
Most Confident Result Least Confident Result
rank
SLIDE 18
Failures
SLIDE 19
Mean Summary Stats (⁰) (Lower Better) Median RMSE % Good Pixels (Higher Better) 11.25⁰ 22.5⁰ 30⁰ 3D Primitives 33.0 28.3 18.8 40.7 52.4 40.0 Karsch et al. 40.8 37.8 7.9 25.8 38.2 46.9 Hoiem et al. 41.2 9.0 31.7 43.9 49.3 34.8 Singh et al. 35.0 32.4 11.2 32.1 45.8 40.6 Saxena et al. 47.1 11.2 28.0 37.4 56.3 42.3 RF + Dense SIFT 36.0 11.4 31.1 44.2 41.7 33.4
RMSE
SLIDE 20
More general environments?
SLIDE 21
KITTI Dataset: Geiger, Lenz, Urtasun, ‘12
SLIDE 22
- Large regions without surface interpretation
- Fewer linear/planar structures to anchor
- Irregular distribution of 3D training data
SLIDE 23
SLIDE 24
SLIDE 25
Discovered Primitives (Examples)
747/203
SLIDE 26
SLIDE 27
Contact points
SLIDE 28
Object surfaces + Contact points
SLIDE 29
Failures
SLIDE 30
Failures
SLIDE 31
Digression
SLIDE 32
Style and structure
SLIDE 33
Style vs. structure?
Tenenbaum & Freeman. Separating Style and Content with Bilinear Models. Neural
- Computation. 2000.
Lee, Efros, Hebert. Style-aware Mid-level Representation for Discovering Visual Connections in Space and Time. 2013.
SLIDE 34
Casablanca Hotel, New York
SLIDE 35
SLIDE 36
SLIDE 37
Meritan Apartments Sydney Sheraton Hotels (North America)
SLIDE 38
Using geometric and physical constraints
SLIDE 39
The Story So Far
SLIDE 40
The Story So Far
SLIDE 41
Adding Physical/Geometric Constraints
SLIDE 42
Adding Physical/Geometric Constraints
SLIDE 43
SLIDE 44
Concave ( - ) Convex ( + )
Edges between surfaces
SLIDE 45
Parameterization
vp
1
vp
2
vp
3
SLIDE 46
Parameterization
vp
1
vp
2
vp
3
SLIDE 47
Parameterization
vp
1
vp
2
vp
3
SLIDE 48
Parameterization
32/64
SLIDE 49
Parameterization
SLIDE 50
Parameterization
SLIDE 51
Labeling
: is cell i on?
SLIDE 52
Unary terms
Should cell i be on?
SLIDE 53
Binary Potentials
8o7s
SLIDE 54
Binary terms
SLIDE 55
Binary terms
SLIDE 56
Binary terms
SLIDE 57
Constraints
Gurobi BB
SLIDE 58
Qualitative Results
Projected 3D Primitives 3D Primitives Proposed Input Ground Truth
SLIDE 59
Projected 3D Primitives 3D Primitives Proposed Input Ground Truth
SLIDE 60
Random Qualitative Results
Proposed 3D Primitives
SLIDE 61
Quantitative Results
Proposed Mean Summary Stats (⁰) (Lower Better) % Good Pixels (Higher Better) Median RMSE 11.25⁰ 22.5⁰ 30⁰ 37.5 17.2 41.9 53.9 58.0 53.2 3D Primitives 38.5 19.0 41.7 52.4 56.3 54.2 Hedau et al. 43.2 24.8 39.1 48.8 52.3 59.4 Lee et al. 47.6 43.4 28.1 39.7 43.9 60.6 Karsch et al. 46.6 43.0 5.4 19.9 31.5 53.6 Hoiem et al. 45.6 8.6 30.5 41.0 55.1 38.2
rank
SLIDE 62
SLIDE 63
Now:
Better reasoning Semantic information Less structured environments Coarse-to-fine depth
SLIDE 64
Martial Hebert Abhinav Gupta David Fouhey, Adrien Matricon, Wajahat Hussain