Shape from X Haoqiang Fan fhq@megvii.com Some figures adapted from - - PowerPoint PPT Presentation

▶
shape from x
SMART_READER_LITE
LIVE PREVIEW

Shape from X Haoqiang Fan fhq@megvii.com Some figures adapted from - - PowerPoint PPT Presentation

Dec 15, 2023 34 likes •826 views

Shape from X Haoqiang Fan fhq@megvii.com Some figures adapted from http://cvg.ethz.ch/teaching/2012spring/3dphoto/Slides/3dphoto12_shapeFromX.pdf Perception / Measurement of 3D 3D is vital for survival How to reconstruct / perceive 3D By

slide-1
SLIDE 1

Shape from X

Haoqiang Fan fhq@megvii.com

Some figures adapted from http://cvg.ethz.ch/teaching/2012spring/3dphoto/Slides/3dphoto12_shapeFromX.pdf

slide-2
SLIDE 2

Perception / Measurement of 3D

3D is vital for survival

slide-3
SLIDE 3

How to reconstruct / perceive 3D

By means of visual information

  • > optical, 2D array of input
slide-4
SLIDE 4

Structure from Motion

The most easy-to-understand approach Triangulation

https://cn.mathworks.com/help/vision/ug/structure-from-motion.html

slide-5
SLIDE 5

Triangulation

The epipolar constraint

Stereo and kinect fusion for continuous 3D reconstruction and visual odometry

slide-6
SLIDE 6

Stereo, rectification, disparity

row-to-row correspondence

https://www.slideshare.net/DngNguyn43/stereo-vision-42147593

slide-7
SLIDE 7

Disparity, depth

d=y_right - y_left z=B*F/d

OpenCV: Depth Map from Stereo Images Middlebury Stereo Evaluation

slide-8
SLIDE 8

3D Point Cloud

x=x_screen/F*z y=y_screen/F*z

Bundler: Structure from Motion (SfM) for Unordered Image Collections

slide-9
SLIDE 9

Surface Reconstruction

Integration of oriented point

slide-10
SLIDE 10

Laplacian and Normal

Laplacian = Normal * Mean Curvature

slide-11
SLIDE 11

SfM Scanning

SLAM based positioning

slide-12
SLIDE 12

Depth Sensing: Active Sensors

Structured Light Time of Flight(ToF)

slide-13
SLIDE 13

Structured Light

Static pattern & dynamic pattern

slide-14
SLIDE 14

Time of Flight (ToF)

Pulsed modulation

slide-15
SLIDE 15

Short Baseline Stereo

Phase Detection Autofocus

slide-16
SLIDE 16

Shape from X

Structure from Motion: 3D geometry Are there other possibilities?

slide-17
SLIDE 17

Shape from Shading

Shading as a cue of 3D shape

slide-18
SLIDE 18

The Lambertian Law

slide-19
SLIDE 19

Shape from Shading

Solve for gradient Assuming constant albedo

slide-20
SLIDE 20

Is Shape Uniquely Determined?

bas-relief ambiguity

slide-21
SLIDE 21

Shape from Shading

Data term + Prior

slide-22
SLIDE 22

Shape from Shading

Example

slide-23
SLIDE 23

Photometric Stereo

slide-24
SLIDE 24

Photometric Stereo

Measure the normal direction: the chrome sphere

slide-25
SLIDE 25

Depth from Normals

slide-26
SLIDE 26

Example

Good for near Lambertian material

slide-27
SLIDE 27

Shape from Texture

Solving normal from texture

slide-28
SLIDE 28

Depth from Focus

Focus sweep

slide-29
SLIDE 29

Depth from Defocus

Measure blur, solve depth

slide-30
SLIDE 30

Shape from Shadows

Shadow carving

3D Reconstruction by Shadow Carving: Theory and Practical Evaluation”

slide-31
SLIDE 31

Shape from Specularities

Solve deformation

  • f mirrors.

Toward a Theory of Shape from Specular Flow

slide-32
SLIDE 32

Shape from ?

Shape from Nothing? Object priors!

slide-33
SLIDE 33

3D Reconstruction from Single Image

infer a whole shape, from a single image

slide-34
SLIDE 34

3D Reconstruction from Single Image

slide-35
SLIDE 35

The ShapeNet Dataset

slide-36
SLIDE 36

3D Reconstruction from Single Image

slide-37
SLIDE 37

3D Reconstruction from Single Image

slide-38
SLIDE 38

The issue of representation

slide-39
SLIDE 39

Depth map

slide-40
SLIDE 40

Depth map

slide-41
SLIDE 41

Second depth map

slide-42
SLIDE 42

Second depth map

slide-43
SLIDE 43

The problem of discontinuity

slide-44
SLIDE 44

Volumetric Occupancy

slide-45
SLIDE 45

Problem of viewpoint

slide-46
SLIDE 46

Canonical View

slide-47
SLIDE 47

Volumetric Occupancy

slide-48
SLIDE 48

XML file

slide-49
SLIDE 49

XML file

slide-50
SLIDE 50

XML file

slide-51
SLIDE 51

XML file

slide-52
SLIDE 52

Can we find a representation that is..

flexible structural natural

slide-53
SLIDE 53
slide-54
SLIDE 54

Point-based representation

flexible structural natural

slide-55
SLIDE 55

Implementation details

slide-56
SLIDE 56

Results

slide-57
SLIDE 57

Results

slide-58
SLIDE 58

Results

slide-59
SLIDE 59

Human Performance

slide-60
SLIDE 60
slide-61
SLIDE 61
slide-62
SLIDE 62

A Neural Method to Stereo Matching

slide-63
SLIDE 63

Flownet & Dispnet

Using raw left and right images as input Output disparity map End-to-End training

slide-64
SLIDE 64

Using two stacked images as input

FlownetSimple

slide-65
SLIDE 65

Adding Correlation Layer

Using correlation layer to explicitly provide cross view communication ability

FlownetCorr

slide-66
SLIDE 66

Stereo Matching Cost Convolutional Neural Network

Using CNN to calculate stereo matching cost between patches from different view Following with several post-process: Cross-based cost aggregation Semiglobal matching Left-right consistency check Disparity <-> Depth

slide-67
SLIDE 67

MRF Stereo methods

We estimate f by minimizing the following energy function based on pairwise MRF Data term Smoothness term

slide-68
SLIDE 68

Global Local Stereo Neural Network

Feature visualization

slide-69
SLIDE 69

results

slide-70
SLIDE 70

results

slide-71
SLIDE 71

results

slide-72
SLIDE 72

Implementation details

Entangle two view feature inside network.

slide-73
SLIDE 73

Large Receptive Field Neural Network

SimpleConv Encoder-Decoder ResConv blindingly increasing the receptive field of feature networks may not Improve the performance

simple conv SimpleConv

slide-74
SLIDE 74

PatchMatch Communication Layer

Directly provide the ability of communicating across two views

slide-75
SLIDE 75

Multi-staged Cascade

slide-76
SLIDE 76

Thanks

Q/A

slide-77
SLIDE 77

单击以结束放映

slide-78
SLIDE 78

SemiGlobal Matching

we define an energy function E(D) that depends on the disparity map D NP-Hard !!! But we can solve it through each directions to get an approximate solution by using Dynamic Programming(DP)

slide-79
SLIDE 79

Slanted patch matching

The disparity d_p of each pixel p is over-parameterized by a local disparity plane Each pixels in the same plane has the same parameter (a_p, b_p, c_p) The true disparity maps are approximately piecewise linear We can estimate (a_p, b_p, c_p) for each pixel p instead of directly estimate d_p