Calibrated Image Acquisition for Multi-view 3D Reconstruction - - PowerPoint PPT Presentation

▶
calibrated image acquisition for multi view 3d
SMART_READER_LITE
LIVE PREVIEW

Calibrated Image Acquisition for Multi-view 3D Reconstruction - - PowerPoint PPT Presentation

Dec 06, 2022 418 likes •936 views

Calibrated Image Acquisition for Multi-view 3D Reconstruction Sriram Kashyap M S Guide: Prof. Sharat Chandran Indian Institute of Technology, Bombay April 2009 Sriram Kashyap 3D Reconstruction 1/ 42 Motivation Given pictures of an object,

slide-1
SLIDE 1

Calibrated Image Acquisition for Multi-view 3D Reconstruction

Sriram Kashyap M S Guide: Prof. Sharat Chandran

Indian Institute of Technology, Bombay

April 2009

Sriram Kashyap 3D Reconstruction 1/ 42

slide-2
SLIDE 2

Motivation

Given pictures of an object, how can I place the object in a virtual environment?

Sriram Kashyap 3D Reconstruction 2/ 42

slide-3
SLIDE 3

Motivation

Given pictures of an object, how can I place the object in a virtual environment?

Sriram Kashyap 3D Reconstruction 2/ 42

slide-4
SLIDE 4

Traditional Graphics

Render a 3D world on a 2D screen The world is authored using modeling tools, by artists

Sriram Kashyap 3D Reconstruction 3/ 42

slide-5
SLIDE 5

Traditional Graphics

Render a 3D world on a 2D screen The world is authored using modeling tools, by artists

Sriram Kashyap 3D Reconstruction 3/ 42

slide-6
SLIDE 6

Going backwards: 2D to 3D

Sriram Kashyap 3D Reconstruction 4/ 42

slide-7
SLIDE 7

Going backwards: 2D to 3D

Monocular reconstruction

Salzmann et al. Local deformation models for monocular 3D shape recovery. CVPR 2008 Sriram Kashyap 3D Reconstruction 4/ 42

slide-8
SLIDE 8

Going backwards: 2D to 3D

Stereo Reconstruction

http://www.cs.washington.edu/homes/indria/project/CSE576finalproject/ Sriram Kashyap 3D Reconstruction 5/ 42

slide-9
SLIDE 9

Going backwards: 2D to 3D

Multi-view Reconstruction

vision.middlebury.edu Sriram Kashyap 3D Reconstruction 6/ 42

slide-10
SLIDE 10

Visual Hull

Visual hull is an approximate 3D structure obtained from reconstruction algorithms based on object silhouettes. Given a camera, a silhouette defines a back-projected generalized cone that contains the actual object Each camera gives us one such generalized cone The intersection of these cones is the Visual Hull

Sriram Kashyap 3D Reconstruction 7/ 42

slide-11
SLIDE 11

Visual Hull

Visual hull is an approximate 3D structure obtained from reconstruction algorithms based on object silhouettes. Given a camera, a silhouette defines a back-projected generalized cone that contains the actual object Each camera gives us one such generalized cone The intersections of these cones is the Visual Hull

Sriram Kashyap 3D Reconstruction 8/ 42

slide-12
SLIDE 12

Visual Hull

Visual hull is an approximate 3D structure obtained from reconstruction algorithms based on object silhouettes. Given a camera, a silhouette defines a back-projected generalized cone that contains the actual object Each camera gives us one such generalized cone The intersections of these cones is the Visual Hull

Sriram Kashyap 3D Reconstruction 9/ 42

slide-13
SLIDE 13

Visual Hull

Visual hull is an approximate 3D structure obtained from reconstruction algorithms based on object silhouettes. Given a camera, a silhouette defines a back-projected generalized cone that contains the actual object Each camera gives us one such generalized cone The intersections of these cones is the Visual Hull

Sriram Kashyap 3D Reconstruction 10/ 42

slide-14
SLIDE 14

Simulating multiple cameras

Object is placed on a turntable Camera is fixed Camera captures frames as the table rotates

Sriram Kashyap 3D Reconstruction 11/ 42

slide-15
SLIDE 15

Issues in visual hull construction

Image segmentation: Obtain silhouette information from images Camera calibration: Find the camera projection matrix for each camera

Sriram Kashyap 3D Reconstruction 12/ 42

slide-16
SLIDE 16

Segmentation

Background Subtraction Capture object and background images If background and object image pixels are similar, mark pixel as background Similarity tests performed in RGB and YCbCr color spaces

Sriram Kashyap 3D Reconstruction 13/ 42

slide-17
SLIDE 17

Segmentation

Background Subtraction Capture object and background images If background and object image pixels are similar, mark pixel as background Similarity tests performed in RGB and YCbCr color spaces

Sriram Kashyap 3D Reconstruction 13/ 42

slide-18
SLIDE 18

Segmentation

Segmented image

Sriram Kashyap 3D Reconstruction 14/ 42

slide-19
SLIDE 19

What went wrong?

We are using a webcam Camera exposure, white balance, noise, compression artifacts What is the correct threshold value? This may vary from image to image

Sriram Kashyap 3D Reconstruction 15/ 42

slide-20
SLIDE 20

Adaptive thresholding

Can we provide more information to the system?

Sriram Kashyap 3D Reconstruction 16/ 42

slide-21
SLIDE 21

Adaptive thresholding

Can we provide more information to the system? Ratio of object pixels to total number of pixels remains within certain bounds Provide the expected upper bound for this ratio

Sriram Kashyap 3D Reconstruction 16/ 42

slide-22
SLIDE 22

Adaptive thresholding

Can we provide more information to the system? Ratio of object pixels to total number of pixels remains within certain bounds Provide the expected upper bound for this ratio Start with a low threshold value Increase the threshold till the ratio is below this upper bound

Sriram Kashyap 3D Reconstruction 16/ 42

slide-23
SLIDE 23

Adaptive thresholding

Segmented image (foreground ratio=0.07)

Sriram Kashyap 3D Reconstruction 17/ 42

slide-24
SLIDE 24

Camera Calibration

Calibration Find the projection matrix corresponding to each camera view

Sriram Kashyap 3D Reconstruction 18/ 42

slide-25
SLIDE 25

Camera Calibration

Calibration Find the projection matrix corresponding to each camera view Find a set of 2D to 3D point correspondences Use existing tools to compute matrices from these correspondences

Sriram Kashyap 3D Reconstruction 18/ 42

slide-26
SLIDE 26

Point Correspondences

Calibration Pattern

Sriram Kashyap 3D Reconstruction 19/ 42

slide-27
SLIDE 27

Point Correspondences

Calibration Pattern

Sriram Kashyap 3D Reconstruction 20/ 42

slide-28
SLIDE 28

Point Correspondences

Feature to find: Centers of black squares Centers are easier to find and more robust against errors Fix an absolute ordering of boxes Use colored boxes to find orientation Thresholding to locate the boxes Find centers of these boxes

Sriram Kashyap 3D Reconstruction 21/ 42

slide-29
SLIDE 29

Example

Calibration Pattern

Sriram Kashyap 3D Reconstruction 22/ 42

slide-30
SLIDE 30

Example

Black threshold, first pass

Sriram Kashyap 3D Reconstruction 23/ 42

slide-31
SLIDE 31

Example

Black threshold, second pass

Sriram Kashyap 3D Reconstruction 24/ 42

slide-32
SLIDE 32

Example

Green threshold, first pass

Sriram Kashyap 3D Reconstruction 25/ 42

slide-33
SLIDE 33

Example

Green threshold, second pass

Sriram Kashyap 3D Reconstruction 26/ 42

slide-34
SLIDE 34

Example

Green threshold, final pass

Sriram Kashyap 3D Reconstruction 27/ 42

slide-35
SLIDE 35

Example

Labeled calibration pattern

Sriram Kashyap 3D Reconstruction 28/ 42

slide-36
SLIDE 36

Camera Calibration

In some cases, the algorithm may explicitly fail(vision is uncertain) Discard such views automatically

Sriram Kashyap 3D Reconstruction 29/ 42

slide-37
SLIDE 37

Camera Calibration

In some cases, the algorithm may explicitly fail(vision is uncertain) Discard such views automatically In some cases, the algorithm may fail silently (returns an incorrect, but mathematically valid matrix) Cannot automatically discard such views, although tools can be written to help find bad views

Sriram Kashyap 3D Reconstruction 29/ 42

slide-38
SLIDE 38

Camera Calibration

Visualizing a camera matrix

  216.201 634.052 −44.0839 3922.02 36.6629 −52.2481 394.727 3119.24 −0.69624 0.685745 −0.212142 12.13  

Sriram Kashyap 3D Reconstruction 30/ 42

slide-39
SLIDE 39

Camera Calibration

Visualizing a camera matrix

  591.256 −313.573 −52.8837 3893.9 −5.06723 −59.6318 −395.35 3148.67 −0.0399772 0.974564 −0.220515 11.9781  

Sriram Kashyap 3D Reconstruction 31/ 42

slide-40
SLIDE 40

Results

Actual object

Sriram Kashyap 3D Reconstruction 32/ 42

slide-41
SLIDE 41

Results

Reconstruction

Sriram Kashyap 3D Reconstruction 33/ 42

slide-42
SLIDE 42

Results

Reconstruction

Sriram Kashyap 3D Reconstruction 34/ 42

slide-43
SLIDE 43

Results

Reconstruction

Sriram Kashyap 3D Reconstruction 35/ 42

slide-44
SLIDE 44

Results

Reconstruction

Sriram Kashyap 3D Reconstruction 36/ 42

slide-45
SLIDE 45

Results

Reconstruction

Sriram Kashyap 3D Reconstruction 37/ 42

slide-46
SLIDE 46

Results

Reconstruction

Sriram Kashyap 3D Reconstruction 38/ 42

slide-47
SLIDE 47

Results

Reconstruction

Sriram Kashyap 3D Reconstruction 39/ 42

slide-48
SLIDE 48

Results

Reconstruction

Sriram Kashyap 3D Reconstruction 40/ 42

slide-49
SLIDE 49

Future Work

1 Better calibration using optimization techniques 2 Textured rendering of visual hull 3 Image based Relighting support

Sriram Kashyap 3D Reconstruction 41/ 42

slide-50
SLIDE 50

References

1 OpenCV camera control and calibration:

  • pencv.willowgarage.com/

2 Image based Animation: http://www.cse.iitb.ac.in/

biswarup/projects/Motion/

Sriram Kashyap 3D Reconstruction 42/ 42