3D from Photographs: Introduction Dr. Gianpaolo Palma - - PowerPoint PPT Presentation

3D from Photographs: Introduction

• Dr. Gianpaolo Palma

gianpaolo.palam@isti.cnr.it

3D from Photographs

• 3D from photographs is a technology that allows us to

do a 3D reconstruction of a real-world scene starting from a set of photographs as input.

• We can see it as an alternative to 3D scanning but it

has some important issues:

• It is not MEASURING tool!
• We cannot know the result of a 3D reconstruction

beforehand.

3D from Photographs

• Advantages:
• Fully automatic process.
• Faster for creating models than modeling (e.g.

AutoCAD, Rhinoceros, etc.).

• Good scalability: from tiny (e.g., a toy) to large models

(e.g., an entire city).

• Unskilled users can create 3D models.
• Economically cheap; i.e., a digital camera.

3D from Photographs

• Disadvantages:
• Accuracy may be low; it can be improved with

expensive set-ups.

• Some real-world objects cannot be captured.
• A generated 3D model may not match ground

truth due to skew.

3D from Photographs

• The 3D model is generated using automatic

Computer Vision techniques.

• The process has three main steps.

3D model

3D from Photographs

Automatic Matching of Images

Camera Calibration Dense Matching

Photographs

Surface Reconstruction

3D model

3D from Photographs

Automatic Matching of Images

Camera Calibration Dense Matching

Photographs

Surface Reconstruction

Automatic Matching of Images

• The entire process is based on finding matches

between images.

• This means that you have to shoot pictures not too

far apart, so that the algorithm can match them easily.

Automatic Matching of Images

• For any object in an image, “interesting points” (or

corners) on the object can be extracted to provide a “feature description” (or descriptor) of the corner.

• A descriptor of an object corner (extracted from an

image) can be employed to locate the object in another image containing many other objects.

Automatic Matching of Images

Automatic Matching of Images

Automatic Matching of Images

3D model

3D from Photographs

Automatic Matching of Images

Camera Calibration Dense Matching

Photographs

Surface Reconstruction

Camera Calibration

• No prior knowledge about camera calibration is available.
• All information must be recovered from photographs.

• It is crucial that we have enough information in photographs.

• Important factors:
• Motion of the camera
• General structure of the scene
• Enough overlap: only points that are visible in at least three

images are useful.

• Note that what you want reconstruct and how you get the

photographs have great influence on the final reconstruction!

Camera Calibration

PhotoTourism: having a set of (even heterogeneous) photographs, you can navigate the photo collection in a “spatially coherent” way.

Camera Calibration

PhotoCloud is the Italian alternative to PhotoTourism made by CNR-ISTI. It requires a 3D model.

3D model

3D from Photographs

Automatic Matching of Images

Camera Calibration Dense Matching

Photographs

Surface Reconstruction

Dense Matching

• After recovery of the camera calibration, we can compute

dense depth maps:

• We need a pair of images for each depth map.
• These contain the depth of every pixel and a quality

measure (how confident we are of each particular pixel).

Dense Matching

Depth Map Input

3D model

3D from Photographs

Automatic Matching of Images

Camera Calibration Dense Matching

Photographs

Surface Reconstruction

Surface Reconstruction

Final 3D Model Dense Point Cloud

• To compute an unique 3D surface by integration of the

all the depth maps of each image:

Back to the Camera Model

Camera Model: Image Formation

~ l x ~ nx Real-world Camera Light

Camera Model: Image Formation

~ l x ~ nx Real-world Camera Light

Camera Model: Image Formation

~ l x ~ nx Real-world Camera Light

Camera Model: Pinhole Camera

world-space image-space

Yc Xc Zc f

Image Plane Hole

Yw Xw Zw

C

Camera Model: Pinhole Camera

Yc Xc Zc f

Image Plane Hole

u v

C c0

Camera Model: Image Plane

u v c0 = [u0, v0]> = C0

• Pixels have different height and width; i.e., (ku, kv).
• c0 is called the principal point.
• The image plane has a finite size: w (width) and h (height)

w h

• This can be expressed in a matrix form with a non-

linear projection:

Camera Model: Pinhole Camera

P =   −fku u0 −fkv v0 1   = K[I|0] K =   −fku u0 −fkv v0 1  

m = P · M m0 = m/mz

• The perspective projection is defined as

Camera Model: Pinhole Camera

P = K[I|0]G = K[R|t] t =   t1 t2 t3   R =   r>

1

r>

2

r>

3

 

K =   −fku u0 −fkv v0 1  

m = P · M m0 = m/mz Intrinsic Matrix Extrinsic Matrix

Camera Model: Thin Lens

Optical Axis C F D

Camera Model: Thin Lens

C F D Z Z’ M M’ 1 Z + 1 Z0 = 1 D Image Plane

Camera Model: Thin Lens

m = P · M m0 = m/mz m0 =   u v 1  

Camera Model: Thin Lens

− · r2

d =

✓(u − u0) αu ◆2 + ✓(v − v0) αv ◆2 αu = −f · ku αu = −f · kv ( u0 = (u − u0) · (1 + k1r2

d + k2r4 d + . . . + knr2n d ) + u0

v0 = (v − v0) · (1 + k1r2

d + k2r4 d + . . . + knr2n d ) + v0

✓ ◆ ✓ ◆

n is set maximum to 3.

Camera Model: Thin Lens

Camera Model: Thin Lens

Barrel distortion

Camera Model: Thin Lens

Pincushion distortion

Best Practice

Best Practice

• How do we shoot pictures?
• Practical suggestions and limitations to avoid

failures during reconstruction.

Best Practice: A Good Sequence

• We have to shoot a picture of the same location for

every step made in the shooting sequence.

• Each picture needs to be of the same scene, but

captured from a slightly different point of view.

• We have to walk with the camera in an arc around

the scene and keeping the entire scene all times.

• We have to keep the same focal length, i.e., zoom!

Best Practice: A Good Sequence

📸 📸 📸 📸 📸 📸

Best Practice: A Good Sequence

• We have to capture as many photographs as we

can:

• The more the better.
• We need at least 5-6 photographs for a very

basic reconstruction!

• A reconstruction algorithm can fail if only four

photographs or less are given as input!

Best Practice: Bad Sequences

📸

• We have to avoid “pan”

sequences (panoramas sequences); i.e., capturing photographs

• n a plane.
• These sequences do not

have 3D information.

Best Practice: Bad Sequences

📸

• We have to avoid “pan”

sequences (panoramas sequences); i.e., capturing photographs

• n a plane.
• These sequences do not

have 3D information.

Best Practice: Bad Sequences

📸 📸

• We have to avoid photo

sequences in which we shoot toward/outward the scene to capture!

Best Practice: Bad Sequences

📸 📸

• We have to avoid photo

sequences in which we shoot toward/outward the scene to capture!

Best Practice: Bad Sequences

📸 📸

• If the angle between a

photograph and another is too small, the reconstruction algorithm may fail or produce low quality reconstruction! α = 30°

Best Practice: Bad Sequences

📸 📸

• If the angle between a

photograph and another is too small, the reconstruction algorithm may fail or produce low quality reconstruction! α = 30°

Best Practice: Bad Sequences

📸

• We cannot take

photographs by rotating the person/object using a turnable table!!

Best Practice: Bad Sequences

📸

• We cannot take

photographs by rotating the person/object using a turnable table!!

Best Practice: Planar Objects

• We cannot take

photographs of planar

• bjects!

Best Practices: Not Enough Textures

Best Practices: Non Constant Appearance

Best Practices: Non Constant Appearance

Best Practices: Non Constant Appearance

Best Practices: Dynamic Scenes

Moving people or objects appear/disappear!

Best Practices: Dynamic Scenes

Moving people or objects appear/disappear!

Best Practices: Blurry Photos

• Blurry photos are caused by:
• Movements in the scenes or of the camera; i.e.,

motion blur.

• Camera is out-of-focus
• These photos MUST be avoided!
• They cause issues during reconstruction and/or

degrade the final result!

Blurry Photos

Blurry Photos

Best Practices: Self-Occlusions

• Self-occlusions have to

be treated with care!

• We have to cover all

self-occluded parts.

Best Practices: Lighting Conditions

Cloudy days are ideal because lighting is stable!

Best Practices: Lighting Conditions

Avoid moving shadows!

Software

Software

Free Software:

• Arc3D (discontinued)
• PhotoSynthToolkit

(discontinued)

• Visual SFM (discontinued)
• Multi-View Environment
• Regard 3D

Commercial Software:

• 3DZephyr
• Autodesk 123DCatch
• PhotoScan
• RealityCapture

Software: Arc3D

• Arc3D is a web-based service, which has been

developed with cultural heritage applications in mind.

• It is based on Apero and MicMac.
• It outputs a point cloud without normal:
• Results can be either visualized on MeshLab
• r ARC3D Model Viewer.

Software: Arc3D

Photographs by Maurizio Forte (C), CNR, Roma

Software: Photosynth Toolkit

• Photosynth toolkit is a package that was

developed on the blog Visual Experiments.

• It is based on Bundler (open source) for

photographs matching, camera calibration, and sparse point cloud generation.

• Dense matching relies on CMVS and PMVS.
• This tool works locally.

Software: Photosynth Toolkit

Software: VisualSFM

• A GUI application for 3D reconstruction using

structure from motion.

• It runs fast by exploiting multicore and GPU

parallelism for feature detection, feature matching, and bundle adjustment.

• Dense matching relies on CMVS and PMVS.
• This tool works locally:
• http://ccwu.me/vsfm/

Software: VisualSFM

Software: Multi-View Environment

• End-to-end pipeline for image-based geometry

reconstruction: Structure-from-Motion, Multi-View Stereo, and Surface Reconstruction.

• Command line applications, but most features

are also available from our user interface UMVE.

• Open-source (C++).
• This tool works locally.
• External tool for high resolution texturing:
• http://www.gcc.tu-darmstadt.de/home/proj/mve/

Software: Multi-View Environment

Software: Regard3D

• GUI application that integrates different open-

source libraries and tools (MVE, OpenMVG, and CMVS/PMVS).

• Complete pipeline; from the Structure-from-

Motion to the Surface Reconstruction.

• Open-source (C++).
• This tool works locally.
• http://www.regard3d.org/index.php

Software: 3DZephyr

• It is a local software that requires an one time

fee:

• $149 Lite version.
• $3,200 Pro version.
• It creates high quality 3d meshes.

Software: 123DCatch

• It is a web service by Autodesk:

http://www.123dapp.com/catch

• Main advantages:
• It is robust; it works even with challenging

photo sequences.

• It creates high quality results.
• It works as mobile app, free for now…
• It exports 3D models with color.

Software: 123DCatch

Software: PhotoScan

• It is a local software by Agisoft that requires an
• ne time fee:
• $59 for students
• $179 standard version
• It is fast and creates high quality models with

color.

Software: RealityCapture

• It is a local software by Capture Reality with a

subscription fee:

• $99 for 3 months, $7500 for one year
• $15000 forever
• The fastest software (recently a real-time capture

and reconstruction inside Unity), but very expensive.

Acknowledgements

• Some images on work by:
• Dr. Matteo Dellepiane:
• http://vcg.isti.cnr.it/~dellepiane/