3D Fusion 3D photography course schedule Topic Feb 21 Introduction - - PowerPoint PPT Presentation

3D Registration 3D Fusion

3D photography course schedule

Topic

Feb 21 Introduction Feb 28 Lecture: Geometry, Camera Model, Calibration Mar 7 Lecture: Features & Correspondences Mar 14 Project Proposals Mar 21 Lecture: Epipolar Geometry Mar 28 Depth Estimation + 2 papers Apr 4 Single View Geometry + 2 papers Apr 11 Active Ranging and Structured Light + 2 papers Apr 18 Project Updates

• Apr. 25
• -- Easter ---

May 2 SLAM + 2 papers May 9 3D & Registration + 2 papers May 16 SfM/Self Calibration + 2 papers May 23 Shape from Silhouettes + 2 papers May 30 Final Projects (if not demo day)

3D modeling

• Aligning range images
• Pairwise
• Globally
• Surface reconstruction
• Single range image
• Merged

(some slides from S. Rusinkiewicz, J. Ponce,…)

Aligning 3D Data

• If correct correspondences are known,

it is possible to find correct relative rotation/translation

q = a +  q is a quaternion, a  R is its real part, and   R3 is its imaginary part.

• Sum of quaternions:
• Multiplication by a scalar:
• Quaternion product:
• Conjugate: )

Operations on quaternions:

• Norm:

Note:

Intermezzo: quaternions

Let R denote the rotation of angle  about the unit vector u. Define Then for any vector , Reciprocally, if q = a + ( b, c, d )T is a unit quaternion, the corresponding rotation matrix is:

Intermezzo: quaternions and rotations

Problem: Find the rotation matrix R and the vector t that minimize

i=1

E =  | xi’ – R xi – t |2 .

n

At a minimum: 0 = E/ t = –2  ( xi’ – R xi – t ) .

n i=1

Or.. t = x’ – R x. If yi = xi –x and yi’ = xi’ –x’, the error is (at a minimum):

i=1

E =  | yi’ – R yi |2

n i=1

=  | yi’ q – q yi |2

n i=1

=  | yi’ – q yi q|2|q|2

n

Or E = Linear least squares !!

Estimate rigid transformation

Aligning 3D Data

• How to find corresponding points?
• Previous systems based on user input,

feature matching, surface signatures, etc.

Spin Images

• Johnson and Hebert
• “Signature” that captures local shape
• Similar shapes  similar spin images

Computing Spin Images

• Start with a point on a 3D model
• Find (averaged) surface normal at that

point

• Define coordinate system centered at this

point, oriented according to surface normal and two (arbitrary) tangents

• Express other points (within some

distance) in terms of the new coordinates

Computing Spin Images

• Compute histogram of locations of other

points, in new coordinate system, ignoring rotation around normal:

n ˆ   p  n ˆ   p  n ˆ   p  n ˆ   p 

“radial dist.” “elevation”

Computing Spin Images

“radial dist.” “elevation”

Spin Image Parameters

• Size of neighborhood
• Determines whether local or global shape

is captured

• Big neighborhood: more discriminatory power
• Small neighborhood: resistance to clutter
• Size of bins in histogram:
• Big bins: less sensitive to noise
• Small bins: captures more detail

Alignment with Spin Image

• Compute Spin Image for each point / subset of

points in both sets

• Find similar spin images => potential

correspondences

• Compute alignment from correspondences

Same problems as with image matching:

• Robustness of descriptor vs. discriminative power
• Mismatches => robust estimation required

Solving 3D puzzles with VIPs

SIFT features

• Extracted from 2D images
• Variation due to viewpoint

VIP features

• Extracted from 3D model
• Viewpoint invariant

15

(Wu et al., CVPR08)

Presented afterwards !

Aligning 3D Data

Alternative: assume closest points correspond

to each other, compute the best transform…

Aligning 3D Data

… and iterate to find alignment

Iterated Closest Points (ICP) [Besl & McKay 92]

Converges if starting position “close enough“

ICP Variant – Point-to-Plane Error Metric

• Using point-to-plane distance instead of point-to-

point lets flat regions slide along each other more easily [Chen & Medioni 91]

Finding Corresponding Points

• Finding closest point is most expensive stage of ICP
• Brute force search – O(n)
• Spatial data structure (e.g., k-d tree) – O(log n)
• Voxel grid – O(1), but large constant, slow preprocessing

Finding Corresponding Points

• For range images, simply project point [Blais 95]
• Constant-time, fast
• Does not require precomputing a spatial data structure

Efficient ICP

• “Efficient Variants of the ICP algorithm”

[Rusinkiewicz & Levoy, 3DIM 2001]

=> Presented afterwards

3D Global Registration

3D Global registration

The problem:

• Given: n scans around an object
• Goal: align them all
• First attempt: ICP each scan to one other

• Want method for distributing accumulated

error among all scans

3D Global registration

Approach #1: Avoid the Problem

• In some cases have 1 scan that covers

large part of surface (e.g., cylindrical scan)

• Align all other scans to this “anchor”
• Disadvantage: not always practical to
• btain anchor scan

Approach #2: The Greedy Solution

• Align each new scan to all previous scans
• Disadvantages:
• Order dependent
• Doesn’t spread out error

Approach #3: Brute-Force Solution

• While not converged:
• For each scan:
• For each point:

– For every other scan » Find closest point

• Minimize error w.r.t. transforms of all scans
• Disadvantage:
• Solve (np)(np) matrix equation, where n is

number of scans and p is number of points per scan

Approach #3a: Slightly Less Brute- Force

• While not converged:
• For each scan:
• For each point:

– For every other scan » Find closest point

• Minimize error w.r.t. transform of this scan
• Faster than previous method

(matrices are pp)

Graph Methods

• Many globalreg algorithms create a graph
• f pairwise alignments between scans

Scan 1 Scan 1 Scan 5 Scan 5 Scan 4 Scan 4 Scan 3 Scan 3 Scan 2 Scan 2 Scan 6 Scan 6

Pulli’s Algorithm

• Perform pairwise ICPs, record sample

(e.g. 200) of corresponding points

• For each scan, starting w. most

connected

• Align scan to existing set
• While (change in error) > threshold
• Align each scan to others

Sharp et al. Algorithm

• Perform pairwise ICPs, record only optimal

rotation/translation for each

• Decompose alignment graph into cycles
• While (change in error) > tolerance
• For each cycle:
• Spread out error equally among all scans in the cycle
• For each scan belonging to more than 1 cycle:
• Assign average transform to scan

Lu and Milios Algorithm

• Perform pairwise ICPs, record optimal

rotation/translation and covariance for each

• Least squares simultaneous minimization
• f all errors (covariance-weighted)
• Requires linearization of rotations

Open Questions in Global Registration

• Best way to do correctly-weighted

globalreg without linearizing rotations?

• How to prevent bias (if many scans in one

area, few scans in another)?

• Robust outlier detection
• One bad ICP can throw off the entire model

Surface Reconstruction

Problems With Reconstruction from General Point Clouds

Surface Reconstruction from Range Images

• Often an easier problem than

reconstruction from arbitrary point clouds

• Implicit information about adjacency,

connectivity

• Roughly uniform spacing

Surface Reconstruction From Range Images

• First, construct surface from each range

image

• Then, merge resulting surfaces
• Obtain average surface in overlapping regions
• Control point density

Range Image Tesselation

• Given a range image, connect up the neighbors

3D surface model

Depth image Triangle mesh Texture image Textured 3D Wireframe model

Range Image Tesselation

• Caveat #1: can’t be too aggressive
• Introduce distance threshold for tesselation

• Caveat #2: Which way to triangulate?
• Possible heuristics:
• Shorter diagonal
• Dihedral angle closer to 180
• Maximize smallest angle in both triangles
• Always the same way (best triangle strips)

Range Image Tesselation

Scan Merging Using Zippering

• Turk & Levoy, 1994
• Erode geometry in overlapping areas
• Stitch scans together along seam
• Re-introduce all data
• Weighted average

Zippering

Point Weighting

• Higher weights to points facing the

camera

• Favor higher sampling rates

Point Weighting

• Lower weights

(tapering to 0) near boundaries

• Smooth blends

between views

Volumetric Reconstruction

• Implicit function defined volumetrically
• Usually stored sampled on a 3D grid
• Can be compressed (e.g., using RLE)
• Another possibility: hierarchical data

structures

• Can extract isosurface (i.e., subset of

space where implicit function = some constant)

Volumetric Reconstruction Overview

• Generate signed distance function (or

something close to it) for each scan

• Compute average (possibly weighted)
• Extract isosurface

Volumetric integration

(Curless and Levoy, Siggraph´96)

sensor range surfaces volume distance depth weight (~accuracy) signed distance to surface surface1 surface2 combined estimate

• use voxel space
• new surface as zero-crossing

(find using marching cubes)

• least-squares estimate

(zero derivative=minimum)

Volumetric Reconstruction Benefits

• Always generates a manifold surface
• Can control sampling density
• Averaging of signed distance functions

corresponds to averaging the surfaces

Volumetric Reconstruction Drawbacks

• Represent a 3D entity rather than 2D
• Running time
• Storage
• Resampling step – bandlimits the function
• Generates consistent topology, but not

always the topology you wanted

• Problems with very thin surfaces

From volume to mesh: Marching Cubes

“Marching Squares” in 2D

“Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.

From volume to mesh: Marching Cubes

“Marching Cubes: A High Resolution 3D Surface Construction Algorithm”, William E. Lorensen and Harvey E. Cline, Computer Graphics (Proceedings of SIGGRAPH '87), Vol. 21, No. 4, pp. 163-169.

Multiple depth images Volumetric integration

Volumetric 3D integration

Presentations

• Efficient Variants of the ICP

algorithm

• Model Matching with Viewpoint

Invariant Patches