Self-calibration and unordered SfM 3D photography course schedule - - PowerPoint PPT Presentation

Self-calibration and unordered SfM

3D photography course schedule

Topic

Feb 21 Introduction Feb 28 Lecture: Geometry, Camera Model, Calibration Mar 7 Lecture: Features & Correspondences Mar 14 Project Proposals Mar 21 Lecture: Epipolar Geometry Mar 28 Depth Estimation + 2 papers Apr 4 Single View Geometry + 2 papers Apr 11 Active Ranging and Structured Light + 2 papers Apr 18 Project Updates

• Apr. 25
• -- Easter ---

May 2 SLAM + 2 papers May 9 3D & Registration + 2 papers May 16 SfM/Self Calibration + 2 papers May 23 Shape from Silhouettes + 2 papers May 30 Final Projects

Evaluation

• Today: May, 16th, 2011
• Course ID: 252-0579-00G

Unordered/Uncalibrated Structure from Motion

Scenarios:

• “folders” with pictures, photo collections
• Unknown cameras/photos

Similar “multiple view geometry” as SLAM, but Challenges:

• Finding Corresponding Images/Features
• Self-Calibration

Simpler Case: Unord./Uncalib. Panorama

• Folder with photos from same position
• Estimate orientation, focal length for each

image (assume defaults for other params)

• Stitch images

(Brown/Lowe, ICCV’03, IJCV’07)

Simpler Case: Unord./Uncalib. Panorama

• SIFT features -> kd-tree
• Find nearest neighbors in descriptor space
• Pick image pair with highest #matches
• Robust estimation of homography (R,f1,f2)
• (Bundle Adjustment)

(Brown/Lowe, ICCV’03, IJCV’07)

Unordered SfM

• Finding Corresponding Images/Features

similar as in pano

• Self-Calibration
• ften simplified model K=diag(f,f,1)
• initial f from pairs,
• optimize in bundle adjustment

(Schaffalitzky/Zisserman ECCV02) (Snavely et al. SIGGRAPH06) (Brown/Lowe 3DIM05)

• > presented afterwards !

Building Rome on a cloudless day

• GIST & clustering (1h35)

SIFT & Geometric verification (11h36) SfM & Bundle (8h35) Dense Reconstruction (1h58) Some numbers

• 1PC
• 2.88M images
• 100k clusters
• 22k SfM with 307k images
• 63k 3D models
• Largest model 5700 images
• Total time 23h53

(Frahm et al. ECCV10)

Self-calibration

• Self-calibration
• Dual Absolute Quadric
• Critical Motion Sequences

Motivation

• Avoid explicit calibration procedure
• Complex procedure
• Need for calibration object
• Need to maintain calibration

Motivation

• Allow flexible acquisition
• No prior calibration necessary
• Possibility to vary intrinsics
• Use archive footage

Projective ambiguity

Reconstruction from uncalibrated images  projective ambiguity on reconstruction ´M´ M) )( ( M m

1

P T PT P   



Stratification of geometry

15 DOF 12 DOF plane at infinity parallelism More general More structure Projective Affine Metric 7 DOF absolute conic angles, rel.dist.

Constraints ?

Scene constraints

• Parallellism, vanishing points, horizon, ...
• Distances, positions, angles, ...

Unknown scene  no constraints

Camera extrinsics constraints

–Pose, orientation, ...

Unknown camera motion  no constraints

Camera intrinsics constraints

–Focal length, principal point, aspect ratio & skew

Perspective camera model too general  some constraints

Euclidean projection matrix

 

t R R K P

T T

 

           1

y y x x

u f u s f K

Factorization of Euclidean projection matrix Intrinsics: Extrinsics: 



t , R

Note: every projection matrix can be factorized, but only meaningful for euclidean projection matrices (camera geometry) (camera motion)

Constraints on intrinsic parameters

Constant

e.g. fixed camera:

Known

e.g. rectangular pixels: square pixels: principal point known:

  

2 1

K K

 s            1

y y x x

u f u s f K

,   s f f

y x

 

       2 , 2 , h w u u

y x

Self-calibration

Upgrade from projective structure to metric

structure using constraints on intrinsic camera parameters

• Constant intrinsics
• Some known intrinsics, others varying
• Constraints on intrinsics and restricted motion

(e.g. pure translation, pure rotation, planar motion)

(Faugeras et al. ECCV´92, Hartley´93, Triggs´97, Pollefeys et al. PAMI´99, ...) (Heyden&Astrom CVPR´97, Pollefeys et al. ICCV´98,...) (Moons et al.´94, Hartley ´94, Armstrong ECCV´96, ...)

A counting argument

• To go from projective (15DOF) to metric

(7DOF) at least 8 constraints are needed

• Minimal sequence length should satisfy
• Independent of algorithm
• Assumes general motion (i.e. not critical)

     

8 # 1 #      fixed m known m

Outline

• Introduction
• Self-calibration
• Dual Absolute Quadric
• Critical Motion Sequences

The Dual Absolute Quadric

        I

* T

The absolute dual quadric Ω*

∞ is a fixed conic under

the projective transformation H iff H is a similarity

1. 8 dof 2. plane at infinity π∞ is the nullvector of Ω∞ 3. Angles:

  

2 * 2 1 * 1 2 * 1

π π π π π π cos

  

   

T T T



Absolute Dual Quadric and Self-calibration

Eliminate extrinsics from equation Equivalent to projection of Dual Abs.Quadric

) )( Ω )( ( Ω

* 1 * T T T T T

P T T T PT P P KK

   

 

Dual Abs.Quadric also exists in projective world

T

´ Ω´ ´

* P

P





Transforming world so that reduces ambiguity to similarity

* *

Ω Ω´

 

* *

Absolute conic = calibration object which is always present but can only be observed through constraints on the intrinsics

T i i T i i i

Ω ω K K P P  

 

Absolute Dual Quadric and Self-calibration

Projection equation:

Translate constraints on K through projection equation to constraints on *

Constraints on *

               



1 ω

2 2 2 2 2 * y x y y y y x y x y x y x x

c c c c f c c sf c c c sf c s f

Zero skew quadratic

m

Principal point linear

2m

Zero skew (& p.p.) linear

m

Fixed aspect ratio (& p.p.& Skew) quadratic

m-1

Known aspect ratio (& p.p.& Skew) linear

m

Focal length (& p.p. & Skew) linear

m

* 23 * 13 * 33 * 12

ω ω ω ω  ω ω

* 23 * 13

  ω*

12  * 11 * 22 * 22 * 11

ω' ω ω' ω 

* 22 * 11

ω ω 

* 11 * 33

ω ω 

condition constraint type #constraints

Linear algorithm

Assume everything known, except focal length

         

Ω Ω Ω Ω Ω

23 T 13 T 12 T 22 T 11 T

    

    

P P P P P P P P P P

(Pollefeys et al.,ICCV´98/IJCV´99)

T

P P

* 2 2 *

1 ˆ ˆ ω              f f

Yields 4 constraint per image Note that rank-3 constraint is not enforced

Linear algorithm revisited

         

Ω Ω Ω Ω Ω

23 T 13 T 12 T 22 T 11 T

    

    

P P P P P P P P P P

           1 ˆ ˆ

2 2

f f

T

KK

9 1 9 1

) 3 log( ) 1 log( ) ˆ log(   f ) 1 . 1 log( ) 1 log( ) log( ˆ

ˆ

 

y x

f f

1 . 0  

x

c 1 . 0  

y

c  s

       

Ω Ω Ω Ω

33 T 22 T 33 T 11 T

   

   

P P P P P P P P

(Pollefeys et al., ECCV‘02)

1 . 11 . 1 01 . 1 2 . 1

assumptions

Weighted linear equations

Projective to metric

Compute T from using eigenvalue decomposition of and then obtain metric reconstruction as

        

 

~ with Ω ~

• r

Ω ~

* * T T

• 1
• T

I I T I T T T I

M and T PT-1

Ω*



Alternatives:

(Dual) image of absolute conic

• Equivalent to Absolute Dual Quadric
• Practical when H can be computed first
• Pure rotation (Hartley’94, Agapito et al.’98,’99)
• Vanishing points, pure translations, modulus

constraint, …

T * *

ω ω

    

H H

T

P P

* *

Ω ω

  

T y x y y y y x x y x x x

KK c c c c f c c c c c c f              



1 ω

2 2 2 2 *

1 2 2 2 2 2 2 2 2 2 2 2 2 2 2

) ( 1 ω

 

                 

T y x x y y x y x x y y x x x y y y x

KK c f c f f f c f c f c f f c f f f f

Note that in the absence of skew the IAC can be more practical than the DIAC!

Kruppa equations

Limit equations to epipolar geometry Only 2 independent equations per pair But independent of plane at infinity

       

T * T T * T *

ω e' ω e' e' ω e' F F H H

        

 

Refinement

• Metric bundle adjustment

Enforce constraints or priors

• n intrinsics during minimization

(this is „self-calibration“ for photogrammetrist)

Outline

• Introduction
• Self-calibration
• Dual Absolute Quadric
• Critical Motion Sequences

Critical motion sequences

• Self-calibration depends on camera motion
• Motion sequence is not always general enough
• Critical Motion Sequences have more than one

potential absolute conic satisfying all constraints

• Possible to derive classification of CMS

(Sturm, CVPR´97, Kahl, ICCV´99, Pollefeys,PhD´99)

Critical motion sequences: constant intrinsic parameters

Most important cases for constant intrinsics

Critical motion type ambiguity pure translation affine transformation (5DOF) pure rotation arbitrary position for  (3DOF)

• rbital motion

proj.distortion along rot. axis (2DOF) planar motion scaling axis  plane (1DOF)

Critical motion sequences: varying focal length

Most important cases for varying focal length (other parameters known)

Critical motion type ambiguity

pure rotation arbitrary position for  (3DOF) forward motion proj.distortion along opt. axis (2DOF) translation and

• rot. about opt. axis

scaling optical axis (1DOF) hyperbolic and/or elliptic motion

• ne extra solution

Critical motion sequences: algorithm dependent

Additional critical motion sequences can exist

for some specific algorithms

• when not all constraints are enforced

(e.g. not imposing rank 3 constraint)

• Kruppa equations/linear algorithm: fixating a

point

Some spheres also project to circles located in the image and hence satisfy all the linear/kruppa self-calibration constraints

• Inst. f. Visual Computing: Open Lab
• Next week:

Wednesday, 25th, 16-18

• Thesis opportunities
• Talk to researchers

Projects

Project Presentations (in two weeks !) must include

• Goals / chosen solution (potential own ideas)
• who did what
• software used from others / web / … vs. own parts
• Results: live or video [+ other offline results]
• [+ whatever is interesting]

Plan for 10 min./team incl. discussion/questions Report per team (latest by Sun., June 5th)

• Same content as above
• + citations (main works built upon)
• Style? Use final ICCV latex/doc template (4-8 pages) from

http://www.iccv2011.org/paper-submission (but don’t worry too much about formatting details)

• Email as pdf to us

Presentations

• Modelling the World from Internet Photo Collections
• Live Dense Reconstruction with a single Moving Camera