Computational Photography Si Lu Spring 2018 - - PowerPoint PPT Presentation

Computational Photography

Si Lu

Spring 2018

http://web.cecs.pdx.edu/~lusi/CS510/CS510_Computati

• nal_Photography.htm

04/26/2018

Last Time

• Panorama

n Overview n Feature detection n Feature matching

2

With slides by Prof. C. Dyer and K. Grauman

Today

• Panorama

n Homography estimation n Blending n Multi-perspective panoramas

3

Stitching Recipe

• Align pairs of images

n Feature Detection n Feature Matching n Homography Estimation

• Align all to a common frame
• Adjust (Global) & Blend

4

What can be globally aligned?

• In image stitching, we seek for a model to

globally warp one image into another. Are any two images of the same scene can be aligned this way? n Images captured with the same center of projection n A planar scene or far-away scene

Credit: Y.Y. Chuang

A pencil of rays contains all views

real camera synthetic camera Can generate any synthetic camera view as long as it has the same center of projection!

Credit: Y.Y. Chuang

Mosaic as an image reprojection

mosaic projection plane

• The images are reprojected onto a common plane
• The mosaic is formed on this plane
• Mosaic is a synthetic wide-angle camera

Credit: Y.Y. Chuang

Changing camera center

Does it still work?

PP1 PP2

Credit: Y.Y. Chuang

Planar scene (or a faraway one)

• PP3 is a projection plane of both centers of projection, so we

are OK!

• This is how big aerial photographs are made

PP1 PP3 PP2

Credit: Y.Y. Chuang

Motion models

• Parametric models as the assumptions on the

relation between two images.

Credit: Y.Y. Chuang

2D Motion models

Credit: Y.Y. Chuang

Motion models

Translation 2 unknowns Affine 6 unknowns Perspective 8 unknowns 3D rotation 3 unknowns

Credit: Y.Y. Chuang

Determine pairwise alignment?

• Feature-based methods: only use feature

points to estimate parameters

• We will study the “Recognising panorama”

paper published in ICCV 2003

• Run SIFT (or other feature algorithms) for

each image, find feature matches.

Credit: Y.Y. Chuang

Determine pairwise alignment

• p’=Mp, where M is a transformation matrix, p

and p’ are feature matches

• It is possible to use more complicated models

such as affine or perspective

• For example, assume M is a 2x2 matrix
• Find M with the least square error

 







n i

p Mp

1 2

'                          y x m m m m y x

22 21 12 11

' '

Credit: Y.Y. Chuang

Determine pairwise alignment

                         y x m m m m y x

22 21 12 11

' '

' 1 22 1 21 1 ' 1 12 1 11 1

y m y m x x m y m x                                                           

' ' ' 2 ' 1 ' 1 22 21 12 11 2 2 1 1 1 1 n n n n n n

y x x y x m m m m y x y x y x y x y x     

Over-determined system

Credit: Y.Y. Chuang

Normal equation

Given an over-determined system

b Ax  b A Ax A

T T



the normal equation is that which minimizes the sum of the square differences between left and right sides Why?

Credit: Y.Y. Chuang

Normal equation

       

 



 



   

b b x b A x b A Ax A x b b b A x Ax b Ax A x b Ax b A x b Ax b Ax b Ax b Ax b Ax

T T T T T T T T T T T T T T T T

                  

T T T

E

2

b A Ax A x

T T

2 2    E

Credit: Y.Y. Chuang

Determine pairwise alignment

• p’=Mp, where M is a transformation matrix, p

and p’ are feature matches

• For translation model, it is easier.
• What if the match is false? Avoid impact of
• utliers.

   

 





     

n i i i i i

y y m x x m E

1 2 ' 2 2 ' 1 1

m E   

Credit: Y.Y. Chuang

RANSAC [Fischler and Bolles 81]

• RANSAC = Random Sample Consensus
• An algorithm for robust fitting of models in the

presence of many data outliers

n Compare to robust statistics

• Given N data points xi, assume that majority of

them are generated from a model with parameters , try to recover .

• M. A. Fischler and R. C. Bolles. "Random Sample Consensus: A Paradigm for Model Fitting with

Applications to Image Analysis and Automated Cartography". Comm. of the ACM 24 (6): 381–395 Credit: Y.Y. Chuang

RANSAC algorithm

Run k times: (1) draw n samples randomly (2) fit parameters  with these n samples (3) for each of other N-n points, calculate its distance to the fitted model, count the number of inlier points, c Output  with the largest c

How many times? How big? Smaller is better How to define? Depends on the problem.

Credit: Y.Y. Chuang

How to determine k

p: probability of real inliers P: probability of success after k trials

k n

p P ) 1 ( 1   

n samples are all inliers a failure failure after k trials

) 1 log( ) 1 log(

n

p P k   

n p k 3 0.5 35 6 0.6 97 6 0.5 293

for P=0.99

Credit: Y.Y. Chuang

Example: line fitting

Credit: Y.Y. Chuang

Example: line fitting

n=2

Credit: Y.Y. Chuang

Model fitting

Credit: Y.Y. Chuang

Measure distances

Credit: Y.Y. Chuang

Count inliers

c=3

Credit: Y.Y. Chuang

Another trial

c=3

Credit: Y.Y. Chuang

The best model

c=15

Credit: Y.Y. Chuang

RANSAC for Homography

Credit: Y.Y. Chuang

RANSAC for Homography

Credit: Y.Y. Chuang

RANSAC for Homography

Credit: Y.Y. Chuang

A case study: cylindrical panorama

• What if you want a 360 field of view?

mosaic projection cylinder

Credit: Y.Y. Chuang

Cylindrical panoramas

• Steps

n Reproject each image onto a cylinder n Blend n Output the resulting mosaic

Credit: Y.Y. Chuang

Cylindrical panorama

• 1. Take pictures on a tripod (or handheld)
• 2. Warp to cylindrical coordinate
• 3. Compute pair-wise alignments
• 4. Fix up the end-to-end alignment
• 5. Blending
• 6. Crop the result and import into a viewer

It is required to do radial distortion correction for better stitching results!

Credit: Y.Y. Chuang

Taking pictures

Kaidan panoramic tripod head

Credit: Y.Y. Chuang

Where should the synthetic camera be

• The projection plan of some camera
• Onto a cylinder

real camera synthetic camera

Credit: Y.Y. Chuang

Cylindrical projection

Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htm

Credit: Y.Y. Chuang

Cylindrical projection

Credit: Y.Y. Chuang

Cylindrical projection

Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htm

Credit: Y.Y. Chuang

Input images

Credit: Y.Y. Chuang

Cylindrical warping

Credit: Y.Y. Chuang

Alignment

• a rotation of the camera is a translation of the

cylinder!

   

                           

     

y x y y x x y x y y x y x y x y x y x x

y x I y x J I y x I y x J I v u I I I I I I

, , , 2 , , , 2

) , ( ) , ( ) , ( ) , (

Credit: Y.Y. Chuang

Alignment

Credit: Y.Y. Chuang

Blending

• Why blending: parallax, lens distortion, scene

motion, exposure difference

• Alpha-blending
• Poisson blending
• Adelson’s pyramid blending

Blending

Credit: Y.Y. Chuang

Linear Blending

Credit: Y.Y. Chuang

Linear Blending

Credit: Y.Y. Chuang

Linear Blending

48

Image from http://www.cs.ubc.ca/~lowe/425/slides/11-PanoramasAR.pdf

Multi-band Blending [BURT and ADELSON 83]

49

Linear blending Multi-band blending

A multi-resolution spline with application to image mosaics. Peter J. Burt and Edward Adelson. ACM Transactions on Graphics, 1983.

Multi-band Blending

• 1. Laplacian pyramids LA and LB are

constructed for images A and B respectively.

• 2. A third Laplacian pyramid LS is constructed by

copying nodes from the left half of LA to the corresponding nodes of LS, and nodes in the right half of LB to the right half of LS

• 3. The final image S is obtained by expanding

and summing the levels of LS.

50

Low frequency (l > 2 pixels) High frequency (l < 2 pixels)

2-band Blending

Credit: Y.Y. Chuang

Linear Blending

2-band Blending

Assembling the panorama

• Stitch pairs together, blend, then crop

Credit: Y.Y. Chuang

Problem: Drift

• Error accumulation

n small errors accumulate over time

Credit: Y.Y. Chuang

Problem: Drift

• Solution

n add another copy of first image at the end n there are a bunch of ways to solve this problem

• add displacement of (y1 – yn)/(n -1) to each image after the first
• compute a global warp: y’ = y + ax
• run a big optimization problem, incorporating this constraint

n best solution, but more complicated n known as “bundle adjustment”

(x1,y1)

• copy of first

image (xn,yn)

Credit: Y.Y. Chuang

End-to-end alignment and crop

Credit: Y.Y. Chuang

Viewer: panorama

example:

http://www.cs.washington.edu/education/courses/cse590ss/01wi/projects/project1/students/dougz/index.html

Credit: Y.Y. Chuang

Demo

http://graphics.stanford.edu/courses/cs178/applets/projection.html

59

Multi-perspective Panorama

60

Reprint from Seitz and Kim 03

Multi-perspective Panorama

61

• A. Agarwala, M. Agrawala, M. Cohen, D. Salesin, R. Szeliski. Photographing long

scenes with multi-viewpoint panoramas. SIGGRAPH 2006

A Good Multiple Perspective Panorama

• Each object in the scene is rendered from a viewpoint

roughly in front of it to avoid perspective distortion.

• The panoramas are composed of large regions of

linear perspective seen from a viewpoint where a person would naturally stand (for example, a city block is viewed from across the street, rather than from some faraway viewpoint).

• Local perspective effects are evident; objects closer to

the image plane are larger than objects further away, and multiple vanishing points can be seen.

• The seams between these perspective regions do not

draw attention

62

Picture Surface Selection

63

A plan view (xz slice) of a scene. The extracted camera locations are shown in red, and the recovered 3D scene points in black. The blue polyline of the picture surface is drawn by the user to follow the building facades. The y-axis of the scene extends out of the page; the polyline is swept up and down the y-axis to form the picture surface. z x

Reprint from Agarwala et al. 2006

Project Images onto Picture Surface

64

Reprint from Agarwala et al. 2006

View Point Selection

65

1.Each image is a view 2.Pick a good view for each pixel

Reprint from Agarwala et al. 2006

Examples

66

http://grail.cs.washington.edu/projects/multipano/

Next Week

• Mid-term project presentation

n Speak clearly and slowly n It is just a warm up n Relax and enjoy

67