Computational Photography Si Lu Spring 2018 - PowerPoint PPT Presentation

Computational Photography Si Lu Spring 2018 http://web.cecs.pdx.edu/~lusi/CS510/CS510_Computati onal_Photography.htm 04/26/2018

Last Time o Panorama n Overview n Feature detection n Feature matching With slides by Prof. C. Dyer and K. Grauman 2

Today o Panorama n Homography estimation n Blending n Multi-perspective panoramas 3

Stitching Recipe o Align pairs of images Feature Detection n Feature Matching n n Homography Estimation o Align all to a common frame o Adjust (Global) & Blend 4

What can be globally aligned? o In image stitching, we seek for a model to globally warp one image into another. Are any two images of the same scene can be aligned this way? n Images captured with the same center of projection n A planar scene or far-away scene Credit: Y.Y. Chuang

A pencil of rays contains all views real synthetic camera camera Can generate any synthetic camera view as long as it has the same center of projection! Credit: Y.Y. Chuang

Mosaic as an image reprojection mosaic projection plane o The images are reprojected onto a common plane o The mosaic is formed on this plane o Mosaic is a synthetic wide-angle camera Credit: Y.Y. Chuang

Changing camera center Does it still work? PP1 PP2 Credit: Y.Y. Chuang

Planar scene (or a faraway one) PP3 PP1 PP2 o PP3 is a projection plane of both centers of projection, so we are OK! o This is how big aerial photographs are made Credit: Y.Y. Chuang

Motion models o Parametric models as the assumptions on the relation between two images. Credit: Y.Y. Chuang

2D Motion models Credit: Y.Y. Chuang

Motion models Perspective 3D rotation Affine Translation 2 unknowns 6 unknowns 8 unknowns 3 unknowns Credit: Y.Y. Chuang

Determine pairwise alignment? o Feature-based methods: only use feature points to estimate parameters o We will study the “Recognising panorama” paper published in ICCV 2003 o Run SIFT (or other feature algorithms) for each image, find feature matches. Credit: Y.Y. Chuang

Determine pairwise alignment o p’=Mp, where M is a transformation matrix, p and p’ are feature matches o It is possible to use more complicated models such as affine or perspective o For example, assume M is a 2x2 matrix       x ' m m x      11 12         y ' m m y       21 22 o Find M with the least square error n    2  Mp p '  i 1 Credit: Y.Y. Chuang

Determine pairwise alignment   '     m m   x m y m x x ' x      11 12   1 11 1 12 1       y ' m m y   '       x m y m y 21 22 1 21 1 22 1     x y 0 0 ' x     1 1 1   ' 0 0 x y m     y 1 1   11 1     x y 0 0 ' m   x     2 2 12  2       m        21       ' x y 0 0 m x       n n 22 n     0 0 x y ' y     n n n Over-determined system Credit: Y.Y. Chuang

Normal equation Given an over-determined system Ax  b the normal equation is that which minimizes the sum of the square differences between left and right sides  T T A Ax A b Why? Credit: Y.Y. Chuang

Normal equation   2   E Ax b     T    Ax b Ax b       T    T Ax b Ax b        T T T x A b Ax b     T T T T T T x A Ax b Ax x A b b b     T T     T T T T T x A Ax A b x A b x b b  E   T T 2 A Ax 2 A b  x Credit: Y.Y. Chuang

Determine pairwise alignment o p’=Mp, where M is a transformation matrix, p and p’ are feature matches o For translation model, it is easier.       n  2 2       ' ' E m x x m y y 1 i i 2 i i  i 1  E  0  m 1 o What if the match is false? Avoid impact of outliers. Credit: Y.Y. Chuang

RANSAC [Fischler and Bolles 81] o RANSAC = Random Sample Consensus o An algorithm for robust fitting of models in the presence of many data outliers n Compare to robust statistics o Given N data points x i , assume that majority of them are generated from a model with parameters  , try to recover  . M. A. Fischler and R. C. Bolles. "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography". Comm. of the ACM 24 (6): 381–395 Credit: Y.Y. Chuang

RANSAC algorithm Run k times: How many times? How big? Smaller is better (1) draw n samples randomly (2) fit parameters  with these n samples (3) for each of other N-n points, calculate its distance to the fitted model, count the number of inlier points, c Output  with the largest c How to define? Depends on the problem. Credit: Y.Y. Chuang

How to determine k p : probability of real inliers P : probability of success after k trials    n k P 1 ( 1 p ) n samples are all inliers a failure n p k failure after k trials 3 0.5 35  log( 1 P ) 6 0.6 97  k for P=0.99 6 0.5 293  n log( 1 p ) Credit: Y.Y. Chuang

Example: line fitting Credit: Y.Y. Chuang

Example: line fitting n=2 Credit: Y.Y. Chuang

Model fitting Credit: Y.Y. Chuang

Measure distances Credit: Y.Y. Chuang

Count inliers c=3 Credit: Y.Y. Chuang

Another trial c=3 Credit: Y.Y. Chuang

The best model c=15 Credit: Y.Y. Chuang

RANSAC for Homography Credit: Y.Y. Chuang

RANSAC for Homography Credit: Y.Y. Chuang

RANSAC for Homography Credit: Y.Y. Chuang

A case study: cylindrical panorama o What if you want a 360  field of view? mosaic projection cylinder Credit: Y.Y. Chuang

Cylindrical panoramas o Steps n Reproject each image onto a cylinder n Blend n Output the resulting mosaic Credit: Y.Y. Chuang

Cylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute pair-wise alignments 4. Fix up the end-to-end alignment 5. Blending 6. Crop the result and import into a viewer It is required to do radial distortion correction for better stitching results! Credit: Y.Y. Chuang

Taking pictures Kaidan panoramic tripod head Credit: Y.Y. Chuang

Where should the synthetic camera be real synthetic camera camera o The projection plan of some camera o Onto a cylinder Credit: Y.Y. Chuang

Cylindrical projection Credit: Y.Y. Chuang Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htm

Cylindrical projection Credit: Y.Y. Chuang

Cylindrical projection Credit: Y.Y. Chuang Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htm

Input images Credit: Y.Y. Chuang

Cylindrical warping Credit: Y.Y. Chuang

Alignment o a rotation of the camera is a translation of the cylinder!           2 I I I I J ( x , y ) I ( x , y ) x x y x   u     x , y x , y  x , y              2 I J ( x , y ) I ( x , y ) I I I v      y x y y     x , y x , y x , y Credit: Y.Y. Chuang

Alignment Credit: Y.Y. Chuang

Blending o Why blending: parallax, lens distortion, scene motion, exposure difference o Alpha-blending o Poisson blending o Adelson’s pyramid blending

Blending Credit: Y.Y. Chuang

Linear Blending Credit: Y.Y. Chuang

Linear Blending Credit: Y.Y. Chuang

Linear Blending Image from http://www.cs.ubc.ca/~lowe/425/slides/11-PanoramasAR.pdf 48

Multi-band Blending [BURT and ADELSON 83] Linear blending Multi-band blending A multi-resolution spline with application to image mosaics. Peter J. Burt and Edward Adelson. ACM Transactions on Graphics, 1983. 49

Multi-band Blending 1. Laplacian pyramids LA and LB are constructed for images A and B respectively. 2. A third Laplacian pyramid LS is constructed by copying nodes from the left half of LA to the corresponding nodes of LS, and nodes in the right half of LB to the right half of LS 3. The final image S is obtained by expanding and summing the levels of LS. 50

2-band Blending Low frequency (l > 2 pixels) High frequency (l < 2 pixels) Credit: Y.Y. Chuang

Linear Blending

2-band Blending

Assembling the panorama o Stitch pairs together, blend, then crop Credit: Y.Y. Chuang

Problem: Drift o Error accumulation n small errors accumulate over time Credit: Y.Y. Chuang