Computational Photography Si Lu Spring 2018 http://web.cecs.pdx.edu/~lusi/CS510/CS510_Computati onal_Photography.htm 04/24/2018
Last Time o Relighting n Tone Mapping n HDR 2
Today o Panorama n Overview n Feature detection n Feature matching o Mid-term project presentation n Not real mid-term n 8 minutes presentation n Schedule o May 1 and 3 With slides by Prof. C. Dyer and K. Grauman 3
Panorama Building: History Along the River During Ching Ming Festival by Z.D Zhang (1085-1145 ) San Francisco from Rincon Hill, 1851, by Martin Behrmanx 4
Panorama Building: A Concise History o The state of the art and practice is good at assembling images into panoramas n Mid 90s –Commercial Players (e.g. QuicktimeVR) n Late 90s –Robust stitchers(in research) Early 00s –Consumer stitching common n Mid 00s –Automation n 5
Stitching Recipe o Align pairs of images o Align all to a common frame o Adjust (Global) & Blend 6
Stitching Images Together 7
When do two images “stitch”? 8
Images can be transformed to match 9
Images are related by Homographies 10
Compute Homographies 11
Automatic Feature Points Matching o Match local neighborhoods around points o Use descriptors to efficiently compare: SIFT n [Lowe 04] most common choice 12
Stitching Recipe o Align pairs of images o Align all to a common frame o Adjust (Global) & Blend 13
Wide Baseline Matching • Images taken by cameras that are far apart make the correspondence problem very difficult • Feature-based approach: Detect and match feature points in pairs of images Credit: C. Dyer
Matching with Features • Detect feature points • Find corresponding pairs Credit: C. Dyer
Matching with Features o Problem 1: n Detect the same point independently in both images no chance to match! We need a repeatable detector Credit: C. Dyer
Matching with Features o Problem 2: n For each point correctly recognize the corresponding point ? We need a reliable and distinctive descriptor Credit: C. Dyer
Properties of an Ideal Feature o Local: features are local, so robust to occlusion and clutter (no prior segmentation) o Invariant (or covariant) to many kinds of geometric and photometric transformations o Robust: noise, blur, discretization, compression, etc. do not have a big impact on the feature o Distinctive: individual features can be matched to a large database of objects o Quantity: many features can be generated for even small objects o Accurate: precise localization o Efficient: close to real-time performance Credit: C. Dyer
Problem 1: Detecting Good Feature Points [Image from T. Tuytelaars ECCV 2006 tutorial] Credit: C. Dyer
Feature Detectors o Hessian o Harris o Lowe: SIFT (DoG) o Mikolajczyk & Schmid: Hessian/Harris-Laplacian/Affine o Tuytelaars & Van Gool: EBR and IBR o Matas: MSER o Kadir & Brady: Salient Regions o Others Credit: C. Dyer
Harris Corner Point Detector C. Harris, M. Stephens, “A Combined Corner and Edge Detector,” 1988 Credit: C. Dyer
Harris Detector: Basic Idea o We should recognize the point by looking through a small window o Shifting a window in any direction should give a large change in response Credit: C. Dyer
Harris Detector: Basic Idea “flat” region: “edge”: “corner”: no change in no change along significant change all directions the edge direction in all directions Credit: C. Dyer
Harris Detector: Derivation Change of intensity for a (small) shift by [ u,v ] in image I : 2 E u v ( , ) w x y ( , ) I x ( u y , v ) I x y ( , ) x y , Weighting Shifted Intensity function intensity Weighting function w ( x,y ) = or 1 in window, 0 outside Gaussian Credit: R. Szeliski
Harris Detector Apply 2 nd order Taylor series expansion: 2 2 E ( u , v ) Au 2 Cuv Bv A C u E u v ( , ) u v 2 A w ( x , y ) I ( x , y ) C B v x x , y 2 B w ( x , y ) I ( x , y ) y I x I ( x , y ) / x x , y I y I ( x , y ) / y C w ( x , y ) I ( x , y ) I ( x , y ) x y x , y Credit: R. Szeliski
Harris Corner Detector Expanding E ( u , v ) in a 2 nd order Taylor series, we have, for small shifts, [ u,v ], a bilinear approximation: u E u v ( , ) u v , M v where M is a 2 2 matrix computed from image derivatives: 2 I I I I x I ( x , y ) / x x x y M w x y ( , ) I y I ( x , y ) / y 2 I I I x y , x y y Note: Sum computed over small neighborhood around given pixel Credit: R. Szeliski
Harris Corner Detector Intensity change in shifting window: eigenvalue analysis u 1 , 2 – eigenvalues of M E u v ( , ) u v , M v direction of the fastest change Ellipse E ( u , v ) = const direction of the slowest change ( max ) -1/2 ( min ) -1/2 Credit: R. Szeliski
Selecting Good Features Image patch SSD surface 1 and 2 both large Credit: C. Dyer
Selecting Good Features SSD surface large 1 , small 2 Credit: C. Dyer
Selecting Good Features SSD surface small 1 , small 2 Credit: C. Dyer
Harris Corner Detector 2 “Edge” 2 >> 1 “Corner” Classification of 1 and 2 both large, image points using 1 ~ 2 ; eigenvalues of M : E increases in all directions 1 and 2 are small; “Edge” E is almost constant “Flat” 1 >> 2 in all directions region 1 Credit: C. Dyer
Harris Corner Detector Measure of corner response: 2 R det M k trace M det M 1 2 trace M 1 2 k is an empirically-determined constant; e.g., k = 0.05 Credit: C. Dyer
Harris Corner Detector 2 “Edge” “Corner” • R depends only on R < 0 eigenvalues of M • R is large for a corner R > 0 • R is negative with large magnitude for an edge • | R | is small for a flat region “Flat” “Edge” |R| small R < 0 1 Credit: C. Dyer
Harris Corner Detector: Algorithm o Algorithm: 1. Find points with large corner response function R (i.e., R > threshold) 2. Take the points of local maxima of R (for localization) by non- maximum suppression Credit: C. Dyer
Harris Detector: Example Credit: C. Dyer
Harris Detector: Example Compute corner response R = 1 2 – k ( 1 + 2 ) 2 Credit: C. Dyer
Harris Detector: Example Find points with large corner response: R > threshold Credit: C. Dyer
Harris Detector: Example Take only the points of local maxima of R Credit: C. Dyer
Harris Detector: Example Credit: C. Dyer
Harris Detector: Example Interest points extracted with Harris (~ 500 points) Credit: C. Dyer
Harris Detector: Example Credit: C. Dyer
Harris Detector: Summary o Average intensity change in direction [ u,v ] can be expressed in bilinear form: u E u v ( , ) u v , M v o Describe a point in terms of eigenvalues of M: measure of corner response: 2 R k 1 2 1 2 o A good (corner) point should have a large intensity change in all directions , i.e., R should be a large positive value Credit: C. Dyer
Harris Detector Properties o Rotation invariance Ellipse rotates but its shape (i.e., eigenvalues) remains the same Corner response R is invariant to image rotation Credit: C. Dyer
Harris Detector Properties o But not invariant to image scale Fine scale: All points will Coarse scale: Corner be classified as edges Credit: C. Dyer
Harris Detector Properties o Quality of Harris detector for different scale changes Repeatability rate: # correct correspondences # possible correspondences C. Schmid et al., “Evaluation of Interest Point Detectors,” IJCV 2000 Credit: C. Dyer
Invariant Local Features o Goal: Detect the same interest points regardless of image changes due to translation, rotation, scale, viewpoint 46
Models of Image Change o Geometry n Rotation n Similarity (rotation + uniform scale) n Affine (scale dependent on direction) valid for: orthographic camera, locally planar object o Photometry n Affine intensity change ( I a I + b ) Credit: C. Dyer 47
SIFT Detector [Lowe ’04] o Difference-of-Gaussian (DoG) is an approximation of the Laplacian-of-Gaussian (LoG) = Lowe, D. G., “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision, 60, 2, pp. 91-110, 2004 Credit: C. Dyer
SIFT Detector Credit: C. Dyer
SIFT Detector Credit: C. Dyer
R B a s e l r u l e p S u b t r a c t m SIFT Detector Algorithm Summary o Detect local maxima in position and scale of squared values of difference-of-Gaussian o Fit a quadratic to surrounding values for sub-pixel and sub- scale interpolation o Output = list of ( x , y , ) points Credit: C. Dyer
Recommend
More recommend
