Outline Types of transformations and invariance Scale invariance - PDF document

Outline • Types of transformations and invariance – Scale invariance Lecture 13: Local invariant features • Local features: detectors and descriptors – SIFT Tuesday, Oct 30 Prof. Kristen Grauman Geometric transformations • What would we like our image descriptions to be invariant to? Figure from T. Tuytelaars ECCV 2006 tutorial Photometric transformations And other nuisances… • Noise • Blur • Compression artifacts • Appearance variation for a category Figure from T. Tuytelaars ECCV 2006 tutorial

Classes of transformations Exhaustive search A multi-scale approach • Euclidean/rigid : Translation + rotation • Similarity : Translation + rotation + uniform scale Projective transformation Similarity transformation Translation and Scaling Affine transformation Translation • Affine : Similarity + shear – Valid for orthographic camera, locally planar object • (Projective : Affine + projective warps) • Photometric : affine intensity change – I -> aI + b Slide from T. Tuytelaars ECCV 2006 tutorial Exhaustive search Exhaustive search A multi-scale approach A multi-scale approach Slide from T. Tuytelaars ECCV 2006 tutorial Slide from T. Tuytelaars ECCV 2006 tutorial Exhaustive search Key idea of invariance A multi-scale approach We want to extract the patches from each image independently . Slide from T. Tuytelaars ECCV 2006 tutorial Slide adapted from T. Tuytelaars ECCV 2006 tutorial

Invariant local features (Good) invariant local features • Reliably detected Subset of local feature types designed to be invariant to y 1 • Distinctive y 2 – Scale … – Translation • Robust to noise, blur, etc. y d – Rotation • Description normalized properly – Affine transformations – Illumination x 1 x 2 1) Detect distinctive interest points … x d 2) Extract invariant descriptors [Mikolajczyk & Schmid, Matas et al., Tuytelaars & Van Gool, Lowe, Kadir et al.,… ] Review: corner detection as an interest Interest points: From stereo to recognition operator • Feature detectors previously used for stereo, motion tracking • Now also for recognition – Schmid & Mohr 1997 • Harris corners to select interest points • Rotationally invariant descriptor of local image “flat” region: “edge”: “corner”: regions no change in no change along significant change • Identify consistent clusters of matched features all directions the edge direction in all directions to do recognition C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988 [Slide credit: Darya Frolova and Denis Simakov] Review: Harris Detector Workflow Review: Harris Detector Workflow Compute corner response R

Review: Harris Detector Workflow Review: Harris Detector Workflow Find points with large corner response: R> threshold Take only the points of local maxima of R Review: Harris Detector Workflow Harris Detector • Rotation invariance Ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner response R is invariant to image rotation But, for corner detection we must search windows at a pre-determined scale . Scale space (Witkin 83) Scale space first derivative peaks contours of f’’ = 0 in scale-space Scale space insights: • edge position may shift with increasing scale ( σ ) • two edges may merge with increasing scale (edges can disappear) larger • an edge may not split into two with increasing scale (new edges do not appear) x Gaussian filtered 1d signal Adapted from Steve Seitz, UW

Scale Invariant Detection Scale Invariant Detection • Consider regions of different sizes around a • The problem: how do we choose point corresponding circles independently in each image? • At the right scale, regions of corresponding content will look the same in both images [Slides by Darya Frolova and Denis Simakov] Scale Invariant Detection Scale Invariant Detection • Solution: • Common approach: – Design a function on the region (circle), which is Take a local maximum of this function “scale invariant” ( the same for corresponding regions, even if they are at different scales ) Observation: region size, for which the maximum is achieved, should be invariant to image scale. Example: average intensity. For corresponding regions (even of different sizes) it will be the same. Important: this scale invariant region size – For a point in one image, we can consider it as a function of region size (circle radius) is found in each image independently! Image 1 f Image 1 f f Image 2 f Image 2 scale = 1/2 scale = 1/2 s 1 s 2 region size region size region size region size Scale Invariant Detection Following example was created by T. [Images from T. Tuytelaars] Tuytelaars, ECCV 2006 tutorial

Scale space Scale Invariant Detection • A “good” function for scale detection: Scale space insights: has one stable sharp peak • edge position may shift with increasing scale ( σ ) f f • two edges may merge with increasing scale f Good ! bad (edges can disappear) bad region size region size • an edge may not split into two with increasing region size scale (new edges do not appear) • For usual images: a good function would be a one which responds to contrast (sharp local What could be an approximation of intensity change) an image’s scale space? Scale invariant detection Scale selection principle Requires a method to repeatably select points in • Intrinsic scale is the scale at which location and scale: normalized derivative assumes a – Only reasonable scale-space kernel is a Gaussian maximum -- marks a feature containing (Koenderink, 1984; Lindeberg, 1994) interesting structure . (T. Lindeberg ’94) – An efficient choice is to detect peaks in the difference u l B r r u l B c a r t b u t a S t c r t b u S of Gaussian pyramid (Burt & Adelson, 1983; Crowley & Parker, 1984) � Maxima/minima of Laplacian – Difference-of-Gaussian is a close approximation to Laplacian Slide adapted from David Lowe, UBC

Scale Invariant Detection Kernels: f = ∗ Kernel Image ( ) = σ σ + σ 2 L G ( , , x y ) G ( , , x y ) xx yy (Laplacian) = σ − σ DoG G x y k ( , , ) G x y ( , , ) (Difference of Gaussians) where Gaussian 2 + 2 x y − σ = σ 2 G x y ( , , ) 1 e 2 πσ Scale space images: repeatedly Adjacent Gaussian images 2 convolve with Gaussian subtracted [Slide by Darya Frolova and Denis Simakov] SIFT: Key point localization SIFT: Example of keypoint detection Threshold on value at DOG peak and on ratio of principle n Detect maxima and minima curvatures (Harris approach) of difference-of-Gaussian in scale space (a) 233x189 image (b) 832 DOG extrema n Then reject points with low (c) 729 left after peak contrast (threshold) value threshold (d) 536 left after testing n Eliminate edge responses B l u r ratio of principle t S c a t b u r curvatures (use ratio of principal curvatures) Candidate keypoints: list of (x,y, σ ) Adapted from David Lowe, UBC Slide from David Lowe, UBC Scale Invariant Detectors Scale Invariant Detection: Summary • Experimental evaluation of detectors w.r.t. scale change • Given: two images of the same scene with a large scale difference between them • Goal: find the same interest points Repeatability rate: independently in each image # correspondences • Solution: search for maxima of suitable # possible correspondences functions in scale and in space (over the image) K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001

Affine Invariant Detection Affine Invariant Detection • Above we considered: • Intensity-based regions (IBR): Similarity transform (rotation + uniform scale) – Start from a local intensity extrema – Consider intensity profile along rays – Select maximum of invariant function f(t) along each ray – Connect local maxima – Fit an ellipse • Now we go on to: Affine transform (rotation + non-uniform scale) T.Tuytelaars, L.V.Gool. “Wide Baseline Stereo Matching Based on Local, Affinely Invariant Regions”. BMVC 2000. Point Descriptors Affine Invariant Detection • We know how to detect points • Next question: How to describe them for matching? • Maximally Stable Extremal Regions (MSER) – Threshold image intensities: ? I > I 0 – Extract connected components (“Extremal Regions”) – Seek extremal regions that remain “Maximally Stable” under range of thresholds Point descriptor should be: 1. Invariant 2. Distinctive Matas et al. Robust Wide Baseline Stereo from Maximally Stable Extremal Regions. BMVC 2002. Rotation Invariant Descriptors Rotation Invariant Descriptors • Find local orientation • Harris corner response measure: Dominant direction of gradient depends only on the eigenvalues of the matrix M • Rotate description relative to dominant orientation 1 K.Mikolajczyk, C.Schmid. “Indexing Based on Scale Invariant Interest Points”. ICCV 2001 C.Harris, M.Stephens. “A Combined Corner and Edge Detector”. 1988 2 D.Lowe. “Distinctive Image Features from Scale-Invariant Keypoints”. Accepted to IJCV 2004