Harris Corners 16-385 Computer Vision (Kris Kitani) Carnegie Mellon - PowerPoint PPT Presentation

Harris Corners 16-385 Computer Vision (Kris Kitani) Carnegie Mellon University

How do you find a corner? [Moravec 1980] Easily recognized by looking through a small window Shifting the window should give large change in intensity

Easily recognized by looking through a small window Shifting the window should give large change in intensity “edge”: 
 “flat” region: 
 “corner”: 
 no change along the edge no change in all significant change in all direction directions directions [Moravec 1980]

Design a program to detect corners (hint: use image gradients)

Finding corners (a.k.a. PCA) I y = ∂ I I x = ∂ I ∂ y ∂ x 1.Compute image gradients over small region 2.Subtract mean from each image gradient 3.Compute the covariance matrix P P 2 I x I x I x I y 3 p ∈ P p ∈ P 4 5 P P I y I x I y I y 4.Compute eigenvectors and p ∈ P p ∈ P eigenvalues 5.Use threshold on eigenvalues to detect corners

1. Compute image gradients over a small region (not just a single pixel)

1. Compute image gradients over a small region (not just a single pixel) array of x gradients I x = ∂ I ∂ x array of y gradients I y = ∂ I ∂ y

visualization of gradients image X derivative Y derivative

I y = ∂ I I y = ∂ I I y = ∂ I ∂ y ∂ y ∂ y I x = ∂ I I x = ∂ I I x = ∂ I ∂ x ∂ x ∂ x What does the distribution tell you about the region?

I y = ∂ I I y = ∂ I I y = ∂ I ∂ y ∂ y ∂ y I x = ∂ I I x = ∂ I I x = ∂ I ∂ x ∂ x ∂ x distribution reveals edge orientation and magnitude

I y = ∂ I I y = ∂ I I y = ∂ I ∂ y ∂ y ∂ y I x = ∂ I I x = ∂ I I x = ∂ I ∂ x ∂ x ∂ x How do you quantify orientation and magnitude?

2. Subtract the mean from each image gradient

2. Subtract the mean from each image gradient plot intensities constant intensity gradient intensities along the line

2. Subtract the mean from each image gradient plot intensities constant intensity gradient intensities along the line I y = ∂ I ∂ y subtract mean I x = ∂ I ∂ x plot of image gradients

2. Subtract the mean from each image gradient plot intensities constant intensity gradient intensities along the line I y = ∂ I I y = ∂ I ∂ y ∂ y subtract mean I x = ∂ I I x = ∂ I ∂ x ∂ x data is centered plot of image gradients (‘DC’ offset is removed)

3. Compute the covariance matrix

3. Compute the covariance matrix P P 2 I x I x I x I y 3 p ∈ P p ∈ P 4 5 P P I y I x I y I y p ∈ P p ∈ P I x = ∂ I I y = ∂ I ∂ x ∂ y =sum( ) *. P I x I y p ∈ P P array of x gradients array of y gradients Where does this covariance matrix come from?

Some mathematical background… Error function Change of intensity for the shift [ u,v ]: Error Window Shifted Intensity function function intensity Window function w(x,y) = or 1 in window, 0 outside Gaussian

Error function approximation Change of intensity for the shift [ u,v ]: Second-order Taylor expansion of E(u,v) about (0,0) (bilinear approximation for small shifts): first derivative second derivative

Bilinear approximation For small shifts [ u,v ] we have a ‘bilinear approximation’: Change in appearance for a shift [u,v] where M is a 2 × 2 matrix computed from image derivatives: ‘second moment’ matrix ‘structure tensor’ M

By computing the gradient covariance matrix… P P I x I x I x I y 2 3 p ∈ P p ∈ P 4 5 P P I y I x I y I y p ∈ P p ∈ P we are fitting a quadratic to the gradients over a small image region

Visualization of a quadratic The surface E ( u , v ) is locally approximated by a quadratic form

Which error surface indicates a good image feature? What kind of image patch do these surfaces represent?

flat edge corner ‘line’ ‘dot’

4. Compute eigenvalues and eigenvectors

eig(M)

4. Compute eigenvalues and eigenvectors eigenvalue M e = λ e ( M − λ I ) e = 0 eigenvector

4. Compute eigenvalues and eigenvectors eigenvalue M e = λ e ( M − λ I ) e = 0 eigenvector M − λ I 1. Compute the determinant of (returns a polynomial)

4. Compute eigenvalues and eigenvectors eigenvalue M e = λ e ( M − λ I ) e = 0 eigenvector M − λ I 1. Compute the determinant of (returns a polynomial) 2. Find the roots of polynomial (returns eigenvalues) det( M − λ I ) = 0

4. Compute eigenvalues and eigenvectors eigenvalue M e = λ e ( M − λ I ) e = 0 eigenvector M − λ I 1. Compute the determinant of (returns a polynomial) 2. Find the roots of polynomial (returns eigenvalues) det( M − λ I ) = 0 3. For each eigenvalue, solve ( M − λ I ) e = 0 (returns eigenvectors)

Visualization as an ellipse Since M is symmetric, we have We can visualize M as an ellipse with axis lengths determined by the eigenvalues and orientation determined by R direction of the fastest change Ellipse equation: direction of the slowest change ( λ max ) -1/2 ( λ min ) -1/2

interpreting eigenvalues λ 2 λ 2 >> λ 1 What kind of image patch does each region represent? λ 1 ∼ 0 λ 1 >> λ 2 λ 2 ∼ 0 λ 1

interpreting eigenvalues λ 2 corner ‘horizontal’ edge λ 2 >> λ 1 λ 1 ~ λ 2 λ 1 >> λ 2 flat ‘vertical’ edge λ 1

interpreting eigenvalues λ 2 corner ‘horizontal’ edge λ 2 >> λ 1 λ 1 ~ λ 2 λ 1 >> λ 2 flat ‘vertical’ edge λ 1

5. Use threshold on eigenvalues to detect corners

5. Use threshold on eigenvalues to detect corners λ 2 Think of a function to score ‘cornerness’ flat λ 1

5. Use threshold on eigenvalues to detect corners λ 2 strong corner Think of a function to score ‘cornerness’ flat λ 1

���������������� 5. Use threshold on eigenvalues to detect corners ^ λ 2 corner Use the smallest eigenvalue as the response function R = min( λ 1 , λ 2 ) flat λ 1

���������������� 5. Use threshold on eigenvalues to detect corners ^ λ 2 corner Eigenvalues need to be bigger than one. R = λ 1 λ 2 − κ ( λ 1 + λ 2 ) 2 Can compute this more efficiently… flat λ 1

���������������� 5. Use threshold on eigenvalues to detect corners ^ λ 2 corner R < 0 R > 0 R = det( M ) − κ trace 2 ( M ) R < 0 R ⌧ 0 flat λ 1

Harris & Stephens (1988) R = det( M ) − κ trace 2 ( M ) Kanade & Tomasi (1994) R = min( λ 1 , λ 2 ) Nobel (1998) det( M ) R = trace( M ) + �

Harris Detector C.Harris and M.Stephens. “A Combined Corner and Edge Detector.”1988. 1. Compute x and y derivatives of image x y I G I I G I = ∗ = ∗ x y σ σ 2. Compute products of derivatives at every pixel I I I I I I I I I = ⋅ = ⋅ = ⋅ x x y y xy x y 2 2 y x 3. Compute the sums of the products of derivatives at each pixel S G I S G I S G I = ∗ = ∗ = ∗ ' 2 2 ' 2 2 y σ y xy ' xy σ x x σ

Harris Detector C.Harris and M.Stephens. “A Combined Corner and Edge Detector.”1988. 4. Define the matrix at each pixel S ( x , y ) S ( x , y ) & # xy 2 x M ( x , y ) = $ ! S ( x , y ) S ( x , y ) $ ! xy 2 % " y Compute the response of the detector at each pixel 5. 2 ( ) R det M k trace M = − Threshold on value of R; compute non-max 6. suppression.

Corner response

Thresholded corner response

Non-maximal suppression

Harris corner response is rotation invariant Ellipse rotates but its shape ( eigenvalues ) remains the same Corner response R is invariant to image rotation

intensity changes Partial invariance to affine intensity change ü Only derivatives are used => invariance to intensity shift I → I + b ü Intensity scale: I → a I R R threshold x (image coordinate) x (image coordinate)

The Harris detector not invariant to changes in …