f*g=? Box filter Gaussian filter Impact of scale / width of - PDF document

CS 376 Spring 2018 - Lecture 3 1/25/2018 Reminders • Piazza for assignment help • Office hours on homepage • Reminder: no laptops, phones, tablets, etc. open Image gradients and edges in class. Thurs Jan 25, 2018 Kristen Grauman UT-Austin Review Last time Filter f = 1/9 x [ 1 1 1 1 1 1 1 1 1] • Various models for image “noise” • Linear filters and convolution useful for – Image smoothing, removing noise f*g=? • Box filter • Gaussian filter • Impact of scale / width of smoothing filter • Separable filters more efficient • Median filter: a non-linear filter, edge-preserving original image h filtered Review Image filtering Filter f = 1/9 x [ 1 1 1 1 1 1 1 1 1] T • Compute a function of the local neighborhood at each pixel in the image – Function specified by a “filter” or mask saying how to combine values from neighbors. f*g=? • Uses of filtering: – Enhance an image (denoise, resize, etc) – Extract information (texture, edges, etc) Today original image h filtered – Detect patterns (template matching) Adapted from Derek Hoiem 1

CS 376 Spring 2018 - Lecture 3 1/25/2018 What causes an edge? Edge detection • Goal : map image from 2d array of pixels to a set of curves or line segments or contours. Depth discontinuity: Reflectance change: object boundary • Why? appearance information, texture Cast shadows Figure from J. Shotton et al., PAMI 2007 Change in surface orientation: shape • Main idea : look for strong gradients, post-process Derivatives and edges Edges/gradients and invariance An edge is a place of rapid change in the image intensity function. intensity function image (along horizontal scanline) first derivative edges correspond to extrema of derivative Source: L. Lazebnik Derivatives with convolution Partial derivatives of an image For 2D function, f(x,y), the partial derivative is:     f ( x , y ) f ( x , y ) f ( x , y )  lim   x   0   f ( x , y ) f ( x , y ) For discrete data, we can approximate using finite  x  y differences:    f ( x , y ) f ( x 1 , y ) f ( x , y )   ? x 1 -1 1 -1 1 or 1 -1 To implement above as convolution, what would be the associated filter? Which shows changes with respect to x? (showing filters for correlation) 2

CS 376 Spring 2018 - Lecture 3 1/25/2018 Image gradient Assorted finite difference filters The gradient of an image: The gradient points in the direction of most rapid change in intensity The gradient direction (orientation of edge normal) is given by: >> My = fspecial(‘sobel’); >> outim = imfilter(double(im), My); The edge strength is given by the gradient magnitude >> imagesc(outim); >> colormap gray; Slide credit Steve Seitz Effects of noise Solution: smooth first Consider a single row or column of the image • Plotting intensity as a function of position gives a signal Where is the edge? Where is the edge? Look for peaks in Slide credit Steve Seitz Derivative theorem of convolution Derivative of Gaussian filters      ( I g ) h I ( g h ) Differentiation property of convolution.   0.0030 0.0133 0.0219 0.0133 0.0030   0.0133 0.0596 0.0983 0.0596 0.0133  1  1 0.0219 0.0983 0.1621 0.0983 0.0219 0.0133 0.0596 0.0983 0.0596 0.0133 0.0030 0.0133 0.0219 0.0133 0.0030 Slide credit Steve Seitz 3

CS 376 Spring 2018 - Lecture 3 1/25/2018 Derivative of Gaussian filters Laplacian of Gaussian Consider Laplacian of Gaussian y -direction x -direction operator Where is the edge? Zero-crossings of bottom graph Source: L. Lazebnik Slide credit: Steve Seitz 2D edge detection filters Smoothing with a Gaussian Recall: parameter σ is the “scale” / “width” / “spread” of the Gaussian kernel, and controls the amount of smoothing. Laplacian of Gaussian … Gaussian derivative of Gaussian • is the Laplacian operator: Slide credit: Steve Seitz Effect of σ on derivatives So, what scale to choose? It depends what we’re looking for. σ = 1 pixel σ = 3 pixels The apparent structures differ depending on Gaussian’s scale parameter. Larger values: larger scale edges detected Smaller values: finer features detected 4

CS 376 Spring 2018 - Lecture 3 1/25/2018 Mask properties Seam carving: main idea • Smoothing – Values positive – Sum to 1  constant regions same as input – Amount of smoothing proportional to mask size – Remove “high-frequency” components; “low-pass” filter • Derivatives – ___________ signs used to get high response in regions of high contrast – Sum to ___  no response in constant regions – High absolute value at points of high contrast [Shai & Avidan, SIGGRAPH 2007] Seam carving: main idea Seam carving: main idea Content-aware resizing Traditional resizing [Shai & Avidan, SIGGRAPH 2007] Real image example Seam carving: main idea Content-aware resizing Intuition: • Preserve the most “interesting” content  Prefer to remove pixels with low gradient energy • To reduce or increase size in one dimension, remove irregularly shaped “seams”  Optimal solution via dynamic programming. 5

CS 376 Spring 2018 - Lecture 3 1/25/2018 Seam carving: main idea Seam carving: algorithm s 1 s 2 s 3 s 4 s 5   Energy ( f ) Energy ( f ) Let a vertical seam s consist of h positions that • Want to remove seams where they won’t be very form an 8-connected path. noticeable: h  – Measure “energy” as gradient magnitude  Let the cost of a seam be: Cost ( s ) Energy ( f ( s )) i • Choose seam based on minimum total energy i  1  Optimal seam minimizes this cost: s * min s Cost ( s ) path across image, subject to 8-connectedness. Compute it efficiently with dynamic programming. Seam carving: algorithm How to identify the minimum cost seam? • Compute the cumulative minimum energy for all possible • First, consider a greedy approach: connected seams at each entry (i,j) :          M ( i , j ) Energy ( i , j ) min M ( i 1 , j 1 ), M ( i 1 , j ), M ( i 1 , j 1 ) 1 3 0 row i-1 j-1 j j+1 2 8 9 j row i 5 2 6 Energy matrix M matrix: (gradient magnitude) cumulative min energy (for vertical seams) • Then, min value in last row of M indicates end of the minimal connected vertical seam. • Backtrack up from there, selecting min of 3 above in M . Energy matrix (gradient magnitude) Example Example                   M ( i , j ) Energy ( i , j ) min M ( i 1 , j 1 ), M ( i 1 , j ), M ( i 1 , j 1 ) M ( i , j ) Energy ( i , j ) min M ( i 1 , j 1 ), M ( i 1 , j ), M ( i 1 , j 1 ) 1 3 0 1 3 0 1 3 0 1 3 0 2 8 9 2 8 9 3 8 9 3 8 9 5 2 6 5 2 6 8 5 14 8 5 14 Energy matrix Energy matrix M matrix M matrix (gradient magnitude) (gradient magnitude) (for vertical seams) (for vertical seams) 6

CS 376 Spring 2018 - Lecture 3 1/25/2018 Real image example Real image example Energy Map Original Image Blue = low energy Red = high energy Example results from prior classes Other notes on seam carving • Analogous procedure for horizontal seams • Can also insert seams to increase size of image in either dimension – Duplicate optimal seam, averaged with neighbors • Other energy functions may be plugged in – E.g., color-based, interactive,… • Can use combination of vertical and horizontal seams Results from Eunho Yang Conventional resize Conventional resize Original image Original image Seam carving result Seam carving result Results from Martin Becker Results from Martin Becker 7

CS 376 Spring 2018 - Lecture 3 1/25/2018 Removal of a marked object Conventional resize (399 by 599) Original image (599 by 799) Seam carving (399 by 599) Results from Jay Hennig Results from Donghyuk Shin Removal of a marked object “Failure cases” with seam carving Results from Eunho Yang By Donghyuk Shin “Failure cases” with seam carving Gradients -> edges Primary edge detection steps: 1. Smoothing: suppress noise 2. Edge enhancement: filter for contrast 3. Edge localization Determine which local maxima from filter output are actually edges vs. noise • Threshold, Thin By Suyog Jain 8

CS 376 Spring 2018 - Lecture 3 1/25/2018 Original image Thresholding • Choose a threshold value t • Set any pixels less than t to zero (off) • Set any pixels greater than or equal to t to one (on) Gradient magnitude image Thresholding gradient with a lower threshold Thresholding gradient with a higher threshold Canny edge detector • Filter image with derivative of Gaussian • Find magnitude and orientation of gradient • Non-maximum suppression : – Thin wide “ridges” down to single pixel width • Linking and thresholding ( hysteresis ): – Define two thresholds: low and high – Use the high threshold to start edge curves and the low threshold to continue them • MATLAB: edge(image, ‘canny’); • >>help edge Source: D. Lowe, L. Fei-Fei 9