Efficient Edge Estimation Instructor - Simon Lucey 16-623 - - PowerPoint PPT Presentation

Efficient Edge Estimation Instructor - Simon Lucey 16-623 - Designing Computer Vision Apps

Today • Motivation. • What is an Edge? • Oriented Filters. • Learning Efficient Edges.

Persistent versus Occluding Edges • Texture and geometric edges are “persistent” across viewpoints. • Occluding edges are “unique” to viewpoint, but potentially provide rich information about a 3D object. Taken from Ham, Singh and Lucey “Occlusions are Fleeting - Texture is Forever: Moving Past Brightness Constancy”

Persistent versus Occluding Edges (a) Rendered Glass (b) Non-persistent Edges (c) Persistent + 3D Poly Edges Taken from Ham, Singh and Lucey “Occlusions are Fleeting - Texture is Forever: Moving Past Brightness Constancy”

Taken from Ham, Singh and Lucey “Occlusions are Fleeting - Texture is Forever: Moving Past Brightness Constancy”

Today • Motivation. • What is an Edge? • Oriented Filters. • Learning Efficient Edges.

1968 Canny edge detector original image human annotator Taken from Isola et al. “Crisp Boundary Detection Using Pointwise Mutual Information”

Edges are Semantic • No mathematical definition of an edge, contour of boundary. • Classic problem in computer vision. • By definition they are semantic. • Early work focussed on oriented filters. • e.g. Canny edge detectors. • Developed by John F. Canny in 1986. “John Canny”

D.H. Hubel & T.N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1):106, 1962.

D.H. Hubel & T.N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1):106, 1962.

D.H. Hubel & T.N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1):106, 1962.

D.H. Hubel & T.N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1):106, 1962.

Olshausen & Field 1996

  . . . M × N . . . . . . X =   . . . . x 1 x 2 x N − 1 x N . .

0.25 0.8 0.7 0.2 0.6 p ( x ) 0.5 0.15 0.4 0.1 0.3 0.2 0.05 0.1 0 0 -6 -6 -4 -4 -2 -2 0 0 2 2 4 4 6 6 x n X 3.19 bits 1.41 bits H ( x ) = − p ( x n ) · log[ p ( x n )] = n =1 Olshausen & Field 1996

�� �� �� �� c) + = × ✏ X D Z Olshausen & Field 1996 Not Always Zero Always Zero H. Lee, A. Ng, et al. 2007

Visualizing CNNs

Today • Motivation. • What is an Edge? • Oriented Filters. • Learning Efficient Edges.

1D Filter ∗

1D Filter ∗

1D Filter ∗

2D Filter     1 , 0 , − 1 1 , 0 , − 1 1 , 0 , − 1 2 , 0 , − 2     ∗ 1 , 0 , − 1 1 , 0 , − 1 (Prewitt) (Sobel)

2D Filter + =

�� �� �� �� 2D Filter c)

99.6% sparse per patch

�� �� �� �� �� �� �� �� + . . . + ∗ + ∗ = c) d 1 d K ✏ z 1 z K x Not Always Zero Always Zero M. Zeiler, et al. 2010

Edges Adapted from: Elder “Are Edges Incomplete?” IJCV 1999.

�� �� Naive Approach? �� �� c) ∗ How do we recover edges?

Canny Edge Detector Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Canny Edge Detector Compute horizontal and vertical gradient images h and v Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Oriented Filters • Seems inefficient to have to search all possible orientations, + = • Instead one can express all orientations as a linear combination of x- and y- gradient filters.

Canny Edge Detector Quantize to 4 directions Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Canny Edge Detector Non-maximal suppression Adapted from: Computer vision: models, learning and inference. Simon J.D. Prince

Non-Max Suppression • Interesting, view of non-max suppression in terms of sparse coding. = × X D Z • Non-max suppression attempts to enforce that Z is sparse.

Canny Edge Detector Hysteresis Thresholding

Classic Edge Detector - Problems Texture Edge Filtering Response Simple Step Edge Filtering Response Appearance Edges Found Original Color Image by Linear Filtering (a) (b) Taken from: “Occlusion Boundaries: Low-Level Detection to High-Level Reasoning” - A. Stein (Ph.D. Thesis)

Today • Motivation. • What is an Edge? • Oriented Filters. • Learning Efficient Edges.

What defines an edge? • Color. • Brightness. • Texture • Continuity • Symmetry

Edge Estimation as a Learning Problem Taken from Martin et al. “Learning to Detect Natural Image Boundaries Using Local Brightness, Color, and Texture Cues“

Current State of the Art Sobel & Feldman Arbeláez et al. 1968 2011 (gPb) Isola et al. ez et al. Dollár & Zitnick Our method Human labelers 2013 (SE) 2014 Taken from Isola et al. “Crisp Boundary Detection Using Pointwise Mutual Information”

Current State of the Art - Speed ODS OIS AP R50 FPS Human .80 .80 - - - Canny .60 .63 .58 .75 15 Felz-Hutt [16] .61 .64 .56 .78 10 Normalized Cuts [10] .64 .68 .45 .81 - Mean Shift [9] .64 .68 .56 .79 - .62 † Hidayat-Green [23] - - - 20 .66 † BEL [13] - - - 1/10 Gb [30] .69 .72 .72 .85 1/6 .70 † 1/2 ‡ gPb + GPU [8] - - - “Requires GPU!!!” 1/30 ‡ ISCRA [42] .72 .75 .46 .89 gPb-owt-ucm [1] .73 .76 .73 .89 1/240 Sketch Tokens [31] .73 .75 .78 .91 1 1/5 ‡ DeepNet [27] .74 .76 .76 - SCG [41] .74 .76 .77 .91 1/280 SE+multi-ucm [2] .75 .78 .76 .91 1/15 SE .73 .75 .77 .90 30 SE+SH .74 .76 .79 .93 12.5 Dollar & Zitnick SE+MS .74 .76 .78 .90 6 SE+MS+SH .75 .77 .80 .93 2.5 Taken from Dollar & Zitnick “Fast Edge Detection Using Structured Forests”

Edge Detection as Classification { 0, 1 } Hard! positives Taken from Dollar & Zitnick “Fast Edge Detection Using Structured Forests”

Edges have Structure Taken from Dollar & Zitnick “Fast Edge Detection Using Structured Forests”

Structured Random Forests Taken from Dollar & Zitnick “Fast Edge Detection Using Structured Forests”

Structured Prediction Taken from Kontschieder “Structured Class-Labels in Random Forests for Semantic Image Labelling”

Structured versus Pixel Prediction pixel output ☹ structured output ☺ Taken from Dollar & Zitnick “Fast Edge Detection Using Structured Forests”

Multiscale Detection Taken from Dollar & Zitnick “Fast Edge Detection Using Structured Forests”

Taken from Dollar & Zitnick “Fast Edge Detection Using Structured Forests”

Taken from Dollar & Zitnick “Fast Edge Detection Using Structured Forests”

Taken from Dollar & Zitnick “Fast Edge Detection Using Structured Forests”

Things to try in iOS - GPUImage