Edge detection Nicolas ROUGON ARTEMIS Department - - PowerPoint PPT Presentation

Institut Mines-Télécom

Edge detection

Nicolas ROUGON

ARTEMIS Department

Nicolas.Rougon@telecom-sudparis.eu

IMA4103

Extraction d’Information Multimédia

Institut Mines-Télécom

Overview

IMA 4103 - Nicolas ROUGON

■ Singularities in images ■ Basic image denoising ■ Edge detection

• Gradient-based approaches
• Laplacian-based approaches
• Canny-Deriche filtering

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

■ We hereafter review basic tools for detecting edges

in gray level still images

• Edges originate from fast local variations / discontinuities of

properties of the imaged 3D scene

• Since image sensors have continuous transfer functions,

the latter generate fast local variations / discontinuities of image features, referred to as image edges

• This generic terminology is therefore polysemic

► Various types of edges can be defined depending on the image feature / 3D scene property under consideration

Institut Mines-Télécom

Problem statement

IMA 4103 - Nicolas ROUGON

■ Edge formation

Image edges convey various information about 3D scene structure  depth discontinuity  surface normal discontinuity  surface texture discontinuity  reflectance discontinuity  illumination discontinuity

• Object ordering
• Object properties
• Light sources

Institut Mines-Télécom

■ Edge categorization

• Boundaries between image regions

 depth of scene objects / background

► edge-based image segmentation

• Singular lines within image regions

 shape| texture | reflection of scene objects

► object recognition

• Light artifacts

cast shadows | lighting variations | …  scene illumination

Problem statement

IMA 4103 - Nicolas ROUGON

Institut Mines-Télécom

■ Illusory edges

• No local variations of image luminance

► cannot be addressed by luminance-based approaches

• Dealing with illusory edges requires using image-extrinsic priors

► perceptual grouping techniques

Problem statement

IMA 4103 - Nicolas ROUGON

Kanizsa patterns Ehrenstein illusions

Institut Mines-Télécom

Image edges

IMA 4103 - Nicolas ROUGON

• Smooth / low-texture regions

■ Photometric characterization

Image regions divide into 2 types with specific luminance profiles

• High-texture regions

luminance line profile

Institut Mines-Télécom

Image edges

IMA 4103 - Nicolas ROUGON

• Deterministic luminance model

= smooth function L

• Continuous variations of L 

Low spatial frequencies

■ Smooth / low-texture regions

• Fast local variation /

discontinuity of L  High spatial frequency

■ Low-texture region boundary

= shape edge

Institut Mines-Télécom

Image edges

IMA 4103 - Nicolas ROUGON

• Fast local variations of L 

High spatial frequencies ► confusion with shape edges

• Statistical luminance model

= random variable L

• Spatially homogeneous statistics

■ High-texture regions

• Fast local variation /

discontinuity of statistics of L

■ High-texture region boundary

= texture edge

Institut Mines-Télécom

• Luminance is modeled as a scalar almost-everywhere (a.e.)

smooth function L over the image domain Ω  Rn (n = 2,3)

• Spatial variations of L (i.e. Cm-continuity) are then described

by image derivatives DmL ► The compact Einstein indexed notation will be used hereafter

Image edges

IMA 4103 - Nicolas ROUGON

■ We hereafter focus our attention on shape edges

• Edges correspond to discontinuities

Institut Mines-Télécom

Image singularities

IMA 4103 - Nicolas ROUGON

• D0, D1, D2

► shape edge related to object shape (silhouette / edges)

• D3

► edge junction

related to occlusions between solid objects

• D4

► edge junction

related to occlusions between transparent objects

■ Image derivative discontinuities inform on 3D scene structure



Ω

X-junction

Ω



T-junction

 Standard edge models

L

x step edge roof edge slope edge

  

Institut Mines-Télécom

Image singularities

IMA 4103 - Nicolas ROUGON

■ Image derivative discontinuities inform on 3D scene structure

D1, D2 D3 D4

Institut Mines-Télécom

Image singularities

IMA 4103 - Nicolas ROUGON

• Multichannel images

► color edge

 vector image derivatives  scalar image component derivatives may also be meaningful e.g. hue edge

• Image sequences

► motion edge

 spatio-temporal image derivatives

• 3D objects

► crest line / valley

 manifold derivatives

■ The connection between edges and derivative discontinuities

holds for all visual contents

Institut Mines-Télécom

Image geometry

IMA 4103 - Nicolas ROUGON

■ Image derivatives allows for describing local image geometry

• Image structure is encoded in its topographic map

level set

► Image geometry can be described via its level lines

• Differential geometry provides a toolbox

for characterizing their local geometry

• rientation | curvature | …

Institut Mines-Télécom

Image geometry

IMA 4103 - Nicolas ROUGON

■ Image derivatives allows for describing local image geometry

Assume a parameterized (= explicit) representation of level lines

• Orientation

► unit Frenet frame (t, n)

− tangent − normal

 Using chain rule for differentiation

n t

Institut Mines-Télécom

Image geometry

IMA 4103 - Nicolas ROUGON

■ Image derivatives allows for describing local image geometry

 Level line curvature

Expanded as a function of 1st- & 2nd-order

image derivatives

• Curvature

► rate of orientation variation of Frenet frame for unit-speed tangent motion n

Institut Mines-Télécom

Image geometry

IMA 4103 - Nicolas ROUGON

■ Image derivatives allows for describing local image geometry

• A gray level image can also be viewed as a surface in Rn+1 (n = 2,3)

(= graph of the luminance function)

• The luminance surface is natively parameterized

by pixel coordinates x = (xi) (i = 1...n) (= Monge patch)

• Differential geometry provides a toolbox

for characterizing its local geometry

• rientation | curvature | …

Institut Mines-Télécom

Image geometry

IMA 4103 - Nicolas ROUGON

■ Image derivatives allows for describing local image geometry

• Orientation

► unit Frenet frame (t1,…,tn, n)

− tangent space − normal

t1 t2 n

• Curvature

► rates of orientation variation tensor

• f Frenet frame for

unit-speed tangent motion

> Functions of 1st-order image derivatives > Function of 1st- & 2nd-order image derivatives

Institut Mines-Télécom

Differentiating noisy signals

IMA 4103 - Nicolas ROUGON

• Example: additive noise model

unnoisy (unobserved) image L0 | i.i.d. noise n

■ Images are noisy

► Polynomial noise amplification  increasing with differentiation order  degrading the high frequencies of DpL

Institut Mines-Télécom

Differentiating noisy signals

IMA 4103 - Nicolas ROUGON

■ Images are noisy

Differentiation is (highly) sensitive to noise ► mathematically ill-posed problem

• Geometric image feature extraction is a hard problem
• To ensure robust results

 Differentiation order must be kept as low as possible  Low-pass filtering (i.e. smoothing) must be performed prior to or combined with differentiation ► Based on SNR, a trade-off between smoothing ( robustness)

• vs. discontinuity-preservation ( accuracy) must be set

Institut Mines-Télécom

Basic image denoising

IMA 4103 - Nicolas ROUGON

Institut Mines-Télécom

Linear smoothing filters

IMA 4103 - Nicolas ROUGON

■ Linear filters

• Continuous case
• Discrete case

 kernel integer-sampling over bounded support (usually a square window) ≡ convolution matrix > weighted sum over pixel neighborhood

shift at x0 shift at xij = (i,j) kernel

Institut Mines-Télécom

Linear smoothing filters

IMA 4103 - Nicolas ROUGON

■ Linear filters

• Convolution theorem
• Spectral implementation results in truncation error-free schemes

 direct / inverse image Fourier transform is computed efficiently using nD Fast Fourier Transform (FFT)

► O( ) spatial integration translates into O(1) pointwise product in Fourier space

− quasilinear

O(NlogN) complexity (N = #pixels)

− separable

Institut Mines-Télécom

Linear smoothing filters

IMA 4103 - Nicolas ROUGON

■ Smoothing kernels

• Admissibility conditions

− positive − symmetric − unimodal − unit mass − equidistributed

 continuous case  discrete case

Institut Mines-Télécom

Mean filter

■ Normalized unit kernel

• Strongly-smoothing and isotropic

► non edge-preserving  loss of contrast / sharpness  delocalization These artifacts increase with kernel extension 1 1 1 1 1 1 1 1 1

9 1

M3 =

IMA 4103 - Nicolas ROUGON

(3x3) mean filter

Institut Mines-Télécom

Mean filter

IMA 4103 - Nicolas ROUGON

• riginal

mean 3x3 mean 5x5

Institut Mines-Télécom

Gaussian filter

IMA 4103 - Nicolas ROUGON

• Separable

► implemented as (tensor) product of 1D kernels along grid axes

• Strongly-smoothing and isotropic

► non edge-preserving  loss of contrast / sharpness  delocalization These artifacts increase with kernel extension ( variance)

■ Continuous Gaussian kernel

Institut Mines-Télécom

Gaussian filter

IMA 4103 - Nicolas ROUGON

• Sampled Gaussian kernel over a [-M, M] window

Usual choice: M = Cσ + 1 with C  [3,6] ► significant errors for large windows

• Discrete Gaussian kernel

Solution of the discrete heat equation ► Scale-space theory

• Recursive Gaussian filters

Optimal Infinite Impulse Response (IIR) approximations of Gσ with given orders ► Canny filtering

Deriche | Young | Triggs | Farnebäck

■ Discrete approximations of the Gaussian kernel

> Real-coefficient filters

Institut Mines-Télécom

Gaussian filter

IMA 4103 - Nicolas ROUGON

• Binomial filters

■ Discrete approximations of the Gaussian kernel

> Integer-coefficient filters

n

2 1 Bn = 1 1

n

• Optimal integer approximations over fixed-size windows

4 1 B2 = 2 1 1

 M = 1  M = 2 (a  0.4)

3a 4 1  a 1 1 1-2a 1-2a

Institut Mines-Télécom

Gaussian filter

IMA 4103 - Nicolas ROUGON

• riginal

Gaussian 3x3 | σ=1 Gaussian 5x5 | σ=1

Institut Mines-Télécom

k

Median filter

■ Order-statistics filtering

Denote the kth-smallest pixel value in the neighborhood

• f x  Ω
• Rank filter of rank k

erosion dilation median filter

1

minimum maximum median



• dd

► Erosion/dilation provide the basic operators of mathematical

morphology (a.k.a. image algebra)

IMA 4103 - Nicolas ROUGON

Institut Mines-Télécom

Median filter

IMA 4103 - Nicolas ROUGON

■ Performances

• Efficient denoising

 optimal for moderate impulse noise («salt-and-pepper» noise) # noisy pixels < 20%  edge preservation + contrast enhancement

• Fine details are smoothed

indistinguishable from noise

• Computational bottleneck = local pixel sorting

► quasilinear sorting algorithms with O( ) complexity Heapsort | Block sort | …

Institut Mines-Télécom

Median filter

IMA 4103 - Nicolas ROUGON

• riginal

median 3x3 median 5x5

Institut Mines-Télécom

• Median filtering is restricted to pixels corrupted by impulse noise.

Other pixels are unfiltered > detail preservation + filtering bias reduction

• A local impulse noise test based on order-statistics in pixel

neighborhood is first applied A candidate noisy pixel

 differs from most of its neighbors  can be distinguished from similar neighboring pixels

• Adaptation to image local scale is performed by letting

neighborhood size vary within a predefined range

Adaptive median filter

IMA 4103 - Nicolas ROUGON

■ Key ideas

Institut Mines-Télécom

• Step #1 - Impulse noise test

for (r =1; r  rmax ; r++) {

// grow until noise is detectable

if ( < < )

// most neighbors differ from noise goto Step #2; } // > proceed with filtering return L(x) // noise is undetectable > no filtering

• Step #2 - Denoising

if ( < L(x) < )

// pixel differs from impulse noise return L(x) // > no filtering else return // else median filtering

Adaptive median filter

IMA 4103 - Nicolas ROUGON

■ Algorithm

Denote | | the minimum | median | maximum pixel value in the r-size neighborhood of x  Ω ( 1  r  rmax )

Institut Mines-Télécom

Adaptive median filter

IMA 4103 - Nicolas ROUGON

■ Performances

• Improved denoising for impulse noise > 20%
• Improved edge contrast enhancement
• Fine detail preservation
• riginal

median 3x3 adaptive median

Institut Mines-Télécom

• Iterative filters generating increasingly smoothed images

defined as solutions of nonlinear diffusion PDEs anisotropic diffusion | geometric heat equations | …

Nonlinear smoothing filters

IMA 4103 - Nicolas ROUGON

■ PDE-based filters

• Algebraic filters with monotonicity + idempotence properties

derived by combining morphological erosion and dilation

• pening | closing | alternating sequential filters | …

■ Morphological filters

• Weighted mean filters with weights depending on the similarity

between patches around current and any other pixels non-local (NL) means | …

■ Patch-based filters

Institut Mines-Télécom

Linear vs. nonlinear smoothing

IMA 4103 - Nicolas ROUGON

• riginal

median 3x3 Gaussian 3x3 | σ=1 mean 3x3

• Nonlinear filters allow

for performing distinct intra- / inter-region smoothing ► reduced bias ► better discontinuity preservation + potential contrast enhancement > sharper details

Institut Mines-Télécom

Edge detection methods

IMA 4103 - Nicolas ROUGON

■ Detecting edges from low-order image derivatives

D0L = L D1L = L D2L = Hessian(L) gradient-based Laplacian-based impractical for discrete signals |L| L = Trace(D2L) local maximum zero-crossing edge map

edge criterion

image derivative class of methods

− −

L

x step edge roof edge slope edge

  

companion edge models

     

■ ■ ■ ■ ■ ■

Institut Mines-Télécom

Gradient-based methods

IMA 4103 - Nicolas ROUGON

Institut Mines-Télécom

Gradient-based edge detection

IMA 4103 - Nicolas ROUGON

• Amplitude = contrast

■ Luminance gradient

• Phase = orientation

► level line normal ► level line density along normal n L n

Institut Mines-Télécom

Gradient-based edge detection

IMA 4103 - Nicolas ROUGON

■ Directional derivative

 Consistency with Cartesian derivatives  Consistency with curvilinear derivatives Viewing d as the tangent vector along a curve x(s) with arclength s

• Assessing luminance variations in arbitrary directions requires

to define directional differential operators

• Directional derivative (or Lie derivative) along a unit vector d  Rn

n d

Institut Mines-Télécom

Gradient-based edge detection

IMA 4103 - Nicolas ROUGON

■ Directional derivative

• Contrast = luminance variation along level line normal

n d

Institut Mines-Télécom

Gradient-based edge detection

IMA 4103 - Nicolas ROUGON

n d

An edge point x is a local directional maximum of contrast The direction d defines the edge normal

■ Edge point

An edge point x is a local directional maximum of contrast in some direction d(x) which locally defines the edge normal

• Detecting edge points requires solving for 2 unknowns

i.e. location x and orientation d ► under-constrained problem

• Local edge properties consist of

− location x − orientation d − directional contrast dL

geometry photometry

Institut Mines-Télécom

Gradient-based edge detection

IMA 4103 - Nicolas ROUGON

n d

■ Gradient-based edge detection schemes

2 approaches

• Search for regular local maxima of |L|
• Approximate locally edges as level lines
• Generate edge orientation hypotheses
• Test for local directional maximum
• f |L| in candidate directions

► Discrete estimators

• f dL

► Discrete estimators

• f L

Institut Mines-Télécom

Gradient-based edge detection

IMA 4103 - Nicolas ROUGON

Preprocessing Edge detection Postprocessing

• denoising
• enhancement
• …
• artifacts removal
• thinning
• linking

Edge map estimation |L| or |d L| Detection upper- threshold

►

► ►

• Preprocessing can be built-in into edge detection

> robust edge detectors

• Hyperparameter: contrast threshold

► critical impact ► A trade-off between saliency ( robustness) ( high value)

• vs. level of detail ( sensitivity) ( low value) must be set

based on noise/ scene texture/ lighting conditions

Institut Mines-Télécom

Contrast map

IMA 4103 - Nicolas ROUGON

■ Properties

|Lx| |Ly| |L|

• Lx  vertical + diagonal edges
• Ly  horizontal + diagonal edges
• These maps complement/ reinforce

when combined into the contrast map |L| L

Sobel filter

Institut Mines-Télécom

Contrast map

IMA 4103 - Nicolas ROUGON

■ Gradient norms |L|

L2 L1 L hybrid

unit ball

8-connectivity 4-connectivity

• Isotropy

L L1 L2

hybrid

L L1

R2

• Computational cost

L2

hybrid

natural topology

Institut Mines-Télécom

FD gradient filters

IMA 4103 - Nicolas ROUGON

■ Finite differences

• Finite difference (FD) techniques allow for estimating an arbitrary
• rder-derivative of a function as a linear combination of its values

at given neighboring points ► A derivative is expressed as a discrete convolution against a kernel ( = differential filter )  Setting the kernel size results from on a trade-off between accuracy ( small) vs. robustness ( large) based on SNR and application-dependent computational constraints  FPGA / GPU implementations for real-time applications

Institut Mines-Télécom

FD gradient filters

IMA 4103 - Nicolas ROUGON

■ Finite differences

• FD estimates are derived analytically from Taylor expansions,
• r geometrically from linear fits of the function graph

x

x+dx x-dx

L(x)

 A basic instance for 1st-order derivatives consists in approximating tangents by chords

Institut Mines-Télécom

FD gradient filters

IMA 4103 - Nicolas ROUGON

■ 1st-order FD over 2D grids

kernel expression centered left-sided right-sided FD type

• 1

1 1

• 1

1

• 1

dx dy

• Usual choice: unit pixel size (dx = dy = 1)

Institut Mines-Télécom

FD gradient filters

IMA 4103 - Nicolas ROUGON

■ Roberts filter

• Lateral FD kernels respond at different (subpixel) locations

1

• 1 0

Dx = Dy =

• 1

1 1

• 1
• 1

1  

• 45°-rotation fuses their zero-crossings, yielding the Roberts filter

 small size kernels ► noise/texture-sensitive  tailored to diagonal edges ► directional bias  subpixel edge point location ► interpolation ► location inconsistency in |L|

► ►

Institut Mines-Télécom

FD gradient filters

IMA 4103 - Nicolas ROUGON

■ Robust gradient filters

• Gradient filters with enhanced noise robustness properties

are derived by combining differentiation along a direction with smoothing in the orthogonal direction

• These separable filters are designed as tensor products of

1D differentiation () and smoothing (S) kernels

x x

y y x

S S D    

T T

y x x y y

S S D    

T T

► Discrete (1xn) differentiation / smoothing kernels yield (nxn) gradient kernels

Institut Mines-Télécom

FD gradient filters

IMA 4103 - Nicolas ROUGON

■ Prewitt filter

1 1 1

• 1
• 1
• 1

3 1

Dx = 1

• 1

1 1

• 1 -1

3 1

Dy =

unit kernel

1 1 1 Sx =

• Combining 1st-order centered FD gradient with mean filtering

yields the Prewitt filter

3 1

• 1 0

1

x 

Institut Mines-Télécom

FD gradient filters

IMA 4103 - Nicolas ROUGON

■ Sobel filter

• FD gradient filters with smoother spectral responses are derived

by using smoothing kernels with stronger continuity properties

2 1 1

• 1
• 2
• 1

4 1

Dx = 1

• 1

2 1

• 2 -1

4 1

Dy =

binomial kernel

1 1 1 Sx = 1 2 1

►

• Switching from mean to Gaussian filtering

yields the Sobel filter

4 1 3 1

Institut Mines-Télécom

FD gradient filters

IMA 4103 - Nicolas ROUGON

■ Frei-Chen filter

• Integer-coefficient FD gradient filters have directional bias
• Gradient filters with enhanced isotropy properties are derived

by setting the Euclidean metric over the image grid

L  1 1 L2 1

2

►

 Roberts | Sobel  Prewitt

• Reweighting the Sobel filter accordingly yields the Frei-Chen filter

 real-coefficient filter ► higher computational cost / memory usage

Institut Mines-Télécom

FD gradient filters

IMA 4103 - Nicolas ROUGON

■ Frei-Chen filter

2 2 1

D0° =

2 2 1

D90° = 1 1

• 1
• 1

2 2

1

• 1

1

• 1
• 2

2

2 2 1

D45° =

2 2 1

D135° = 1 1

• 1
• 1

2

1

• 1 0
• 1

2

1

2

• 2
• Formally, the Frei Chen filter arises from a decomposition of

(3x3) image patches into smooth (1D) + edge (4D) + line (4D)

• components. It is derived as a basis of the edge subspace

Institut Mines-Télécom

FD gradient filters

IMA 4103 - Nicolas ROUGON

■ Frei-Chen filter

• The companion edge map is defined as the fraction of the patch

belonging to the edge subspace using the Frobenius norm ( ) over the patch space





2 2

a A

ij

• Standard threshold values ≈ 95%

Institut Mines-Télécom

Template matching

IMA 4103 - Nicolas ROUGON

• Orientation sampling

► candidate directions di

• FD approximation of

► discrete templates Di

• Selection rule

■ Principle

Joint estimation of local image orientation d and contrast dL via hypothesis testing

i

d



► local estimates

− orientation

d ≈ di*

− contrast

dL ≈ |Di*L| ► Computational cost increasing with neighborhood size and angular resolution

Institut Mines-Télécom

Template matching

IMA 4103 - Nicolas ROUGON

■ Standard kernel bases

• Size: 3x3
• Angular resolution: 45°
• Directional kernels Di x 45° (i  [1..7]) are derived from D0°

via circular permutations of peripheral coefficients

90° 45° 135° 0° 180° 225° 270° 315°

Institut Mines-Télécom

Template matching

IMA 4103 - Nicolas ROUGON

■ Standard kernel bases

• Robinson

2 1 1

• 1
• 2
• 1

4 1

D0° = 1 1 1

• 1
• 1
• 1
• 2

15 1

D0° = 1 1

• Prewitt compass

5 5 5

• 3
• 3
• 3

15 1

D0° =

• 3
• 3
• Kirsch
• Many other bases

− larger kernels − finer angular resolution

Nevatia-Babu | Zucker-Hummel Morgenthaler| …

Institut Mines-Télécom

FD gradient filters

IMA 4103 - Nicolas ROUGON

■ Performances

Roberts Kirsch Sobel Prewitt Roberts Kirsch Sobel Prewitt Roberts Kirsch Sobel Prewitt

kernel size (bias )

 

(variance)



kernel type Accuracy Robustness

/ noise, texture

Complexity trade-off criterion

Institut Mines-Télécom

Post-processing

IMA 4103 - Nicolas ROUGON

■ Improving gradient-based edge map

3 steps with specific goals

Hysteresis Thresholding Non-Maximum Suppression Linking

ENMS EHT Eraw E

► ► ► ►

• Reconnect distant

edge fragments

• Reconnect close

edge fragments (= edgels)

• Remove artifacts

 noise points  thick edges

Institut Mines-Télécom

Post-processing

IMA 4103 - Nicolas ROUGON

■ Step #1: Non-maximum suppression

> Removing edge artifacts

• Noise does not exhibit directional consistency
• Thick structures violate the local maximum contrast property
• Hence the idea of filtering both by checking that the definition
• f an edge point holds

Institut Mines-Télécom

Post-processing

IMA 4103 - Nicolas ROUGON

■ Step #1: Non-maximum suppression

Algorithm For all pixel x in the raw edge map Eraw

• Interpolate |L| along L at neighboring

points in the 8-connected neighborhood of x

• Test if |L(x)| is a local maximum along L

If not, remove x from Eraw L(xi,j)

xi,j

► Simultaneous edge thinning

Institut Mines-Télécom

Interpolation

IMA 4103 - Nicolas ROUGON

■ 1D interpolation

x i +1 i

L

u

• 0th-order: nearest-neighbor (NN)

> piecewise-constant interpolant

• 1st-order: linear

> piecewise-linear interpolant

• Higher-order: polynomial

spline | B-spline* | … > piecewise-smooth interpolant

* continuous stitching properties

Institut Mines-Télécom

Interpolation

IMA 4103 - Nicolas ROUGON

■ 2D interpolation

• 0th-order: nearest-neighbor (NN)

> piecewise-constant interpolant

• 1st-order: bilinear

> piecewise-linear interpolant

• Higher-order: polynomial

spline | B-spline | …

x xi,j xi+1,j+1 xi,j+1 xi+1,j

u v

Institut Mines-Télécom

Post-processing

IMA 4103 - Nicolas ROUGON

■ Step #2: Hysteresis thresholding

> Reconnecting close edgels

• Short gaps between edgels in ENMS often correspond to

weakly contrasted edge parts which are not detected due to a too high contrast threshold

• Lowering the threshold can lead to include irrelevant edges

and spurious noise / texture pixels in the edge map

• The latter are usually not connected to shape edges.

Hence the idea of filtering them using a connectivity constraint

Institut Mines-Télécom

Post-processing

IMA 4103 - Nicolas ROUGON

■ Step #2: Hysteresis thresholding

Algorithm Given λhigh > λ low > 0, add to the edge map ENMS any pixel x s.t.

• |L(x)| is a local directional maximum
• |L(x)| ≥ λ low
• x is connected to some y  ENMS s.t. |L(y)| ≥ λhigh
• (λhigh ,λlow) are set empirically or derived from a noise estimate
• Typically:

λ high = kλlow where k [2,3]

• Very efficient in practice

− 15-20 pixel gaps are filled − improved noise robustness

compared to direct threshold

Institut Mines-Télécom

Post-processing

IMA 4103 - Nicolas ROUGON

■ Step #3: Linking

Key idea: from segment endpoint(s), propagate a front over Ω and move along its normal n until reconnection

• Front lines are generated as the level sets of some connection

cost function V to edges defined over Ω | minimal over edges ►

• Shortest connecting path

 starting from endpoint, iterate until reconnection ► requires interpolating V over Rn  unidirectional or bidirectional scheme n

Institut Mines-Télécom

Post-processing

IMA 4103 - Nicolas ROUGON

■ Step #3: Linking

• Connection criteria

cost function V gradient-based morphological distance-based scheme minimal path methods closing with  structuring element size digital distance to binary edge map E

− V1 image cost − V2 edge cost − α

smoothness term

Institut Mines-Télécom

Laplacian-based methods

IMA 4103 - Nicolas ROUGON

Institut Mines-Télécom

Laplacian-based edge detection

IMA 4103 - Nicolas ROUGON

• Isotropic operator

► orientation-free

■ Luminance Laplacian

• 2nd-order operator

► contrast-free ► noise sensitive

• Laplacian zero-crossings (ZC) comprise

 local directional maxima of |L| ► shape edges  local minima of |L|  constant luminance regions ► non generic in natural (= plateau) (noisy) images

Institut Mines-Télécom

Laplacian-based edge detection

■ Laplacian edge map

• Since  is continuous, its ZC consist of closed* curves/surfaces

i.e. geometric sets (as opposed to point sets)

* except along image boundaries

• Salient edge points are filtered using a local contrast criterion

 statistical based on luminance local variance in some neighborhood of x  differential ► requires computing L R2 R1

x

IMA 4103 - Nicolas ROUGON

Institut Mines-Télécom

Laplacian-based edge detection

IMA 4103 - Nicolas ROUGON

Preprocessing Edge detection Postprocessing

• denoising
• enhancement
• …
• artifacts removal
• thinning
• linking

Edge map estimation L Detection zero- crossings

►

► ►

• ZC detection is based on sign change (> parameter-free)

 the 8-connected neighborhood is used (> the Jordan curve theorem holds)  ZC location is computed at subpixel scale via interpolation

• Since edges are defined as level lines, postprocessing is simplified

for they are ensured to be thin and connected

Institut Mines-Télécom

FD Laplacian filters

IMA 4103 - Nicolas ROUGON

■ 2nd-order FD over 2D grids

2nd-order FD estimates are derived by composing 1st-order FD estimates kernel expression centered FD type

• 2

1 1

dx dy

• Usual choice: unit pixel size (dx = dy = 1)

Institut Mines-Télécom

FD Laplacian filters

IMA 4103 - Nicolas ROUGON

■ Standard 2D Laplacian kernels

• 4-connected

1 1

• 4

 = 1 1 1 1 1 1 1 1

• 8

 = 1 1

• 8-connected

2

• 1
• 1
• 1

2

• 1
• 4

 = 2 2

 only separable (3x3) FD Laplacian kernel  many other kernels

Institut Mines-Télécom

FD Laplacian filters

IMA 4103 - Nicolas ROUGON

■ Standard 2D Laplacian kernels

• Small-size Laplacian kernels are noise-sensitive

► not suited to edge detection in natural images

 =

1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

• 48

1 1 1 1 1 1 1

• Estimating a reliable Laplacian

map requires large kernels > from (7x7) to (11x11) ► computational load boundary conditions

• Many kernels are available

in the literature

Institut Mines-Télécom

σ

Kσ

Robust Laplacian filters

IMA 4103 - Nicolas ROUGON

■ Regularized Laplacian filters

Key idea: Improve Laplacian filter robustness w.r.t. noise by performing low-pass filtering prior to differentiation

• Linear low-pass filter
• Convolution theorem

 ► the Laplacian map is smoothed ► not operative since L is noisy  ► the Laplacian operator is smoothed ► regularized Laplacian operator

Institut Mines-Télécom

Robust Laplacian filters

IMA 4103 - Nicolas ROUGON

■ Regularized Laplacian filters

• The kernel defines a band-pass filter
• As for FD Laplacian filters, implementation in the spatial domain

requires large discrete kernels (e.g. obtained by sampling ) ► computational load | boundary conditions ► applicable only when σ is small

• Implementation in the spectral domain using FFT + precomputed

Laplacian of kernel spectrum ► quasilinear complexity | truncation error-free

Institut Mines-Télécom

Robust Laplacian filters

IMA 4103 - Nicolas ROUGON

■ Laplacian of Gaussian (LoG) filter

• Gσ
• Choosing Kσ as a centered Gaussian kernel Gσ with variance σ2

yields the Laplacian of Gaussian (LoG) filter (a.k.a. Marr-Hildreth or Mexican Hat filter) ► The hyperparameter σ is set based on noise/texture properties

Institut Mines-Télécom

Robust Laplacian filters

IMA 4103 - Nicolas ROUGON

■ Laplacian of Gaussian (LoG) filter

• Strongly-smoothing

σ > 0 (Gσ*L)  C(Ω)

► well-posed differentiation: arbitrary-order derivatives are estimated in a robust way (= Gaussian derivatives)

• Separable

► computational efficiency

• Well-localized in both space and frequency

TF (Gσ)  G1/σ

► good trade-off between accuracy vs. smoothing

• Delocalization artifacts ( with σ)

Institut Mines-Télécom

Robust Laplacian filters

IMA 4103 - Nicolas ROUGON

■ LoG kernels

σ =

2 2 2 3 3 5 5 5 3 3 2 2 2 5 5 5 5 3 2 2 5 3 5 2 5 2 3 3 5 2 5 2 3 5 3 3 2 3 3 3 3 5 2 3 3 5 3 3 2 3 3 3 5 3 2 3

• 12
• 40
• 23
• 12
• 12
• 12
• 23
• 23
• 23
• For small values of σ,

LoG kernels are readily derived by sampling ► computational load boundary conditions

Institut Mines-Télécom

L

Laplacian map

IMA 4103 - Nicolas ROUGON

■ FD Laplacian vs. LoG

4-connected Laplacian LoG (σ=1.2)

|L | |L |

Institut Mines-Télécom

Robust Laplacian filters

IMA 4103 - Nicolas ROUGON

■ Regularized Laplacian filters

• The kernel width σ acts as a scale parameter which allows for

hierarchising image structure in terms of level of detail (LoD)

σ high LoD fine edge detail low LoD most salient edges full LoD

Institut Mines-Télécom

Robust Laplacian filters

IMA 4103 - Nicolas ROUGON

■ Difference of Gaussians (DoG) filter

Given σ1 > σ2 > 0, the DoG filter is the linear filter with kernel

• accurately approximates the LoG kernel whenever
• For a given accuracy, the DoG kernel bandwidth is slightly larger

than the LoG kernel bandwidth

• Biological consistency

Retinal cell assemblies of mammals behave as DoG filter banks

Institut Mines-Télécom

Image sharpening

IMA 4103 - Nicolas ROUGON

■ Image sharpening

• Blurring may occur during image acquisition (e.g. defocusing),

scanning or scaling ► low-pass filtering mostly noticeable along edges

• Image sharpening refers to techniques aiming at enhancing

luminance transitions ► performed by amplifying high-frequencies ► reduces effects of blurring

• 2 main approaches to image sharpening

 Laplacian-based  Unsharp masking

Institut Mines-Télécom

Image sharpening

IMA 4103 - Nicolas ROUGON

■ Laplacian-based image sharpening

Key idea: Use the Laplacian both to

− detect edges (ZC) − distinguish edge sides (sign)

L Lx Lxx ► a Laplacian kernel  induces a sharpening kernel

• Edge enhancement is achieved by

subtracting from the image a fraction

• f its Laplacian

Institut Mines-Télécom

Image sharpening

IMA 4103 - Nicolas ROUGON

■ Standard 2D Laplacian-based sharpening kernels (b = 1)

• 4-connected
• 1
• 1

5 KS =

• 1
• 1
• 1
• 1
• 1
• 1
• 1
• 1

9 KS =

• 1
• 1
• 8-connected
• 2

1 1 1

• 2

1 5 KS =

• 2
• 2
• Robust sharpening kernels are build from LoG / DoG kernels

Institut Mines-Télécom

Image sharpening

IMA 4103 - Nicolas ROUGON

■ Laplacian-based image sharpening

• riginal

4-connected Laplacian-based

Institut Mines-Télécom

Image sharpening

IMA 4103 - Nicolas ROUGON

■ Laplacian-based image sharpening

• riginal

4-connected Laplacian-based

Institut Mines-Télécom

Image sharpening

IMA 4103 - Nicolas ROUGON

■ Laplacian-based image sharpening

4-connected Laplacian-based 8-connected Laplacian-based

Institut Mines-Télécom

Image sharpening

IMA 4103 - Nicolas ROUGON

■ Unsharp masking

Key idea: Build a detail mask by subtracting from the image a smoothed version of itself L K*L L – K*L ► a smoothing kernel K induces a sharpening kernel

• Edge enhancement is achieved by

adding to the image a fraction

• f the detail mask

► called highboost filtering if b > 1

Institut Mines-Télécom

Image sharpening

IMA 4103 - Nicolas ROUGON

■ Unsharp masking

• riginal

highboost filtering

Gaussian kernel (σ = 2.0) | b = 1.5

Institut Mines-Télécom

Image sharpening

IMA 4103 - Nicolas ROUGON

■ Laplacian-based sharpening vs. Unsharp masking

• Unsharp masking relies on smoothing and does not involve

differentiation ► intrinsically more robust to noise than Laplacian-based methods (especially when using small-size Laplacian kernels) ► the smoothing kernel bandwidth provides an additional scale hyperparameter allowing for finer performance control

• Using robust Laplacian kernels, Laplacian-based techniques tend

to perform similarly to unsharp masking

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Key ideas

Canny-Deriche filtering provides a class of linear differential filters with arbitrary-order

• Optimized for a noisy edge model
• Based on quantitative performance criteria

► mathematical derivation

• Recursive implementation

► computational efficiency

• J. Canny - A computational approach to edge detection

IEEE Transactions on Pattern Analysis and Machine Intelligence 8(6):679-698, Nov. 1986

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ 1D edge model

Noisy step edge (location x0 | contrast ρ) with additive white Gaussian noise

• Detection using a linear filter with kernel K

► Optimal kernel? x0 ρ

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Performance criteria

Expressed as functions of kernel K and its derivatives

• Good detection

► high specificity

• Good localization

► high accuracy

• Single response at edge points

► no ambiguity

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Optimal detection

Low probability of false alarms (= no detection | wrong detection)

• SNR criterion

► to be maximized

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Optimal localization

Detected edge points must be close to ground truth ► to be maximized

• Localization criterion

Inverse standard deviation of expected edge location

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Single detection

Filter response at edge points must be unique

• Response criterion

Mean distance between zero-crossings of filter response ► to be minimized

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Performance optimization

The 3 elementary performance criteria are combined into a constrained optimization problem over the kernel space

• Maximization of (SL) under the constraint that ξ is minimum
• A closed-form expression for the optimal kernel K is derived

using variational optimization techniques

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Optimal FIR filter

• Accurately approximated by the Gaussian 1st derivative kernel

 Performance: SL = 0.92 ► 6 hyperparameters

• No recursive implementation

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Optimal IIR filter

• Performance assessment

2 2

2 ω α α 

2 2 2 2

5 ω α ω α  

2 2

2 ω α α 

S SL ξ

α 2

L

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Optimal IIR filter

• Performance assessment assuming a = mw

1 5 1

2 2

  m m

SL ξ

1 2

2 

m m

0.44 2

► optimal kernel is obtained letting m  

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Optimal 1D derivation filter

One-parameter exponential kernel (= 1st-order Canny-Deriche filter)

• Exact solution of the performance optimization problem
• IIR filters can be implemented recursively

► accurate implementation via a 2nd-order recursive scheme b1 = -2e-α b2 = e-2α c = (1 - e-α)2

Dx

x y

− causal scan − anti-causal scan − output synthesis

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Optimal 1D arbitrary-order derivation filters

• Integrating Dx yields an optimal 1D smoothing kernel

(= Canny-Deriche smoothing filter)

• Optimal 1D higher-order derivation kernels are obtained by

differentiating Dx (= nth-order Canny-Deriche filters)  2nd-order Canny-Deriche kernel

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Optimal 1D arbitrary-order derivation filters

All these filters can be accurately implemented via 2nd-order recursive schemes x y

− causal scan − anti-causal scan − output synthesis

with filter coefficients (ai, bi, c) depending on α

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Canny-Deriche image filters

Optimal nD differential filters of any order are built by tensor product

• f 1D Canny-Deriche kernels. Robustness is achieved by combining*
• differentiation along target direction(s)
• smoothing along orthogonal direction(s)

2D filter derivative 3D filter

Lx Dx Sy L Dx Sy Sz L Ly Sx Dy L Dx Sy Sz L Lxx Dxx Sy L Dxx Sy Sz L Lxy Dx Dy L Dx Dy Sz L

* commutativity holds

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Canny-Deriche image filters

nD Canny-Deriche filters are implemented in a separable way using 2-pass recursive schemes along image coordinate axes

L

   

► ► ► ► ► ► apply Dx along lines

     

apply Sy along columns

Lx

• Example: estimating 2D image derivative Lx

Institut Mines-Télécom

Canny-Deriche filtering

IMA 4103 - Nicolas ROUGON

■ Canny-Deriche image filters

α low LoD most salient edges high LoD fine edge detail full LoD 0.25 0.5 0.75 1.0

• The hyperparameter α > 0 allows for controlling smoothing / LoD

based on SNR ► LoD increases with α

Institut Mines-Télécom

Shen-Castan filtering

IMA 4103 - Nicolas ROUGON

■ Quasi-optimal 1D derivation filter

One-parameter exponential kernel (= 1st-order Shen-Castan filter)

• Quasi-optimal 1D smoothing/higher-order derivation kernels

are obtained by integration/differentiation

• IIR filters

► accurate implementation via 2nd-order recursive schemes

• The hyperparameter α > 0 controls smoothing / LoD

► LoD increases with α

• Performance equivalent to Canny-Deriche filtering

Institut Mines-Télécom

Edge detection

Nicolas ROUGON

ARTEMIS Department

Nicolas.Rougon@telecom-sudparis.eu

IMA4103

Extraction d’Information Multimédia