Feature Extraction Manos Baltsavias, Haris Papasaika OVERVIEW - - PowerPoint PPT Presentation

Feature Extraction

Manos Baltsavias, Haris Papasaika

OVERVIEW

APPLICATIONS

• Object detection and localisation (2D measurement)
• Matching (3D information)
• Image analysis and scene interpretation , e.g.

Edgels -> edge segments -> closed edges (2D structures) -> 3D structures …. and much more TASKS

• General -> e.g. extract image edges
• Specific -> extract specific object (e.g. circles, crosses)

FEATURE TYPES

• Point

corners, junctions, high curvature gradient, gravity center, line ends, edgels

• Linear

curved or straight edge or line segments, closed edges, polygonal approximations

• Areal

segmented regions, blobs, specific shapes (ellipses, etc.)

• Structures

combination of features and their relations

Examples From left to right: Different feature types, e.g. points, edges, regions

ATTRIBUTES OF FEATURES

Position: 3nd points or center of line, gravity center, polygon nodes Geometric:

• rientation, length, curvature, area, perimeter, line width, quad min/max diameter,

elongation, axes of symmetry, number/position of singular points, compactness Radiometric: edge sign and strength, grey level statistics on either side of edge, multiple spectral channels, autocorrelation Textural: co-occurence matrix, homogeneity, texture gradients, textural filters, grey level moments Topological: above, left, adjacency, common end points, intersect, parallel, vertical, touch, overlap, contains, contained Comparisons: ordering of attributes, seldomness of features or attributes Spectral: Motion: features on static or moving objects Time: change of feature attributes over time Symbolic: set of constituent features/attributes, their relationships descriptions Feature combination -> new features - > symbolic description CHOICE OF ATTRIBUTES Discrimination power: statistical properties of each attribute in all features, information-theoretic measures (e.g. entropy-based) Independence: correlations between attributes in all features

PROPERTIES OF GOOD FEATURES / ATTRIBUTES Detection - Density: appearance in all images, sufficient density Seldomness-Distinctness: unique or seldom in image, distinct in neighbourhood Invariance-Stability: robust with respect to geometric and radiometric transformations, insensitive to noise Localisation: permit precise localisation Interpretability: permit interpretation (for subsequent image analysis), e.g. junctions, edges, regions, YES; edgels, NO Speed: feature extraction speed

IMAGE FEATURES

• Regions (have been treated in previous lecture)
• Edges (will be treated here)
• Points (will be treated here)

EDGES

• Discontinuities in intensity, color, texture, ...

Here, focus on intensity

• 2D edges, 3D (in surfaces), in multispectral images

Here, focus on 2D edges

• Similar terms: edges, lines, contours (special case: straight edges), ribbons = two parallel edges

The role of edges in object detection and reconstruction

Edge models (from Gonzalez and Woods)

• both are too idealistic

Edges (the real life) 3D representation of grey values as surface. Below 3 zoomed parts showing the problems with the edge definition and the noise.

EDGE ATTRIBUTES AND RELATIONS Usual attributes

• Position (center, end points, all points, polygonal approximation)
• Strength (contrast, gradient magnitude)
• Orientation

More complicated attributes possible, e.g.

• Properties of flanking regions
• Relations between neighbouring edges, etc.

RICH FEATURE ATTRIBUTES AND RELATIONS (HENRICSSON, 1996)

• Transformation of RGB to Lab color space
• Detection of photometric and chromatic attributes using the color flanking regions

The construction of a flanking region. The source edge is black, the interfering edges are gray and the region is encoded in gray-values indicating the distance from the source edge. (A) schematically shows the construction of a flanking region, (B-D) the effect of using different region widths and

• ffsets for the same edge, (E) a “curved” region, (F) a uniform region with dark and light

disturbances (outliers), (G) same as (F) after excluding the outliers with robust estimation.

EDGE ATTRIBUTES

EDGE RELATIONS

EDGE RELATIONS Grouping of edges based on similarity:

• position
• orientation
• radiometry
• color properties

Extraction of edge/linear features

Other edge detection approaches possible, e.g.

• Modelling of grey level surface (Haralick), or profiles vertical to edge
• Template matching
• Hough transform

Extraction of edge/linear features

Edge extraction (example with Sobel operator). From left: original image, magnitude of gradients, gradient orientation (0-180 o) coded as grey values (0-255).

Ramp edge examples (from Gonzalez and Woods). From top to bottom, addition of Gaussian noise with mean 0 and sigma=0, 0.1,1 and 10. Left column shows grey values, second the first derivatives, third the second derivatives. The higher the derivatives, the higher the effect of noise.

Preprocessing

• Can change results drastically
• Typical operations:
• reduce noise (but without edge smoothing)
• sharpen edges
• enhance contrast
• Pay attention to saturation (can change edge position)

The edges are detected from the zero crossing of the LoG.

• Difference of Gaussians (DoG)
• DoG looks like a Mexican hat
• When s1 / s2 = 1:1.75, then DoG is approximately equal to the Laplacian of the Gaussian

with s, i.e. first a low-pass filtering (Gaussian), then high-pass filtering (Laplace)

• Result is an enhancement of the edges (zero crossings of the second derivatives) and edge

detection

Examples of edge detection (all refer to gradient magnitude) Results with Canny. From top left to bottom right increasing value of σ.

Examples of edge detection (all refer to gradient magnitude) Results with Canny after deletion of the short edges.

Examples of edge detection (all refer to gradient magnitude) Top: left discrete differentiator, right Roberts. Bottom: left 3x3 Prewitt, right 3x3 Sobel.

Examples of edge detection (all refer to gradient magnitude)

Top: left DoG, 7x7 mask size, excitatory σ =1, inhibitory s =1.6 ; right original – 5x5 local average Bottom: left 3x3 compass with 8 masks ; right Laplace

Examples of edge detection (all refer to gradient magnitude) 3x3 Kirsch with 8 masks.

Computation of threshold for binarising gradients

• Interactively, e.g. by measuring gradients of low-contrast edges that should be

extracted, or visual inspection of results

• Empirically, e.g. if grey level difference between neighbours > 8, then edge
• Automatically from the image, e.g. compute the histogram of the gradients. Choose

the threshold such that N% (e.g. 10%) of the gradients is larger than the threshold, i.e. keep the N% higher gradients

• Compute the noise variance (e.g. using homogeneous regions) and choose the

threshold such that no edges due to noise are detected

• Above approaches can be used globally or locally (in image parts)

Example: edge tracking in a 4-neighbourhood Efficient storage of edgels as chain code.

• Coding of the 8 neighbour directions
• Necessary for storage (instead of x,y coordinates of all edgels): Start

point, number of edgels, direction code for every edgel)

• Example

Original image

• Chain code with 4- and 8-neighbourhood. Start point is marked dark.

4-neighbourhood. Chain code is: Start point: x=3, y=5 Number of points: 44 Directions: 2,2,0,2,0,0,0,6,6,0,6, ......4,2,2,4,2,2 8-neighbourhood. Chain code is: Start point: x=3, y=5 Number of points: 28 Directions: 2,1,1,0,7,6,7,1,2,1,0, .........4,3,4,3,3,3,2,2

Closing of gaps

• Mathematical morphology (e.g. closing)
• closes gaps < M ; M dimensions of Structural Element (SE ; like filter

mask), SE symmetric

• distance between different edges must be > M, otherwise wrong edge

combination

• Heuristic approaches (also for edge tracking), e.g. the gap should be

small, the direction of the detected edge should not change much, i.e. search space is a circle segment

Starting from an edge endpoint send 11 search rays of length 4 within an angle tolerance with respect to the orientation of the last edgel. Two other edge endpoints were detected and were connected to the first

• edge. The original edgels are

dark, the new ones lighter.

• Ring operator (for edge tracking and gap closing)
• define a circle arc (or more in different distances from the edge endpoint)
• examine the grey values or gradients along the arc
• multiple arcs useful to (a) bridge gaps, (b) recognise junctions
• Simpler approach: instead of arcs, use lines vertical to the estimated edge

direction possible gap

• ther edge

search space

Problems of edge tracking (and generally edge detection)

• Poor contrast -> gaps
• Edges close to each other (parallel or belonging to different objects, e.g.

roof edge and road edge)

• Junctions
• are they recognised? How does the algorithm reacts?
• is double usage of a branch recognised?
• Disturbances (shadows, other objects like trees etc.)
• Intensity and contrast changes along the edge
• Influence due to noise, e.g. many, small edges (such edges could be

deleted in a post-processing step)

Semi-automated approaches

• User gives some points at or close to edge, algorithm then extracts the

edge (edge following)

• Manual (seed) points should be typical, e.g. where edge changes
• rientation
• Seed points could be the result of another automated procedure
• Other possibility is to give approximately the form of the edge, see active

edges (snakes), in the segmentation lecture

Example of edge extraction

Feature Detection with Hough Transform

We haven’t still found edge segments, only edge points How can we find and describe more complex features? The Hough transform is a common approach to finding parameterised line segments (here straight lines)

 Option 1:

 Search for the line at every possible position/orientation  What is the cost of this operation?

 Option 2:

 Use a voting scheme: Hough transform

Finding lines in an image

Each straight line in this image can be described by an equation. Each white point if considered in isolation could lie on an infinite number of straight lines.

How do we represent lines?

Any line can be represented by two numbers Here we will represent the yellow line by (w,φ) In other words we define it using

• a line from an agreed origin
• of length w
• at angle φ to the x-axis

φ w

Since we can use (w,φ) to represent any line in the image space We can represent any line in the image space as a point in the plane defined by (w,φ) This is called Hough space

Hough space

φ w

φ=0 φ=180

w=0 w=100

How does a point in image space vote?

φ w

φ=0 φ=180

w=0 w=100

Hough Transform

 There are generalised versions for ellipses,

circles

 For the straight line transform we need to

supress non-local maxima

 The input image could also benefit from edge

thinning

 Single line segments not isolated  Will still fail in the face of certain textures

Line : http://www.dai.ed.ac.uk/HIPR2/houghdemo.html http://www.dis.uniroma1.it/~iocchi/slides/icra2001/java/hough.html Circle : http://www.markschulze.net/java/hough/

Example: Finding a Circle

 Circles have three parameters

 Center (a,b)  Radius r

 Circle f(x,y,r) = (x-a)2+(y-b)2-r2 = 0  Task:  Given an edge point at (x,y) in the image, where

could the center of the circle be? Find the center of a circle with known radius r given an edge image with no gradient direction information (edge location only)

Hough Transform - example

Original image Edge detection Hough transform

Interest points

Geometric features

repeatable under transformations

2D characteristics of the signal

high informational content

Comparison of different detectors [Schmid98] Harris detector

More motivation…

 Feature points are used for:

 Image co-registration  3D reconstruction  Motion tracking  Object recognition  Indexing and database retrieval  Robot navigation  … other

Finding Corners

Edge detectors perform poorly at corners. Corners provide repeatable points for matching, so are worth detecting. Idea:

• Exactly at a corner, gradient is

ill defined.

• However, in the region around

a corner, gradient has two or more different values.

What Is a Corner?

 Corners are local image features characterised by

locations where variations of intensity function f(x,y) in both X and Y directions are high

 Intersection points between two or more edge

segments

Families of Corner Detectors

 Work directly with the values of brightness of

images

 Usually based on the study of derivatives

(orientation, magnitude) of grey-level or color image

 Extract object boundaries first then analyze its

shape

 Use edge detectors first then analyze the curvature

• f boundaries

 Less reliable

Edge Detectors Tend to Fail at Corners

Intuition:

• At a corner, edge gradients are

ill defined.

• Near a corner, gradient has two
• r more different values.

Formula for Finding Corners

C = Ix

2

"

IxIy

"

IxIy

"

Iy

2

"

# $ % % & ' ( (

We look at the matrix:

Sum over a small region, the hypothetical corner Gradient with respect to x times gradient with respect to y Matrix is symmetric

General Case:

From Linear Algebra, it follows that because C is symmetric:

R R C ! " # $ % & =

' 2 1 1

( (

With R a rotation matrix, λ eigenvalues.

So, to detect corners

 Filter image.  Compute magnitude of the

gradient everywhere.

 We construct C in a window.  Use Linear Algebra to find λ1

and λ2.

 If they are both big, we have

a corner.

 C.Harris, M.Stephens. “A Combined Corner and Edge

Detector”. 1988

 We should easily recognize the point by looking through a

small window

 Shifting a window in any direction should give a large

change in intensity.

Harris Operator

“flat” region: no change in all directions “edge”: no change along the edge direction “corner”: significant change in all directions

Harris Detector: Mathematics

[ ]

2 ,

( , ) ( , ) ( , ) ( , )

x y

E u v w x y I x u y v I x y = + + !

"

Change of intensity for the shift [u,v]:

Intensity Shifted intensity Window function

• r

Window function w(x,y) = Gaussian 1 in window, 0 outside

Harris Detector: Mathematics

[ ]

( , ) , u E u v u v M v ! " # $ % & '

For small shifts [u,v] we have a bilinear approximation:

2 2 ,

( , )

x x y x y x y y

I I I M w x y I I I ! " = # $ # $ % &

'

where M is a 2×2 matrix computed from image derivatives:

Harris Detector: Mathematics

[ ]

( , ) , u E u v u v M v ! " # $ % & '

Intensity change in shifting window: eigenvalue analysis λ1, λ2 – eigenvalues of M

direction of the slowest change direction of the fastest change

(λmax)-1/2 (λmin)-1/2

Ellipse E(u,v) = const

Harris Detector: Mathematics

λ1 λ2 “Corner” λ1 and λ2 are large,

λ1 ~ λ2;

E increases in all

directions

λ1 and λ2 are small; E is almost constant

in all directions

“Edge” λ1 >> λ2 “Edge” λ2 >> λ1 “Flat” region Classification of image points using eigenvalues of M:

Harris Detector: Mathematics

Measure of corner response:

( )

2

det trace R M k M = !

1 2 1 2

det trace M M ! ! ! ! = = +

(k – empirical constant, k = 0.04-0.06)

Harris Detector: Mathematics

λ1 λ2 “Corner” “Edge” “Edge” “Flat”

• R depends only on

eigenvalues of M

• R is large for a corner
• R is negative with large

magnitude for an edge

• |R| is small for a flat

region R > 0 R < 0 R < 0 |R| small

Harris Detector: Workflow

Harris Detector: Workflow

Compute corner response R

Harris Detector: Workflow

Find points with large corner response: R>threshold

Harris Detector: Workflow

Take only the points of local maxima of

R

Harris Detector: Workflow