Today Image segmentation methods: Boundaries, Hough transform, - - PowerPoint PPT Presentation

Today

Image segmentation methods: Boundaries, Hough transform, Intelligent Scissors

Slides based on Steve Seitz, Kristen Grauman, and others

Boundaries in Medical Imaging

Image Machine Edge Detection Human Boundary Marking

Boundaries in Medical Imaging

Detection of cancerous regions.

[Foran, Comaniciu, Meer, Goodell, 00]

Boundaries in Medical Imaging

Hard to detect in the presence of large amount of speckle noise

Preprocessing Edge Images

Image Edge detection and Thresholding Noisy edge image Incomplete boundaries Shrink and Expand Thinning

Given: Approximate Location of Boundary Task: Find Accurate Location of Boundary

• Search for STRONG EDGES along normals to approximate boundary.
• Fit curve (eg., polynomials) to strong edges.

Edge Tracking Methods

Adjusting a priori Boundaries:

Fitting Lines to Edges (Least Squares)

c mx y + = c mx y

i i

− − ) , (

i i y

x y x

Given: Many pairs Find: Parameters Minimize: Average square distance: Using: Note:

) , (

i i y

x ) , ( c m

∑

− − =

i i i

N c mx y E

2

) ( & = ∂ ∂ = ∂ ∂ c E m E x m y c − =

∑ ∑

− − − =

i i i i i

x x y y x x m

2

) ( ) )( (

N y y

i i

∑

= N x x

i i

∑

=

Problem with Parameterization

y x

Line that minimizes E!!

Curve Fitting

y x

Find Polynomial: that best fits the given points Minimize: Using: Note: is LINEAR in the parameters (a, b, c, d)

) , (

i i y

x d cx bx ax x f y + + + = =

2 3

) (

∑

+ + + −

i i i i i

d cx bx ax y N

2 2 3

)] ( [ 1

, , , = ∂ ∂ = ∂ ∂ = ∂ ∂ = ∂ ∂ d E c E b E a E

) (x f

Line Grouping Problem

Slide credit: David Jacobs

This is difficult because of:

• Extraneous data: clutter or multiple models

– We do not know what is part of the model? – Can we pull out models with a few parts from much larger amounts of background clutter?

• Missing data: only some parts of model are

present

• Noise
• Cost:

– It is not feasible to check all combinations of features by fitting a model to each possible subset

Hough Transform

Elegant (somewhat slow? method for direct object recognition

• Edges need not be connected
• Complete object need not be visible
• Key Idea: Edges VOTE for the possible model

Image and Parameter Spaces

c mx y + = ) , (

i i y

x y x

Equation of Line: Find: Consider point:

c mx y + = ) , ( c m ) , (

i i y

x

i i i i

y m x c

• r

c mx y + − = + = m c ) , ( c m

Image Space Parameter Space Parameter space also called Hough Space

Connection between image (x,y) and Hough (m,b) spaces – A line in the image corresponds to a point in Hough space – To go from image space to Hough space:

• given a set of points (x,y), find all (m,b) such that y =

mx + b

x y m b m0 b0

image space Hough (parameter) space

Slide credit: Steve Seitz

Image and Parameter Spaces

Connection between image (x,y) and Hough (m,b) spaces – A line in image corresponds to a point in Hough space – To go from image space to Hough space:

• given points (x,y), find all (m,b) such that y = mx + b

– What does a point (x0, y0) in the image space map to?

x y m b

image space Hough (parameter) space

– Answer: the solutions of b = -x0m + y0 – this is a line in Hough space x0 y0

Slide credit: Steve Seitz

Image and Parameter Spaces

What are the line parameters for the line that contains both (x0, y0) and (x1, y1)? – It is the intersection of the lines b = –x0m + y0 and b = –x1m + y1

x y m b

image space Hough (parameter) space

x0 y0

b = –x1m + y1 (x0, y0) (x1, y1)

Image and Parameter Spaces

How can we use this to find the most likely parameters (m,b) for the most prominent line in the image space?

• Let each edge point in image space vote for a set of

possible parameters in Hough space

• Accumulate votes in discrete set of bins; parameters with

the most votes indicate line in image space.

x y m b

image space Hough (parameter) space

Image and Parameter Spaces

Line Detection by Hough Transform

y x ) , ( c m

Parameter Space

1 1 1 1 1 1 2 1 1 1 1 1 1

) , ( c m A

Algorithm:

• Quantize Parameter Space
• Create Accumulator Array
• Set
• For each image edge increment:
• If lies on the line:
• Find local maxima in

) , ( c m ) , ( c m A c m c m A , ) , ( ∀ = ) , (

i i y

x 1 ) , ( ) , ( + = c m A c m A ) , ( c m ) , ( c m A

i i

y m x c + − =

: perpendicular distance from line to origin : angle the perpendicular makes with the x-axis

Point in image space  sinusoid segment in Hough space

d y x = − θ θ sin cos

d

θ

[0,0]

d θ

x

y

Issues with usual (m,b) parameter space: can take on infinite values, undefined for vertical lines.

Image columns Image rows

Kristen Grauman

Better Parameterization

Better Parameterization

NOTE: Large Accumulator More memory and computations Improvement: (Finite Accumulator Array Size) Line equation: Here

∞ ≤ ≤ ∞ − m θ θ sin cos y x d − =

max

2 ρ ρ π θ ≤ ≤ ≤ ≤ ) , (

i i y

x y x

Image Space

ρ θ

Hough Space

?

Image space Votes Horizontal axis is θ, vertical is ρ.

Image space votes

Mechanics of the Hough transform

• Difficulties

– how big should the cells be? (too big, and we merge quite different lines; too small, and noise causes lines to be missed)

• How many lines?

– Count the peaks in the Hough array – Treat adjacent peaks as a single peak

• Which points belong to each

line? – Search for points close to the line – Solve again for line and iterate

Image space edge coordinates Votes Here, everything appears to be “noise”, or random edge points, but we still see peaks in the vote space.

Mechanics of the Hough transform

a0 b0 r (xi,yi) Equation of Circle:

2 i i

r = ) b (y + ) a (x

2 2

− −

If radius r is known:

x y x y a b (xi,yi)

Accumulator array A(a,b)

Finding Circles by Hough Transform

Finding Circles by Hough Transform

Equation of Circle:

2 2 2

) ( ) ( r b y a x

i i

= − + −

If radius is not known: 3D Hough Space! Use Accumulator array

) , , ( r b a A

Key issues:

• can be generalized to fit any object that can

be parameterized

• problems: running time (curse of

dimensionality), will fail for complex objects, size/resolution of cell in accumulator

• Some heuristic improvements are possible

a0 b0 r (xi,yi) x y

Finding Circles by Hough Transform

a0 x y

Circles: Hough Transform

Further reading: Ballard and Brown, hough.pdf

Image: Multi-subject Diffusion MRI Tractography via a Hough Transform Global Approach, Aganj et al.

Example: iris detection

• Hemerson Pistori and Eduardo Rocha Costa

http://rsbweb.nih.gov/ij/plugins/hough-circles.html Gradient+threshold Hough space (fixed radius) Max detections

Kristen Grauman

Hough transform: pros and cons

Pros

• All points are processed independently, so can cope with
• cclusion, gaps due to poor signal contrast
• Some robustness to noise: noise points unlikely to

contribute consistently to any single bin

• Can detect multiple instances of a model in a single pass

Cons

• Complexity of search time increases exponentially with

the number of model parameters

• Non-target shapes can produce spurious peaks in

parameter space

• Quantization: can be tricky to pick a good grid size

Kristen Grauman

• Hough line demo
• http://www.dis.uniroma1.it/~iocchi/slides/icr

a2001/java/hough.html

What if we want to detect arbitrary shapes?

Kristen Grauman

Generalized Hough Transform

What if we want to detect arbitrary shapes? Intuition:

Kristen Grauman

Generalized Hough Transform

Model image

What if we want to detect arbitrary shapes? Intuition:

Kristen Grauman

Generalized Hough Transform

Model image

What if we want to detect arbitrary shapes? Intuition:

• Ref. point

Kristen Grauman

Generalized Hough Transform

Model image

What if we want to detect arbitrary shapes? Intuition:

• Ref. point

Displacement vectors

Kristen Grauman

Generalized Hough Transform

Model image Novel image

What if we want to detect arbitrary shapes? Intuition:

• Ref. point

Displacement vectors

Kristen Grauman

Generalized Hough Transform

Model image Vote space Novel image

What if we want to detect arbitrary shapes? Intuition:

• Ref. point

Displacement vectors

Kristen Grauman

Generalized Hough Transform

Model image Vote space Novel image x

What if we want to detect arbitrary shapes? Intuition:

• Ref. point

Displacement vectors

Kristen Grauman

Generalized Hough Transform

Model image Vote space Novel image x x

What if we want to detect arbitrary shapes? Intuition:

• Ref. point

Displacement vectors

Kristen Grauman

Generalized Hough Transform

Model image Vote space Novel image x x x

What if we want to detect arbitrary shapes? Intuition:

• Ref. point

Displacement vectors

Kristen Grauman

Generalized Hough Transform

Model image Vote space Novel image x x x x

What if we want to detect arbitrary shapes? Intuition:

• Ref. point

Displacement vectors

Kristen Grauman

Generalized Hough Transform

Model image Vote space Novel image x x x x x

What if we want to detect arbitrary shapes? Intuition:

• Ref. point

Displacement vectors

Kristen Grauman

Generalized Hough Transform

Model image Vote space Novel image x x x x x

What if we want to detect arbitrary shapes? Intuition:

• Ref. point

Displacement vectors

Kristen Grauman

Generalized Hough Transform

Model image Vote space Novel image x x x x x

What if we want to detect arbitrary shapes? Intuition:

• Ref. point

Displacement vectors

Kristen Grauman

Generalized Hough Transform

Model image Vote space Novel image x x x x x

Now suppose those colors encode gradient directions… What if we want to detect arbitrary shapes? Intuition:

• Ref. point

Displacement vectors

Kristen Grauman

Generalized Hough Transform

• Define a model shape by its boundary points and a

reference point.

[Dana H. Ballard, Generalizing the Hough Transform to Detect Arbitrary Shapes, 1980]

x a

p1

θ

p2

θ

Offline procedure:

Model shape

Kristen Grauman

Generalized Hough Transform

• Define a model shape by its boundary points and a

reference point.

[Dana H. Ballard, Generalizing the Hough Transform to Detect Arbitrary Shapes, 1980]

x a

p1

θ

p2

θ

At each boundary point, compute displacement vector: r = a – pi. Offline procedure:

Model shape

Kristen Grauman

Generalized Hough Transform

• Define a model shape by its boundary points and a

reference point.

[Dana H. Ballard, Generalizing the Hough Transform to Detect Arbitrary Shapes, 1980]

x a

p1

θ

p2

θ

At each boundary point, compute displacement vector: r = a – pi. Offline procedure:

Model shape

Kristen Grauman

Generalized Hough Transform

• Define a model shape by its boundary points and a

reference point.

[Dana H. Ballard, Generalizing the Hough Transform to Detect Arbitrary Shapes, 1980]

x a

p1

θ

p2

θ

At each boundary point, compute displacement vector: r = a – pi. Store these vectors in a table indexed by gradient orientation θ. Offline procedure:

Model shape

Kristen Grauman

Generalized Hough Transform

• Define a model shape by its boundary points and a

reference point.

[Dana H. Ballard, Generalizing the Hough Transform to Detect Arbitrary Shapes, 1980]

x a

p1

θ

p2

θ

At each boundary point, compute displacement vector: r = a – pi. Store these vectors in a table indexed by gradient orientation θ. Offline procedure:

Model shape

θ θ

… … …

Kristen Grauman

Generalized Hough Transform

p1

For each edge point: Detection procedure:

Assuming translation is the only transformation here, i.e., orientation and scale are fixed.

Novel image

θ θ

… … …

Kristen Grauman

Generalized Hough Transform

p1

For each edge point:

• Use its gradient orientation θ

to index into stored table

Detection procedure:

Assuming translation is the only transformation here, i.e., orientation and scale are fixed.

Novel image

θ θ

… … …

Kristen Grauman

Generalized Hough Transform

p1

θ

For each edge point:

• Use its gradient orientation θ

to index into stored table

Detection procedure:

Assuming translation is the only transformation here, i.e., orientation and scale are fixed.

θ

Novel image

θ θ

… … …

θ

Kristen Grauman

Generalized Hough Transform

p1

θ

For each edge point:

• Use its gradient orientation θ

to index into stored table

• Use retrieved r vectors to

vote for reference point

Detection procedure:

Assuming translation is the only transformation here, i.e., orientation and scale are fixed.

θ

Novel image

θ θ

… … …

θ

Kristen Grauman

Generalized Hough Transform

p1

θ

For each edge point:

• Use its gradient orientation θ

to index into stored table

• Use retrieved r vectors to

vote for reference point

Detection procedure:

Assuming translation is the only transformation here, i.e., orientation and scale are fixed.

θ

Novel image

θ θ

… … …

θ

Kristen Grauman

Generalized Hough Transform

p1

θ

For each edge point:

• Use its gradient orientation θ

to index into stored table

• Use retrieved r vectors to

vote for reference point

Detection procedure:

Assuming translation is the only transformation here, i.e., orientation and scale are fixed.

x

θ

Novel image

θ θ

… … …

θ

x x Kristen Grauman

Generalized Hough Transform

p1

θ

For each edge point:

• Use its gradient orientation θ

to index into stored table

• Use retrieved r vectors to

vote for reference point

Detection procedure:

Assuming translation is the only transformation here, i.e., orientation and scale are fixed.

x

θ

Novel image

θ θ

… … …

θ

x x Kristen Grauman

Generalized Hough Transform

p1

θ θ

For each edge point:

• Use its gradient orientation θ

to index into stored table

• Use retrieved r vectors to

vote for reference point

Detection procedure:

Assuming translation is the only transformation here, i.e., orientation and scale are fixed.

x

θ θ

Novel image

θ θ

… … …

θ

x x Kristen Grauman

Generalized Hough Transform

p1

θ θ

For each edge point:

• Use its gradient orientation θ

to index into stored table

• Use retrieved r vectors to

vote for reference point

Detection procedure:

Assuming translation is the only transformation here, i.e., orientation and scale are fixed.

x

θ θ

Novel image

θ θ

… … …

θ

x x Kristen Grauman

Generalized Hough Transform

p1

θ θ

For each edge point:

• Use its gradient orientation θ

to index into stored table

• Use retrieved r vectors to

vote for reference point

Detection procedure:

Assuming translation is the only transformation here, i.e., orientation and scale are fixed.

x

θ θ

Novel image

θ θ

… … …

θ

x x x x Kristen Grauman

Generalized Hough Transform

p1

θ θ

For each edge point:

• Use its gradient orientation θ

to index into stored table

• Use retrieved r vectors to

vote for reference point

Detection procedure:

Assuming translation is the only transformation here, i.e., orientation and scale are fixed.

x

θ θ

Novel image

θ θ

… … …

θ

x x x x Kristen Grauman

Generalized Hough Transform

Find shortest path from s to t.

s 3 t 2 6 7 4 5

24 18 2 9 14 15 5 30 20 44 16 11 6 19 6

Live wires: shortest path

from Tardos/Kleinberg book

Live Wires: Intelligent Scissors

• Today’s Readings

– Intelligent Scissors, Mortensen et. al, SIGGRAPH 1995

Extracting objects

• How could this be done?

– hard to do manually – hard to do automatically – easy to do semi-automatically

Extracting objects

Intelligent Scissors

• Approach answers a

basic question – Q: how to find a path from seed to mouse that follows

• bject boundary as

closely as possible? – A: define a path that stays as close as possible to edges

• Basic Idea

– Define edge score for each pixel

• edge pixels have low cost

– Find lowest cost path from seed to mouse

seed mouse

Questions

• How to define costs?
• How to find the path?

Intelligent Scissors

seed mouse

Questions

• How to define costs?
• How to find the path?
• Basic Idea

– Define edge score for each pixel

• edge pixels have low cost

– Find lowest cost path from seed to mouse

Intelligent Scissors

Node: pixel Edge: cost of path or cut between two pixels

Intelligent Scissors

• Treat the image as a graph

Graph

• node for every pixel p
• link between every adjacent pair of pixels, p,q
• cost c for each link

Note: each link has a cost

• this is a little different than the figure before where

each pixel had a cost

p q c

Intelligent Scissors

Path Search (basic idea)

• Graph Search Algorithm

– Computes minimum cost path from seed to all other pixels

Note: diagonal “paths” are scaled by a factor .

2

… Why?

After object rotation

L

Should be invariant to image rotation

After object rotation

L

Path’s cost along the top boundary Path’s cost along the top-left boundary

14 7 * 2 = 10 5 * 2 =

After diagonal links are adjusted by 2

14 5 * 2 * 2 =

≠

2 2 2 2 2 2 2 2 2 2 2 2

image gradient scores

Should be invariant to image rotation

• good (low-cost) links follow the intensity edge

– want intensity to change rapidly ⊥ to the link

|intensity of r – intensity of s|

Want to hug image edges: how to define cost of a link? p q c r s

Defining the cost

p q c

• c can be computed using a cross-correlation filter

– assume it is centered at p

Defining the cost

p q c

1

• 1

w

• 1
• 1

1 1

c can be computed using a cross-correlation filter

• assume it is centered at p

A couple more modifications

• Scale the filter response by length of link c.
• Make c positive

– Set c = (max-|filter response|*length) – where max = maximum |filter response|*length over all pixels in the image

Defining the cost

 Is the pixel an edge pixel? Evaluate Laplacian zero crossings  How good is this edge pixel? Gradient magnitude  What is its orientation? Gradient direction

Evaluate for each pair of neighboring pixels and compute the cost accordingly

invert the edge gradients (high gradient -> small cost) variation between the two unit vectors

Other options to calculate edge cost

Dijkstra’s shortest path algorithm

5 3 1 3 3 4 9 2 Algorithm

1. init node costs to ฀, set p = seed point, cost(p) = 0 2. expand p as follows:

for each of p’s neighbors q that are not expanded

» set cost(q) = min( cost(p) + cpq, cost(q) )

link cost

4 1 5 3 3 2 3 9 Algorithm

1. init node costs to ฀, set p = seed point, cost(p) = 0 2. expand p as follows:

for each of p’s neighbors q that are not expanded

» set cost(q) = min( cost(p) + cpq, cost(q) )

» if q’s cost changed, make q point back to p

» put q on the ACTIVE list (if not already there)

5 3 1 3 3 4 9 2 1 1

Dijkstra’s shortest path algorithm

4 1 5 3 3 2 3 9 5 3 1 3 3 4 9 2 1 5 2 3 3 3 2 4 Algorithm

1. init node costs to ฀, set p = seed point, cost(p) = 0 2. expand p as follows:

for each of p’s neighbors q that are not expanded

» set cost(q) = min( cost(p) + cpq, cost(q) )

» if q’s cost changed, make q point back to p

» put q on the ACTIVE list (if not already there)

3. set r = node with minimum cost on the ACTIVE list 4. repeat Step 2 for p = r

Dijkstra’s shortest path algorithm

3 1 5 3 3 2 3 6 5 3 1 3 3 4 9 2 4 3 1 4 5 2 3 3 3 2 4 Algorithm

1. init node costs to ฀, set p = seed point, cost(p) = 0 2. expand p as follows:

for each of p’s neighbors q that are not expanded

» set cost(q) = min( cost(p) + cpq, cost(q) )

» if q’s cost changed, make q point back to p

» put q on the ACTIVE list (if not already there)

3. set r = node with minimum cost on the ACTIVE list 4. repeat Step 2 for p = r

Dijkstra’s shortest path algorithm

A B

Dijkstra algorithm

• processed nodes (distance to A is known)
• active nodes (front)
• active node with the smallest distance value

Dijkstra’s shortest path algorithm

• Properties

– It computes the minimum cost path from the seed to every node in the graph. This set of minimum paths is represented as a tree – Running time, with N pixels:

• O(N2) time if you use an active list
• O(N log N) if you use an active priority queue (heap)
• takes fraction of a second for a typical (640x480) image

– Once this tree is computed once, we can extract the

• ptimal path from any point to the seed in O(N) time.
• it runs in real time as the mouse moves

– What happens when the user specifies a new seed?

Dijkstra’s shortest path algorithm

A B

The arrows indicate the direction of the path with minimum cumulative cost.

image: Antonio Criminisi and Sing Bing Kang

Live wires and Intelligent scissors

Live wires and Intelligent scissors: examples

image: Sharon Eve

Live wires and Intelligent scissors: examples

Live wires and Intelligent scissors: examples

Live wires and Intelligent scissors: examples