[PDF] - Lecture 9: Fitting, Contours Thursday, Sept 27 Announcements PDF Document

SLIDE 1

Lecture 9: Fitting, Contours

Thursday, Sept 27

SLIDE 2

Announcements

Midterm review:

next Wed Oct 4, 12-1 pm, ENS 31NQ

SLIDE 3

Last time

Fitting shape patterns with the Hough

transform and generalized Hough transform

SLIDE 4

Today

Fitting lines (brief)

– Least squares – Incremental fitting, k-means allocation

RANSAC, robust fitting
Deformable contours

SLIDE 5

Line fitting: what is the line?

Assuming all the points that belong to a particular

line are known, solve for line parameters that yield minimal error.

Forsyth & Ponce 15.2.1

SLIDE 6

Line fitting: which point is on which line?

Two possible strategies:

Incremental line fitting
K-means

SLIDE 7

Incremental line fitting

Take connected curves of edge points and

fit lines to runs of points (use gradient directions)

SLIDE 8

Incremental line fitting

SLIDE 9

SLIDE 10

SLIDE 11

SLIDE 12

SLIDE 13

If we have occluded edges, will often result in more than

ne fitted line

SLIDE 14

Allocating points with k-means

Believe there are k lines, each of which

generates some subset of the data points

Best solution would minimize the sum of

the squared distances from points to their assigned lines

Use k-means algorithm
Convergence based on size of change in

lines, whether labels have been flipped.

SLIDE 15

Allocating points with k-means

SLIDE 16

SLIDE 17

SLIDE 18

SLIDE 19

SLIDE 20

SLIDE 21

SLIDE 22

Sensitivity to starting point

SLIDE 23

Outliers

Outliers can result from

– Data collection error – Overlooked case for the model chosen

Squared error terms mean big penalty for

large errors, can lead to significant bias

SLIDE 24

Forsyth & Ponce, Fig 15.7

Outliers affect least squares fit

SLIDE 25

Outliers affect least squares fit

SLIDE 26

Outliers affect least squares fit

SLIDE 27

Least squares and error

( )

θ ,

i i i

x r

∑

Best model minimizes residual error:

Outliers have large influence on the fit

model parameters data point

SLIDE 28

Least squares and error

If we are assuming Gaussian additive noise

corrupts the data points – Probability of noisy point being within distance d of corresponding true point decreases rapidly with d – So, points that are way off are not really consistent with Gaussian noise hypothesis, model wants to fit to them…

SLIDE 29

Robustness

A couple possibilities to handle outliers:

– Give the noise heavier tails – Search for “inliers”

SLIDE 30

M-estimators

Estimate parameters by minimizing modified

residual expression

Reflects a noise distribution that does not vanish

as quickly as Gaussian, i.e., consider outliers more likely to occur

De-emphasizes contribution of distant points

( ) ( )

σ θ ρ ; ,

i i i

x r

∑

residual error parameter determining when function flattens out

SLIDE 31

Example M-estimator

riginal

Looks like distance for small values, Like a constant for large values Non-linear optimization, must be solved iteratively

Impact of sigma on fitting quality?

SLIDE 32

Fit with good choice of

Applying the M-estimator

SLIDE 33

Applying the M-estimator

too small: error for all points similar

SLIDE 34

Applying the M-estimator

too large: error about same as least squares

SLIDE 35

Scale selection

Popular choice: at iteration n during

minimization

SLIDE 36

RANSAC

RANdom Sample Consensus
Approach: we don’t like the impact of
utliers, so let’s look for “inliers”, and use

those only.

SLIDE 37

RANSAC

Choose a small subset uniformly at

random

Fit to that
Anything that is close to result is signal; all
thers are noise
Refit
Do this many times and choose the best

(best = lowest fitting error)

SLIDE 38

RANSAC Reference: M. A. Fischler, R. C. Bolles. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Comm. of the ACM, Vol 24, pp 381-395, 1981.

SLIDE 39

RANSAC Line Fitting Example

Task: Estimate best line

Slide credit: Jinxiang Chai, CMU

SLIDE 40

RANSAC Line Fitting Example

Sample two points

SLIDE 41

RANSAC Line Fitting Example

Fit Line

SLIDE 42

RANSAC Line Fitting Example

Total number of points within a threshold of line.

SLIDE 43

RANSAC Line Fitting Example

Repeat, until get a good result

SLIDE 44

RANSAC Line Fitting Example

Repeat, until get a good result

SLIDE 45

RANSAC Line Fitting Example

Repeat, until get a good result

SLIDE 46

RANSAC application: robust computation

Interest points (Harris corners) in left and right images about 500 pts / image

640x480 resolution Outliers (117) (t=1.25 pixel; 43 iterations) Final inliers (262)

Hartley & Zisserman p. 126

Putative correspondences (268) (Best match,SSD<20) Inliers (151)

SLIDE 47

RANSAC parameters

Number of samples required (n)

– Absolute minimum will depending on model being fit (lines

> 2, circles -> 3, etc)
Number of trials (k)

– Need a guess at probability of a random point being “good” – Choose so that we have high probability of getting one sample free from outliers

Threshold on good fits (t)

– Often trial and error: look at some data fits and estimate average deviations

Number of points that must agree (d)

– Again, use guess of probability of being an outlier; choose d so that unlikely to have one in the group

SLIDE 48

Grouping and fitting

Grouping, segmentation: make a compact

representation that merges similar features

– Relevant algorithms: K-means, hierarchical clustering, Mean Shift, Graph cuts

Fitting: fit a model to your observed features

– Relevant algorithms: Hough transform for lines, circles (parameterized curves), generalized Hough transform for arbitrary boundaries; least squares; assigning points to lines incrementally or with k- means; robust fitting

SLIDE 49

SLIDE 50

Today

Fitting lines (brief)

– Least squares – Incremental fitting, k-means allocation

RANSAC, robust fitting
Deformable contours

SLIDE 51

Towards object level grouping

Low-level segmentation cannot go this far… How do we get these kinds of boundaries? One direction: semi-automatic methods

Give a good but rough initial boundary
Interactively guide boundary placement

Still use image analysis techniques in concert.

SLIDE 52

Deformable contours

Tracking Heart Ventricles (multiple frames)

SLIDE 53

Deformable contours

Given: initial contour (model) near desired object

a.k.a. active contours, snakes

(Single frame)

SLIDE 54

Deformable contours

[Kass, Witkin, Terzopoulos 1987]

Goal: evolve the contour to fit exact object boundary

a.k.a. active contours, snakes

SLIDE 55

Deformable contours

initial intermediate final

a.k.a. active contours, snakes

SLIDE 56

Deformable contours

Elastic band of arbitrary shape, initially

located near image contour of interest

Attracted towards target contour

depending on intensity gradient

Iteratively refined

a.k.a. active contours, snakes

SLIDE 57

Comparison: shape-related methods

Chamfer matching: given two shapes defined by points,

measure average distance from one to the other

(Generalized) Hough transform: given pattern/model

shape, use oriented edge points to vote for likely position

f that pattern in new image
Deformable contours: given initial starting boundary

and priors on preferred shape types, iteratively adjust boundary to also fit observed image

SLIDE 58

Snake Energy

The total energy of the current snake defined as

ex in total

E E E + =

Internal energy encourages smoothness

r any particular shape

Internal energy incorporates prior knowledge about object boundary, which allows a boundary to be extracted even if some image data is missing External energy encourages curve onto image structures (e.g. image edges)

We will want to iteratively minimize this energy for a good fit between the deformable contour and the target shape in the image

Many of the snakes slides are adapted from Yuri Boykov

SLIDE 59

Parametric curve representation

Coordinates given as functions of a parameter

that varies along the curve

For example, for a circle with center (0,0):

parametric form:

x = r sin(s) y = r cos(s)

parameters:

radius r angle 0 <= s < 2pi

(continuous case)

r (0,0)

SLIDE 60

pen curve

closed curve

1 )) ( ), ( ( ) ( ≤ ≤ = s s y s x s ν

Parametric curve representation

(continuous case) Curves parameterized by arc length, the length along the curve

SLIDE 61

Internal energy

Bending energy of a continuous curve

The more the curve bends larger this energy value is.

Elasticity, Tension Stiffness, Curvature

s d d ds d

s s s Ein 2 2 ) ( ) ( )) ( (

2 2

ν ν

β α ν + =

∫

=

1

)) ( ( ds s E E

in in

ν

Internal energy for a curve:

SLIDE 62

External energy

Measures how well the curve matches the

image data, locally

Attracts the curve toward different image

features

– Edges, lines, etc.

SLIDE 63

External energy: edge strength

Image I(x,y)
Gradient images &
External energy at a point is
External energy for the curve:

) , ( y x Gx ) , ( y x Gy

) | )) ( ( | | )) ( ( | ( )) ( (

2 2

s G s G s E

y x ex

ν ν ν + − =

(Negative so that minimizing it forces the curve toward strong edges)

∫

=

1

)) ( ( ds s E E

ex ex

ν

SLIDE 64

Snake Energy (continuous form)

e.g. bending energy e.g. total edge strength under curve

ex in total

E E E + =

∫

=

1

)) ( ( ds s E E

in in

ν

∫

=

1

)) ( ( ds s E E

ex ex

ν

SLIDE 65

Discrete approach

discrete image discrete snake representation discrete optimization (dynamic programming)

SLIDE 66

Parametric curve representation

(discrete case)

Represent the curve with a set of n points

1 ) , ( − = = n i y x

i i i

K ν

…

SLIDE 67

Discrete representation

If the curve is represented by n points

Elasticity, Tension Stiffness Curvature

1 ) , ( − = = n i y x

i i i

K ν

2

1 i i

v ds d ν ν − ≈

+ 1 1 1 1 2 2

2 ) ( ) (

− + − +

+ − = − − − ≈

i i i i i i i

ds d ν ν ν ν ν ν ν ν

∑

− = − + +

+ − + − =

1 2 1 1 2 1

| 2 | | |

n i i i i i i in

E ν ν ν β ν ν α

…

SLIDE 68

68

Simple elastic curve

For a curve represented as a set of points

a simple elastic energy term is

This encourages the closed curve to shrink to a point (like a very small elastic band)

∑

− =

⋅ =

1 2 n i i in

L E α

2 1 1 2 1

) ( ) (

i i n i i i

y y x x − + − ⋅ =

+ − = +

∑

α

SLIDE 69

Encouraging point spacing

To stop the curve from shrinking to a point

– encourages formation of equally spaced chains of points

∑

− =

− ⋅ =

1 2

) ˆ (

n i i i in

L L E α

Average distance between pairs of points – updated at each iteration

SLIDE 70

70

Optional: specify shape prior

If object is some smooth variation on a

known shape, use

where give points of the basic shape

∑

− =

− ⋅ =

1 2

) ˆ (

n i i i in

E ν ν α

} ˆ {

i

ν

SLIDE 71

Edge strength for external energy

An external energy term for a (discrete)

snake based on image edge

2 1 2

| ) , ( | | ) , ( |

i i y n i i i x ex

y x G y x G E

∑

− =

+ − =

SLIDE 72

Summary: simple elastic snake

A simple elastic snake is thus defined by

– A set of n points, – An internal elastic energy term – An external edge based energy term

To use this to locate the outline of an
bject

– Initialize in the vicinity of the object – Modify the points to minimize the total energy

SLIDE 73

Energy minimization

Many algorithms proposed to fit

deformable contours

– Greedy search – Gradient descent – Dynamic programming (for 2d snakes)

SLIDE 74

Greedy minimization

For each point, search window around it

and move to where energy function is minimal

Stop when predefined number of points

have not changed in last iteration

Local minimum

SLIDE 75

75

Synthetic example

(1) (2) (3) (4)

SLIDE 76

Dealing with missing data

The smoothness constraint can deal with

missing data:

[Figure from Kass et al. 1987]

SLIDE 77

Relative weighting

α

large α small α medium α

weight controls internal elasticity

SLIDE 78

Dynamic programming (2d snakes) ∑

− = + −

=

1 1 1

) , ( ) , , (

n i i i i n total

E E ν ν ν ν K

Often snake energy can be rewritten as a

sum of pair-wise interaction potentials

Or sum of triple-interaction potentials.

∑

− = + − −

=

1 1 1 1

) , , ( ) , , (

n i i i i i n total

E E ν ν ν ν ν K

… …

SLIDE 79

Snake energy: pair-wise interactions

2 1 2 1 1

| ) , ( | | ) , ( | ) , , , , , (

i i y n i i i x n n total

y x G y x G y y x x E

∑

− = − −

+ − = K K

2 1 1 2 1

) ( ) (

i i n i i i

y y x x − + − ⋅ +

+ − = +

∑

α

∑

− = −

− =

1 2 1

|| ) ( || ) , , (

n i i n total

G E ν ν ν K

∑

− = + −

⋅ +

1 2 1

|| ||

n i i i

ν ν α

∑

− = + −

=

2 1 1

) , ( ) , , (

n i i i i n total

E E ν ν ν ν K

2 1 2 1

|| || || ) ( || ) , (

+ +

− + − =

i i i i i i

G E ν ν α ν ν ν

where

… … … …

SLIDE 80

1

v

2

v

3

v

4

v

6

v

5

v

control points Energy E is minimized via Dynamic Programming

) , ( ... ) , ( ) , ( ) ,..., , (

1 1 3 2 2 2 1 1 2 1 n n n n

v v E v v E v v E v v v E

− −

+ + + =

First-order interactions (elasticity)

DP Snakes [Amini, Weymouth, Jain, 1990]

SLIDE 81

DP Snakes [Amini, Weymouth, Jain, 1990]

2

v

3

v

4

v

6

v

5

v

control points

Iterate until optimal position for each point is the center of the box, i.e. the snake is optimal in the local search space constrained by boxes

Energy E is minimized via Dynamic Programming

) , ( ... ) , ( ) , ( ) ,..., , (

1 1 3 2 2 2 1 1 2 1 n n n n

v v E v v E v v E v v v E

− −

+ + + =

First-order interactions (elasticity) 1

v

SLIDE 82

DP Viterbi Algorithm

Reuse solutions to subproblems
Introduce intermediate variables

: lowest total energy for the first k-1 vertices of the snake for a given value of vk determine

ptimal position
f predecessor,

for each possible position of self

SLIDE 83

) , ( 4

4 n

v v E ) , (

4 3 3

v v E

) 3 (

3

E ) 4 (

3

E ) 4 (

4

E ) 3 (

4

E ) 2 (

4

E ) 1 (

4

E ) 4 (

n

E ) 3 (

n

E ) 2 (

n

E ) 1 (

n

E ) 2 (

3

E ) 1 (

3

E ) 4 (

2

E ) 3 (

2

E

DP Viterbi Algorithm

) , ( ... ) , ( ) , (

1 1 3 2 2 2 1 1 n n n

v v E v v E v v E

− −

+ + +

) , (

3 2 2

v v E

) 1 (

2

E ) 2 (

2

E

) , (

2 1 1

v v E

) (

2

nm O

Complexity:

) 1 (

1

= E ) 2 (

1

= E ) 3 (

1

= E ) 4 (

1

= E

Considering first-order interactions (elasticity), one minimization iteration

states 1 2 … m sites

1

v

2

v

3

v

4

v

n

v

vs. brute force search ____?

SLIDE 84

Dynamic Programming for a closed snake?

) , ( ... ) , ( ) , (

1 1 3 2 2 2 1 1 n n n

v v E v v E v v E

− −

+ + +

DP can be applied to optimize an open ended snake What about “looped” energy, in the case of a closed snake?

1

ν

n

ν

) , ( ) , ( ... ) , ( ) , (

1 1 1 3 2 2 2 1 1

v v E v v E v v E v v E

n n n n n

+ + + +

− −

1

ν

n

ν

2

ν

1 − n

ν

3

ν

4

ν

SLIDE 85

Problems with snakes

Depends on number and spacing of control

points

Snake may oversmooth the boundary
Not trivial to prevent curve self intersecting
Cannot follow topological changes of objects

SLIDE 86

Problems with snakes

May be sensitive to initialization, get stuck

in local minimum

Accuracy (and computation time) depends
n the convergence criteria used in the

energy minimization technique

SLIDE 87

Problems with snakes

External energy: snake does not really “see”
bject boundaries in the image unless it gets very

close to it.

image gradients are large only directly on the boundary

I ∇

SLIDE 88

Tracking via deformable models

1. Use final contour/model extracted at frame

t as an initial solution for frame t+1

2. Evolve initial contour to fit exact object

boundary at frame t+1

3. Repeat steps 1 and 2 for t = t+1

SLIDE 89

Tracking via deformable models

Acknowledgements: Visual Dynamics Group, Dept. Engineering Science, University of Oxford.

Traffic monitoring Human-computer interaction Animation Surveillance Computer Assisted Diagnosis in medical imaging Applications:

SLIDE 90

Intelligent scissors

[Mortensen & Barrett, SIGGRAPH 1995, CVPR 1999]

Use dynamic programming to compute optimal paths from every point to the seed based on edge-related costs User interactively selects most suitable boundary from set of all optimal boundaries emanating from a seed point

SLIDE 91

Snakes vs. scissors

1 2 3 4

Shortest paths on image-based graph connect seeds placed on object boundary

SLIDE 92

Snakes vs. scissors

Given: initial contour (model) near desirable object

SLIDE 93

Snakes vs. scissors

Given: initial contour (model) near desirable object Goal: evolve the contour to fit exact object boundary

SLIDE 94

Coming up

Stereo
F&P 10.1, 11
Trucco & Verri handout