Segmentation Professor Fei Fei Li Stanford Vision Lab 1 19 Apr 11 - - PowerPoint PPT Presentation

Lecture 8 -

Fei-Fei Li

Segmentation

Professor Fei‐Fei Li Stanford Vision Lab

19‐Apr‐11 1

Lecture 8 -

Fei-Fei Li

Image Segmentation

• Goal: identify groups of pixels that go together

19‐Apr‐11 2

Lecture 8 -

Fei-Fei Li

Success Story

19‐Apr‐11 3

Lecture 8 -

Fei-Fei Li

Gestalt Theory

• Gestalt: whole or group

– Whole is greater than sum of its parts – Relationships among parts can yield new properties/features

• Psychologists identified series of factors that predispose set of

elements to be grouped (by human visual system)

Untersuchungen zur Lehre von der Gestalt,

Psychologische Forschung, Vol. 4, pp. 301-350, 1923

http://psy.ed.asu.edu/~classics/Wertheimer/Forms/forms.htm

“I stand at the window and see a house, trees, sky. Theoretically I might say there were 327 brightnesses and nuances of colour. Do I have "327"?

• No. I have sky, house,

and trees.”

Max Wertheimer

(1880-1943)

19‐Apr‐11 4

Lecture 8 -

Fei-Fei Li

Gestalt Factors

• These factors make intuitive sense, but are very difficult to translate into algorithms.

19‐Apr‐11 5

Lecture 8 -

Fei-Fei Li

K‐Means Clustering

• Basic idea: randomly initialize the k cluster centers, and iterate between

the two steps we just saw.

1.

Randomly initialize the cluster centers, c1, ..., cK

2.

Given cluster centers, determine points in each cluster

• For each point p, find the closest ci. Put p into cluster i

3.

Given points in each cluster, solve for ci

• Set ci to be the mean of points in cluster i

4.

If ci have changed, repeat Step 2 19‐Apr‐11 6

Lecture 8 -

Fei-Fei Li

Expectation Maximization (EM)

• Goal

– Find blob parameters θ that maximize the likelihood function:

• Approach:

1.

E‐step: given current guess of blobs, compute ownership of each point

2.

M‐step: given ownership probabilities, update blobs to maximize likelihood function

3.

Repeat until convergence 19‐Apr‐11 7

Lecture 8 -

Fei-Fei Li

Mean‐Shift Algorithm

• Iterative Mode Search

1.

Initialize random seed, and window W

2.

Calculate center of gravity (the “mean”) of W:

3.

Shift the search window to the mean

4.

Repeat Step 2 until convergence

19‐Apr‐11 8

Lecture 8 -

Fei-Fei Li

Mean‐Shift Segmentation

• Find features (color, gradients, texture, etc)
• Initialize windows at individual pixel locations
• Perform mean shift for each window until convergence
• Merge windows that end up near the same “peak” or mode

19‐Apr‐11 9

Lecture 8 -

Fei-Fei Li

Back to the Image Segmentation Problem…

• Goal: identify groups of pixels that go together
• Up to now, we have focused on ways to group pixels into image

segments based on their appearance…

– Segmentation as clustering.

• We also want to enforce region constraints.

– Spatial consistency – Smooth borders

19‐Apr‐11 10

Lecture 8 -

Fei-Fei Li

What we will learn today?

• Graph theoretic segmentation

– Normalized Cuts – Using texture features

• Segmentation as Energy Minimization

– Markov Random Fields – Graph cuts for image segmentation – Applications

19‐Apr‐11 11

Lecture 8 -

Fei-Fei Li

What we will learn today?

• Graph theoretic segmentation

– Normalized Cuts – Using texture features

• Segmentation as Energy Minimization

– Markov Random Fields – Graph cuts for image segmentation – Applications

19‐Apr‐11 12

Lecture 8 -

Fei-Fei Li

Images as Graphs

• Fully‐connected graph

– Node (vertex) for every pixel – Link between every pair of pixels, (p,q) – Affinity weight wpq for each link (edge)

• wpq measures similarity
• Similarity is inversely proportional to difference

(in color and position…)

q p wpq

w

S lide credit: S teve S eitz

19‐Apr‐11 13

Lecture 8 -

Fei-Fei Li

Segmentation by Graph Cuts

• Break Graph into Segments

– Delete links that cross between segments – Easiest to break links that have low similarity (low weight)

• Similar pixels should be in the same segments
• Dissimilar pixels should be in different segments

w A B C

S lide credit: S teve S eitz

19‐Apr‐11 14

Lecture 8 -

Fei-Fei Li

Measuring Affinity

• Distance
• Intensity
• Color
• Texture

{ }

2

2 1 2

( , ) exp

d

aff x y x y

σ

= − −

{ }

2

2 1 2

( , ) exp ( ) ( )

d

aff x y I x I y

σ

= − −

(some suitable color space distance)

( )

{ }

2

2 1 2

( , ) exp ( ), ( )

d

aff x y dist c x c y

σ

= −

S

• urce: Forsyth & Ponce

{ }

2

2 1 2

( , ) exp ( ) ( )

d

aff x y f x f y

σ

= − −

(vectors of filter outputs)

19‐Apr‐11 15

Lecture 8 -

Fei-Fei Li

Scale Affects Affinity

• Small σ: group only nearby points
• Large σ: group far‐away points

S lide credit: S vetlana Lazebnik

Small σ Medium σ Large σ

19‐Apr‐11 16

Lecture 8 -

Fei-Fei Li

Graph Cut: using Eigenvalues

• Extract a single good cluster

– Where elements have high affinity values with each other

points

Lecture 8 -

Fei-Fei Li

points matrix

Eigenvector associated w/ the largest eigenvalue

Graph Cut: using Eigenvalues

• Extract a single good cluster

Lecture 8 -

Fei-Fei Li

• Extract a single good cluster
• Extract weights for a set of clusters

Graph Cut: using Eigenvalues

Eigenvector associated w/ the largest eigenvalue Eigenvectors associated with other eigenvalues

Lecture 8 -

Fei-Fei Li

Graph Cut: using Eigenvalues (effect of the scaling factor)

Lecture 8 -

Fei-Fei Li

Lecture 8 -

Fei-Fei Li

Graph Cut

• Set of edges whose removal makes a graph

disconnected

• Cost of a cut

– Sum of weights of cut edges:

• A graph cut gives us a segmentation

– What is a “good” graph cut and how do we find one?

S lide credit: S teve S eitz

A B

∑

∈ ∈

=

B q A p q p

w B A cut

, ,

) , (

19‐Apr‐11 22

Lecture 8 -

Fei-Fei Li

Graph Cut

Image S

• urce: Forsyth & Ponce

Here, the cut is nicely defined by the block-diagonal structure of the affinity matrix.

⇒ How can this be generalized?

19‐Apr‐11 23

Lecture 8 -

Fei-Fei Li

Minimum Cut

• We can do segmentation by finding the minimum cut in a graph

– a minimum cut of a graph is a cut whose cutset has the smallest number

• f elements (unweighted case) or smallest sum of weights possible.

– Efficient algorithms exist for doing this

• Drawback:

– Weight of cut proportional to number of edges in the cut – Minimum cut tends to cut off very small, isolated components

Ideal Cut Cuts with lesser weight than the ideal cut

S lide credit: Khurram Hassan-S hafique

19‐Apr‐11 24

Lecture 8 -

Fei-Fei Li

Normalized Cut (NCut)

• A minimum cut penalizes large segments
• This can be fixed by normalizing for size of segments
• The normalized cut cost is:
• The exact solution is NP‐hard but an approximation can be

computed by solving a generalized eigenvalue problem.

assoc(A,V) = sum of weights of all edges in V that touch A

) , ( ) , ( ) , ( ) , ( V B assoc B A cut V A assoc B A cut +

• J. Shi and J. Malik. Normalized cuts and image segmentation. PAMI 2000

19‐Apr‐11 25

Lecture 8 -

Fei-Fei Li

Interpretation as a Dynamical System

• Treat the links as springs and shake the system

– Elasticity proportional to cost – Vibration “modes” correspond to segments

• Can compute these by solving a generalized eigenvector problem

S lide credit: S teve S eitz

19‐Apr‐11 26

Lecture 8 -

Fei-Fei Li

NCuts as a Generalized Eigenvector Problem

• Definitions
• Rewriting Normalized Cut in matrix form:

,

: ( , ) ; : ( , ) ( , ); : {1, 1} , ( ) 1 .

the affinity matrix, the diag. matrix, a vector in

i j j N

W W i j w D D i i W i j x x i i A = = − = ⇔ ∈

∑

S lide credit: Jitendra Malik

(A,B) (A,B) (A,B) (A,V) (B,V) ( , ) (1 ) ( )(1 ) (1 ) ( )(1 ) ; 1 1 (1 )1 1 ( , ) ...

i

T T x T T i

cut cut NCut assoc assoc D i i x D W x x D W x k k D k D D i i

>

= + + − + − − − = + = − =

∑ ∑

19‐Apr‐11 27

Lecture 8 -

Fei-Fei Li

Some More Math…

S lide credit: Jitendra Malik

19‐Apr‐11 28

Lecture 8 -

Fei-Fei Li

NCuts as a Generalized Eigenvalue Problem

• After simplification, we get
• This is a Rayleigh Quotient

– Solution given by the “generalized” eigenvalue problem – Solved by converting to standard eigenvalue problem

• Subtleties

– Optimal solution is second smallest eigenvector – Gives continuous result—must convert into discrete values of y

( ) ( , ) , with {1, }, 1 0.

T T i T

y D W y NCut A B y b y D y Dy − = ∈ − =

S lide credit: Alyosha Efros

This is hard, as y is discrete! Relaxation: continuous y.

,

1 1 1 2 2 2

D (D W)D z λz where z D y

− −

− = =

19‐Apr‐11 29

Lecture 8 -

Fei-Fei Li

NCuts Example

Smallest eigenvectors

Image source: S hi & Malik

NCuts segments

19‐Apr‐11 30

Lecture 8 -

Fei-Fei Li

Discretization

• Problem: eigenvectors take on continuous values

– How to choose the splitting point to binarize the image?

• Possible procedures

a) Pick a constant value (0, or 0.5). b) Pick the median value as splitting point. c) Look for the splitting point that has the minimum NCut value:

1. Choose n possible splitting points. 2. Compute NCut value. 3. Pick minimum. Image Eigenvector NCut scores

19‐Apr‐11 31

Lecture 8 -

Fei-Fei Li

NCuts: Overall Procedure

• 1. Construct a weighted graph G=(V,E) from an image.
• 2. Connect each pair of pixels, and assign graph edge weights,
• 3. Solve for the smallest few eigenvectors. This

yields a continuous solution.

• 4. Threshold eigenvectors to get a discrete cut

– This is where the approximation is made (we’re not solving NP).

• 5. Recursively subdivide if NCut value is below a pre‐specified

value.

( ) D W y Dy λ − =

( , )

• Prob. that and belong to the same region.

W i j i j =

S lide credit: Jitendra Malik

NCuts Matlab code available at http://www.cis.upenn.edu/~jshi/software/ 19‐Apr‐11 32

Lecture 8 -

Fei-Fei Li

Color Image Segmentation with NCuts

Image S

• urce: S

hi & Malik

19‐Apr‐11 33

Lecture 8 -

Fei-Fei Li

Using Texture Features for Segmentation

• Texture descriptor is vector of filter bank outputs
• J. Malik, S. Belongie, T. Leung and J. Shi. "Contour and Texture Analysis for Image Segmentation".

IJCV 43(1),7‐27,2001.

S lide credit: S vetlana Lazebnik

19‐Apr‐11 34

Lecture 8 -

Fei-Fei Li

Using Texture Features for Segmentation

• Texture descriptor is

vector of filter bank

• utputs.
• Textons are found by

clustering.

S lide credit: S vetlana Lazebnik

19‐Apr‐11 35

Lecture 8 -

Fei-Fei Li

Using Texture Features for Segmentation

• Texture descriptor is

vector of filter bank

• utputs.
• Textons are found by

clustering.

• Affinities are given by

similarities of texton histograms over windows given by the “local scale” of the texture .

S lide credit: S vetlana Lazebnik

19‐Apr‐11 36

Lecture 8 -

Fei-Fei Li

Results with Color & Texture

19‐Apr‐11 37

Lecture 8 -

Fei-Fei Li

Summary: Normalized Cuts

• Pros:

– Generic framework, flexible to choice of function that computes weights (“affinities”) between nodes – Does not require any model of the data distribution

• Cons:

– Time and memory complexity can be high

• Dense, highly connected graphs ⇒ many affinity computations
• Solving eigenvalue problem for each cut

– Preference for balanced partitions

• If a region is uniform, NCuts will find the

modes of vibration of the image dimensions

S lide credit: Kristen Grauman

19‐Apr‐11 38

Lecture 8 -

Fei-Fei Li

What we will learn today?

• Graph theoretic segmentation

– Normalized Cuts – Using texture features

• Segmentation as Energy Minimization

– Markov Random Fields – Graph cuts for image segmentation – Applications

19‐Apr‐11 39

Lecture 8 -

Fei-Fei Li

Markov Random Fields

• Allow rich probabilistic models for images
• But built in a local, modular way

– Learn local effects, get global effects out

S lide credit: William Freeman

Observed evidence Hidden “true states” Neighborhood relations

19‐Apr‐11 40

Lecture 8 -

Fei-Fei Li

MRF Nodes as Pixels

Reconstruction from MRF modeling pixel neighborhood statistics Degraded image Original image

S lide credit: Bastian Leibe

19‐Apr‐11 41

Lecture 8 -

Fei-Fei Li

MRF Nodes as Patches

Image Scene Image patches Scene patches

( , )

i i

x y Φ ( , )

i j

x x Ψ

S lide credit: William Freeman

19‐Apr‐11 42

Lecture 8 -

Fei-Fei Li

Network Joint Probability

,

( , ) ( , ) ( , )

i i i j i i j

P x y x y x x = Φ Ψ

∏ ∏

Scene Image Image-scene compatibility function Scene-scene compatibility function Neighboring scene nodes Local

• bservations

S lide credit: William Freeman

19‐Apr‐11 43

Lecture 8 -

Fei-Fei Li

Energy Formulation

• Joint probability
• Taking the log p(.) turns this into an Energy optimization

problem

• This is similar to free‐energy problems in statistical mechanics

(spin glass theory). We therefore draw the analogy and call E an energy function.

• ϕ and ψ are called potentials.

,

( , ) ( , ) ( , )

i i i j i i j

P x y x y x x = Φ Ψ

∏ ∏

, ,

log ( , ) log ( , ) log ( , ) ( , ) ( , ) ( , )

i i i j i i j i i i j i i j

P x y x y x x E x y x y x x ϕ ψ = Φ + Ψ = +

∑ ∑ ∑ ∑

S lide credit: Bastian Leibe

19‐Apr‐11 44

Lecture 8 -

Fei-Fei Li

Energy Formulation

• Energy function
• Single‐node potentials ϕ

– Encode local information about the given pixel/patch – How likely is a pixel/patch to belong to a certain class (e.g. foreground/background)?

• Pairwise potentials ψ

– Encode neighborhood information – How different is a pixel/patch’s label from that of its neighbor? (e.g. based on intensity/color/texture difference, edges)

Pairwise potentials Single-node potentials

( , )

i i

x y ϕ ( , )

i j

x x ψ

,

( , ) ( , ) ( , )

i i i j i i j

E x y x y x x ϕ ψ = +

∑ ∑

S lide credit: Bastian Leibe

19‐Apr‐11 45

Lecture 8 -

Fei-Fei Li

19‐Apr‐11 46 Slides credit: Carsten Rother

GrabCut: Interactive Foreground Extraction

Lecture 8 -

Fei-Fei Li

Fast & Accurate ?

19‐Apr‐11 47

Goal of GrabCut

[Rother et al, 2004]

Lecture 8 -

Fei-Fei Li

User Input Result

Magic Wand

(Adobe, 2002)

Intelligent Scissors

Mortensen and Barrett (1995)

GrabCut

Regions Boundary Regions & Boundary

19‐Apr‐11 48

What GrabCut Does

Lecture 8 -

Fei-Fei Li

GrabCut

19‐Apr‐11 49

What GrabCut Does

User Input Result

1

( , , )

N

z z = z

• 1

( , , ), {0,1}

N n

α α α = ∈ α

• Pixel values:

Segmentation result:

n

α = ?

n

α = ( , , , ) ( , , , ) ( , ) E U V = + α k θ z α k θ z α z Learn a Gaussian mixture model to minimize: , k θ

Lecture 8 -

Fei-Fei Li

Energy Minimization for Segmentation

( , , , ) ( , , , ) ( , ) E U V = + α k θ z α k θ z α z α: Results; θ: Model; z: Image pixels.

( , , , ) U α k θ z : Gaussian Mixture Models (GMM)

[ ]

1

( , , , ) log ( , ) log ( | , , )

N n n n n n n

U k p z k π α α

=

= − −

∑

α k θ z θ

1

1 log ( , ) log ( , ) 2

N n n n n n

k k π α α

=

⎡ = − + Σ ⎣

∑

1

1 ( ( , )) ( , ) ( ( , )) 2

T n n n n n n n n

z k k z k μ α α μ α

−

⎤ + − Σ − ⎦ { ( , ), ( , ), ( , ), 0,1, 1, , } k k k k K π α μ α α α = Σ = = θ

• mixture weighting

Mean and variance

• f Gaussian dist.

# mixture components

Lecture 8 -

Fei-Fei Li

Energy Minimization for Segmentation

( , ) V α z : Smoothness term

2 ( , )

( , ) 1 exp( )

n m

m n m n

V z z

α α

γ β

≠

= ⋅ − −

∑

α z

( )

1 2 ( , )

2

m n m n

z z β

−

⎛ ⎞ = − ⎜ ⎟ ⎝ ⎠

∑

n m

α α ≠

n m

α α =

( , , , ) ( , , , ) ( , ) E U V = + α k θ z α k θ z α z α: Results; θ: Model; z: Image pixels.

Lecture 8 -

Fei-Fei Li

Gaussian Mixture Model for Colour Distributions

?

GrabCut – Iterative Optimization

19‐Apr‐11 52

Foreground & Background Background

Initialization

,

, argmin ( , , , ) U =

θ k

θ k α k θ z

( , , , ) ( , , , ) ( , ) E U V = + α k θ z α k θ z α z α: Results; θ: Model; z: Image pixels.

• Update k: GMM component assignment;
• Update θ: Gaussian parameter estimation.

Lecture 8 -

Fei-Fei Li

Gaussian Mixture Model for Colour Distributions Graph cuts to infer the segmentation

?

GrabCut – Iterative Optimization

19‐Apr‐11 53

Foreground & Background Background Foreground Background

Initialization

argmin ( , , , ) E =

α

α α k θ z

( , , , ) ( , , , ) ( , ) E U V = + α k θ z α k θ z α z α: Results; θ: Model; z: Image pixels.

• Only update the pixels that are not intialized.
• Max‐flow/min‐cut optimization

Lecture 8 -

Fei-Fei Li

1 2 3 4

Energy after each Iteration Result

19‐Apr‐11 54

GrabCut – Iterative Optimization

Lecture 8 -

Fei-Fei Li

19‐Apr‐11 55

Further Editing

Automatic Segmentation Automatic Segmentation

Lecture 8 -

Fei-Fei Li

… GrabCut completes automatically

19‐Apr‐11 56

More Results

Lecture 8 -

Fei-Fei Li

GrabCut Boykov and Jolly (2001)

Error Rate: 0.72% Error Rate: 1.87% Error Rate: 1.81% Error Rate: 1.32% Error Rate: 1.25% Error Rate: 0.72%

User Input Result

Conventional Graph Cut vs. GrabCut

19‐Apr‐11 57

Lecture 8 -

Fei-Fei Li

GrabCut: live demo

• Included in MS Office 2010 (let’s try it)

19‐Apr‐11 58

Lecture 8 -

Fei-Fei Li

GraphCut Image Synthesis Results

59

Source: Vivek Kwatra

19‐Apr‐11

Lecture 8 -

Fei-Fei Li

Application: Texture Synthesis in the Media

• Currently, still done manually…

S lide credit: Kristen Grauman

19‐Apr‐11 60

Lecture 8 -

Fei-Fei Li

Improving Efficiency of Segmentation

• Problem: Images contain many pixels

– Even with efficient graph cuts, an MRF formulation has too many nodes for interactive results.

• Efficiency trick: Superpixels

– Group together similar‐looking pixels for efficiency of further processing. – Cheap, local oversegmentation – Important to ensure that superpixels do not cross boundaries

• Several different approaches possible

– Superpixel code available here

– http://www.cs.sfu.ca/~mori/research/superpixels/

Image source: Greg Mori

19‐Apr‐11 61

Lecture 8 -

Fei-Fei Li

Superpixels for Pre‐Segmentation

Speedup Graph structure

19‐Apr‐11 62

Lecture 8 -

Fei-Fei Li

Summary: Graph Cuts Segmentation

• Pros

– Powerful technique, based on probabilistic model (MRF). – Applicable for a wide range of problems. – Very efficient algorithms available for vision problems. – Becoming a de‐facto standard for many segmentation tasks.

• Cons/Issues

– Graph cuts can only solve a limited class of models

• Submodular energy functions
• Can capture only part of the expressiveness of MRFs

– Only approximate algorithms available for multi‐label case

63

19‐Apr‐11

Lecture 8 -

Fei-Fei Li

What we have learned today

• Graph theoretic segmentation

– Normalized Cuts – Using texture features

• Segmentation as Energy Minimization

– Markov Random Fields – Graph cuts for image segmentation – Applications

19‐Apr‐11 64