S I GGRAPH 2004 I nte rac tive I mag e Cuto ut Se pa ra te a - - PDF document

5/22/2006 1

S I GGRAPH 2004

L azy Snapping Gr abCut

Yin Li Jian Sun Chi-Keung Tang Heung-Yeung Shum

Microsoft Research Asia Microsoft Research Asia Hong Kong University Hong Kong University

Carsten Rother Vladimir Kolmogorov Andrew Blake

Microsoft Research Cambridge Microsoft Research Cambridge-

• UK

UK

I nte rac tive I mag e Cuto ut

Se pa ra te a n o b je c t fro m its b a c kg ro und Co mpo se the o b je c t o n a no the r ima g e

Image

I nte rac tive Graph Cut

(Bo yko v & Jo lly. I CCV’ 01) Optimize d b y s-t min-c ut a lg o rithm

and background and background Draw foreground Draw foreground Graph Cut Segmentation Graph Cut Segmentation

I nte rac tive Graph Cut

(Bo yko v & Jo lly. I CCV’ 01)

I nte rac tive Graph Cut

(Bo yko v e t a l. I CCV’ 01)

Hard Co nstraints

" " " " bkg x B i

• bj

x O i

i i

= ∈ ∀ = ∈ ∀ X : Se g me nta tio n.

Ha rd Co nstra int:

} " " , " {" bkg

• bj

xi ∈

5/22/2006 2

S

• ft Co nstraints

Minimize the E

ne rg y:

E1 : Re g io n: Co lo r diffe re nc e to use r ma rks E2 : Bo unda ry: Co lo r simila rity b e twe e n pixe ls

∑ ∑

≠ ∈ ∈

+ =

j i

x x E j i j i V i i

x x E x E X E

, 2 1

) , ( ) ( ) ( λ

I mag e as a We ig hte d Graph

Image Image Min Cut Min Cut

Graph: source & sink, n-links & t-links Cut=Segmentation: Separate ‘source‘ & ‘sink‘ Energy of cut: sum wieghts of edges Min-Cut Max-Flow: Global minimal enegry in polynomial time

Foreground Foreground (source S) (source S) Background Background (sink T) (sink T)

We ig hts

: } , { : } , { S i T i B i ∞ ⇒ ∈ ) ( ) (

1 i x i

I h x E U i

i

= ⇒ ∈

) ) I – (I

• exp(

x x E

2 2 1 j i

∝ ) , (

2

t-links n-links

Min Cut = Minimize Soft Constr aints ke e ping Har d Constr aints

L azy S napping

L i e t al. S I GGRAPH’ 04

L azy S napping

• L

a zy Sna pping fo r L a zy Use rs

• 2 Ste ps UI

:

• 1. Co a rse Ste p:

Ob j/ Bkg Ma rking => Gra ph Cut

L azy S napping

• 2. F

ine Ste p:

a . Bo rde r Brush b . Pixe l E diting

=> Gra ph-Cut

• n b o rde r

5/22/2006 3

We ig hts

E1 : Co lo r diffe re nc e to use r ma rks

Inte nsitie s -> Co lo rs Histo g ra m -> “K

• me a ns” c luste ring

E2 : Co lo r simila rity b e twe e n pixe ls

F

• r ne ig hb o ring pixe ls o f diffe re nt xi

c e ntro id c luste r c lo se st to dist RGB_ ) " " (

1

∝ = obj x E

i

C

• C

1 1 ) , (

2 j i 2

+ =

j i x

x E

Pe r-Pix Graph Cut Pre -S e g me ntatio n Graph Cut o n Re g io ns Graph Cut o n Re g io ns Graph Cut o n Re g io ns

5/22/2006 4 Graph Cut Alg o rithm

Re g io n c o lo r diffe re nc e Co lo r diffe re nc e Re g io n me a n c o lo r Pixe l c o lo r Re g io n c o nne c tio n Ne ig hb o rs Sma ll re g io ns Pixe ls Re g io n b a se d me tho d Pe r-pixe l me tho d

Adva nta g e s

Mo re tha n 10 time s fe we r no de s I

nsta nt fe e db a c k o f c uto ut re sult

Pre -pro c e ssing o ve rhe a d

2~3 se c o nds b a c kg ro und pro c e ssing

Re g io n-base d Graph Cut

Divide and Co nque r

Input Input Image Image Coarse Coarse Boundary Boundary Refined Refined Boundary Boundary

First Step: Object Marking Second Steps: Boundary Editing

Quickly identify the object Quickly identify the object Control the detail boundary Control the detail boundary

Po lyg o n F itting

F

irst ve rte x – b o rde r pixe l with hig he st c urvature

Ne xt ve rtic e s: furthe st b o unda ry pixe l Sto p whe n dista nc e < thre sh

Bo rde r E diting

Brush - Re pla c e po lyg o n se g me nt Ve rte x E

diting : Mo ve / Ad d/ De le te => Gra ph Cut o n b o rde r pixe ls

Band o f Unc e rtainty Optimizatio n in the Band

Pixel Based Graph Cut Segmentation

5/22/2006 5

E dit the Po lyg o n Ve rtic e s E dit the Po lyg o n Ve rtic e s L

• w Co ntrast E

xample Bo undary E diting Bo undary E diting

F

• r L
• w Co ntra st c a se :

I

n E2 - Add a te rm to re fle c t dista nc e fro m po lyg o n

Ha rd Ve rte x c o nstra int

Adjust g ra ph so c ut pa sse s thro ug h ve rte x

Vide o De mo (L e ft boy)

5/22/2006 6

Vide o De mo (R ight Boy)

S ummary: T wo S te ps

Pre Pre-

• Segmentation

Segmentation Input Input Image Image Small Small Regions Regions Coarse Coarse Boundary Boundary Editable Editable Polygon Polygon Refined Refined Boundary Boundary Region Based Region Based Graph Cut Graph Cut Polygon Fitting Polygon Fitting Band Pixels Band Pixels Graph Cut Graph Cut

First Step: Object Marking Second Steps: Boundary Editing

GrabCut GrabCut

I nte rac tive F

• re gro und E

xtrac tio n I nte rac tive F

• re gro und E

xtrac tio n using I te rate d Graph Cuts using I te rate d Graph Cuts

Pho to mo ntag e Pho to mo ntag e I te rate d Graph Cut I te rate d Graph Cut

GMM estimation for learning colour distributions Graph cuts to infer the segmentation

?

User Initialization

Gaussian Mixture Mo de ls ( Gaussian Mixture Mo de ls (GMMs GMMs) )

GMM inste a d o f Histo g ra m (Co lo r mo de l) Assume distrib utio n is a mixture o f Ga ussia ns E M a lg o rithm – find b e st fo r the g ive n se t o f sa mple s Gra b Cut – Diffe re nt a ppro a c h

1 ) ( ) (

, ,

= = −

∑ ∑

Σ = Σ k K 1 k k

w x G w GMM(x) Gaussian x G

k k

μ μ k k k

w Σ , ,μ

5/22/2006 7

I te rate d Graph Cuts I te rate d Graph Cuts

• E1 – GMMs (E2 – No c ha ng e )
• Alg o rithm:

1. I nitia lize I nitia lize GMMs 2. Re pe a t (until c o nsta nt e ne rg y)

a . a ssig n b e st Gk => 2K c luste rs b . F

• r e a c h c luste r c a lc ula te => 2 GMMs

c . F ind Min Cut => U d e c re a se s

3. Apply b o rde r ma tting 4. E na b le use r e diting & re pe a t

φ = = F B U B , ,

k k k

w Σ , ,μ

U p∈ ∀

k k k

w Σ , ,μ

I nc o mple te L abe ling

Use r spe c ifie s b o rde r => F po pula te s thro ug h ite ra tio ns So me F pixe ls c a n b e re tra c te d. B c a nno t

E diting (I

n c a se o f e rro r):

Use r a dds F, B (b rush) Re -c o mpute Gra ph Cut c a n b e re use d. φ = = F B U B , ,

1 2 3 4

I te rate d Graph Cuts I te rate d Graph Cuts

Energy after each Iteration Result Guaranteed to converge

Gaussian S e paratio n Gaussian S e paratio n

Ga ussia n Mixture Mo de l (typic a lly K =5)

Foreground & Background Background Foreground Background

G R G R

Iterated graph cut

Mo de rate ly straig htfo rward Mo de rate ly straig htfo rward e xample s e xample s Diffic ult E xample s Diffic ult E xample s

Camouflage & L

• w Contr

ast No te le pathy F ine str uc tur e Initial Rectangle Initial Result

5/22/2006 8

E valuatio n E valuatio n – – L abe lle d L abe lle d Database Database Co mpariso n Co mpariso n

GrabCut Boykov and Jolly (2001)

Error Rate: 0.72% Error Rate: 1.87% Error Rate: 1.81% Error Rate: 1.32% Error Rate: 1.25% Error Rate: 0.72%

User Input Result

Bo rde r Matting Bo rde r Matting

Hard Segmentation Band of Uncertainty Soft Segmentation

E xtra c t α-va lue s a lo ng b o rde r

F B

Baye s Matting - Chuang e t. al. (2001)

Cre a te U b a nd L

• c a l re c ta ng le

E

stima te GF , GB

• F

ind α tha t ma ximize s GU with re spe c t to pixe ls in U U w ±

) G( G U

U B F α α α

μ α μ α μ α μ Σ = − + = , ) ( ) 1 ( : α

Noisy alpha-profile

Bo rde r Matting Bo rde r Matting -

• GrabCut

GrabCut

1 Foreground Mix Back- ground Foreground Mix Background

σ Δ

Fit a smooth alpha-profile with parameters

σ , Δ

Result using DP Border Matting

DP

t

Dynamic Pro g ramming Dynamic Pro g ramming

Noisy alpha-profile Regularisation

t+1

∑

= − −

− + Δ − Δ Σ

T t t t t t

Min G Max

1 2 1 2 1

) ( ) ( : ) , ( : σ σ μ

α α

5/22/2006 9

Input Bayes Matting (no regularization) GrabCut (with regularization)

S ummary S ummary

• sho uld ma tc h U pixe ls

α sho uld c ha ng e like a so ft ste p func tio n Ste p func tio n sho uld c ha ng e smo o thly

a lo ng c o nto ur

) (α

U

G

Matting Re sults Matting Re sults L azy S napping vs. Grab Cut L azy S napping vs. Grab Cut

F ast F ully inte r a c tive Inc lude s Pr e - Pr

• c e ssing

Pe r for manc e Ite r ative Gr aph Cut R e gion- ba se d Gr a ph Cut Bor de r pixe l Graph Cut Algor ithm R e c tangle / lasso – BG only Mar king br ush - [optiona l] Mar king br ush – F G + BG Ove r riding br ush Ve r te x e diting Use r Inte r fac e Bor de r Matting Bor de r E diting Bor de r Gr a bCut L azy Sna pping

Thank You