Digital Matting Digital Matting Introduction to Digital Matting - - PDF document

SLIDE 1

1

Digital Matting Digital Matting

Presenting: Presenting: Alon Alon Gamliel Gamliel, Tel , Tel-

• Aviv University, May 2006

Aviv University, May 2006

Outline Outline

1. 1.

Introduction to Digital Matting Introduction to Digital Matting

2. 2.

Bayesian Matting Bayesian Matting

3. 3.

Poisson Matting Poisson Matting

4. 4.

A Closed Form Solution to Matting A Closed Form Solution to Matting

Original Photo Object with new background

Goal: Separate a foreground object from the background

Introduction to Digital Matting Introduction to Digital Matting

Problem: Intricate boundaries Problem: Intricate boundaries – – hair hair strands and fur. strands and fur.

Introduction to Digital Matting Introduction to Digital Matting

Solution: Opacity Solution: Opacity

Introduction to Digital Matting Introduction to Digital Matting

Alpha Matte Image

= * * +

C = α * F + (1- α) * B Compositing equation:

SLIDE 2

2

C = αF + (1- α)B

Solve: Given C, find (F,B,α) for each pixel The Problem: 3 equations, 7 unknowns

CR = α * FR + ( 1 – α ) * BR CG = α * FG + ( 1 – α ) * BG CB = α * FB + ( 1 – α ) * BB

Compositing equation: Blue Screen Matting Chroma Keying, “Blue Screen” Matting

• Known Background

Known Background

• Controlled Scene

Controlled Scene

Statistical Methods

Berman et al. 2000, Berman et al. 2000, Ruzon Ruzon and and Tomasi Tomasi 2000, 2000, Chuang Chuang et al. 2001 et al. 2001

Input Alpha Matte Image Compositing User Supplied Trimap

Calculate P(B) Calculate P(F) Given image Unknown region

Statistical Methods

• 2. solve for F, B,
• 2. solve for F, B, α

α in the unknown region in the unknown region Rely on good separation in color space Rely on good separation in color space

• 1. gather statistical information from user
• 1. gather statistical information from user

trimap trimap

Color Model Acquisition - Summary

Blue Screen Geometric Model Natural Matting Geometric Model Natural Matting Probabilistic Model Natural Matting Bayesian Model

SLIDE 3

3

Bayesian Matting

“ “Video Matting of Complex Scenes Video Matting of Complex Scenes” ”, SIGGRAPH 2002 , SIGGRAPH 2002 “ “A Bayesian Approach to Digital Matting A Bayesian Approach to Digital Matting” ”, CVPR 2001 , CVPR 2001 Chuang Chuang, , Agarwala Agarwala, , Curless Curless, , Salesin Salesin, , Szeliski Szeliski

Bayesian Matting Overview Bayesian Matting Overview

• 1. Start with a user
• 1. Start with a user trimap

trimap

• 2. Solve for boundaries of the unknown region

Estimate F,B,α using probabilistic framework, relying

• n nearest pixels from trimap
• 3. Refine trimap
• 4. Back to (2)
• Find most likely

Find most likely (F, B, (F, B, α α) values for each ) values for each pixel. pixel.

• Apply

Apply Bayes Bayes’ ’ Rule Rule

• P(C) is constant with

P(C) is constant with F,B, F,B, α α

• Use log

Use log-

• likelihood

likelihood

) | , , ( max arg ) , , (

, ,

C B F P B F

B F

α α

α

= ) ( / ) ( ) ( ) ( ) , , | ( max arg

, ,

C P P B P F P B F C P

B F

α α

α

= ) ( ) ( ) ( ) , , | ( max arg

, ,

α α

α

P B P F P B F C P

B F

= ) ( ) ( ) ( ) , , | ( max arg

, ,

α α

α

L B L F L B F C L

B F

+ + + =

Bayesian Matting Bayesian Matting – Details

• Estimating L(

Estimating L(C| C|F,B, F,B,α α) )

This log-likelihood models error in the measurement

• f C and corresponds to a gaussian probability

distribution centered at μ=αF+(1- α)B with standard deviation σC

Estimating Color Correspondence

• The probability that color c agree with

The probability that color c agree with gaussian gaussian color model, parameterized with color model, parameterized with μ μ, , Σ Σ

• Taking the maximum logarithm (in respect to c)

Taking the maximum logarithm (in respect to c) ( ) ( ) ( )

1

1 ( , ) exp 2 2 det( )

T d

c c f c μ μ μ π

−

⎛ ⎞ − Σ − Σ = − ⎜ ⎟ ⎜ ⎟ Σ ⎝ ⎠

( ) ( )

1

( , )

T

L c c c μ μ μ

−

Σ = − − Σ −

Bayesian Matting – Details

• Estimating L(F),L(B)

Estimating L(F),L(B)

• How well estimated F,B

How well estimated F,B correspond to F/B color correspond to F/B color model model

L(F) Nearest known F points (weighted)

SLIDE 4

4

Bayesian Matting – Details

• Eliminating L(

Eliminating L(α α) )

• With no other good assumption, assume L(

With no other good assumption, assume L(α α) is ) is constant for every possible constant for every possible α α

• Solving:

Solving:

• Derive from

Derive from argmax( argmax(L( L(C| C|F,B F,B, ,α α)+L(F)+L(B)) )+L(F)+L(B)) two sets of linear equations two sets of linear equations

• Solve iteratively, alternating the sets, assuming that

Solve iteratively, alternating the sets, assuming that α α

• r
• r F/B are constant each iteration

F/B are constant each iteration

Bayesian Matting – Results

Video Matting Video Matting

• Problem

Problem

• Bayesian matting requires a trimap for each frame

Bayesian matting requires a trimap for each frame

• Drawing

Drawing trimaps trimaps is labor is labor-

• intensive

intensive

• Solution

Solution

• Draw

Draw trimaps trimaps only for key

• nly for key-
• frames

frames

• Interpolate

Interpolate trimaps trimaps between key between key-

• frames

frames

• How to interpolate?

How to interpolate? Optical Flow Optical Flow

Optical Flow Optical Flow

• C(x

C(x) : color at pixel ) : color at pixel coord

• coord. x

. x

• u(x

u(x) : velocity of the pixel at x ) : velocity of the pixel at x – – called called flow field flow field

) ( ) (

1

u x x + =

+ i i

C C

Video Result Video Result Poisson Matting Poisson Matting

“ “Poisson Matting Poisson Matting” ”, SIGGRAPH 2004 , SIGGRAPH 2004 Sun, Sun, Jia Jia, Tang, Shum , Tang, Shum

SLIDE 5

5

Poisson Matting Overview Poisson Matting Overview

• Assumption: The matting gradients mimic the image

Assumption: The matting gradients mimic the image gradients gradients

• 1. Start with a user
• 1. Start with a user trimap

trimap

• 2. Estimate alpha values in unknown area, based on
• 2. Estimate alpha values in unknown area, based on

image gradients and known alpha values image gradients and known alpha values

• 3. Refine
• 3. Refine trimap

trimap

• 4. Back to (2)
• 4. Back to (2)

Initializing Iterative Process Initializing Iterative Process

• Start from a user

Start from a user-

• defined

defined trimap trimap

• For each p in

For each p in Ω Ω, measure F&B from the nearest pixels , measure F&B from the nearest pixels in in Ω ΩF

F &

& Ω ΩB

B

Ω ΩF ΩB

Estimating Matting Field Estimating Matting Field

• Assume smooth foreground & background

Assume smooth foreground & background

B F B F I ∇ − + ∇ + ∇ − = ∇ ) 1 ( ) ( α α α

α α α ∇ − << ∇ − + ∇ ) ( ) 1 ( B F B F

• We get:

We get:

I = I = α αF F + (1 + (1-

• α

α)B )B

I B F ∇ − ≈ ∇ 1 α

Calculating Calculating α

α

• Ω

Ω – – unknown area unknown area

• Ω

ΩF

F –

– trimap trimap foreground area foreground area

• Ω

ΩB

B–

– trimap trimap background area background area

• Solve:

Solve:

• F

Fp

p/B

/Bp

p are taken from nearest pixels at

are taken from nearest pixels at Ω ΩF

F /

/ Ω ΩB

B

Calculating Calculating α

α

• Minimizing is equivalent to solving the Poisson

Minimizing is equivalent to solving the Poisson equation: equation:

• Assuming

Assuming Dirichlet Dirichlet boundary condition boundary condition

• F, B and are measured in grayscale

F, B and are measured in grayscale

• Solve linear system

Solve linear system

⎟ ⎠ ⎞ ⎜ ⎝ ⎛ − ∇ = ∂ ∂ + ∂ ∂ = Δ B F I div y x

2 2 2 2

α α α

I ∇

Refinement Refinement

• Ω

ΩF+

F+ = { p in

= { p in Ω Ω, , s.t s.t. . α

αP

P > 0.95 (I

> 0.95 (IP

P ~ F

~ FP

P) }

) }

• Ω

ΩB+

B+ = { p in

= { p in Ω Ω, , s.t s.t. . α

αP

P < 0.05 (I

< 0.05 (IP

P ~ B

~ BP

P) }

) }

• Generate a new

Generate a new trimap trimap

• Iterate for

Iterate for “ “few few” ” iterations iterations

SLIDE 6

6

Results Results

• Bayesian

Bayesian

• Poisson

Poisson

• Given Image

Given Image

Results Results

Error Effects Error Effects

( ) ( )

D I A B F I B F − ∇ = ∇ − − ∇ − ∇ − = ∇ ) 1 ( 1 α α α

B F B F I ∇ − + ∇ + ∇ − = ∇ ) 1 ( ) ( α α α

α α α ∇ − << ∇ − + ∇ ) ( ) 1 ( B F B F

Calculating Calculating α

α in the local case in the local case

• Ω

Ω – – unknown area unknown area

• Ω

ΩF

F –

– trimap trimap foreground area foreground area

• Ω

ΩB

B–

– trimap trimap background area background area

• Ω

ΩL

L–

– user selected area user selected area

• Solve:

Solve:

B F A − = 1 B F D ∇ − + ∇ = ) 1 ( α α

Calculating Calculating α

α in the local case in the local case

• Assuming

Assuming Dirichlet Dirichlet boundary condition boundary condition

• α

αg

g –

– was calculated at the global case was calculated at the global case

• No local discontinuity can be seen

No local discontinuity can be seen

Results Results Results Results

SLIDE 7

7

Results Results

A Closed Form Solution to Matting

“ “A Closed Form Solution to Natural Image Matting A Closed Form Solution to Natural Image Matting” ”, CVPR 2006 , CVPR 2006 Levin, Levin, Lischinski Lischinski, Weiss , Weiss

Recall Poisson Matting Recall Poisson Matting… …

• Assume smooth foreground & background

Assume smooth foreground & background

B F B F I ∇ − + ∇ + ∇ − = ∇ ) 1 ( ) ( α α α

α α α ∇ − << ∇ − + ∇ ) ( ) 1 ( B F B F

• We get:

We get:

I = I = α αF F + (1 + (1-

• α

α)B )B

I B F ∇ − ≈ ∇ 1 α

Estimating Matting Field Estimating Matting Field

• Assume smooth foreground & background

Assume smooth foreground & background

I = I = α αF F + (1 + (1-

• α

α)B )B For every window W For every window W α αi

i ≈

≈ aI aIi

i + b

+ b for each i in W for each i in W

I1 I2 I3 I4 I5 I6 I7 I8 I9

a = 1/(F a = 1/(F-

• B)

B) b = b = -

• B/(F

B/(F-

• B)

B)

W W

Estimating Matting Field Estimating Matting Field

• Minimize J:

Minimize J:

• ε

ε is a regulation term is a regulation term

Closed Form Solution Closed Form Solution

• It can be proven that:

It can be proven that:

• Where L is

Where L is NxN NxN matrix, and computed directly: matrix, and computed directly:

• δ

δ – – delta function delta function

• μ

μ, , σ σ – – window parameters window parameters

SLIDE 8

8

Moving to Color Images Moving to Color Images

• Derive

Derive a aR

R,

, a aG

G,

, a aB

B, b from :

, b from :

• This relaxes the assumption that foreground &

This relaxes the assumption that foreground & background are smooth background are smooth

For every window W For every window W α αi

i ≈

≈ a aR

RI

IR

R i i +

+ a aG

GI

IG

G i i +

+ a aB

BI

IB

B i i + b

+ b for each i in W for each i in W

I1 I2 I3 I4 I5 I6 I7 I8 I9

W W

F = F = β βF F1

1 + (1

+ (1-

• β

β)F )F2

2

B = B = γ γB B1

1 + (1

+ (1-

• γ

γ)B )B2

2

• L is still

L is still NxN NxN matrix, but computed differently: matrix, but computed differently:

• δ

δ – – delta function delta function

• μ

μ, , Σ Σ– – window parameters window parameters

Moving to Color Images Moving to Color Images Solving for Solving for α α, F, B , F, B

• Solving for

Solving for α α, based on user constraints (scribbles) , based on user constraints (scribbles)

• Can be transformed to

Can be transformed to A Aα

α = B linear problem

= B linear problem

Solving for Solving for α α, F, B , F, B

• To find F,B, solve a set of linear equations

To find F,B, solve a set of linear equations

• Correspondence term (for each pixel)

Correspondence term (for each pixel)

• Smoothness terms (for each pixel)

Smoothness terms (for each pixel)

2

( (1 ) ) F B I α α + − −

2 2 2 2

( , ) ( 1, ) ( , ) ( , 1) (1 ) ( , ) ( 1, ) (1 ) ( , ) ( 1, ) F x y F x y F x y F x y B x y B x y B x y B x y α α α α − + + − + + − − + + − − +

Results Results

Scribbles Bayesian This Scribbles

Results Results

Poisson Scribbles This

SLIDE 9

9

More details More details… …

• We wish to solve:

We wish to solve:

• Which is like:

Which is like:

• Differentiate:

Differentiate:

• D is a diagonal matrix, where

D is a diagonal matrix, where d dii

ii == 1

== 1 iff iff i in S i in S

• B is a vector, where B

B is a vector, where Bi

i==known alpha value

==known alpha value iff iff i in S i in S

• Cholesky

Cholesky Decomposition: Decomposition:

*

arg min( )

T i i

L s i S α α α α ⎧ = ⎨ = ∀ ∈ ⎩

2

arg min ( )

T i i i S

L s α α λ α λ

∈

⎛ ⎞ + − → ∞ ⎜ ⎟ ⎝ ⎠

∑

( )

L D B λ α λ + =

T

RR B α λ =

Future Work (my thesis) Future Work (my thesis)

• Interactive Matting

Interactive Matting

• Incorporate additional user scribbles directly into

Incorporate additional user scribbles directly into A Aα

α = B

= B linear problem linear problem

• Integrating global color model

Integrating global color model

• Statistical analysis of user scribbles

Statistical analysis of user scribbles

• Based on this analysis add more alpha constraints

Based on this analysis add more alpha constraints

• Solving for video

Solving for video

• Goal:

Goal: Keep a minimal user interface framework Keep a minimal user interface framework

• Suggestion:

Suggestion: Feature Feature-

• based expansion of user scribbles

based expansion of user scribbles

• Goal:

Goal: Retrieving coherent results between frames Retrieving coherent results between frames

• Suggestion:

Suggestion: Globally solve for frame pairs Globally solve for frame pairs

• Live presentation

Live presentation… …

Summary Summary

Scribbles Scribbles Trimap Trimap Trimap Trimap User Data User Data ~60 ~60 secs secs ~5 ~5 secs secs ~30 ~30 secs secs Computation Computation Time (640x480) Time (640x480) Direct Direct Iterative Iterative Iterative Iterative Solving Method Solving Method Locally linear Locally linear foreground and foreground and background background Locally smooth Locally smooth foreground and foreground and background background Color separation, Color separation, camera quality camera quality Assumptions Assumptions Image windows Image windows Image gradients Image gradients Local color Local color analysis analysis Hints Hints Closed Form Closed Form Poisson Poisson Bayesian Bayesian