Joint Optimization of Segmentation and Appearance Models David Mandle, Sameep Tandon April 29, 2013 David Mandle, Sameep Tandon (Stanford) April 29, 2013 1 / 19
Overview 1 Recap: Image Segmentation 2 Optimization Strategy 3 Experimental David Mandle, Sameep Tandon (Stanford) April 29, 2013 2 / 19
Recap: Image Segmentation Problem Segment an image into foreground and background Figure: Left: Input image. Middle: Segmentation by EM (GrabCut). Right: Segmentation by the method covered today David Mandle, Sameep Tandon (Stanford) April 29, 2013 3 / 19
Recap: Image Segmentation as Energy Optimization Recall Grid Structured Markov Random Field: Latent variables x i ∈ { 0 , 1 } corresponding to foreground/background Observations z i . Take to be RGB pixel values Edge potentials Φ( x i , z i ), Ψ( x i , x j ) David Mandle, Sameep Tandon (Stanford) April 29, 2013 4 / 19
Recap: Image Segmentation as Energy Optimization The Graphical Model encodes the following (unnormalized) probability distribution: David Mandle, Sameep Tandon (Stanford) April 29, 2013 5 / 19
Recap: Image Segmentation as Energy Optimization Goal: find x to maximize P ( x , z ) ( z is observed) David Mandle, Sameep Tandon (Stanford) April 29, 2013 6 / 19
Recap: Image Segmentation as Energy Optimization Goal: find x to maximize P ( x , z ) ( z is observed) Taking logs: � � E ( x , z ) = φ ( x i , z i ) + ψ ( x i , x j ) i i , j David Mandle, Sameep Tandon (Stanford) April 29, 2013 6 / 19
Recap: Image Segmentation as Energy Optimization Goal: find x to maximize P ( x , z ) ( z is observed) Taking logs: � � E ( x , z ) = φ ( x i , z i ) + ψ ( x i , x j ) i i , j Unary potential φ ( x i , z i ) encodes how likely it is for a pixel or patch y i to belong to segmentation x i . David Mandle, Sameep Tandon (Stanford) April 29, 2013 6 / 19
Recap: Image Segmentation as Energy Optimization Goal: find x to maximize P ( x , z ) ( z is observed) Taking logs: � � E ( x , z ) = φ ( x i , z i ) + ψ ( x i , x j ) i i , j Unary potential φ ( x i , z i ) encodes how likely it is for a pixel or patch y i to belong to segmentation x i . Pairwise potential ψ ( x i , x j ) encodes neighborhood info about pixel/patch segmentation labels David Mandle, Sameep Tandon (Stanford) April 29, 2013 6 / 19
Recap: GrabCut Model Unary Potentials: log of Gaussian Mixture Model ◮ But to deal with tractability, we assign each x i to component k i φ ( x i , k i , θ | z i ) = − log π ( x i , k i ) + log N ( z i ; µ ( k i ) , Σ( k i )) David Mandle, Sameep Tandon (Stanford) April 29, 2013 7 / 19
Recap: GrabCut Model Unary Potentials: log of Gaussian Mixture Model ◮ But to deal with tractability, we assign each x i to component k i φ ( x i , k i , θ | z i ) = − log π ( x i , k i ) + log N ( z i ; µ ( k i ) , Σ( k i )) Pairwise Potentials: ψ ( x i , x j | z i , z j ) = [ x i � = x j ] exp( − β − 1 � z i − z j � 2 ) where β = 2 · avg( � z i − z j � 2 ) David Mandle, Sameep Tandon (Stanford) April 29, 2013 7 / 19
Recap: GrabCut Optimization Strategy GrabCut EM Algorithm 1 Initialize Mixture Models David Mandle, Sameep Tandon (Stanford) April 29, 2013 8 / 19
Recap: GrabCut Optimization Strategy GrabCut EM Algorithm 1 Initialize Mixture Models 2 Assign GMM components: k i = arg min k φ ( x i , k i , θ | z i ) David Mandle, Sameep Tandon (Stanford) April 29, 2013 8 / 19
Recap: GrabCut Optimization Strategy GrabCut EM Algorithm 1 Initialize Mixture Models 2 Assign GMM components: k i = arg min k φ ( x i , k i , θ | z i ) 3 Get GMM parameters: � θ = arg min φ ( x i , k i , θ | z i ) θ i David Mandle, Sameep Tandon (Stanford) April 29, 2013 8 / 19
Recap: GrabCut Optimization Strategy GrabCut EM Algorithm 1 Initialize Mixture Models 2 Assign GMM components: k i = arg min k φ ( x i , k i , θ | z i ) 3 Get GMM parameters: � θ = arg min φ ( x i , k i , θ | z i ) θ i 4 Perform segmentation using reduction to min-cut: x = arg min x E ( x , z ; k , θ ) David Mandle, Sameep Tandon (Stanford) April 29, 2013 8 / 19
Recap: GrabCut Optimization Strategy GrabCut EM Algorithm 1 Initialize Mixture Models 2 Assign GMM components: k i = arg min k φ ( x i , k i , θ | z i ) 3 Get GMM parameters: � θ = arg min φ ( x i , k i , θ | z i ) θ i 4 Perform segmentation using reduction to min-cut: x = arg min x E ( x , z ; k , θ ) 5 Iterate from step 2 until converged David Mandle, Sameep Tandon (Stanford) April 29, 2013 8 / 19
New Model Let’s consider a simpler model. This will be useful soon Unary terms: Histograms David Mandle, Sameep Tandon (Stanford) April 29, 2013 9 / 19
New Model Let’s consider a simpler model. This will be useful soon Unary terms: Histograms ◮ K bins, b i is bin of pixel z i David Mandle, Sameep Tandon (Stanford) April 29, 2013 9 / 19
New Model Let’s consider a simpler model. This will be useful soon Unary terms: Histograms ◮ K bins, b i is bin of pixel z i ◮ θ 0 , θ 1 ∈ [0 , 1] K represent color models (distributions) over foreground/background David Mandle, Sameep Tandon (Stanford) April 29, 2013 9 / 19
New Model Let’s consider a simpler model. This will be useful soon Unary terms: Histograms ◮ K bins, b i is bin of pixel z i ◮ θ 0 , θ 1 ∈ [0 , 1] K represent color models (distributions) over foreground/background ◮ φ ( x i , b i , θ ) = − log θ x i b i David Mandle, Sameep Tandon (Stanford) April 29, 2013 9 / 19
New Model Let’s consider a simpler model. This will be useful soon Unary terms: Histograms ◮ K bins, b i is bin of pixel z i ◮ θ 0 , θ 1 ∈ [0 , 1] K represent color models (distributions) over foreground/background ◮ φ ( x i , b i , θ ) = − log θ x i b i Pairwise Potentials ψ ( x i , x j ) = w ij | x i − x j | We will define w ij later; for now, consider pairwise equal to Grabcut David Mandle, Sameep Tandon (Stanford) April 29, 2013 9 / 19
New Model Let’s consider a simpler model. This will be useful soon Unary terms: Histograms ◮ K bins, b i is bin of pixel z i ◮ θ 0 , θ 1 ∈ [0 , 1] K represent color models (distributions) over foreground/background ◮ φ ( x i , b i , θ ) = − log θ x i b i Pairwise Potentials ψ ( x i , x j ) = w ij | x i − x j | We will define w ij later; for now, consider pairwise equal to Grabcut Total Energy: � � E ( x , θ 0 , θ 1 ) = − log P ( z p | θ x p ) + w pq | x p − x q | p ∈ V ( p , q ) ∈ N P ( z p | θ x p ) = θ x p b p David Mandle, Sameep Tandon (Stanford) April 29, 2013 9 / 19
EM under new model 1 Initialize histograms θ 0 , θ 1 . David Mandle, Sameep Tandon (Stanford) April 29, 2013 10 / 19
EM under new model 1 Initialize histograms θ 0 , θ 1 . 2 Fix θ . Perform segmentation using reduction to min-cut: x E ( x , θ 0 , θ 1 ) x = arg min David Mandle, Sameep Tandon (Stanford) April 29, 2013 10 / 19
EM under new model 1 Initialize histograms θ 0 , θ 1 . 2 Fix θ . Perform segmentation using reduction to min-cut: x E ( x , θ 0 , θ 1 ) x = arg min 3 Fix x. Compute θ 0 , θ 1 (via standard parameter fitting). David Mandle, Sameep Tandon (Stanford) April 29, 2013 10 / 19
EM under new model 1 Initialize histograms θ 0 , θ 1 . 2 Fix θ . Perform segmentation using reduction to min-cut: x E ( x , θ 0 , θ 1 ) x = arg min 3 Fix x. Compute θ 0 , θ 1 (via standard parameter fitting). 4 Iterate from step 2 until converged David Mandle, Sameep Tandon (Stanford) April 29, 2013 10 / 19
Optimization Goal: Minimize Energy min x E ( x ) � � h k ( n 1 + h ( n 1 ) E ( x ) = k ) + w pq | x p − x q | � �� � k ( p , q ) ∈ N E 2 ( x ) � �� � E 1 ( x ) k = � p ∈ V k x p and n 1 = � where n 1 p ∈ V x p David Mandle, Sameep Tandon (Stanford) April 29, 2013 11 / 19
Optimization Goal: Minimize Energy min x E ( x ) � � h k ( n 1 + h ( n 1 ) E ( x ) = k ) + w pq | x p − x q | � �� � k ( p , q ) ∈ N E 2 ( x ) � �� � E 1 ( x ) k = � p ∈ V k x p and n 1 = � where n 1 p ∈ V x p This is hard! David Mandle, Sameep Tandon (Stanford) April 29, 2013 11 / 19
Optimization Goal: Minimize Energy min x E ( x ) � � h k ( n 1 + h ( n 1 ) E ( x ) = k ) + w pq | x p − x q | � �� � k ( p , q ) ∈ N E 2 ( x ) � �� � E 1 ( x ) k = � p ∈ V k x p and n 1 = � where n 1 p ∈ V x p This is hard! But efficient strategies for optimizing E 1 ( x ) and E 2 ( x ) separately David Mandle, Sameep Tandon (Stanford) April 29, 2013 11 / 19
Optimization via Dual Decomposition Consider an optimization of the form min x f 1 ( x ) + f 2 ( x ) David Mandle, Sameep Tandon (Stanford) April 29, 2013 12 / 19
Optimization via Dual Decomposition Consider an optimization of the form min x f 1 ( x ) + f 2 ( x ) where optimizing f ( x ) = f 1 ( x ) + f 2 ( x ) is hard David Mandle, Sameep Tandon (Stanford) April 29, 2013 12 / 19
Optimization via Dual Decomposition Consider an optimization of the form min x f 1 ( x ) + f 2 ( x ) where optimizing f ( x ) = f 1 ( x ) + f 2 ( x ) is hard But min x f 1 ( x ) and min x f 2 ( x ) are easy problems David Mandle, Sameep Tandon (Stanford) April 29, 2013 12 / 19
Recommend
More recommend
