Efficient Inference in Fully Connected CRFs with Gaussian Edge - PowerPoint PPT Presentation

Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials Philipp Kr¨ ahenb¨ uhl Vladlen Koltun philkr@stanford.edu vladlen@stanford.edu Department of Computer Science, Stanford University December 14, 2011

Multi-class image segmentation Assign a class label to each pixel in the image background table chair P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 2 / 29

CRF models in multi-class image segmentation � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � j ∈N i i i unary term pairwise term MAP inference in conditional random field Unary term ◮ From classifier ◮ TextonBoost [Shotton et al. 09] Pairwise term ◮ Consistent labeling P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 3 / 29

CRF models in multi-class image segmentation � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � j ∈N i i i unary term pairwise term MAP inference in conditional random field Unary term ◮ From classifier ◮ TextonBoost [Shotton et al. 09] Pairwise term ◮ Consistent labeling P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 3 / 29

CRF models in multi-class image segmentation � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � j ∈N i i i unary term pairwise term MAP inference in conditional random field Unary term ◮ From classifier ◮ TextonBoost [Shotton et al. 09] Pairwise term ◮ Consistent labeling P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 3 / 29

Adjacency CRF models � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � j ∈N i i i unary term pairwise term Pairwise term ◮ Neighboring pixels ◮ Color-sensitive Potts model � � � � −| I i − I j | 2 w (1) exp + w (2) ψ p ( x i , x j ) = 1 [ x i � = x j ] 2 θ 2 β P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 4 / 29

Adjacency CRF models � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � j ∈N i i i unary term pairwise term grid crf Efficient inference ◮ 1 second for 50 ′ 000 variables Limited expressive power sky Only local interactions Excessive smoothing of object boundaries tree ◮ Shrinking bias grass P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 5 / 29

Adjacency CRF models � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � j ∈N i i i unary term pairwise term grid crf Efficient inference ◮ 1 second for 50 ′ 000 variables Limited expressive power sky Only local interactions Excessive smoothing of object boundaries tree ◮ Shrinking bias grass P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 5 / 29

Adjacency CRF models � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � j ∈N i i i unary term pairwise term Efficient inference ◮ 1 second for 50 ′ 000 variables Limited expressive power Only local interactions Excessive smoothing of object boundaries ◮ Shrinking bias P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 5 / 29

Adjacency CRF models � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � j ∈N i i i unary term pairwise term grid crf Efficient inference ◮ 1 second for 50 ′ 000 variables Limited expressive power sky Only local interactions Excessive smoothing of object boundaries tree ◮ Shrinking bias grass P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 5 / 29

Fully connected CRF � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � i i j > i unary term pairwise term Every node is connected to every other node ◮ Connections weighted differently P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 6 / 29

Fully connected CRF � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � i i j > i unary term pairwise term Every node is connected to every other node ◮ Connections weighted differently P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 6 / 29

Fully connected CRF � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � i i j > i unary term pairwise term fully connected sky Long-range interactions No more shrinking bias tree grass P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 7 / 29

Fully connected CRF � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � i i j > i unary term pairwise term Long-range interactions No more shrinking bias P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 7 / 29

Fully connected CRF � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � i i j > i unary term pairwise term fully connected sky Long-range interactions No more shrinking bias tree grass P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 7 / 29

Fully connected CRF � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � i i j > i unary term pairwise term Region-based [Rabinovich et al. 07, fully connected Galleguillos et al. 08, Toyoda & Hasegawa 08, Payet & Todorovic 10] sky ◮ Tractable up to hundreds of variables Pixel-based ◮ Tens of thousands of variables ⋆ Billions of edges tree ◮ Computationally expensive grass P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 8 / 29

Fully connected CRF � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � i i j > i unary term pairwise term Region-based [Rabinovich et al. 07, fully connected Galleguillos et al. 08, Toyoda & Hasegawa 08, Payet & Todorovic 10] sky ◮ Tractable up to hundreds of variables Pixel-based ◮ Tens of thousands of variables ⋆ Billions of edges tree ◮ Computationally expensive grass P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 8 / 29

Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials Inference in 0 . 2 seconds ◮ 50 ′ 000 variables ◮ MCMC inference: 36 hrs fully connected Pairwise potentials: linear combinations of bench Gaussians tree road grass P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 9 / 29

Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials Inference in 0 . 2 seconds ◮ 50 ′ 000 variables ◮ MCMC inference: 36 hrs fully connected Pairwise potentials: linear combinations of bench Gaussians tree road grass P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 9 / 29

Model definition � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � i i j > i unary term pairwise term Gaussian edge potentials K � w ( m ) k ( m ) ( f i , f j ) ψ p ( x i , x j ) = µ ( x i , x j ) m =1 Label compatibility function µ Linear combination of Gaussian kernels k ( m ) ( f i , f j ) = exp( − 1 2( f i − f j )Σ ( m ) ( f i − f j )) Arbitrary feature space f i P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 10 / 29

Model definition � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � i i j > i unary term pairwise term Gaussian edge potentials K � w ( m ) k ( m ) ( f i , f j ) ψ p ( x i , x j ) = µ ( x i , x j ) m =1 Label compatibility function µ Linear combination of Gaussian kernels k ( m ) ( f i , f j ) = exp( − 1 2( f i − f j )Σ ( m ) ( f i − f j )) Arbitrary feature space f i P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 10 / 29

Model definition � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � i i j > i unary term pairwise term Gaussian edge potentials K � w ( m ) k ( m ) ( f i , f j ) ψ p ( x i , x j ) = µ ( x i , x j ) m =1 Label compatibility function µ Linear combination of Gaussian kernels k ( m ) ( f i , f j ) = exp( − 1 2( f i − f j )Σ ( m ) ( f i − f j )) Arbitrary feature space f i P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 10 / 29

Model definition � � � E ( x ) = ψ u ( x i ) + ψ p ( x i , x j ) � �� � � �� � i i j > i unary term pairwise term Gaussian edge potentials K � w ( m ) k ( m ) ( f i , f j ) ψ p ( x i , x j ) = µ ( x i , x j ) m =1 Label compatibility function µ Linear combination of Gaussian kernels k ( m ) ( f i , f j ) = exp( − 1 2( f i − f j )Σ ( m ) ( f i − f j )) Arbitrary feature space f i P. Kr¨ ahenb¨ uhl (Stanford) Efficient Inference in Fully Connected CRFs December 2011 10 / 29