Image Segmentation with a Bounding Box Prior Victor Lempitsky, - PowerPoint PPT Presentation

Image Segmentation with a Bounding Box Prior Victor Lempitsky, Pushmeet Kohli, Carsten Rother, Toby Sharp Microsoft Research Cambridge Dylan Rhodes and Jasper Lin 1

Presentation Overview ● Segmentation problem description ● Background and Previous Work ● Problems and Proposed Solutions ○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution ● Experiments and Results 2

Presentation Overview ● Segmentation problem description ● Background and Previous Work ● Problems and Proposed Solutions ○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution ● Experiments and Results 3

Segmentation Problem How does one separate the foreground from the background with minimal user input? 4

Bounding Box ● Allows the algorithm to focus on subimage ● Desired segmentation is close to sides of bounding box 5

Bounding Box ● Allows the algorithm to focus on subimage ● Desired segmentation is close to sides of bounding box 6

Presentation Overview ● Segmentation problem description ● Background and Previous Work ● Problems and Proposed Solutions ○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution ● Experiments and Results 7

Basic Formulation 8

Basic Formulation B is the set of pixels within the bounding box 9

Basic Formulation E is the set of adjacent pixels within the bounding box 10

Basic Formulation p and q are pixel indices 11

Basic Formulation x_p can take a label of 1 for foreground or 0 for background 12

Basic Formulation Unary potentials encode preference for foreground or background 13

Basic Formulation Pairwise potentials enforce smoothness of the solution 14

Related Work ● Nowozin and Lampert derived framework for segmentation under connectivity constraint ● Relax NP-hard integer problem and solve resulting LP 15

Nowozin and Lampert 16

Nowozin and Lampert 17

Nowozin and Lampert 18

Presentation Overview ● Segmentation problem description ● Background and Previous Work ● Problems and Proposed Solutions ○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution ● Experiments and Results 19

Presentation Overview ● Segmentation problem description ● Background and Previous Work ● Problems and Proposed Solutions ○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution ● Experiments and Results 20

Why tightness? 21

Tightness Definition 22

Corollary A shape x is strongly tight if and only if its intersection with the middle box has a connected component touching all four sides of the middle box 23

Energy Minimization Problem 24

Presentation Overview ● Segmentation problem description ● Background and Previous Work ● Problems and Proposed Solutions ○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution ● Experiments and Results 25

Energy Minimization Problem 26

Energy Minimization Problem 27

Convex Continuous Relaxation 28

Convex Continuous Relaxation 29

Continuous Optimization 30

Continuous Optimization 31

Additional Approximation Intuition: Solve LP with a subset Γ' of the constraints in 3c activated 32

Calculating Γ' 1. Begin with Γ' = ∅ 2. Solve the LP 3. Pick a group of crossing paths from Γ \ Γ' which are violated by more than a small tolerance 4. Add these paths to Γ' 5. Repeat steps 2 through 4 until all paths in Γ are satisfied within the tolerance 33

Final Form 34

Presentation Overview ● Segmentation problem description ● Background and Previous Work ● Problems and Proposed Solutions ○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution ● Experiments and Results 35

Pinpointing Algorithm Normally, output of LP is rounded to integer solution 36

Pinpointing Algorithm ● Pinpoint set Π contains pixels hard-assigned to foreground 37

Pinpoint Algorithm 38

Pinpoint Algorithm 39

Challenges ● Existing methods perform energy-driven shrinking over bounding box ○ No guarantees optimization won’t shrink excessively ○ Stuck at poor local minima ○ Discretization of approximate solution is noisy 40

Paper’s Contributions ● Common methods initialize foreground region and perform energy-driven shrinking ○ No guarantees optimization won’t shrink excessively Solution: new tightness prior ○ Stuck at poor local minima ○ Discretization of approximate solution is noisy 41

Paper’s Contributions ● Common methods initialize foreground region and perform energy-driven shrinking ○ No guarantees optimization won’t shrink excessively Solution: new tightness prior ○ Stuck at poor local minima Solution: new approximation strategies ○ Discretization of approximate solution is noisy 42

Paper’s Contributions ● Common methods initialize foreground region and perform energy-driven shrinking ○ No guarantees optimization won’t shrink excessively Solution: new tightness prior ○ Stuck at poor local minima Solution: new approximation strategy ○ Discretization of approximate solution is noisy Solution: new pinpointing algorithm 43

Presentation Overview ● Segmentation problem description ● Background and Previous Work ● Problems and Proposed Solutions ○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution ● Experiments and Results 44

Experiments ● Evaluated over 50 image GrabCut dataset ○ Each image comes with bounding box ● Comparison with competing methods and initialization strategies 45

GrabCut Dataset ● 50 natural images with bounding box annotations ○ Includes background, outside strip, and foreground bounding boxes 46

GrabCut Dataset ● 50 natural images with bounding box annotations ○ Includes background, outside strip, and foreground bounding boxes 47

Unary and Pairwise Terms ● Pairwise terms over 8-connected edge set 48

Relative Performance Error rate - mislabeled pixels inside bounding box Optimum Rank - average rank of energy of final integer program solutions 49

Relative Performance 50

Iterative Process ● Compare the following algorithms on the segmentation task: ○ GrabCut with standard graph cut minimization for all segmentation steps ○ GrabCut which enforces the tightness prior for all segmentation steps ● 5 iterations each 51

Initialization Strategies ● Compare two methods for initializing foreground/background GMMs: ○ InitThirds = same as Experiment 1 (outside strip + best matches vs. poor matches) ○ InitFullBox which sets background GMM to outside strip and foreground to whole interior of bounding box 52

Iterative Process 53

Effect of Margin Thickness Error rates as function of margin thickness 54

Strong vs. Weak Tightness ● Strong and weak tightness lead to similar error rates in general ○ same error rate (3.7%) for best model (GrabCut- Pinpoint/InitThirds) 55

Iterative Process Comparisons 56

Conclusions ● New bounding-box based prior for interactive image segmentation 57

Conclusions ● New bounding-box based prior for interactive image segmentation ● Demonstrated segmentation tasks under this prior can be formulated as integer programs 58

Conclusions ● New bounding-box based prior for interactive image segmentation ● Demonstrated segmentation tasks under this prior can be formulated as integer programs ● Developed new optimization approaches for approximate solution of these NP-hard problems ○ Can be applied to other computer vision problems e.g. other image segmentation or silhouettes in multi-view stereo 59

Thank you! 60