Image Segmentation with a Bounding Box Prior Victor Lempitsky, - - PowerPoint PPT Presentation

▶
image segmentation with a bounding box prior
SMART_READER_LITE
LIVE PREVIEW

Image Segmentation with a Bounding Box Prior Victor Lempitsky, - - PowerPoint PPT Presentation

Feb 18, 2024 454 likes •1.07k views

Image Segmentation with a Bounding Box Prior Victor Lempitsky, Pushmeet Kohli, Carsten Rother, Toby Sharp Microsoft Research Cambridge Dylan Rhodes and Jasper Lin 1 Presentation Overview Segmentation problem description Background

slide-1
SLIDE 1

Image Segmentation with a Bounding Box Prior

Victor Lempitsky, Pushmeet Kohli, Carsten Rother, Toby Sharp Microsoft Research Cambridge

Dylan Rhodes and Jasper Lin

1

slide-2
SLIDE 2

Presentation Overview

  • Segmentation problem description
  • Background and Previous Work
  • Problems and Proposed Solutions

○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution

  • Experiments and Results

2

slide-3
SLIDE 3

Presentation Overview

  • Segmentation problem description
  • Background and Previous Work
  • Problems and Proposed Solutions

○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution

  • Experiments and Results

3

slide-4
SLIDE 4

Segmentation Problem

How does one separate the foreground from the background with minimal user input?

4

slide-5
SLIDE 5

Bounding Box

  • Allows the algorithm to focus on subimage
  • Desired segmentation is close to sides of bounding

box

5

slide-6
SLIDE 6

Bounding Box

  • Allows the algorithm to focus on subimage
  • Desired segmentation is close to sides of bounding

box

6

slide-7
SLIDE 7

Presentation Overview

  • Segmentation problem description
  • Background and Previous Work
  • Problems and Proposed Solutions

○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution

  • Experiments and Results

7

slide-8
SLIDE 8

Basic Formulation

8

slide-9
SLIDE 9

Basic Formulation

B is the set of pixels within the bounding box

9

slide-10
SLIDE 10

Basic Formulation

E is the set of adjacent pixels within the bounding box

10

slide-11
SLIDE 11

Basic Formulation

p and q are pixel indices

11

slide-12
SLIDE 12

Basic Formulation

x_p can take a label of 1 for foreground or 0 for background

12

slide-13
SLIDE 13

Basic Formulation

Unary potentials encode preference for foreground or background

13

slide-14
SLIDE 14

Basic Formulation

Pairwise potentials enforce smoothness of the solution

14

slide-15
SLIDE 15

Related Work

  • Nowozin and Lampert derived framework for

segmentation under connectivity constraint

  • Relax NP-hard integer problem and solve

resulting LP

15

slide-16
SLIDE 16

Nowozin and Lampert

16

slide-17
SLIDE 17

Nowozin and Lampert

17

slide-18
SLIDE 18

Nowozin and Lampert

18

slide-19
SLIDE 19

Presentation Overview

  • Segmentation problem description
  • Background and Previous Work
  • Problems and Proposed Solutions

○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution

  • Experiments and Results

19

slide-20
SLIDE 20

Presentation Overview

  • Segmentation problem description
  • Background and Previous Work
  • Problems and Proposed Solutions

○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution

  • Experiments and Results

20

slide-21
SLIDE 21

Why tightness?

21

slide-22
SLIDE 22

Tightness Definition

22

slide-23
SLIDE 23

Corollary

A shape x is strongly tight if and only if its intersection with the middle box has a connected component touching all four sides of the middle box

23

slide-24
SLIDE 24

Energy Minimization Problem

24

slide-25
SLIDE 25

Presentation Overview

  • Segmentation problem description
  • Background and Previous Work
  • Problems and Proposed Solutions

○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution

  • Experiments and Results

25

slide-26
SLIDE 26

Energy Minimization Problem

26

slide-27
SLIDE 27

Energy Minimization Problem

27

slide-28
SLIDE 28

Convex Continuous Relaxation

28

slide-29
SLIDE 29

Convex Continuous Relaxation

29

slide-30
SLIDE 30

Continuous Optimization

30

slide-31
SLIDE 31

Continuous Optimization

31

slide-32
SLIDE 32

Additional Approximation

Intuition: Solve LP with a subset Γ' of the constraints in 3c activated

32

slide-33
SLIDE 33

Calculating Γ'

  • 1. Begin with Γ' = ∅
  • 2. Solve the LP
  • 3. Pick a group of crossing paths from Γ \ Γ' which are

violated by more than a small tolerance

  • 4. Add these paths to Γ'
  • 5. Repeat steps 2 through 4 until all paths in Γ are

satisfied within the tolerance

33

slide-34
SLIDE 34

Final Form

34

slide-35
SLIDE 35

Presentation Overview

  • Segmentation problem description
  • Background and Previous Work
  • Problems and Proposed Solutions

○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution

  • Experiments and Results

35

slide-36
SLIDE 36

Pinpointing Algorithm

Normally, output of LP is rounded to integer solution

36

slide-37
SLIDE 37

Pinpointing Algorithm

  • Pinpoint set Π contains pixels hard-assigned

to foreground

37

slide-38
SLIDE 38

Pinpoint Algorithm

38

slide-39
SLIDE 39

Pinpoint Algorithm

39

slide-40
SLIDE 40

Challenges

  • Existing methods perform energy-driven

shrinking over bounding box

○ No guarantees optimization won’t shrink excessively ○ Stuck at poor local minima ○ Discretization of approximate solution is noisy

40

slide-41
SLIDE 41

Paper’s Contributions

  • Common methods initialize foreground

region and perform energy-driven shrinking

○ No guarantees optimization won’t shrink excessively

Solution: new tightness prior

○ Stuck at poor local minima ○ Discretization of approximate solution is noisy

41

slide-42
SLIDE 42

Paper’s Contributions

  • Common methods initialize foreground

region and perform energy-driven shrinking

○ No guarantees optimization won’t shrink excessively

Solution: new tightness prior

○ Stuck at poor local minima

Solution: new approximation strategies

○ Discretization of approximate solution is noisy

42

slide-43
SLIDE 43

Paper’s Contributions

  • Common methods initialize foreground

region and perform energy-driven shrinking

○ No guarantees optimization won’t shrink excessively

Solution: new tightness prior

○ Stuck at poor local minima

Solution: new approximation strategy

○ Discretization of approximate solution is noisy

Solution: new pinpointing algorithm

43

slide-44
SLIDE 44

Presentation Overview

  • Segmentation problem description
  • Background and Previous Work
  • Problems and Proposed Solutions

○ Formalizing tightness ○ Defining tractable optimization problem for segmentation ○ Discretizing continuous approximation of solution

  • Experiments and Results

44

slide-45
SLIDE 45

Experiments

  • Evaluated over 50 image GrabCut dataset

○ Each image comes with bounding box

  • Comparison with competing methods and

initialization strategies

45

slide-46
SLIDE 46

GrabCut Dataset

  • 50 natural images with bounding box

annotations

○ Includes background, outside strip, and foreground bounding boxes

46

slide-47
SLIDE 47

GrabCut Dataset

  • 50 natural images with bounding box

annotations

○ Includes background, outside strip, and foreground bounding boxes

47

slide-48
SLIDE 48

Unary and Pairwise Terms

  • Pairwise terms over 8-connected edge set

48

slide-49
SLIDE 49

Relative Performance

Error rate - mislabeled pixels inside bounding box Optimum Rank - average rank of energy of final integer program solutions

49

slide-50
SLIDE 50

Relative Performance

50

slide-51
SLIDE 51

Iterative Process

  • Compare the following algorithms on the

segmentation task: ○ GrabCut with standard graph cut minimization for

all segmentation steps

○ GrabCut which enforces the tightness prior for all

segmentation steps

  • 5 iterations each

51

slide-52
SLIDE 52

Initialization Strategies

  • Compare two methods for initializing

foreground/background GMMs: ○ InitThirds = same as Experiment 1 (outside strip +

best matches vs. poor matches)

○ InitFullBox which sets background GMM to

  • utside strip and foreground to whole interior of

bounding box

52

slide-53
SLIDE 53

Iterative Process

53

slide-54
SLIDE 54

Effect of Margin Thickness

Error rates as function of margin thickness

54

slide-55
SLIDE 55

Strong vs. Weak Tightness

  • Strong and weak tightness lead to similar

error rates in general

○ same error rate (3.7%) for best model (GrabCut- Pinpoint/InitThirds)

55

slide-56
SLIDE 56

Iterative Process Comparisons

56

slide-57
SLIDE 57

Conclusions

  • New bounding-box based prior for interactive

image segmentation

57

slide-58
SLIDE 58

Conclusions

  • New bounding-box based prior for interactive

image segmentation

  • Demonstrated segmentation tasks under this

prior can be formulated as integer programs

58

slide-59
SLIDE 59

Conclusions

  • New bounding-box based prior for interactive

image segmentation

  • Demonstrated segmentation tasks under this

prior can be formulated as integer programs

  • Developed new optimization approaches for

approximate solution of these NP-hard problems

○ Can be applied to other computer vision problems e.g.

  • ther image segmentation or silhouettes in multi-view

stereo

59

slide-60
SLIDE 60

Thank you!

60