Interactive image segmentation with integrated use of the markers - - PowerPoint PPT Presentation

Watershed transform Integrated approach SegmentIt

Interactive image segmentation with integrated use

• f the markers and the hierarchical watershed

approaches

Bruno Klava and Nina Sumiko Tomita Hirata

Institute of Mathematics and Statistics University of S˜ ao Paulo Brazil

06 February, 2009

This work is supported by CNPq, Brazil.

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Outline

1

Watershed transform Classical watershed Watershed from markers Hierarchical watershed

2

Integrated approach Interactive segmentation From markers to the hierarchical watershed From hierarchical watershed to markers Integrated approach

3

SegmentIt Demonstration

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Watershed transform [Beucher and Meyer, 1993]

Classical watershed - Flooding simulation from the regional minima

(Loading animation...)

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Classical watershed

Oversegmentation problem - Primitive catchment basins

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Classical watershed

Oversegmentation problem - Primitive catchment basins

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Watershed from markers

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Watershed from markers

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Watershed from markers

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Watershed from markers

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Watershed from markers

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Watershed on graphs

WT can be formulated as a graph optimization problem: WS(GI,M) GI = (V,E,ω): a graph representing the input image I

Granularities:

Pixel Adjacency Graph PAGI Region Adjacency Graph RAGI

ω: dissimilarity measure

between the vertices

M: the set of markers

93 98 98 98 60 27 22 19 166 169 169 169 124 100 22 19 140 169 159 155 155 155 33 15 170 170 90 162 168 168 83 14 170 170 55 139 168 168 139 19 30 152 168 168 85 119 170 170 170 170 170 46 156 156 152 32 140 152 159 159 61 167 167 167 164 164 164 102 157 157 137 129 139 147 179 179 169 169 163 157 115 115 158 158 168 168 104 33 76 140 152 152 152 152 120 120 133 133 157 164 167 170 170 170 170 170 169 166 169 169 159 90 119 170 170 167 164 163 169 169 102 61 55 55 72 145 169 169 124 155 162 162 139 85 61 102 169 169 179 179 159 159 168 168 168 168 155 100 22 33 83 139 152 152 152 137 147 104 139 139 140 140 32 30 19 15 19 19 19 19 14 19 30 30 140 140 129 129

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Watershed on graphs

WT can be formulated as a graph optimization problem: WS(GI,M) GI = (V,E,ω): a graph representing the input image I

Granularities:

Pixel Adjacency Graph PAGI Region Adjacency Graph RAGI

ω: dissimilarity measure

between the vertices

M: the set of markers

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Watershed on graphs

P = WS(GI,M) is obtained by propagating the label of the markers for all the vertices of GI using minimum cost paths; WS(GI,M) can have multiple optimal solutions (minimum cost paths can be not unique), the obtained solution depends on the implementation details (queue policies, processing order of vertices, neighbors and markers, etc.)

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Hierarchical watershed

Tree of Critical Lakes (TCL) [Meyer, 1996]

Hierarchy: nested set of partitions, where the finest partition is given by the classical watershed.

(a) Original image

C E D A B

(b) Finest partition

(simplified) A B C D E

(c) MST

• f

the RAG

A B C D E F H G I

(d) TCL

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Hierarchical watershed

Construction of the Tree of Critical Lakes (TCL) A C B C A B

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Hierarchical watershed

Construction of the Tree of Critical Lakes (TCL) D A C D A B

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Hierarchical watershed

Construction of the Tree of Critical Lakes (TCL) C D E A B E

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Hierarchical watershed

Limitation of partitions that can be found on the TCL A C B C D E A B

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Hierarchical watershed

Navigating through the partitions of the TCL [Zanoguera et al., 1999]

F C G A B C D E F H G I

(a) Threshold

F C E D A B C D E F H G I

(b) Split

E H D A B C D E F H G I

(c) Merge

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Interactive segmentation

Systems based on automatic segmentation are usually restricted to specific domains and for two classes of objects (foreground / background) Formal specification of the parameters of a segmentation algorithm can be very difficult; A post-processing of the partition (obtained through automatic techniques) may be necessary to achieve a satisfactory result; The concept of a good partition may depend on the purpose of its use and may be highly subjective.

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Watershed from markers

Strong points easy to segment a specific region works on both granularities (pixels and regions) easy to “force” inexistent borders Weak points a marker has to be drawn for each region of interest high interaction effort to design markers for complex regions

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Hierarchical watershed

Strong points easy to segment several objects according to their metrics works on the regions granularity (efficiency) Weak points works on the regions granularity (borders are always borders of the primitive catchment basins) the partitions are limited due to the structure of the TCL hard to segment a specific region

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Proposal

Integrate the watershed from markers and the hierarchical watershed in order to allow the full use of the strengths of both approaches.

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Problem

From markers to the hierarchical watershed Given: Set of markers M Partition P = WS(GI,M) of the image I We want: Hierarchy of partitions of I which contains P From hierarchical watershed to markers Given: Partition P selected from a hierarchy of partitions of the image I We want: Set of markers M such that WS(GI,M) = P

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From markers to the hierarchical watershed

1

The primitive regions must be atomic in the partition

28 27 36 19 15 3 36 17 18 2 2 22 2 2 6 27 35 14 30 22 11 4 26 35 41 36 14

Desired partition with 3 regions of interest (ROI)

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From markers to the hierarchical watershed

1

The primitive regions must be atomic in the partition

28 27 36 19 15 3 36 17 18 2 2 22 2 2 6 27 35 14 30 22 11 4 26 35 41 36 14

RAG

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From markers to the hierarchical watershed

1

The primitive regions must be atomic in the partition

28 27 36 19 15 3 36 17 18 2 2 22 2 2 6 27 35 14 30 22 11 4 26 35 41 36 14

Desired partition is incompatible with the granularity of the hierarchy

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From markers to the hierarchical watershed

1

The primitive regions must be atomic in the partition

28 27 36 19 15 3 36 17 18 2 2 22 2 2 6 27 35 14 30 22 11 4 26 35 41 36 14 28 27 36 19 15 3 36 17 18 2 2 22 2 2 6 27 35 14 30 22 11 4 26 35 41 36 14

Desired partition is incompatible with the granularity of the hierarchy

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From markers to the hierarchical watershed

2

Each ROI must be connected in the spanning tree from which the hierarchy is constructed

19 15 17 18 2 2 6 27 35 14 11 26 36 14

Arbitrary MST is incompatible

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From markers to the hierarchical watershed

2

Each ROI must be connected in the spanning tree from which the hierarchy is constructed

27 19 17 18 2 35 14 26 35 36 14

Spanning forest composed by MSTs of each ROI

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From markers to the hierarchical watershed

2

Each ROI must be connected in the spanning tree from which the hierarchy is constructed

11 15 27 19 17 18 2 35 14 26 35 36 14

Resultant spanning tree

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From hierarchical watershed to markers

Minimal seed set (MSS): inverse of the segmentation by watershed problem, corresponding to finding a minimal set of seeds (markers) that are necessary to recover a given partition by the watershed Receptive regions (RR): regions that must contain markers in

• rder to recover the considered partition

Granularities:

• n the RAG [Lotufo and Silva, 2002]
• n the PAG [Audigier and Lotufo, 2007]

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From hierarchical watershed to markers

MRAGI: MSS computed on the RAG of I MPAGI: MSS computed on the PAG of I P = WS(RAGI,MRAGI) = WS(PAGI,MPAGI)

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From hierarchical watershed to markers

MSS on the RAG vs. MSS on the PAG

Partition P selected on the hierarchy (6 ROIs)

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From hierarchical watershed to markers

MSS on the RAG vs. MSS on the PAG

RRs computed on the RAG (one RR for each ROI)

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From hierarchical watershed to markers

MSS on the RAG vs. MSS on the PAG (a) 39 RRs (b) 47 RRs (c) 61 RRs (d) 62 RRs (e) 87 RRs (f) 127 RRs

RRs computed on the PAG

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From hierarchical watershed to markers

MSSs on the RAG vs. MSS on the PAG

P′ = WS(PAGI,MRAGI)

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From hierarchical watershed to markers

MSS on the RAG vs. MSS on the PAG

Differences between P and P′ (9 pixels divergent from the desired labeling) due to the distinct granularities

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From hierarchical watershed to markers

MSS on the RAG vs. MSS on the PAG

In the paper we have a proof that: P is also an optimal solution of WS(PAGI,MRAGI)

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

From hierarchical watershed to markers

MSS on the RAG vs. MSS on the PAG

Advantages of using MRAGI instead of MPAGI reduced computational cost to obtain can be localized far from the borders: more appropriate for user edition!

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Integrated approach

MSS computed on the RAG (independent of the granularity used in the markers approach); coarse to fine approach.

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

SegmentIt

Demonstration

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Contributions of our work

Integration of the markers and the hierarchical watershed approaches; segmentation tool integrating several interaction possibilities described in the literature; new result concerning the granularities on the MSS problem; use of the hierarchy limited to a region, making easier the segmentation of regions of the same image that have distinct characteristics; arbitrary welding to overcome the limitations of the TCL.

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Future works

Case studies to evaluate the interaction effort reduction due to the integrated approach; provide means to easily place markers on both sides of the borders of interest; extension of the color image segmentation support (currently is rather limited); improvement of the hierarchy representativity by taking the user interaction in machine learning techniques.

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

References

Beucher, S. and Meyer, F . (1993). The morphological approach to segmentation: the watershed transformation. In Dougherty, E., editor, Mathematical morphology in image processing, chapter 12, pages 433–481. M. Dekker. Meyer, F . (1996). The dynamics of minima and contours. In Maragos, P ., Schafer, R. W., and Butt, M. A., editors, Mathematical Morphology and its Applications to Image and Signal Processing, pages 329–336. Kluwer Academic Publishers. Zanoguera, F ., Marcotegui, B., and Meyer, F . (1999). A toolbox for interactive segmentation based on nested partitions. In Proceedings of the International Conference on Image Processing, volume 1, pages 21–25. Lotufo, R. A. and Silva, W. (2002). Minimal set of markers for the watershed. In Talbot, H. and Beare, R., editors, 6th International Symposium on Mathematical Morphology, pages 359–368. CSIRO Publications. Audigier, R. and Lotufo, R. A. (2007). Seed-relative segmentation robustness of watershed and fuzzy connectedness approaches. In SIBGRAPI ’07: Proceedings of the XX Brazilian Symposium on Computer Graphics and Image Processing, pages 61–70, Washington, DC, USA. IEEE Computer Society. Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .

Watershed transform Integrated approach SegmentIt

Thank you!

http://watershed.sourceforge.net/

Bruno Klava and Nina Sumiko Tomita Hirata Interactive image segmentation with integrated use of the markers . . .