Intrinsic Quality Analysis of Binary Partition Trees Jimmy Francky - - PowerPoint PPT Presentation

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives

Intrinsic Quality Analysis of Binary Partition Trees

Jimmy Francky Randrianasoa1 Camille Kurtz2 Pierre Gançarski1 Éric Desjardin3 Nicolas Passat3

1 Université de Strasbourg, ICube, France 2 Université Paris-Descartes, LIPADE, France 3 Université de Reims Champagne-Ardenne, CReSTIC, France

ICPRAI 2018

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 1

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives

Plan

1

Introduction

2

Intrinsic evaluation of the quality of a BPT

3

Experimental study

4

Conclusions and Perspectives

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 2

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Context Problematic and purpose

Plan

1

Introduction

2

Intrinsic evaluation of the quality of a BPT

3

Experimental study

4

Conclusions and Perspectives

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 3

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Context Problematic and purpose

Hierarchical image representation

Hierarchical segmentation Better consideration of complex and heterogeneous objects Construction of hierarchical models as image representation (e.g., min-tree, max-tree, inclusion-tree, BPT, ...) Application of a segmentation method on the hierarchical model (e.g., cut, node selection, ...) Various segmentations for different level scales

Figure: Different level scales

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 4

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Context Problematic and purpose

Hierarchical image representation

Binary Partition Tree (BPT) [Salembier and Garrido, 2000] A hierarchical representation of the regions contained in an image Leaves: elementary regions defined by an initial partition L (pixels, pre-segmentation) Nodes: fusion of two neighbouring regions according to a metric of similarity W Root: node representing the image support The BPT is a hierarchical structure, frequently involved in image segmentation procedures.

Figure: Creation based on a bottom-up algorithm

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 5

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Context Problematic and purpose

Metric of similarity W involved during the BPT construction

Figure: One image ⇒ various BPTs according to the metrics W used.

Prior information Defined by the user (expert) Homogeneity criterion (radiometric, geometric,. . . ) Metric W: valuation function defining the similarity between two neighbouring regions ⇒ The metric W guides the node merging process.

Radiometric Intensity Geometric Elongation Feature Neighbouring regions Metric Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 6

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Context Problematic and purpose

Image segmentation from a BPT

Image segmentation by cutting a BPT Extraction of homogeneous relevant areas (partitioning an image)

Image Partition Borders

cut BPT

Figure: Example of a segmentation result by partitioning a remote sensing image.

Various cutting methods Horizontal cut Optimal cut ...

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 7

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Context Problematic and purpose

Impact of the quality of a hierarchical image representation for further image segmentation

For the intrinsic image models (e.g., component trees) The quality of the result depends on the segmentation method (e.g., cut on the tree) For the mixed hierarchical structures (e.g., BPT) The quality of the result does not only depend on the segmentation method (e.g., cut

• n the tree)

The quality of the result depends also on the quality of the BPT ⇒ A good segmentation method should be applied on a good BPT. Quality of a BPT

Capacity of a BPT to provide relevant segmentation results for the user Depending on the construction of the BPT according to a given metric W

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 8

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Context Problematic and purpose

Problematic and purpose

Problematic How to evaluate the capacity of a BPT to provide relevant segmentation results?

cut cut cut cut cut cut cut

?

?

?

? ?

result

Is the BPT "good" ? How to evaluate ?

How to chose a good metric / feature to build a BPT ? result result result result result result

Purpose Intrinsic evaluation of the quality of a BPT or, equivalently, its construction process ⇒ help the user to choose a right BPT (i.e., BPT evaluation) but not to use it the right way (i.e., not a segmentation evaluation) Evaluation based on comparisons to a set of segments of reference (a.k.a. ground-truth examples)

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 9

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Related works New approach for the evaluation of the quality of a BPT Intrinsic analysis of the quality of a BPT

Plan

1

Introduction

2

Intrinsic evaluation of the quality of a BPT

3

Experimental study

4

Conclusions and Perspectives

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 10

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Related works New approach for the evaluation of the quality of a BPT Intrinsic analysis of the quality of a BPT

Related works on the evaluation of the quality of a BPT

Segmentation quality evaluation Unsupervised approaches: computation of properties (homogeneity, etc.) on the segments [Troya-Galvis et al, 2015] Supervised approaches: comparison of regions to some segments of reference

[Vojodi and Moghadam, 2012] [Pont-Tuset and Marques, 2016] Metrics based on the overlapping of a region N and a segment of reference G:

Jaccard index [Jaccard, 1912] Dice Coefficient (a.k.a F-mesure) [Dice, 1945]

Other approaches: object oriented metrics and border oriented metrics

Related works on the evaluation of hierarchical models [Pont-Tuset et al, 2012] Selecting, in the tree, a set of segments matching an ideal partition that is forced to be in the hierarchy [Perret et al., 2017] Evaluation of hierarchical watersheds [Randrianasoa et al, 2017] Extrinsic quality evaluation of binary partition trees

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 11

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Related works New approach for the evaluation of the quality of a BPT Intrinsic analysis of the quality of a BPT

New approach for the evaluation of the quality of a BPT

Difficulty of the adaptation of the classical methods Several possible results from one BPT Quality of a BPT depending on the expectation of the user Choice of the segments of reference

Built area Forest area Herbaceous area Shadow

Ground-truth map

Semantic labels Problem of uncertain borders

Image

Expert

Supervised approach for the evaluation of the quality of a BPT Intrinsic analysis: evaluation from the internal coherence of the hierarchical structure Evaluate the quality of a BPT (or equivalently its construction process) Reliable clues for such quality analysis can be obtained directly investigating the BPT structure with respect to partial ground-truth (GT) examples.

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 12

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Related works New approach for the evaluation of the quality of a BPT Intrinsic analysis of the quality of a BPT

Steps of the intrinsic BPT quality analysis relying on GT examples

(a) G1 (b) (c) G9 (d) (e) G11 (f) (g) G12 (h)

Figure: Examples of segments of reference G (partial ground-truth (GT) examples)

Direct observation of the intern structure of the BPT Choice of the partial ground-truth examples by the user Step 1: extraction of a subtree TG (intersecting the segment of reference G) Step 2: verification of the relevance of the intrinsic BPT quality analysis Step 3: Intrinsic analysis: directly investigate the BPT structure for the quality evaluation

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 13

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Related works New approach for the evaluation of the quality of a BPT Intrinsic analysis of the quality of a BPT

Step 1: extraction of a subtree “example-based”

Families of nodes Intersecting the segment of reference G Not intersecting the segment of reference G

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 14

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Related works New approach for the evaluation of the quality of a BPT Intrinsic analysis of the quality of a BPT

Step 2: relevance of the quality intrinsic analysis

Choice of the initial partition L pixels, pre-segmentation (super-pixels, flat zones. . . ) Granularity γ: ration between the size of G (number of points) and the size of LG (number of leaves) γ = 1 ⇒ the number of leaves that intersect G is equal to the number of points of G Discordance δ : relative error on the size of G induced by LG The lowest the discordance, the most relevant the intrinsic quality analysis carried out

• n TG

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 15

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Related works New approach for the evaluation of the quality of a BPT Intrinsic analysis of the quality of a BPT

Step 3: intrinsic quality analysis

pure: totally inside the segment of reference impure: partially outside the segment of reference Classification of the leaves LG of the subtree TG pure leaves: LG ⊆ G (lp: number of pure leaves) impure leaves: LG G (li: number of impure leaves) ⇒ l = lp + li: number of the leaves of the subtree TG Classification of the nodes NG of the subtree TG pure node: pure-pure fusion impure node: pure-pure fusion pure-impure

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 16

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Related works New approach for the evaluation of the quality of a BPT Intrinsic analysis of the quality of a BPT

Step 3: intrinsic quality analysis

Combinatorial analysis: key elements

Table: Definitions of ui, bpp, bii and bpi.

ui number of unary nodes bpp number of binary nodes built from pure-pure couples bii number of binary nodes built from impure-impure couples bpi number of binary nodes built from pure-impure couples A «good» BPT Maximizing the pure node / leaves fusions Minimizing the impure node / leaves fusions A «bad» BPT Minimizing the pure node / leaves fusions Maximizing the impure node / leaves fusions

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 17

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives Related works New approach for the evaluation of the quality of a BPT Intrinsic analysis of the quality of a BPT

Step 3: intrinsic quality analysis

Quantitative analysis: key elements The lowest set including G: root NG of the subtree TG The greatest set Np included in G: union of all the pure nodes of NG whose parents are impure

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 18

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives

Plan

1

Introduction

2

Intrinsic evaluation of the quality of a BPT

3

Experimental study

4

Conclusions and Perspectives

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 19

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives

Experimental study

Purpose Giving an intiuition on the intrinsic quality analysis for the evaluation of hierarchical structure such as the BPT. Protocol Construction of BPTstd, BPTndvi, BPTndwi (3 different metrics W) Initial partitions (L1 (pixels), L2 (18 750 regions), L3 (12 500 regions) , L4 (6 250 regions)) Combinatorial analysis (score entre [0, 1])

cb1 = (bpp + bii)/(ui + bpp + bii + bpi) cb2 = bpp/(bpp + bpi)

Quantitative

qt1 = |NG \ G| − |(

L∈LG L) \ G| = |NG \ L∈LG L|

qt2 = |NG \ G|/|(

L∈LG L) \ G|

qt3 = (|G| − νp) − (|G| − λp) = λp − νp qt4 = |Np|

νp = (

N∈Np |N|) and λp = ( L∈Lp |L|)

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 20

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives

Experimental study: remote sensing image

Figure: Image of Strasbourg captured by P´ eiades satellites in 2012 (courtesy LIVE, UMR CNRS 7263).

(a) G1 (b) (c) G9 (d) (e) G11 (f) (g) G12 (h)

Examples of segments of reference building areas water canals roads vegetations

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 21

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives

Experimental study: combinatorial analysis for each segment of reference

Table: Scores of BPTstds built from L1 (γ = 1, δ = 0, li = 0).

cb1 = (bpp + bii)/(ui + bpp + bii + bpi) cb2 = bpp/(bpp + bpi) lp ui bpp bii bpi cb1 cb2 G1 182 17 (0) 166 (181) 5 (0) 10 (0) 0.86 0.94 G9 22378 378 (0) 22019 (22377) 129 (0) 229 (0) 0.97 0.99 G11 566 105 (0) 534 (565) 14 (0) 17 (0) 0.82 0.97 G12 541 48 (0) 512 (540) 11 (0) 17 (0) 0.89 0.97 Results of an intrinsic combinatorial analysis for each segment of reference Scores above 0.5 Estimation of a good capacity of the BPTstd to provide nodes better corresponding to G1, G9, G11 and G12 The values of ui, bpp, bii and bpi are not far from the optimal values (in brackets)

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 22

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives

Experimental study: combinatorial analysis for each initial partition

Table: Scores for BPTstds built from L1 à L4, according to G1.

γ δ li lp ui bpp bii bpi cb1 cb2 L1 1.00 0.00 182 17 (0) 166 (181) 5 (0) 10 (0) 0.86 0.94 L2 0.06 0.06 5 6 8 (0) 2 (5) 5 (4) 3 (1) 0.39 0.40 L3 0.04 0.06 5 3 7 (0) 1 5 (4) 1 (1) 0.43 0.50 L4 0.02 0.09 3 1 5 (0) 0 (0) 3 (2) 0 (1) 0.38 0.00 Results of an intrinsic combinatorial analysis for each initial partition the BPTstds built from L1 an L3 are interesting (scores above 0.5) non-optimal δ for L2, L3 et L4

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 23

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives

Experimental study: combinatorial analysis for each type of BPT

Table: Scores of three types of BPTs built from L1, for G1.

BPT ui bpp bii bpi cb1 cb2 BPTstd 17 (0) 166 (181) 5 (0) 10 (0) 0.864 0.943 BPTndvi 83 (0) 131 (181) 16 (0) 34 (0) 0.557 0.794 BPTndwi 82 (0) 133 (181) 12 (0) 36 (0) 0.551 0.787 Results of the intrinsic combinatorial analysis for each type of BPT Subtree extracted from BPTstd more interesting (best scores) less unary nodes more pure node fusions (good merging choices during the BPT construction) less useless fusions

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 24

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives

Plan

1

Introduction

2

Intrinsic evaluation of the quality of a BPT

3

Experimental study

4

Conclusions and Perspectives

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 25

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives

Conclusions and Perspectives

Conclusions New scheme for BPT quality analysis Observation of the intrinsic information contained in the BPT Helping to better understand the potential results that we can obtain from the BPTs with respect to a set of user-defined ground truth segments The relevance of the analysis highly depends on the initial partition used during the BPT construction Perspectives Extending the proposed strategy to multiple GT examples Adapting the proposed method to other kind of hierarchical data-structures

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 26

Introduction Intrinsic evaluation of the quality of a BPT Experimental study Conclusions and Perspectives

Thank you for your attention!

Tianatahina Jimmy Francky Randrianasoa ICPRAI 2018 - page 27