A Color Tree of Shapes with Illustrations on Filtering, - - PowerPoint PPT Presentation

▶
a color tree of shapes with illustrations on filtering
SMART_READER_LITE
LIVE PREVIEW

A Color Tree of Shapes with Illustrations on Filtering, - - PowerPoint PPT Presentation

Sep 02, 2023 193 likes •564 views

A Color Tree of Shapes with Illustrations on Filtering, Simplification, and Segmentation Edwin Carlinet 1,2 , Thierry G eraud 1 1 EPITA Research and Development Laboratory (LRDE) 2 Laboratoire dInformatique Gaspard-Monge (LIGM) firstname .

slide-1
SLIDE 1

A Color Tree of Shapes with Illustrations on Filtering, Simplification, and Segmentation

Edwin Carlinet1,2, Thierry G´ eraud1

1EPITA Research and Development Laboratory (LRDE) 2Laboratoire d’Informatique Gaspard-Monge (LIGM)

firstname.lastname@lrde.epita.fr

May 2015, 27th

slide-2
SLIDE 2

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Context

About morphological tree representations:

  • versatile and efficient → many apps;
  • (very) easy to compute/manipulate [5, 2, 4],
  • implicit multiscale analysis,
  • some of them feature (very) desirable properties:
  • contrast change invariance,
  • self-duality. . .

Not convinced? Let’s see. . .

A Color Tree of Shapes 2/ 31

  • E. Carlinet, T. G´

eraud

slide-3
SLIDE 3

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Grain filters [3](1/2)

Method overview

16 6 3 2 1 8 7 4 1 1

λ < 4

16 6

3 2 1

8 7 4

1 1

Tree pruning

  • 1. Compute the size attribute over the tree.
  • 2. Threshold and collapse.

A Color Tree of Shapes 3/ 31

  • E. Carlinet, T. G´

eraud

slide-4
SLIDE 4

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Grain filters (2/2): Document layout extraction

A Color Tree of Shapes 4/ 31

  • E. Carlinet, T. G´

eraud

slide-5
SLIDE 5

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Interactive segmentation (1/2)

Method overview

Color ToS Computation Markers (User Input) Markers on the tree Tree Node Classification Image Classification

A Color Tree of Shapes 5/ 31

  • E. Carlinet, T. G´

eraud

slide-6
SLIDE 6

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Interactive segmentation (2/2)

A Color Tree of Shapes 6/ 31

  • E. Carlinet, T. G´

eraud

slide-7
SLIDE 7

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Document detection in videos (ICDAR SmartDoc’15)

Text

10 1 6 3 4 7 2 5 9 8 Energy computation 10 1 6 3 4 7 2 5 9 8 Shape selection

Text

  • 1. Valuate an energy adpated to the object to detect.
  • 2. Retrieve the shape with the lowest energy.

A Color Tree of Shapes 7/ 31

  • E. Carlinet, T. G´

eraud

slide-8
SLIDE 8

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Document detection in videos (ICDAR SmartDoc’15)

Leakages Shadows Specular effects

A Color Tree of Shapes 8/ 31

  • E. Carlinet, T. G´

eraud

slide-9
SLIDE 9

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Natural Image Simplification[7]

  • Principle. Mumford-Shah energy optimization constrained to the tree.

+∞

  • 13

1

  • 3
  • 5

1 3

  • 9

3

  • 11

2 2 4

∆ Energy

+∞ 2 4 1 3

  • 9

3

  • 11

2 2 4

Iteration 1

+∞ 4 2 1 3 2 4

· · · Iteration n

A Color Tree of Shapes 9/ 31

  • E. Carlinet, T. G´

eraud

slide-10
SLIDE 10

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Natural Image Simplification[7]

Image simplification: the simplified images have less than 100 nodes (original: ∼80k nodes)

A Color Tree of Shapes 10/ 31

  • E. Carlinet, T. G´

eraud

slide-11
SLIDE 11

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Document Image Simplification[7]

(a) Original (113k nodes). (b) Strong simplification (1000 nodes). (c) Drastic simplification (285 nodes).

# nodes ÷100 # nodes ÷1000

A Color Tree of Shapes 11/ 31

  • E. Carlinet, T. G´

eraud

slide-12
SLIDE 12

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

These applications use a single image representation:

The Color Tree of Shapes

A Color Tree of Shapes 12/ 31

  • E. Carlinet, T. G´

eraud

slide-13
SLIDE 13

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Outline

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion

A Color Tree of Shapes 13/ 31

  • E. Carlinet, T. G´

eraud

slide-14
SLIDE 14

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

What is the Tree of Shapes? (1/2)

As the fusion of the min- and max- trees

A B C F D E

Ω

≥ 0 < 4

A

< 2

B

< 1

D

≥ 2

E

≥ 2

C

≥ 2

F

< 2

The Tree of shapes (ToS) of u, formed by cavity-filled connected components of the min- and max- trees (self-dual representation)

A Color Tree of Shapes 14/ 31

  • E. Carlinet, T. G´

eraud

slide-15
SLIDE 15

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

What is the Tree of Shapes? (2/2)

As the inclusion tree of the level lines

u and its level lines (every 5 levels)

  • The ToS also encodes the inclusion of the image level lines,
  • They are the contours of shapes.

A Color Tree of Shapes 15/ 31

  • E. Carlinet, T. G´

eraud

slide-16
SLIDE 16

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Properties of the ToS

We have:

  • Invariance by contrast change:

T(g(u)) = T(u) for any increasing function g

→ it handles low-contrasted objects

  • Invariance by contrast inversion:

T(∁u) = T(u)

→ it represents light objects over dark background and the contrary, in a symmetric way

  • A way to get self-dual connected operators:

→ they do not shift object boundaries

A Color Tree of Shapes 16/ 31

  • E. Carlinet, T. G´

eraud

slide-17
SLIDE 17

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

We would like the same “kind” of properties for color images.

→ Yet, the ToS requires a total order on colors (does one make sense?)

A Color Tree of Shapes 17/ 31

  • E. Carlinet, T. G´

eraud

slide-18
SLIDE 18

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Outline

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion

A Color Tree of Shapes 18/ 31

  • E. Carlinet, T. G´

eraud

slide-19
SLIDE 19

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Independant Marginal contrast change & inversion. Local contrast change What do these images have in common?

They share an exact same representation: the Color Tree of Shapes

A Color Tree of Shapes 19/ 31

  • E. Carlinet, T. G´

eraud

slide-20
SLIDE 20

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

General Overview

What do we want?

  • Given M = {S1, S2, . . . , Sn}, where (Si, ⊆) is a tree, we note

S = Si the primary shape set.

  • We aim at defining a new set of shapes S such that:

(P1) Tree structure: every two shapes are either nested or disjoint. (P2) Maximal shape preservation: any shape that does not overlap with any

  • ther shape should exist in the final shape set.

It implies the Scalar ToS equivalence if u is scalar. (P3) Marginal contrast change/inversion invariance: invariant to any strictly monotonic functions applied independently to u’s channels. (Q) A “well-formed” tree: #nodes ≃ #pixels and not degenerated.

A Color Tree of Shapes 20/ 31

  • E. Carlinet, T. G´

eraud

slide-21
SLIDE 21

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

General Overview

Scheme of the method

ToS T1 ToS T2 Graph of Shapes G ToS T3 ρ computation on G + ω reconstruction 67 3 40 96 20 66 86 86 46 Hole-filled maxtree of ω

Graph of Shapes Construction Tree Extraction

  • 1. Get the primary shape set S from the marginal ToS.
  • 2. Compute the Graph of Shapes G = (S, ⊆)

A Color Tree of Shapes 21/ 31

  • E. Carlinet, T. G´

eraud

slide-22
SLIDE 22

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

General Overview

Scheme of the method

ToS T1 ToS T2 Graph of Shapes G ToS T3 ρ computation on G + ω reconstruction 78 46 91 17 54 8 27 10 92 Hole-filled maxtree of ω

Graph of Shapes Construction Tree Extraction

  • 1. Compute the depth attribute ρ over G,
  • 2. Reconstruct the attribute map ω (in the image space),
  • 3. Compute the cavity-filled maxtree of ω

A Color Tree of Shapes 22/ 31

  • E. Carlinet, T. G´

eraud

slide-23
SLIDE 23

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

A B C D E F (a) Input u Ω A B C Ω D E F Ω 0 A 1 B 2 C 3 E 4 D 2 F 3 T1 T2 G (b) Graph of Shapes + ρ 1 2 3 2 4 3 (c) ω map Ω A B ∪ D C E F Tω (d) Cavity-filled Maxtree Tω

A Color Tree of Shapes 23/ 31

  • E. Carlinet, T. G´

eraud

slide-24
SLIDE 24

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Justification

There is no magic! In gray level: The ToS of u is related to the maxtree of the depth map (cf. paper).

  • Furthermore. . .

It fulfills the properties. (Proofs in an upcoming paper) You can get effective results. (you’ve already seen that!)

A Color Tree of Shapes 24/ 31

  • E. Carlinet, T. G´

eraud

slide-25
SLIDE 25

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Outline

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion

A Color Tree of Shapes 25/ 31

  • E. Carlinet, T. G´

eraud

slide-26
SLIDE 26

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Comparing on image simplification with classical approaches

A Color Tree of Shapes 26/ 31

  • E. Carlinet, T. G´

eraud

slide-27
SLIDE 27

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Original Ours Gray-level Marginal Total preorder [6]

A Color Tree of Shapes 27/ 31

  • E. Carlinet, T. G´

eraud

slide-28
SLIDE 28

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Conclusion (1/2)

Key Idea. A method where the ordering is not based on colors (values), but on inclusion of shapes (components). What has been done?

  • 1. A proposal for a Color Tree of Shapes
  • 2. An a-posteriori validation: get convincing results for simplification,
  • segmentation. . .

A Color Tree of Shapes 28/ 31

  • E. Carlinet, T. G´

eraud

slide-29
SLIDE 29

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Conclusion (2/2)

Perspectives: Use it! Reproductible research: http://publications.lrde.epita.fr/carlinet.15.itip → Source code, binaries, and extra results. By the way. . . It’s quite fast (2s on a 512 × 512 pixels image).

A Color Tree of Shapes 29/ 31

  • E. Carlinet, T. G´

eraud

slide-30
SLIDE 30

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Plant a tree!

Questions?

A Color Tree of Shapes 30/ 31

  • E. Carlinet, T. G´

eraud

slide-31
SLIDE 31

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

[1] E. Aptoula and S. Lef` evre. A comparative study on multivariate mathematical morphology. Pattern Recognition, 40(11):2914–2929, 2007. [2] E. Carlinet and T. G´ eraud. A comparative review of component tree computation algorithms. IEEE Transactions on Image Processing, 23(9):3885–3895, September 2014. [3] V. Caselles and P. Monasse. Grain filters. Journal of Mathematic Imaging and Vision, 17(3):249–270, November 2002. [4] S. Crozet and T. G´ eraud. A first parallel algorithm to compute the morphological tree of shapes of nD images. In Proc. of IEEE Intl. Conf. on Image Processing (ICIP), pages 2933–2937, Paris, France, 2014. [5] T. G´ eraud, E. Carlinet, S/ Crozet, and L. Najman. A quasi-linear algorithm to compute the tree of shapes of nD images. In Proc. of Intl. Symp. on Mathematical Morphology (ISMM), volume 7883 of LNCS, pages 98–110, Heidelberg, 2013. Springer. [6] O. L´ ezoray and A. Elmoataz. Nonlocal and multivariate mathematical morphology. In Proc. of IEEE Intl. Conf. on Image Processing (ICIP), pages 129–132, Orlando, USA, 2012. [7] Y. Xu, T. G´ eraud, and L. Najman. Salient level lines selection using the Mumford-Shah functional. In Proc. of IEEE Intl. Conf. on Image Processing (ICIP), pages 1227–1231, Merlbourne, Australia, 2013.

A Color Tree of Shapes 31/ 31

  • E. Carlinet, T. G´

eraud

slide-32
SLIDE 32

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Rationale (1/2)

Idea 1. T + dec. attribute ρ + restitution ωρ + Maxtree Tωρ = T Input Tree T 28 17 97 17 29 93 78 77 90 ρ computation on T + ω reconstruction Maxtree of ω Tω T = Tω

A Color Tree of Shapes 32/ 31

  • E. Carlinet, T. G´

eraud

slide-33
SLIDE 33

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Rationale (2/2)

Idea 1. T + dec. attribute ρ + restitution ωρ + Maxtree Tωρ = T Idea 2. u level lines = ωTV level lines (TV from the border). = ωCV level lines (Counted variations). → ToS of u = Maxtree of ωCV

Ω,3 2 A 4 B 1 C 3 D (a) u and its level lines. (Ω,3) (A,2) (B,4) 3 2 (C,1) (D,3) 4 3 2 2 1 1 (b) The ToS of u and ρCV

(orange).

Ω,0 1 A 2 B 2 C 3 D (c) The level lines of ωCV.

  • Conclusion. Use the depth attribute on G and reconstruct.

ωCV (x) stands for: The number of marginal level lines (that are nested) along the path from the border to the deepest shape that contains x.

A Color Tree of Shapes 33/ 31

  • E. Carlinet, T. G´

eraud

slide-34
SLIDE 34

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Differences with “Shape” component-graphs

Ω A C B D E

Image u

green red a b c d e f g ⊤ g b f c e d a

Lattice of the values

Ω ABCDE BCDE CDE CE E DE ACDE ˙ G Ω BCDE CE E ACDE DE ¨ G

“Shape” Component graphs

Ω ABCDE BDCE CE E Tred Ω BACDE ACDE DE E Tgreen Ω ABCDE BCDE CE E ACDE DE Graph of Shapes

The graph of shapes

A Color Tree of Shapes 34/ 31

  • E. Carlinet, T. G´

eraud

slide-35
SLIDE 35

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

On the need of the saturation

A B C

(a) Original.

Ω A Ω B C Ω A B C T1 T2 G

(b) Marginal ToS and GoS.

A B C

(c) ω map.

Ω (A ∪ B) C

(d) Maxtree of ω

(w/o cavity filling).

Ω H(A ∪ B) C Tω

(e) Final Color ToS

(with cavity filling).

A Color Tree of Shapes 35/ 31

  • E. Carlinet, T. G´

eraud

slide-36
SLIDE 36

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Effect of noise

(a) House (b) House (red

channel) + Gaussian Noise (σ = 20, green channel)

(c) Level lines of the

tos of (a). Level lines: 24k, avg. depth: 37, max. depth: 124.

(d) Level lines of

the ctos of (b). Level lines: 48k,

  • avg. depth: 48,
  • max. depth: 127.

A Color Tree of Shapes 36/ 31

  • E. Carlinet, T. G´

eraud

slide-37
SLIDE 37

What for? Why is a Color ToS challenging? Proposal for a Color ToS Comparison and Conclusion Bibliography Boni

Effect of the dynamic

(a) Peppers (only

red/green)

(b) Peppers (only

red/green) with green sub-quant. to 10 levels

(c) Level lines of the red

channel of (a) and (b)

(d) Level lines of

the green channel (a)

(e) Level lines of the

green channel (b)

(f) Level lines of the

ctos of (a)

(g) Level lines of

the ctos of (b)

A Color Tree of Shapes 37/ 31

  • E. Carlinet, T. G´

eraud