Binary Foreground Map Evaluation Deng-Ping Fan Nankai University of - - PowerPoint PPT Presentation

▶
binary foreground map evaluation
SMART_READER_LITE
LIVE PREVIEW

Binary Foreground Map Evaluation Deng-Ping Fan Nankai University of - - PowerPoint PPT Presentation

May 26, 2023 343 likes •1.02k views

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan Nankai University of Media Computing Lab IJCAI2018 Oral Presentation http://dpfan.net/e-measure Outline Overview of Binary Foreground Maps Previous Work

slide-1
SLIDE 1

Enhanced-alignment Measure for Binary Foreground Map Evaluation

Deng-Ping Fan Nankai University of Media Computing Lab

IJCAI’2018 Oral Presentation http://dpfan.net/e-measure

slide-2
SLIDE 2

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Outline

  • Overview of Binary Foreground Maps
  • Previous Work
  • Enhanced-alignment Measure
  • Experiments
slide-3
SLIDE 3

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Binary Foreground Map

(a) Image (b) Binary foreground map 1

slide-4
SLIDE 4

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Binary Foreground Map

 The binary foreground map consists of values of either 0 or 1.  1 denotes foreground, 0 for background.

(a) Image (b) Binary foreground map 1

slide-5
SLIDE 5

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Application

 Object Segmentation

(a) Image (d) GT

slide-6
SLIDE 6

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Application

 Object Segmentation

(a) Image (d) GT (b) MDF (CVPR’15) (c) DISC (TNNLS’16) (e) DCL (CVPR’16) (f) Noise

slide-7
SLIDE 7

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Evaluation

 Similarity evaluation is very important.

slide-8
SLIDE 8

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Evaluation

 Similarity evaluation is very important.

VS VS

slide-9
SLIDE 9

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Outline

  • Overview of Binary Foreground Maps
  • Previous Work
  • Enhanced-alignment Measure
  • Experiments
slide-10
SLIDE 10

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Intersection-over-Union (IoU)

slide-11
SLIDE 11

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Intersection-over-Union (IoU)

A

slide-12
SLIDE 12

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Intersection-over-Union (IoU)

B

slide-13
SLIDE 13

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Intersection-over-Union (IoU)

A B

slide-14
SLIDE 14

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Intersection-over-Union (IoU)

𝐽𝑝𝑉 = 𝐵∩𝐶 𝐵 ∪ 𝐶

slide-15
SLIDE 15

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Intersection-over-Union (IoU)

𝐽𝑝𝑉 = 𝐵∩𝐶 𝐵 ∪ 𝐶

A∩B

slide-16
SLIDE 16

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Intersection-over-Union (IoU)

𝐽𝑝𝑉 = 𝐵∩𝐶 𝐵 ∪ 𝐶

A∩B

slide-17
SLIDE 17

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Intersection-over-Union (IoU)

𝐽𝑝𝑉 = 𝐵∩𝐶 𝐵 ∪ 𝐶

A∩B A ∪ B

slide-18
SLIDE 18

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Contour Mapping (CM)[1]

[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010.

slide-19
SLIDE 19

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Contour Mapping (CM)[1]

[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010.

slide-20
SLIDE 20

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Contour Mapping (CM)[1]  Weighted 𝐺

𝛾-measure (Fbw)[2]

[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010. [2] Margolin et al. How to evaluate foreground maps? CVPR, 2014. [3] Shi et al. Visual quality evaluation of image object segmentation: Subjective assessment and objective measure. TIP, 2015. [4] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.

slide-21
SLIDE 21

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Contour Mapping (CM)[1]  Weighted 𝐺

𝛾-measure (Fbw)[2]

Introducing weight to the 𝐺

𝛾-measure (related to IoU) framework.

[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010. [2] Margolin et al. How to evaluate foreground maps? CVPR, 2014. [3] Shi et al. Visual quality evaluation of image object segmentation: Subjective assessment and objective measure. TIP, 2015. [4] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.

𝐽𝑝𝑉 = 𝐺1 2 − 𝐺1

slide-22
SLIDE 22

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Contour Mapping (CM)[1]  Weighted 𝐺

𝛾-measure (Fbw)[2]

Introducing weight to the 𝐺

𝛾-measure (related to IoU) framework.

 Visual Quality (VQ)[3]

[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010. [2] Margolin et al. How to evaluate foreground maps? CVPR, 2014. [3] Shi et al. Visual quality evaluation of image object segmentation: Subjective assessment and objective measure. TIP, 2015. [4] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.

slide-23
SLIDE 23

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Contour Mapping (CM)[1]  Weighted 𝐺

𝛾-measure (Fbw)[2]

Introducing weight to the 𝐺

𝛾-measure (related to IoU) framework.

 Visual Quality (VQ)[3] Considering psychological function based on the IoU.

[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010. [2] Margolin et al. How to evaluate foreground maps? CVPR, 2014. [3] Shi et al. Visual quality evaluation of image object segmentation: Subjective assessment and objective measure. TIP, 2015. [4] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.

slide-24
SLIDE 24

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Contour Mapping (CM)[1]  Weighted 𝐺

𝛾-measure (Fbw)[2]

Introducing weight to the 𝐺

𝛾-measure (related to IoU) framework.

 Visual Quality (VQ)[3] Considering psychological function based on the IoU.  Structure measure (S-measure)[4]

[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010. [2] Margolin et al. How to evaluate foreground maps? CVPR, 2014. [3] Shi et al. Visual quality evaluation of image object segmentation: Subjective assessment and objective measure. TIP, 2015. [4] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.

slide-25
SLIDE 25

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Previous Work

 Contour Mapping (CM)[1]  Weighted 𝐺

𝛾-measure (Fbw)[2]

Introducing weight to the 𝐺

𝛾-measure (related to IoU) framework.

 Visual Quality (VQ)[3] Considering psychological function based on the IoU.  Structure measure (S-measure)[4] Mainly focus on the non-binary maps evaluation.

[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010. [2] Margolin et al. How to evaluate foreground maps? CVPR, 2014. [3] Shi et al. Visual quality evaluation of image object segmentation: Subjective assessment and objective measure. TIP, 2015. [4] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.

slide-26
SLIDE 26

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Measure Year Pros Cons IoU

190 1 easy to calculate losing image level statistics

CM [1] 2010

considering both region and contour noise sensitive

Fbw [2] 2014

assigning different weights for errors error location sensitive, complicated

VQ [3] 2015

weighting errors by psychological function subjective measure

S-measure [4] 2017

considering structure similarity focusing on non-binary map properties

  • Table1. Current evaluation measure summary

Previous Work

[1] Movahedi and Elder. Design and perceptual validation of performance measures for salient object segmentation. CVPRW, 2010. [2] Margolin et al. How to evaluate foreground maps? CVPR, 2014. [3] Shi et al. Visual quality evaluation of image object segmentation: Subjective assessment and objective measure. TIP, 2015. [4] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017.

slide-27
SLIDE 27

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Problem

(a) Image (b) GT (c) Foreground map (d) Noise

slide-28
SLIDE 28

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Problem

(a) Image (b) GT (c) Foreground map (d) Noise  Almost all of current measure (e.g., IoU, CM, Fbw, VQ) prefer the Noise map.

slide-29
SLIDE 29

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Problem

(a) Image (b) GT (c) Foreground map (d) Noise  Almost all of current measure (e.g., IoU, CM, Fbw, VQ) prefer the Noise map.  They are either edge-based(local details) or region-based (global infromation).

slide-30
SLIDE 30

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Problem

(a) Image (b) GT (c) Foreground map (d) Noise  Almost all of current measure (e.g., IoU, CM, Fbw, VQ) prefer the Noise map.  They are either edge-based(local details) or region-based (global information).  None of them consider both local and global simultaneously.

slide-31
SLIDE 31

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Outline

  • Overview of Binary Foreground Maps
  • Previous Work
  • Enhanced-alignment Measure
  • Experiments
slide-32
SLIDE 32

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Motivation

COGN COGNIVIS VISION ION

http://knowledgelotus.info/human-brain-facts/ 1.Global information can be captured by the eye movement.

  • 2. Local details

recorded by focusing the special image region.

slide-33
SLIDE 33

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Example

1.Global information (a) Image (b) GT

slide-34
SLIDE 34

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Example

1.Global information (a) Image (b) GT (c) Noise

slide-35
SLIDE 35

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Example

1.Global information (a) Image (b) GT (c) Noise (d) Map1

slide-36
SLIDE 36

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Example

1.Global information (a) Image (b) GT (c) Noise (d) Map1

slide-37
SLIDE 37

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Example

1.Global information (a) Image (b) GT (c) Noise (d) Map1

slide-38
SLIDE 38

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Example

1.Global information & 2.Local Details (a) Image (b) GT (c) Map1 (d) Map2

slide-39
SLIDE 39

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Example

1.Global information & 2.Local Details (a) Image (b) GT (c) Map1 (d) Map2

slide-40
SLIDE 40

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Example

1.Global information & 2.Local Details (a) Image (b) GT (c) Map1 (d) Map2

slide-41
SLIDE 41

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Example

1.Global information & 2.Local Details (a) Image (b) GT (d) Map1 (c) Map2

slide-42
SLIDE 42

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

1.Global information Firstly, we compute the global mean of the input map to capture global information.

slide-43
SLIDE 43

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

1.Global information Firstly, we compute the global mean of the input map to capture global information. (a) Map 𝑌𝑗𝑘 (b) Global mean 𝑣 = ෍

𝑘=1 𝑁

෍

𝑗=1 N

𝑌𝑗𝑘 = 5

slide-44
SLIDE 44

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

1.Global information Firstly, we compute the global mean of the input map to capture global information. (a) Map 𝑌𝑗𝑘 (b) Global mean 𝑣 = ෍

𝑘=1 𝑁

෍

𝑗=1 N

𝑌𝑗𝑘 = 5 2.Local details Then, we treat each pixel in the map as the local details. (e.g., 𝑌12 = 6)

slide-45
SLIDE 45

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

1.Global information Firstly, we compute the global mean of the input map to capture global information. (a) Map 𝑌𝑗𝑘 (b) Global mean 𝑣 = ෍

𝑘=1 𝑁

෍

𝑗=1 N

𝑌𝑗𝑘 = 5 2.Local details Then, we treat each pixel in the map as the local details. (e.g., 𝑌12 = 6) 3.Combine global information with local details Finally, we need to combine them simultaneously. Thus, we introduce a bias matrix which can be treated as the signal centering by removing the mean from the signal.

slide-46
SLIDE 46

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

1.Global information Firstly, we compute the global mean of the input map to capture global information. (a) Map 𝑌𝑗𝑘 (b) Global mean 𝑣 = ෍

𝑘=1 𝑁

෍

𝑗=1 N

𝑌𝑗𝑘 = 5 2.Local details Then, we treat each pixel in the map as the local details. (e.g., 𝑌12 = 6) 3.Combine global information with local details Finally, we need to combine them simultaneously. Thus, we introduce a bias matrix which can be treated as the signal centering by removing the mean from the signal. 𝜒𝑗𝑘 = 𝑌𝑗𝑘 − 𝑣 ∗ 𝐽 (c) Bias matrix

slide-47
SLIDE 47

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

1.Global information Firstly, we compute the global mean of the input map to capture global information. (a) Map 𝑌𝑗𝑘 (b) Global mean 𝑣 = ෍

𝑘=1 𝑁

෍

𝑗=1 N

𝑌𝑗𝑘 = 5 2.Local details Then, we treat each pixel in the map as the local details. (e.g., 𝑌12 = 6) 3.Combine global information with local details Finally, we need to combine them simultaneously. Thus, we introduce a bias matrix which can be treated as the signal centering by removing the mean from the signal. (c) Bias matrix

slide-48
SLIDE 48

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

4.Alignment matrix

slide-49
SLIDE 49

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

4.Alignment matrix

slide-50
SLIDE 50

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

4.Alignment matrix alignment matrix [1][2]

[1] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017. [2] Wang et al. Image quality assessment: from error visibility to structural similarity. TIP, 2004

slide-51
SLIDE 51

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

4.Alignment matrix

slide-52
SLIDE 52

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

4.Alignment matrix

slide-53
SLIDE 53

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

4.Alignment matrix

slide-54
SLIDE 54

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Alignment term

  • 5. Enhanced alignment matrix
slide-55
SLIDE 55

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Outline

  • Overview of Binary Foreground Maps
  • Previous Work
  • Enhanced-alignment Measure
  • Experiments
slide-56
SLIDE 56

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Experiments

  • 1. Meta-Measure 1: Application Ranking
slide-57
SLIDE 57

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Experiments

  • 2. Meta-Measure 2: SOTA vs. Generic Maps
slide-58
SLIDE 58

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Experiments

  • 2. Meta-Measure 2: SOTA vs. Generic Maps
  • 3. Meta-Measure 3: SOTA vs. Random Noise
slide-59
SLIDE 59

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Experiments

  • 2. Meta-Measure 2: SOTA vs. Generic Maps
  • 3. Meta-Measure 3: SOTA vs. Random Noise
slide-60
SLIDE 60

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Experiments

  • 4. Meta-Measure 4: Human Ranking

[Tips]: The human ranked datasets can be download through our website: http://dpfan.net/e-measure

slide-61
SLIDE 61

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Experiments

  • 5. Meta-Measure 5: Ground truth Switch
slide-62
SLIDE 62

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Experiments

  • 5. Meta-Measure 5: Ground truth Switch
  • 6. Results
slide-63
SLIDE 63

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Experiments

  • 5. Meta-Measure 5: Ground truth Switch
  • 6. Results

9.08%-19.65% improvement.

slide-64
SLIDE 64

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

Example

slide-65
SLIDE 65

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7

[1] Fan et al. Structure-measure: A New Way to Evaluate Foreground Maps. ICCV, 2017. [2] Fan et al. Face Sketch Synthesis Style Similarity: A New Structure Co-occurrence Texture Measure. arXive, 2018 [3] Fan et al. Salient Objects in Clutter: Bringing Salient Object Detection to the Foreground. ECCV, 2018. [4] Margolin et al. How to evaluate the foreground maps? CVPR, 2014. [5] Wang et al. Image quality assessment: from error visibility to structural similarity. TIP, 2004.

Related papers

slide-66
SLIDE 66

Enhanced-alignment Measure for Binary Foreground Map Evaluation Deng-Ping Fan, http://dpfan.net/, 2018/7/7