Superpixel Segmentation using Depth Information David Stutz - - PowerPoint PPT Presentation

Superpixel Segmentation using Depth Information

David Stutz

October 7th, 2014

David Stutz | October 7th, 2014 Superpixel Segmentation using Depth Information David Stutz | October 7th, 2014 1

1

Introduction

2

Goals

3

Related Work

4

SEEDS

5

SEEDS with Depth

6

Evaluation Qualitative Quantitative Runtime

7

Conclusion

Table of Contents

David Stutz | October 7th, 2014 2

1

Introduction

2

Goals

3

Related Work

4

SEEDS

5

SEEDS with Depth

6

Evaluation Qualitative Quantitative Runtime

7

Conclusion

Introduction

Table of Contents

David Stutz | October 7th, 2014 3

The term superpixel was coined by Ren and Malik [RM03] to describe a group of pixels perceptually belonging together: – Color similarity – Spatial proximity Why are we interested in superpixels? – Pixels are only a result of discretization. – The number of primitives is highly reduced.

Introduction

Introduction – Superpixels

David Stutz | October 7th, 2014 4

The term superpixel was coined by Ren and Malik [RM03] to describe a group of pixels perceptually belonging together: – Color similarity – Spatial proximity Why are we interested in superpixels? – Pixels are only a result of discretization. – The number of primitives is highly reduced.

Introduction

Introduction – Superpixels

David Stutz | October 7th, 2014 4

Figure: Example for a superpixel segmentation with exactly 400 superpixels.

Introduction

Introduction – Superpixels

David Stutz | October 7th, 2014 5

1

Introduction

2

Goals

3

Related Work

4

SEEDS

5

SEEDS with Depth

6

Evaluation Qualitative Quantitative Runtime

7

Conclusion

Goals

Table of Contents

David Stutz | October 7th, 2014 6

Two main goals:

• 1. An analysis of using depth information for superpixel segmentation by

extending the algorithm called SEEDS [vdBBR+12];

• 2. A thorough evaluation of several superpixel algorithms in order to

provide an overview of existing approaches.

Goals

Goals

David Stutz | October 7th, 2014 7

Two main goals:

• 1. An analysis of using depth information for superpixel segmentation by

extending the algorithm called SEEDS [vdBBR+12];

• 2. A thorough evaluation of several superpixel algorithms in order to

provide an overview of existing approaches.

Goals

Goals

David Stutz | October 7th, 2014 7

1

Introduction

2

Goals

3

Related Work

4

SEEDS

5

SEEDS with Depth

6

Evaluation Qualitative Quantitative Runtime

7

Conclusion

Related Work

Table of Contents

David Stutz | October 7th, 2014 8

Literature on superpixel algorithms is quite extensive. Therefore, we focus on four out of thirteen evaluated approaches: FH – Felzenswalb & Huttenlocher [FH04]. SLIC – Simple Linear Iterative Clustering [ASS+10]. SEEDS – Superpixels Extracted via Energy-Driven Sampling. VCCS – Voxel-Cloud Connectivity Segmentation [PASW13].

Related Work

Related Work – Superpixel Algorithms

David Stutz | October 7th, 2014 9

Literature on superpixel algorithms is quite extensive. Therefore, we focus on four out of thirteen evaluated approaches: FH – Felzenswalb & Huttenlocher [FH04]. SLIC – Simple Linear Iterative Clustering [ASS+10]. SEEDS – Superpixels Extracted via Energy-Driven Sampling. VCCS – Voxel-Cloud Connectivity Segmentation [PASW13].

Related Work

Related Work – Superpixel Algorithms

David Stutz | October 7th, 2014 9

Literature on superpixel algorithms is quite extensive. Therefore, we focus on four out of thirteen evaluated approaches: FH – Felzenswalb & Huttenlocher [FH04]. SLIC – Simple Linear Iterative Clustering [ASS+10]. SEEDS – Superpixels Extracted via Energy-Driven Sampling. VCCS – Voxel-Cloud Connectivity Segmentation [PASW13].

Related Work

Related Work – Superpixel Algorithms

David Stutz | October 7th, 2014 9

Literature on superpixel algorithms is quite extensive. Therefore, we focus on four out of thirteen evaluated approaches: FH – Felzenswalb & Huttenlocher [FH04]. SLIC – Simple Linear Iterative Clustering [ASS+10]. SEEDS – Superpixels Extracted via Energy-Driven Sampling. VCCS – Voxel-Cloud Connectivity Segmentation [PASW13].

Related Work

Related Work – Superpixel Algorithms

David Stutz | October 7th, 2014 9

1

Introduction

2

Goals

3

Related Work

4

SEEDS

5

SEEDS with Depth

6

Evaluation Qualitative Quantitative Runtime

7

Conclusion

SEEDS

Table of Contents

David Stutz | October 7th, 2014 10

Remember: SEEDS refines an initial superpixel segmentation based on color histograms by: – Exchanging blocks of pixels between neighboring superpixels. – Exchanging single pixels between neighboring superpixels. The initial superpixel segmentation is given by a uniform grid.

SEEDS

SEEDS – Idea

David Stutz | October 7th, 2014 11

Figure: Initial superpixel segmentation: 400 superpixels.

SEEDS

SEEDS – Initial Superpixels

David Stutz | October 7th, 2014 12

Figure: Superpixel segmentation after exchanging biggest blocks.

SEEDS

SEEDS – Block Updates

David Stutz | October 7th, 2014 13

Figure: Superpixel segmentation after exchanging small blocks.

SEEDS

SEEDS – Block Updates

David Stutz | October 7th, 2014 14

Figure: Superpixel segmentation after exchanging smallest blocks.

SEEDS

SEEDS – Block Updates

David Stutz | October 7th, 2014 15

Figure: Superpixel segmentation after running pixel updates.

SEEDS

SEEDS – Pixel Updates

David Stutz | October 7th, 2014 16

Figure: Superpixel segmentation after running pixel updates with an additional compactness constraint – SEEDS*.

SEEDS

SEEDS – Pixel Updates

David Stutz | October 7th, 2014 17

1

Introduction

2

Goals

3

Related Work

4

SEEDS

5

SEEDS with Depth

6

Evaluation Qualitative Quantitative Runtime

7

Conclusion

SEEDS with Depth

Table of Contents

David Stutz | October 7th, 2014 18

Block updates provide a good initial superpixel segmentation for pixel updates. Goal: Integrate depth information into block updates. Ideas: – Depth histograms – Normal histograms – Mean based block updates (plane fitting) Unfortunately, these attempts did not result in increased performance.

SEEDS with Depth

SEEDS – Depth Information

David Stutz | October 7th, 2014 19

Block updates provide a good initial superpixel segmentation for pixel updates. Goal: Integrate depth information into block updates. Ideas: – Depth histograms – Normal histograms – Mean based block updates (plane fitting) Unfortunately, these attempts did not result in increased performance.

SEEDS with Depth

SEEDS – Depth Information

David Stutz | October 7th, 2014 19

Pixel updates seem to have the most influence on the final superpixel segmentation. Goal: Integrate depth information into pixel updates. Ideas: – 3D point coordinates – Normal information

SEEDS with Depth

SEEDS – Depth Information

David Stutz | October 7th, 2014 20

Figure: Superpixel segmentation generated by SEEDS*. Image taken from the NYU Depth Dataset [SHKF12].

SEEDS with Depth

SEEDS – Depth Information

David Stutz | October 7th, 2014 21

Figure: Superpixel segmentation generated by SEEDS3D, a variant of SEEDS using 3D point coordinates for pixel updates.

SEEDS with Depth

SEEDS – Depth Information

David Stutz | October 7th, 2014 22

Figure: Superpixel segmentation generated by SEEDS3D using normal information.

SEEDS with Depth

SEEDS – Depth Information

David Stutz | October 7th, 2014 23

Unfortunately, few of our efforts resulted in significantly better superpixel segmentations. Possible explanations: – SEEDS performs well even without depth information – little room for improvement. – Images from the NYU Depth Dataset [SHKF12] are difficult because

• f clutter and bad lighting.

→ Noisy depth images, unreliable normal information.

SEEDS with Depth

SEEDS – Depth Information

David Stutz | October 7th, 2014 24

Unfortunately, few of our efforts resulted in significantly better superpixel segmentations. Possible explanations: – SEEDS performs well even without depth information – little room for improvement. – Images from the NYU Depth Dataset [SHKF12] are difficult because

• f clutter and bad lighting.

→ Noisy depth images, unreliable normal information.

SEEDS with Depth

SEEDS – Depth Information

David Stutz | October 7th, 2014 24

Figure: Difficult image from the NYU Depth Dataset [SHKF12].

SEEDS with Depth

SEEDS – Depth Information

David Stutz | October 7th, 2014 25

Figure: Corresponding raw depth image.

SEEDS with Depth

SEEDS – Depth Information

David Stutz | October 7th, 2014 26

Figure: Pre-processed depth image.

SEEDS with Depth

SEEDS – Depth Information

David Stutz | October 7th, 2014 27

Figure: Computed normals (color coded) using the Point Cloud Library [RC11].

SEEDS with Depth

SEEDS – Depth Information

David Stutz | October 7th, 2014 28

1

Introduction

2

Goals

3

Related Work

4

SEEDS

5

SEEDS with Depth

6

Evaluation Qualitative Quantitative Runtime

7

Conclusion

Evaluation

Table of Contents

David Stutz | October 7th, 2014 29

Remember, the algorithms: FH – Felzenswalb & Huttenlocher [FH04]. SLIC – Simple Linear Iterative Clustering [ASS+10]. SEEDS – Superpixels Extracted via Energy-Driven Sampling. VCCS – Voxel-Cloud Connectivity Segmentation [PASW13].

Evaluation

Evaluation

David Stutz | October 7th, 2014 30

Used datasets: – Berkeley Segmentation Dataset (BSDS500) [AMFM11]: 500 natural images. – NYU Depth Dataset (NYUV2) [SHKF12]: 1449 images of indoor scenes with depth information.

Figure: Images and corresponding ground truth segmentations from the BSDS500 and the NYUV2.

Evaluation

Evaluation

David Stutz | October 7th, 2014 31

Parameters have been optimized on training sets with respect to: – Boundary Recall Rec: the fraction of boundary pixels in the ground truth segmentation correctly detected in the superpixel segmentation.

→ 100% is best.

– Undersegmentation Error UE: the error made when comparing the ground truth segmentation to the superpixel segmentation.

→ 0% is best.

Qualitative and quantitative comparison on test sets.

Evaluation

Evaluation

David Stutz | October 7th, 2014 32

Parameters have been optimized on training sets with respect to: – Boundary Recall Rec: the fraction of boundary pixels in the ground truth segmentation correctly detected in the superpixel segmentation.

→ 100% is best.

– Undersegmentation Error UE: the error made when comparing the ground truth segmentation to the superpixel segmentation.

→ 0% is best.

Qualitative and quantitative comparison on test sets.

Evaluation

Evaluation

David Stutz | October 7th, 2014 32

Parameters have been optimized on training sets with respect to: – Boundary Recall Rec: the fraction of boundary pixels in the ground truth segmentation correctly detected in the superpixel segmentation.

→ 100% is best.

– Undersegmentation Error UE: the error made when comparing the ground truth segmentation to the superpixel segmentation.

→ 0% is best.

Qualitative and quantitative comparison on test sets.

Evaluation

Evaluation

David Stutz | October 7th, 2014 32

1

Introduction

2

Goals

3

Related Work

4

SEEDS

5

SEEDS with Depth

6

Evaluation Qualitative Quantitative Runtime

7

Conclusion

Evaluation • Qualitative

Table of Contents

David Stutz | October 7th, 2014 33

Figure: Superpixel segmentations generated by FH.

Evaluation • Qualitative

Qualitative Comparison – FH

David Stutz | October 7th, 2014 34

Figure: Superpixel segmentations generated by SLIC.

Evaluation • Qualitative

Qualitative Comparison – SLIC

David Stutz | October 7th, 2014 35

Figure: Superpixel segmentations generated by oriSEEDS.

Evaluation • Qualitative

Qualitative Comparison – oriSEEDS

David Stutz | October 7th, 2014 36

Figure: Superpixel segmentations generated by reSEEDS*.

Evaluation • Qualitative

Qualitative Comparison – reSEEDS*

David Stutz | October 7th, 2014 37

Figure: Superpixel segmentations generated by SEEDS3D.

Evaluation • Qualitative

Qualitative Comparison – SEEDS3D

David Stutz | October 7th, 2014 38

Figure: Superpixel segmentations generated by VCCS.

Evaluation • Qualitative

Qualitative Comparison – VCCS

David Stutz | October 7th, 2014 39

1

Introduction

2

Goals

3

Related Work

4

SEEDS

5

SEEDS with Depth

6

Evaluation Qualitative Quantitative Runtime

7

Conclusion

Evaluation • Quantitative

Table of Contents

David Stutz | October 7th, 2014 40

500 1,000 0.9 0.92 0.94 0.96 0.98 1 Superpixels Rec 500 1,000 0.03 0.04 0.06 0.08 0.1 Superpixels UE BSDS500:

• riSEEDS

reSEEDS*

Evaluation • Quantitative

Quantitative Comparison – BSDS500

David Stutz | October 7th, 2014 41

500 1,000 0.9 0.92 0.94 0.96 0.98 1 Superpixels Rec 500 1,000 0.03 0.04 0.06 0.08 0.1 Superpixels UE BSDS500: SLIC

• riSEEDS

reSEEDS*

Evaluation • Quantitative

Quantitative Comparison – BSDS500

David Stutz | October 7th, 2014 42

500 1,000 0.9 0.92 0.94 0.96 0.98 1 Superpixels Rec 500 1,000 0.03 0.04 0.06 0.08 0.1 Superpixels UE BSDS500: FH SLIC

• riSEEDS

reSEEDS*

Evaluation • Quantitative

Quantitative Comparison – BSDS500

David Stutz | October 7th, 2014 43

500 1,000 1,500 0.91 0.92 0.94 0.96 0.98 1 Superpixels Rec 500 1,000 1,500 0.07 0.08 0.1 0.12 0.14 0.16 0.18 0.19 Superpixels UE NYUV2:

• riSEEDS

reSEEDS* SEEDS3D

Evaluation • Quantitative

Quantitative Comparison – NYUV2

David Stutz | October 7th, 2014 44

500 1,000 1,500 0.91 0.92 0.94 0.96 0.98 1 Superpixels Rec 500 1,000 1,500 0.07 0.08 0.1 0.12 0.14 0.16 0.18 0.19 Superpixels UE NYUV2: FH SLIC

• riSEEDS

reSEEDS* SEEDS3D

Evaluation • Quantitative

Quantitative Comparison – NYUV2

David Stutz | October 7th, 2014 45

500 1,000 1,500 0.91 0.92 0.94 0.96 0.98 1 Superpixels Rec 500 1,000 1,500 0.07 0.08 0.1 0.12 0.14 0.16 0.18 0.19 Superpixels UE NYUV2: FH SLIC

• riSEEDS

reSEEDS* SEEDS3D VCCS

Evaluation • Quantitative

Quantitative Comparison – NYUV2

David Stutz | October 7th, 2014 46

1

Introduction

2

Goals

3

Related Work

4

SEEDS

5

SEEDS with Depth

6

Evaluation Qualitative Quantitative Runtime

7

Conclusion

Evaluation • Runtime

Table of Contents

David Stutz | October 7th, 2014 47

Runtime is an important aspect, especially for realtime applications. Runtime (in seconds) based on: – i7 @ 3.4GHz with 16GB RAM. – No multi-threading and no GPU. Pixel counts: – BSDS500: 481 · 321 = 154401 pixels. – NYUV2: 608 · 448 = 272384 pixels.

Evaluation • Runtime

Comparison – Runtime

David Stutz | October 7th, 2014 48

500 1,000 0.05 0.1 0.2 0.3 0.35 Superpixels t BSDS500 500 1,000 1,500 0.05 0.1 0.2 0.3 0.4 0.45 Superpixels t

• riSEEDS

reSEEDS* SEEDS3D NYUV2

Evaluation • Runtime

Comparison – Runtime

David Stutz | October 7th, 2014 49

500 1,000 0.05 0.1 0.2 0.3 0.35 Superpixels t BSDS500 500 1,000 1,500 0.05 0.1 0.2 0.3 0.4 0.45 Superpixels t FH SLIC

• riSEEDS

reSEEDS* SEEDS3D NYUV2

Evaluation • Runtime

Comparison – Runtime

David Stutz | October 7th, 2014 50

500 1,000 0.05 0.1 0.2 0.3 0.35 Superpixels t BSDS500 500 1,000 1,500 0.05 0.1 0.2 0.3 0.4 0.45 Superpixels t FH SLIC

• riSEEDS

reSEEDS* SEEDS3D VCCS NYUV2

Evaluation • Runtime

Comparison – Runtime

David Stutz | October 7th, 2014 51

FH is pretty fast with ∼ 60ms on the BSDS500. – Cannot be sped up further. However, SLIC and SEEDS can be sped up: – SLIC and SEEDS run iteratively.

→ Reduce number of iterations T.

– Reduce the size Q of the color histograms used by SEEDS.

Evaluation • Runtime

Comparison – Runtime – Discussion

David Stutz | October 7th, 2014 52

500 1,000 0.03 0.05 0.1 0.2 0.3 0.35 Superpixels t BSDS500 500 1,000 1,500 0.05 0.1 0.2 0.4 0.45 Superpixels t T = 10: SLIC T = 2, Q = 73:

• riSEEDS

reSEEDS* NYUV2

Evaluation • Runtime

Comparison – Runtime

David Stutz | October 7th, 2014 53

500 1,000 0.03 0.05 0.1 0.2 0.3 0.35 Superpixels t BSDS500 500 1,000 1,500 0.05 0.1 0.2 0.4 0.45 Superpixels t T = 10: SLIC T = 1: SLIC T = 2, Q = 73:

• riSEEDS

reSEEDS* T = 1, Q = 33:

• riSEEDS

reSEEDS* NYUV2

Evaluation • Runtime

Comparison – Runtime

David Stutz | October 7th, 2014 54

500 1,000 0.9 0.92 0.94 0.96 0.98 1 Superpixels Rec 500 1,000 0.03 0.04 0.06 0.08 0.1 0.12 Superpixels UE BSDS500: T = 10: SLIC T = 2, Q = 73:

• riSEEDS

reSEEDS*

Evaluation • Runtime

Comparison – Runtime

David Stutz | October 7th, 2014 55

500 1,000 0.9 0.92 0.94 0.96 0.98 1 Superpixels Rec 500 1,000 0.03 0.04 0.06 0.08 0.1 0.12 Superpixels UE BSDS500: T = 10: SLIC T = 1: SLIC T = 2, Q = 73:

• riSEEDS

reSEEDS* T = 1, Q = 33:

• riSEEDS

reSEEDS*

Evaluation • Runtime

Comparison – Runtime

David Stutz | October 7th, 2014 56

500 1,000 1,500 0.9 0.92 0.94 0.96 0.98 1 Superpixels Rec 500 1,000 1,500 0.08 0.1 0.12 0.14 0.16 0.18 Superpixels UE NYUV2: T = 10: SLIC T = 2, Q = 73:

• riSEEDS

reSEEDS*

Evaluation • Runtime

Comparison – Runtime

David Stutz | October 7th, 2014 57

500 1,000 1,500 0.9 0.92 0.94 0.96 0.98 1 Superpixels Rec 500 1,000 1,500 0.08 0.1 0.12 0.14 0.16 0.18 Superpixels UE NYUV2: T = 10: SLIC T = 1: SLIC T = 2, Q = 73:

• riSEEDS

reSEEDS* T = 1, Q = 33:

• riSEEDS

reSEEDS*

Evaluation • Runtime

Comparison – Runtime

David Stutz | October 7th, 2014 58

1

Introduction

2

Goals

3

Related Work

4

SEEDS

5

SEEDS with Depth

6

Evaluation Qualitative Quantitative Runtime

7

Conclusion

Conclusion

Table of Contents

David Stutz | October 7th, 2014 59

The conclusion is split up into three observations. Conclusion 1: Our implementation of SEEDS offers state-of-the-art performance while providing realtime! In addition: – Number of superpixels is controllable. – Compactness is adjustable. – Allows to trade performance for runtime.

Conclusion

Conclusion – First Part

David Stutz | October 7th, 2014 60

Conclusion 2: Using depth information for superpixel segmentation does not show significant performance increase. – At least for SEEDS. Possible explanations: – Performance of SEEDS leaves little room for improvement. – Scenes from the NYUV2 are highly cluttered and provided depth images have low quality.

Conclusion

Conclusion – Second Part

David Stutz | October 7th, 2014 61

Conclusion 3: Many superpixel algorithms show state-of-the-art performance. Therefore, other aspects become important: – Runtime – Ease-of-use (implementation, parameters etc.) – Control over the number of superpixels – Compactness parameter Based on these considerations, our implementation of SEEDS is an excellent choice.

Conclusion

Conclusion – Third Part

David Stutz | October 7th, 2014 62

Conclusion 3: Many superpixel algorithms show state-of-the-art performance. Therefore, other aspects become important: – Runtime – Ease-of-use (implementation, parameters etc.) – Control over the number of superpixels – Compactness parameter Based on these considerations, our implementation of SEEDS is an excellent choice.

Conclusion

Conclusion – Third Part

David Stutz | October 7th, 2014 62

Thank you for your attention.

david.stutz@rwth-aachen.de

Questions?

Conclusion

The End – Thanks

David Stutz | October 7th, 2014 63

Input: image I, block size w × h, levels L, histogram size Q Output: superpixel segmentation S 1. // Initialization: 2. group w × h pixels to form blocks at level l = 1 3. for l = 2 to L 4. group 2 × 2 blocks at level (l − 1) to form blocks at level l 5. for l = 1 to L 6. // For l = L, these are the initial superpixels. 7. for each block B(l)

i

at level l 8. // hB(l)

i (q) is the fraction of pixels in B(l)

i

falling in bin q. 9. compute color histogram hB(l)

i Appendix

Appendix – SEEDS

David Stutz | October 7th, 2014 64

Input: image I, block size w(1) × h(1), levels L, histogram size Q Output: superpixel segmentation S

• 10. // Block updates:
• 11. for l = L − 1 to 1

12. for each block B(l)

i

at level l 13. let Sj be the superpixel B(l)

i

belongs to 14. if a neighboring block belongs to a different superpixel Sk 15. // ∩(h, h′) = Q

q=1 min(h(q), h′(q)).

16. then if ∩(hB(l)

i , hSk) > ∩(hB(l) i , hSj−B(l) i )

17. then assign B(l)

i

to superpixel Sk

Appendix

Appendix – SEEDS

David Stutz | October 7th, 2014 65

Input: image I, block size w(1) × h(1), levels L, histogram size Q Output: superpixel segmentation S

• 18. // Pixel updates:
• 19. for n = 1 to N

20. let Sj be the superpixel xn belongs to 21. if a neighboring pixel belongs to a different superpixel Sk 22. // h(xn) denotes the bin of pixel xn. 23. then if hSk(h(xn)) > hSj(h(xn)) 24. then assign xn to superpixel Sk

• 25. return S

Appendix

Appendix – SEEDS

David Stutz | October 7th, 2014 66

Input: image I, block size w(1) × h(1), levels L, histogram size Q Output: superpixel segmentation S

• 19. // Mean pixel updates:
• 20. for n = 1 to N

21. let Sj be the superpixel xn belongs to 22. if a neighboring pixel belongs to a different superpixel Sk 23. // d(xn, Sj) = I(xn) − I(Sj)2 + βxn − µ(Sj)2. 24. then if d(xn, Sk) < d(xn, Sj) 25. then assign xn to superpixel Sk

• 26. return S

Appendix

Appendix – SEEDS

David Stutz | October 7th, 2014 67

500 1,000 0.91 0.92 0.94 0.96 0.98 1 Superpixels Rec 500 1,000 0.03 0.04 0.06 0.08 0.1 Superpixels UE NYUV2: FH TP SLIC ERS

• riSEEDS

reSEEDS*

Appendix

Appendix – Comparison – BSDS500

David Stutz | October 7th, 2014 68

500 1,000 1,500 0.91 0.92 0.94 0.96 0.98 1 Superpixels Rec 500 1,000 1,500 0.07 0.08 0.1 0.12 0.14 0.16 0.18 0.19 Superpixels UE NYUV2: FH TP SLIC ERS

• riSEEDS

reSEEDS* SEEDS3D DASP VCCS

Appendix

Appendix – Comparison – NYUV2

David Stutz | October 7th, 2014 69

It can be shown that SEEDS runs linear in the number of pixels N:

O(QTN)

(1) with – Q the number of histogram bins, – T the number of iterations at each level. However, in practice, the runtime also depends on the number of levels L!

Appendix

Appendix – Runtime

David Stutz | October 7th, 2014 70

It can be shown that SEEDS runs linear in the number of pixels N:

O(QTN)

(1) with – Q the number of histogram bins, – T the number of iterations at each level. However, in practice, the runtime also depends on the number of levels L!

Appendix

Appendix – Runtime

David Stutz | October 7th, 2014 70

500 1,000 0.03 0.05 0.1 0.2 0.3 0.35 Superpixels t BSDS500 500 1,000 1,500 0.05 0.1 0.2 0.4 0.45 Superpixels t T = 10: SLIC T = 1: SLIC T = 2, Q = 73:

• riSEEDS

reSEEDS* T = 1, Q = 33:

• riSEEDS

reSEEDS* NYUV2

Appendix

Appendix – Runtime

David Stutz | October 7th, 2014 71

Let G be a ground truth segmentation and S be a superpixel segmentation. Some definitions [NP12]: – True Positives TP(G, S): The number of boundary pixels in G for which there is a boundary pixel in S in range r. – False Negatives FN(G, S): The number of boundary pixels in G for which there is no boundary pixel in S in range r. Boundary Recall is defined as

Rec(G, S) = TP(G, S) TP(G, S) + FN(G, S).

(2)

Appendix

Appendix – Boundary Recall

David Stutz | October 7th, 2014 72

Let G be a ground truth segmentation, S be a superpixel segmentation and N be the total number of pixels. Undersegmentation Error is defined as

UE(G, S) = 1 N  

Gi∈G

• Sj∩Gi=∅

min(|Sj ∩ Gi|, |Sj − Gi|)   .

(3)

Appendix

Appendix – Undersegmentation Error

David Stutz | October 7th, 2014 73

P . Arbeláez, M. Maire, C. Fowlkes, and J. Malik. From contours to regions: An empirical evaluation. In Computer Vision and Pattern Recognition, Conference on, pages 2294–2301, Miami, Florida, June 2009. P . Arbeláez, M. Maire, C. Fowlkes, and J. Malik. Contour detection and hierarchical image segmentation. Pattern Analysis and Machine Intelligence, Transactions on, 33(5):898–916, May 2011.

• R. Achanta, A. Shaji, K. Smith, A. Lucchi, P

. Fua, and S. Süsstrunk. SLIC superpixels. Technical report, École Polytechnique Fédérale de Lausanne, Lusanne, Switzerland, June 2010.

• R. Achanta, A. Shaji, K. Smith, A. Lucchi, P

. Fua, and S. Süsstrunk. SLIC superpixels compared to state-of-the-art superpixel methods. Pattern Analysis and Machine Intelligence, Transactions on, 34(11):2274–2281, November 2012.

David Stutz | October 7th, 2014 73

• C. Bishop.

Pattern Recognition and Machine Learning. Springer Verlag, New York, 2006.

• J. Borovec and J. Kybic.

jSLIC: Superpixels in ImageJ. In Computer Vision Winter Workshop, 2014.

• A. Barla, F. Odone, and A. Verri.

Histogram intersection kernel for image classification. In Image Processing, International Conference on, volume 3, pages 513–516, Barcelona, Spain, September 2003.

• G. Bradski.

The OpenCV Library.

• Dr. Dobb’s Journal of Software Tools, 2000.

❤tt♣✿✴✴♦♣❡♥❝✈✳♦r❣✴.

• Y. Boykov, O. Veksler, and R. Zabih.

Fast approximate energy minimization via graph cuts.

David Stutz | October 7th, 2014 73

Pattern Analysis and Machine Intelligence, Transactions on, 23(11):1222–1239, November 2001.

• T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein.

Introduction to Algorithms. MIT Press, Cambridge, 2009.

• C. Conrad, M. Mertz, and R. Mester.

Contour-relaxed superpixels. In Energy Minimization Methods in Computer Vision and Pattern Recognition, volume 8081 of Lecture Notes in Computer Science, pages 280–293. Springer Berlin Heidelberg, 2013.

• J. Chang, D. Wei, and J. W. Fisher.

A video representation using temporal superpixels. In Computer Vision and Pattern Recognition, Conference on, pages 2051–2058, Portland, Oregon, June 2013.

• F. Drucker and J. MacCormick.

Fast superpixels for video analysis.

David Stutz | October 7th, 2014 73

In Motion and Video Computing, Workshop on, pages 1–8, Snowbird, Utah, December 2009.

• H. Fu, X. Cao, D. Tang, Y. Han, and D. Xu.

Regularity preserved superpixels and supervoxels. Multimedia, Transactions on, 16(4):1165–1175, June 2014. P . F. Felzenswalb and D. P . Huttenlocher. Efficient graph-based image segmentation. Computer Vision, International Journal of, 59(2), 2004.

• D. Forsyth and J. Ponce.

Computer Vision: A Modern Approach. Prentice Hall Professional Technical Reference, New Jersey, 2002.

• S. Gupta, P

. Arbeláez, and J. Malik. Perceptual organization and recognition of indoor scenes from RGB-D images. In Computer Vision and Pattern Recognition, Conference on, pages 564–571, Portland, Oregon, June 2013.

David Stutz | October 7th, 2014 73

• S. Holzer, R. B. Rusu, M. Dixon, S. Gedikli, and N. Navab.

Adaptive neighborhood selection for real-time surface normal estimation from organized point cloud data using integral images. In Intelligent Robots and Systems, International Conference on, pages 2684–2689, Vilamoura, Portugal, October 2012.

• V. Jain, S. C. Turaga, K. L. Briggman, M. N. Helmstaedter, W. Denk,

and H. S. Seung. Learning to agglomerate superpixel hierarchies. In Neural Information Processing Systems, Conference on, pages 648–656. Curran Associates, December 2011.

• K. Klasing, D. Althoff, D. Wollherr, and M. Buss.

Comparison of surface normal estimation methods for range sensing applications. In Robotics and Automation, International Conference on, pages 3206–3211, Kobe, Japan, May 2009.

• A. Levinshtein, A. Stere, K. N. Kutulakos, D. J. Fleet, S. J. Dickinson,

and K. Siddiqi.

David Stutz | October 7th, 2014 73

TurboPixels: Fast superpixels using geometric flows. Pattern Analysis and Machine Intelligence, Transactions on, 31(12):2290–2297, December 2009.

• M. Y. Lui, O. Tuzel, S. Ramalingam, and R. Chellappa.

Entropy rate superpixel segmentation. In Computer Vision and Pattern Recognition, Converence on, pages 2097–2104, Providence, Rhode Island, June 2011.

• R. Mester, C. Conrad, and A. Guevara.

Multichannel segmentation using contour relaxation: Fast super-pixels and temporal propagation. In Image Analysis, volume 6688 of Lecture Notes in Computer Science, pages 250–261. Springer Berlin Heidelberg, 2011.

• M. Meilˇ

a. Comparing clusterings by the variation of information. In Learning Theory and Kernel Machines, volume 2777 of Lecture Notes in Computer Science, pages 173–187. Springer Berlin Heidelberg, 2003.

David Stutz | October 7th, 2014 73

• M. Meilˇ

a. Comparing clusterings: an axiomatic view. In Machine Learning, International Conference on, pages 577–584, Bonn, Germany, 2005.

• D. Martin, C. Fowlkes, and J. Malik.

Learning to detect natural image boundaries using local brightness, color, and texture cues. Pattern Analysis and Machine Intelligence, Transactions on, 26(5):530–549, May 2004.

• D. Martin, C. Fowlkes, D. Tal, and J. Malik.

A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Computer Vision, International Conference on, volume 2, pages 416–423, Vancouver, Canada, July 2001.

• G. Mori.

Guiding model search using segmentation.

David Stutz | October 7th, 2014 73

In Computer Vision, International Conference on, volume 2, pages 1417–1423, Beijing, China, October 2005.

• A. P

. Moore, S. J. D. Prince, J. Warrell, U. Mohammed, and G. Jones. Superpixel lattices. In Computer Vision and Pattern Recognition, Conference on, pages 1–8, Anchorage, Alaska, June 2008.

• A. P

. Moore, S. J. D. Prince, and J. Warrell. Lattice cut - constructing superpixels using layer constraints. In Computer Vision and Pattern Recognition, Conference on, pages 2117–2124, San Francisco, California, June 2010.

• G. Mori, X. Ren, A. A. Efros, and J. Malik.

Recovering human body configurations: combining segmentation and recognition. In Computer Vision and Pattern Recognition, Conference on, volume 2, pages 326–333, Washington, D.C., June 2004. P . Neubert and P . Protzel.

David Stutz | October 7th, 2014 73

Superpixel benchmark and comparison. In Forum Bildverarbeitung, Regensburg, Germany, November 2012.

• S. Osher and R. Fedkiw.

Level Set Methods and Dynamic Implicit Surfaces. Springer Verlag, New York, 2003.

• J. Papon, A. Abramov, M. Schoeler, and F. Wörgötter.

Voxel cloud connectivity segmentation - supervoxels for point clouds. In Computer Vision and Pattern Recognition, Conference on, pages 2027–2034, Portland, Oregon, June 2013.

• F. Perbet and A. Maki.

Homogeneous superpixels from random walks. In Machine Vision and Applications, Conference on, pages 26–30, Nara, Japan, June 2011.

• W. M. Rand.

Objective criteria for the evaluation of clustering methods. American Statistical Association, Journal of the, 66(336):846–850, 1971.

David Stutz | October 7th, 2014 73

• X. Ren and L. Bo.

Discriminatively trained sparse code gradients for contour detection. In Advances in Neural Information Processing Systems, volume 25, pages 584–592. Curran Associates, 2012.

• R. B. Rusu, N. Blodow, and M. Beetz.

Fast point feature histograms (FPFH) for 3D registration. In Robotics and Automation, International Conference on, pages 3212–3217, Kobe, Japan, May 2009.

• X. Ren, L. Bo, and D. Fox.

RGB-(D) scene labeling: Features and algorithms. In Computer Vision and Pattern Recognition, Conference on, pages 2759–2766, Providence, Rhode Island, June 2012.

• R. B. Rusu and S. Cousins.

3D is here: Point Cloud Library (PCL). In Robotics and Automation, International Conference on, Shanghai, China, May 2011.

David Stutz | October 7th, 2014 73

• C. Rohkohl and K. Engel.

Efficient image segmentation using pairwise pixel similarities. In Pattern Recognition, volume 4713 of Lecture Notes in Computer Science, pages 254–263. Springer Berlin Heidelberg, 2007.

• M. Reso, J. Jachalsky, B. Rosenhahn, and J. Ostermann.

Temporally consistent superpixels. In Computer Vision, International Conference on, pages 385–392, Sydney, Australia, December 2013.

• X. Ren and J. Malik.

Learning a classification model for segmentation. In Computer Vision, International Conference on, pages 10–17, Nice, France, October 2003.

• C. Y. Ren and I. Reid.

gSLIC: a real-time implementation of SLIC superpixel segmentation. Technical report, University of Oxford, Oxford, England, 2011.

• B. C. Russell, A. Torralba, K. P

. Murphy, and W. T. Freeman.

David Stutz | October 7th, 2014 73

LabelMe: A database and web-based tool for image annotation. Computer Vision, International Journal of, 77(1-3):157–173, 2008.

• R. B. Rusu.

Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments. PhD thesis, Technische Universität München, Munich, Germany, 2009.

• N. Silberman and R. Fergus.

Indoor scene segmentation using a structured light sensor. In Computer Vision Workshops, International Conference on, pages 601–608, Barcelona, Spain, November 2011.

• A. Schick, M. Fischer, and R. Stiefelhagen.

Measuring and evaluating the compactness of superpixels. In Pattern Recognition, International Conference on, pages 930–934, Tsukuba, Japan, November 2012.

• N. Silberman, D. Hoiem, P

. Kohli, and R. Fergus. Indoor segmentation and support inference from RGBD images.

David Stutz | October 7th, 2014 73

In Computer Vision, European Conference on, volume 7576 of Lecture Notes in Computer Science, pages 746–760. Springer Berlin Heidelberg, 2012.

• J. Shi and J. Malik.

Normalized cuts and image segmentation. Pattern Analysis and Machine Intelligence, Transactions on, 22(8):888–905, August 2000.

• J. Shotton, J. Winn, C. Rother, and A. Criminisi.

TextonBoost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. Computer Vision, International Journal of, 81(1):2–23, 2009.

• D. Tang, H. Fu, and X. Cao.

Topology preserved regular superpixel. In Multimedia and Expo, International Conference on, pages 765–768, Melbourne, Australia, July 2012.

• J. Tighe and S. Lazebnik.

David Stutz | October 7th, 2014 73

SuperParsing: Scalable nonparametric image parsing with superpixels. In Computer Vision, European Conference on, volume 6315 of Lecture Notes in Computer Science, pages 352–365. Springer Berlin Heidelberg, 2010.

• R. Unnikrishnan, C. Pantofaru, and M. Hebert.

Toward objective evaluation of image segmentation algorithms. Pattern Analysis and Machine Intelligence, Transactions on, 29(6):929–944, June 2007.

• O. Veksler, Y. Boykov, and P

. Mehrani. Superpixels and supervoxels in an energy optimization framework. In Computer Vision, European Conference on, volume 6315 of Lecture Notes in Computer Science, pages 211–224. Springer Berlin Heidelberg, 2010.

• M. van den Bergh, X. Boix, G. Roig, B. de Capitani, and L. van Gool.

SEEDS: Superpixels extracted via energy-driven sampling.

David Stutz | October 7th, 2014 73

In Computer Vision, European Conference on, volume 7578 of Lecture Notes in Computer Science, pages 13–26. Springer Berlin Heidelberg, 2012.

• M. van den Bergh, X. Boix, G. Roig, B. de Capitani, and L. van Gool.

SEEDS: Superpixels extracted via energy-driven sampling. Computing Research Repository, abs/1309.3848, 2013.

• M. van den Bergh, G. Roig, X. Boix, S. Manen, and L. van Gool.

Online video seeds for temporal window objectness. In Computer Vision, International Conference on, pages 377–384, Sydney, Australia, December 2013.

• A. Vedaldi and B. Fulkerson.

VLFeat: An open and portable library of computer vision algorithms.

❤tt♣✿✴✴✇✇✇✳✈❧❢❡❛t✳♦r❣✴, 2008.

• L. Vincent and P

. Soille. Watersheds in digital spaces: an efficient algorithm based on immersion simulations.

David Stutz | October 7th, 2014 73

Pattern Analysis and Machine Intelligence, Transactions on, 13(6):583–598, June 1991.

• A. Vedaldi and S. Soatto.

Quick shift and kernel methods for mode seeking. In Computer Vision, European Conference on, volume 5305 of Lecture Notes in Computer Science, pages 705–718. Springer Berlin Heidelberg, 2008.

• D. Weikersdorfer.

Efficiency by Sparsity: Depth-Adaptive Superpixels and Event-based SLAM. PhD thesis, Technische Universität München, Munich, Germany, 2014.

• D. Weikersdorfer, D. Gossow, and M. Beetz.

Depth-adaptive superpixels. In Pattern Recognition, International Conference on, pages 2087–2090, Tsukuba, Japan, November 2012.

• S. Wang, H. Lu, F. Yang, and M.-H. Yang.

David Stutz | October 7th, 2014 73

Superpixel tracking. In Computer Vision, International Conference on, pages 1323–1330, Barcelona, Spain, November 2011.

• D. Weikersdorfer, A. Schick, and D. Cremers.

Depth-adaptive supervoxels for RGB-D video segmentation. In Image Processing, International Converence on, pages 2708–2712, Melbourne, Australia, September 2013.

• F. Yang, H. Lu, and M.-H. Yang.

Robust superpixel tracking. Image Processing, Transactions on, 23(4):1639–1651, April 2014.

• Y. Zhang, R. Hartley, J. Mashford, and S. Burn.

Superpixels via pseudo-boolean optimization. In Computer Vision, International Conference on, pages 1387–1394, Barcelona, Spain, November 2011. Yuhang Zhang, R. Hartley, J. Mashford, and S. Burn. Superpixels, occlusion and stereo.

David Stutz | October 7th, 2014 73

In Digital Image Computing Techniques and Applications, International Conference on, pages 84–91, Noosa, Australia, December 2011.

• G. Zeng, P

. Wang, J. Wang, R. Gan, and H. Zha. Structure-sensitive superpixels via geodesic distance. In Computer Vision, International Conference on, pages 447–454, Barcelona, Spain, November 2011.

David Stutz | October 7th, 2014 73