Is Crowdsourcing feasible for optical flow Ground Truth generation? - - PowerPoint PPT Presentation

▶
is crowdsourcing feasible for optical flow ground truth
SMART_READER_LITE
LIVE PREVIEW

Is Crowdsourcing feasible for optical flow Ground Truth generation? - - PowerPoint PPT Presentation

Apr 08, 2024 225 likes •683 views

Is Crowdsourcing feasible for optical flow Ground Truth generation? Axel Donath, Daniel Kondermann HCI Heidelberg ICVS 2013, St.Petersburg Crowdsourcing for Ground Truth generation ICVS 2013 1 Overview 1.Introduction 3.Experiments &

slide-1
SLIDE 1

Crowdsourcing for Ground Truth generation ICVS 2013 1

Is Crowdsourcing feasible for optical flow Ground Truth generation?

Axel Donath, Daniel Kondermann – HCI Heidelberg

ICVS 2013, St.Petersburg

slide-2
SLIDE 2

Crowdsourcing for Ground Truth generation ICVS 2013 2

Overview

1.Introduction 2.Ground Truth via Mechanical T urk 3.Experiments & Results 4.Conclusion

slide-3
SLIDE 3

Crowdsourcing for Ground Truth generation ICVS 2013 3

1.Introduction

slide-4
SLIDE 4

Crowdsourcing for Ground Truth generation ICVS 2013 4

Motivation

Start

Sequence taken from [3]

slide-5
SLIDE 5

Crowdsourcing for Ground Truth generation ICVS 2013 5

Large scale dynamic outdoor scene

Frame 1

slide-6
SLIDE 6

Crowdsourcing for Ground Truth generation ICVS 2013 6

Large scale dynamic outdoor scene

Frame 2

slide-7
SLIDE 7

Crowdsourcing for Ground Truth generation ICVS 2013 7

Large scale dynamic outdoor scene

Frame 3

slide-8
SLIDE 8

Crowdsourcing for Ground Truth generation ICVS 2013 8

Large scale dynamic outdoor scene

Frame 4

slide-9
SLIDE 9

Crowdsourcing for Ground Truth generation ICVS 2013 9

Large scale dynamic outdoor scene

Frame 5

slide-10
SLIDE 10

Crowdsourcing for Ground Truth generation ICVS 2013 10

Large scale dynamic outdoor scene

End

Sequence taken from [3]

slide-11
SLIDE 11

Crowdsourcing for Ground Truth generation ICVS 2013 11

Flow field estimated by algorithm

Optical flow algorithm: Classic+NL [5] Color legend:

slide-12
SLIDE 12

Crowdsourcing for Ground Truth generation ICVS 2013 12

Principles to obtain Ground Truth

(1) Measurement with suitable setups

E.g. Middlebury dataset [2]

(2) Simulate data with computer graphics

E.g. Sintel dataset [4] and Middlebury dataset [2]

(3) Data can be annotated by humans

„Human assisted motion annotation“ with Motion-Annotation-T

  • ol,

proposed by Liu et. al [1]

slide-13
SLIDE 13

Crowdsourcing for Ground Truth generation ICVS 2013 13

Start

Manual labeling and tracking

Sequence labeled with Motion T

  • ol [1]
slide-14
SLIDE 14

Crowdsourcing for Ground Truth generation ICVS 2013 14

Manual labeling and tracking

Frame 1

slide-15
SLIDE 15

Crowdsourcing for Ground Truth generation ICVS 2013 15

Frame 2

Manual labeling and tracking

slide-16
SLIDE 16

Crowdsourcing for Ground Truth generation ICVS 2013 16

Frame 3

Manual labeling and tracking

slide-17
SLIDE 17

Crowdsourcing for Ground Truth generation ICVS 2013 17

Frame 4

Manual labeling and tracking

slide-18
SLIDE 18

Crowdsourcing for Ground Truth generation ICVS 2013 18

Frame 5

Manual labeling and tracking

slide-19
SLIDE 19

Crowdsourcing for Ground Truth generation ICVS 2013 19

End

Manual labeling and tracking

Sequence labeled with Motion T

  • ol [1]
slide-20
SLIDE 20

Crowdsourcing for Ground Truth generation ICVS 2013 20

Idea

Outsource manual correction of outlines and finding of feature points to Mechanical T urk

slide-21
SLIDE 21

Crowdsourcing for Ground Truth generation ICVS 2013 21

2.Ground Truth via Mechanical T urk

slide-22
SLIDE 22

Crowdsourcing for Ground Truth generation ICVS 2013 22

General workflow

Initial segmentation Tracking of outlines Correction of outlines Selection of feature points Selection of motion models

Ground Truth

Laymen via Mechanical T urk Trained user Trained user

slide-23
SLIDE 23

Crowdsourcing for Ground Truth generation ICVS 2013 23

Mechanical T urk workflow

Correction of outlines Selection of feature points

Laymen via Mechanical T urk

5 „HIT s“ per

  • utline

Download & Review results Blur outlines Merge & import outlines

Webinterface DEMO

slide-24
SLIDE 24

Crowdsourcing for Ground Truth generation ICVS 2013 24

Mechanical T urk workflow

Correction of outlines Selection of feature points

Laymen via Mechanical T urk

8 points per patch Download & Review results Divide image into patches Import feature points

Webinterface DEMO

slide-25
SLIDE 25

Crowdsourcing for Ground Truth generation ICVS 2013 25

3.Experiments and Results

slide-26
SLIDE 26

Crowdsourcing for Ground Truth generation ICVS 2013 26

Outline correction of simple scenes

Outlines before... ...and after correction by the workers

slide-27
SLIDE 27

Crowdsourcing for Ground Truth generation ICVS 2013 27

Results on simple scenes I

AEE = 0.37 AEE = 0.79 AEE = 0.37 AEE = 0.47 AEE = 0.51 AEE = 0.63 All images are normalized to max. endpoint error of 2 pix Endpoint error of six runs on the „Rubber Whale“ sequence:

slide-28
SLIDE 28

Crowdsourcing for Ground Truth generation ICVS 2013 28

Results on simple scenes II

Endpoint error with overlapping patches: AEE = 0.19 AEE = 0.38

  • Overlapping patches tend to result in better AEE!
  • Largest deviation in region of backgound fabric due to

non rigid motion

  • Bias due to bad correspondences
slide-29
SLIDE 29

Crowdsourcing for Ground Truth generation ICVS 2013 29

Results on simple scenes III

Endpoint error with high resolution image: No significant improvement in endpoint error. AEE = 0.20

slide-30
SLIDE 30

Crowdsourcing for Ground Truth generation ICVS 2013 30

Results on complex scenes I

AEE = 0.86 AEE = 1.13 Endpoint error on „Dimetrodon“ and „Urban“ sequences :

  • Larger AEE due to non rigid motion (Dimetrodon)
  • Error due to single layer building in the foreground

(Urban)

slide-31
SLIDE 31

Crowdsourcing for Ground Truth generation ICVS 2013 31

Outline correction of complex scenes

Outlines before... ...and after correction by the workers

slide-32
SLIDE 32

Crowdsourcing for Ground Truth generation ICVS 2013 32

Results on complex scenes II

AEE = 0.46 Endpoint error on Sintel [4] sequence:

  • Larger deviations in complex regions (hair)
slide-33
SLIDE 33

Crowdsourcing for Ground Truth generation ICVS 2013 33

Flow field estimated by crowdsourcing

Estimated accuracy of 1 pixel

Color legend:

slide-34
SLIDE 34

Crowdsourcing for Ground Truth generation ICVS 2013 34

Time effort

Initial segmentation Tracking of outlines Correction of outlines Selection of feature points Selection of motion models

Ground Truth

1h -2h 1h – 2h (simple scenes)

Trained user: 2h – 3h MT urk workers:

  • 1 – 2d in total
  • 2 - 4min. per HIT
slide-35
SLIDE 35

Crowdsourcing for Ground Truth generation ICVS 2013 35

Costs

Costs [$/frame] Outline correction Feature points

10 20 Simple scene (Rubber Whale) Complex scene (Sintel) Trained user (simple scene) 10 $/frame 25 $/frame 20 $/frame 10 $/frame 17 $/frame 3.5 $

slide-36
SLIDE 36

Crowdsourcing for Ground Truth generation ICVS 2013 36

4.Conclusion

slide-37
SLIDE 37

Crowdsourcing for Ground Truth generation ICVS 2013 37

Summary

  • Accuracy is around 1 pixel
  • Reduced accuracy when non rigid motion is

present, due to improper motion models

  • Reduced precision but similar accuracy

compared to trained workers on simple scenes

  • Savings up to 40% per frame

Suitable method, where otherwise no flow estimation at all would be available and pixel accuray is sufficient

slide-38
SLIDE 38

Crowdsourcing for Ground Truth generation ICVS 2013 38

Future work

  • Replace work of trained user:

↳Automatic estimation of flow field ↳Let MT urk workers do the initial segmentation

  • Better and more suitable motion models

Thanks for your attention!

slide-39
SLIDE 39

Crowdsourcing for Ground Truth generation ICVS 2013 39

Future work

  • Replace work of trained user:

↳Automatic estimation of flow field ↳Let MT urk workers do the initial segmentation

  • Better and more suitable motion models

Thanks for your attention!

We can generate cheap ground truth for you! Ask Daniel!

slide-40
SLIDE 40

Crowdsourcing for Ground Truth generation ICVS 2013 40

References

[1] Liu, C., Freeman, W.T., Adelson, E.H., Weiss, Y .: Human-assisted motion annotation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (CVPR08) 0 (2008) 1–8 [2] Baker, S., Scharstein, D., Lewis, J.P ., Roth, S., Black, M.J., Szeliski, R.: A database and evaluation methodology for optical flow. International Journal of Computer Vision 92(1) (2011) 1–31 [3] Meister, S., Jähne, B., Kondermann, D.: Outdoor stereo camera system for the generation of real-world benchmark data sets. Optical Engineering 51 (2012) [4] Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.), ed.: European Conf. onComputer Vision (ECCV). Part IV, LNCS 7577, Springer-Verlag (October 2012)611–625 [5] Sun, D., Roth, S., Black, M.J.: Secrets of optical flow estimation and their principles. In: Porc. IEEE Computer Society COnference on COmputer Vision and Pattern Recognition, (CVPR10), IEEE (2010) 2432–2439

slide-41
SLIDE 41

Crowdsourcing for Ground Truth generation ICVS 2013 41

Spare slides

slide-42
SLIDE 42

Crowdsourcing for Ground Truth generation ICVS 2013 42

Experiments

General procedure:

  • T

est method on datasets with known ground truth to evaluate accuracy

  • Perform multiple runs to evaluate precision
  • T

est on real as well as synthetic data

  • T

est on simple as well as complicated scenes to find out limitations of human perception

  • Accuracy is measured with average

endpoint error (AEE) compared to GT

slide-43
SLIDE 43

Crowdsourcing for Ground Truth generation ICVS 2013 43

Segmentation webinterface

slide-44
SLIDE 44

Crowdsourcing for Ground Truth generation ICVS 2013 44

Feature points webinterface