Is Crowdsourcing feasible for optical flow Ground Truth generation? Axel Donath, Daniel Kondermann – HCI Heidelberg ICVS 2013, St.Petersburg Crowdsourcing for Ground Truth generation ICVS 2013 1
Overview 1.Introduction 3.Experiments & Results 2.Ground Truth via Mechanical T urk 4.Conclusion Crowdsourcing for Ground Truth generation ICVS 2013 2
1.Introduction Crowdsourcing for Ground Truth generation ICVS 2013 3
Motivation Start Sequence taken from [3] Crowdsourcing for Ground Truth generation ICVS 2013 4
Large scale dynamic outdoor scene Frame 1 Crowdsourcing for Ground Truth generation ICVS 2013 5
Large scale dynamic outdoor scene Frame 2 Crowdsourcing for Ground Truth generation ICVS 2013 6
Large scale dynamic outdoor scene Frame 3 Crowdsourcing for Ground Truth generation ICVS 2013 7
Large scale dynamic outdoor scene Frame 4 Crowdsourcing for Ground Truth generation ICVS 2013 8
Large scale dynamic outdoor scene Frame 5 Crowdsourcing for Ground Truth generation ICVS 2013 9
Large scale dynamic outdoor scene End Sequence taken from [3] Crowdsourcing for Ground Truth generation ICVS 2013 10
Flow field estimated by algorithm Optical flow algorithm: Classic+NL [5] Color legend: Crowdsourcing for Ground Truth generation ICVS 2013 11
Principles to obtain Ground Truth (1) Measurement with suitable setups E.g. Middlebury dataset [2] (2) Simulate data with computer graphics E.g. Sintel dataset [4] and Middlebury dataset [2] (3) Data can be annotated by humans „Human assisted motion annotation“ with Motion-Annotation-T ool, proposed by Liu et. al [1] Crowdsourcing for Ground Truth generation ICVS 2013 12
Manual labeling and tracking Start Sequence labeled with Motion T ool [1] Crowdsourcing for Ground Truth generation ICVS 2013 13
Manual labeling and tracking Frame 1 Crowdsourcing for Ground Truth generation ICVS 2013 14
Manual labeling and tracking Frame 2 Crowdsourcing for Ground Truth generation ICVS 2013 15
Manual labeling and tracking Frame 3 Crowdsourcing for Ground Truth generation ICVS 2013 16
Manual labeling and tracking Frame 4 Crowdsourcing for Ground Truth generation ICVS 2013 17
Manual labeling and tracking Frame 5 Crowdsourcing for Ground Truth generation ICVS 2013 18
Manual labeling and tracking End Sequence labeled with Motion T ool [1] Crowdsourcing for Ground Truth generation ICVS 2013 19
Idea Outsource manual correction of outlines and finding of feature points to Mechanical T urk Crowdsourcing for Ground Truth generation ICVS 2013 20
2.Ground Truth via Mechanical T urk Crowdsourcing for Ground Truth generation ICVS 2013 21
General workflow Initial segmentation Trained user Tracking of outlines Correction of outlines Laymen via Mechanical T urk Selection of feature points Trained user Selection of motion models Ground Truth Crowdsourcing for Ground Truth generation ICVS 2013 22
Mechanical T urk workflow Download & Merge & 5 „HIT s“ per Blur outlines Review results import outlines outline Correction of outlines Laymen via Mechanical T urk Selection of feature points Webinterface DEMO Crowdsourcing for Ground Truth generation ICVS 2013 23
Mechanical T urk workflow Webinterface DEMO Correction of outlines Laymen via Mechanical T urk Selection of feature points Divide image Download & Import feature 8 points per into patches Review results points patch Crowdsourcing for Ground Truth generation ICVS 2013 24
3.Experiments and Results Crowdsourcing for Ground Truth generation ICVS 2013 25
Outline correction of simple scenes Outlines before... ...and after correction by the workers Crowdsourcing for Ground Truth generation ICVS 2013 26
Results on simple scenes I Endpoint error of six runs on the „Rubber Whale“ sequence: AEE = 0.79 AEE = 0.51 AEE = 0.37 AEE = 0.63 AEE = 0.37 AEE = 0.47 All images are normalized to max. endpoint error of 2 pix Crowdsourcing for Ground Truth generation ICVS 2013 27
Results on simple scenes II Endpoint error with overlapping patches: AEE = 0.19 AEE = 0.38 • Overlapping patches tend to result in better AEE ! • Largest deviation in region of backgound fabric due to non rigid motion • Bias due to bad correspondences Crowdsourcing for Ground Truth generation ICVS 2013 28
Results on simple scenes III Endpoint error with high resolution image: AEE = 0.20 No significant improvement in endpoint error. Crowdsourcing for Ground Truth generation ICVS 2013 29
Results on complex scenes I Endpoint error on „Dimetrodon“ and „Urban“ sequences : AEE = 0.86 AEE = 1.13 • Larger AEE due to non rigid motion (Dimetrodon) • Error due to single layer building in the foreground (Urban) Crowdsourcing for Ground Truth generation ICVS 2013 30
Outline correction of complex scenes Outlines before... ...and after correction by the workers Crowdsourcing for Ground Truth generation ICVS 2013 31
Results on complex scenes II Endpoint error on Sintel [4] sequence: AEE = 0.46 • Larger deviations in complex regions (hair) Crowdsourcing for Ground Truth generation ICVS 2013 32
Flow field estimated by crowdsourcing Color legend: Estimated accuracy of 1 pixel Crowdsourcing for Ground Truth generation ICVS 2013 33
Time effort Initial segmentation 1h – 2h (simple Tracking of outlines scenes) Trained user: 2h – 3h Correction of outlines MT urk workers : • 1 – 2d in total • 2 - 4min. per HIT Selection of feature points 1h -2h Selection of motion models Ground Truth Crowdsourcing for Ground Truth generation ICVS 2013 34
Costs Outline correction Feature points Simple scene (Rubber Whale) 3.5 $ 10 $/frame 17 $/frame Complex scene (Sintel) 25 $/frame Trained user (simple scene ) 10 $/frame 20 $/frame 10 Costs [$/frame] 20 Crowdsourcing for Ground Truth generation ICVS 2013 35
4.Conclusion Crowdsourcing for Ground Truth generation ICVS 2013 36
Summary • Accuracy is around 1 pixel • Reduced accuracy when non rigid motion is present, due to improper motion models • Reduced precision but similar accuracy compared to trained workers on simple scenes • Savings up to 40% per frame Suitable method , where otherwise no flow estimation at all would be available and pixel accuray is sufficient Crowdsourcing for Ground Truth generation ICVS 2013 37
Future work • Replace work of trained user: ↳Automatic estimation of flow field ↳ Let MT urk workers do the initial segmentation • Better and more suitable motion models Thanks for your attention! Crowdsourcing for Ground Truth generation ICVS 2013 38
Future work • Replace work of trained user: ↳Automatic estimation of flow field ↳ Let MT urk workers do the initial segmentation • Better and more suitable motion models We can generate cheap ground truth for you! Ask Daniel! Thanks for your attention! Crowdsourcing for Ground Truth generation ICVS 2013 39
References [1] Liu, C., Freeman, W.T., Adelson, E.H., Weiss, Y . : Human-assisted motion annotation. IEEE Computer Society Conference on Computer Vision and Pattern Recognition, (CVPR08) 0 (2008) 1–8 [2] Baker, S., Scharstein, D., Lewis, J.P ., Roth, S., Black, M.J., Szeliski, R. : A database and evaluation methodology for optical flow. International Journal of Computer Vision 92(1) (2011) 1–31 [3] Meister, S., Jähne, B., Kondermann, D. : Outdoor stereo camera system for the generation of real-world benchmark data sets. Optical Engineering 51 (2012) [4] Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J. : A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.), ed.: European Conf. onComputer Vision (ECCV). Part IV, LNCS 7577, Springer-Verlag (October 2012)611–625 [5] Sun, D., Roth, S., Black, M.J. : Secrets of optical flow estimation and their principles. In: Porc. IEEE Computer Society COnference on COmputer Vision and Pattern Recognition, (CVPR10), IEEE (2010) 2432–2439 Crowdsourcing for Ground Truth generation ICVS 2013 40
Spare slides Crowdsourcing for Ground Truth generation ICVS 2013 41
Experiments General procedure: • T est method on datasets with known ground truth to evaluate accuracy • Perform multiple runs to evaluate precision • T est on real as well as synthetic data • T est on simple as well as complicated scenes to find out limitations of human perception • Accuracy is measured with average endpoint error (AEE) compared to GT Crowdsourcing for Ground Truth generation ICVS 2013 42
Segmentation webinterface Crowdsourcing for Ground Truth generation ICVS 2013 43
Feature points webinterface Crowdsourcing for Ground Truth generation ICVS 2013 44
Recommend
More recommend
