DC Flow Jia Xu, Intel Labs May 24, 2017 Joint work with Ren Ranftl - PowerPoint PPT Presentation

DC Flow Jia Xu, Intel Labs May 24, 2017 Joint work with René Ranftl and Vladlen Koltun DC Flow, Intel Visual Computing Lab

Optical Flow Input Output Dense correspondence for each pixel between two frames

Optical Flow Key building block for many computer vision systems: - Video processing and analytics: motion detection, object tracking, action recognition, video segmentation, etc. - Robotics: visual odometry - Autonomous driving

Optical Flow Key building block for many computer vision systems: - Video processing and analytics: motion detection, object tracking, action recognition, video segmentation, etc. - Robotics: visual odometry - Autonomous driving Challenges: - large displacements, textureless regions, motion blur, and non-rigid deformation.

Stereo v.s. Optical Flow Stereo -256 0 256 1-D displacement Left image Right image Optical Flow -256 -256 0 256 256 First image 2-D displacement Second image

Prior Work Sparse-to-dense regime: - Finding matches with hand-crafted feature: Brox and Malik 2014, EpicFlow (Revaud et al. 2015), DiscreteFlow (Menze et al. 2015), FlowFields (Bailer et al. 2015), CPM (Hu et al. 2016), FullFlow (Chen and Koltun, 2016) - Approximation with nearest neighbor search or coarse- to-fine schemes: Brox and Malik 2014, DiscreteFlow (Menze et al. 2015), FlowFields (Bailer et al. 2015), CPM (Hu et al. 2016) Learning based methods : FlowNet (Dosovitskiy et al. 2015), PatchBatch (Gadot and Wolf 2016), DeepDiscreteFlow (Guney and Geige, 2016) Domain specific methods : SOF (Sevilla-Lara et al. 2016), JHS (Hur and Roth 2016), SDF (Bai et al. 2016)

Our Idea Direct 4-D cost volume processing 𝑁× 𝑂 × 𝑆× 𝑆 First frame Second frame

Overview

Learning a Pixel-level Feature Embedding xp xa xn Learning with triplet loss norm norm norm conv conv conv ... ... ... conv+relu conv+relu conv+relu conv+relu conv+relu conv+relu Positive patch Anchor patch Negative patch

Direct Cost Volume Processing Build 4-D cost volume: dot products then stored with 8 bits Semi-Global Matching for optical flow Four cardinal path directions • 16-bit filtered cost volume • Smallest cost gives the final correspondence • Forward and backward matching •

Post-processing

Post-processing

Sintel Test Set 250 DeepDiscreteFlow FlowFields+ SPM-BPv2 200 FlowFields CPM-Flow Runtime (sec) FullFlow 150 Ours 100 50 0 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 6 EPE-all

Sintel Test Set

KITTI 2015 Test Set

KITTI 2015 Test Set

Run-time

Qualitative Result - Sintel

Qualitative Results – KITTI 2015

Ablation Study - Components

Ablation Study - Dimensionality

Summary An optical flow estimation approach that directly constructs • and processes the 4-D cost volume A step towards unifying optical flow and stereo estimation • Our approach combines high accuracy with competitive • runtimes, outperforming prior methods on standard benchmarks by significant margins More details • http://pages.cs.wisc.edu/~jiaxu/dcflow/ •

Thank you! Acknowledgements: Qifeng Chen(Intel VCL), Alexey Dosovitskiy(Intel VCL)