DC Flow Jia Xu, Intel Labs May 24, 2017 Joint work with Ren Ranftl - - PowerPoint PPT Presentation

β–Ά
dc flow
SMART_READER_LITE
LIVE PREVIEW

DC Flow Jia Xu, Intel Labs May 24, 2017 Joint work with Ren Ranftl - - PowerPoint PPT Presentation

May 28, 2023 396 likes β€’663 views

DC Flow Jia Xu, Intel Labs May 24, 2017 Joint work with Ren Ranftl and Vladlen Koltun DC Flow, Intel Visual Computing Lab Optical Flow Input Output Dense correspondence for each pixel between two frames Optical Flow Key building block

slide-1
SLIDE 1

DC Flow

May 24, 2017

Joint work with RenΓ© Ranftl and Vladlen Koltun

Jia Xu, Intel Labs

DC Flow, Intel Visual Computing Lab

slide-2
SLIDE 2

Optical Flow

Dense correspondence for each pixel between two frames Input Output

slide-3
SLIDE 3

Optical Flow

Key building block for many computer vision systems:

  • Video processing and analytics: motion detection, object

tracking, action recognition, video segmentation, etc.

  • Robotics: visual odometry
  • Autonomous driving
slide-4
SLIDE 4

Optical Flow

Key building block for many computer vision systems:

  • Video processing and analytics: motion detection, object

tracking, action recognition, video segmentation, etc.

  • Robotics: visual odometry
  • Autonomous driving

Challenges:

  • large displacements, textureless regions, motion blur, and

non-rigid deformation.

slide-5
SLIDE 5

Stereo v.s. Optical Flow

Stereo Left image Right image

  • 256 0 256

1-D displacement Optical Flow

  • 256 0 256
  • 256

256

First image Second image 2-D displacement

slide-6
SLIDE 6

Prior Work

Sparse-to-dense regime:

  • Finding matches with hand-crafted feature: Brox and

Malik 2014, EpicFlow (Revaud et al. 2015), DiscreteFlow (Menze et al. 2015), FlowFields (Bailer et al. 2015), CPM (Hu et al. 2016), FullFlow (Chen and Koltun, 2016)

  • Approximation with nearest neighbor search or coarse-

to-fine schemes: Brox and Malik 2014, DiscreteFlow (Menze et al. 2015), FlowFields (Bailer et al. 2015), CPM (Hu et al. 2016) Learning based methods: FlowNet (Dosovitskiy et al. 2015), PatchBatch (Gadot and Wolf 2016), DeepDiscreteFlow (Guney and Geige, 2016) Domain specific methods: SOF (Sevilla-Lara et al. 2016), JHS (Hur and Roth 2016), SDF (Bai et al. 2016)

slide-7
SLIDE 7

Our Idea

Direct 4-D cost volume processing 𝑁× 𝑂 Γ— 𝑆× 𝑆 First frame Second frame

slide-8
SLIDE 8
slide-9
SLIDE 9

Overview

slide-10
SLIDE 10

Learning a Pixel-level Feature Embedding

Positive patch Anchor patch Negative patch conv+relu conv+relu ... conv norm conv+relu conv+relu ... conv norm conv+relu conv+relu ... conv norm Learning with triplet loss xp xa xn

slide-11
SLIDE 11

Direct Cost Volume Processing

Build 4-D cost volume: dot products then stored with 8 bits Semi-Global Matching for optical flow

  • Four cardinal path directions
  • 16-bit filtered cost volume
  • Smallest cost gives the final correspondence
  • Forward and backward matching
slide-12
SLIDE 12

Post-processing

slide-13
SLIDE 13

Post-processing

slide-14
SLIDE 14

Sintel Test Set

50 100 150 200 250 5.1 5.2 5.3 5.4 5.5 5.6 5.7 5.8 5.9 6 Runtime (sec) EPE-all

DeepDiscreteFlow FlowFields+ SPM-BPv2 FlowFields CPM-Flow FullFlow Ours

slide-15
SLIDE 15

Sintel Test Set

slide-16
SLIDE 16

KITTI 2015 Test Set

slide-17
SLIDE 17

KITTI 2015 Test Set

slide-18
SLIDE 18

Run-time

slide-19
SLIDE 19

Qualitative Result - Sintel

slide-20
SLIDE 20

Qualitative Results – KITTI 2015

slide-21
SLIDE 21

Ablation Study - Components

slide-22
SLIDE 22

Ablation Study - Dimensionality

slide-23
SLIDE 23

Summary

  • An optical flow estimation approach that directly constructs

and processes the 4-D cost volume

  • A step towards unifying optical flow and stereo estimation
  • Our approach combines high accuracy with competitive

runtimes, outperforming prior methods on standard benchmarks by significant margins

  • More details
  • http://pages.cs.wisc.edu/~jiaxu/dcflow/
slide-24
SLIDE 24

Thank you!

Acknowledgements: Qifeng Chen(Intel VCL), Alexey Dosovitskiy(Intel VCL)