Dense Optical Flow Prediction from a Static Image Jacob Walker, - - PowerPoint PPT Presentation

Dense Optical Flow Prediction from a Static Image

Jacob Walker, Abhinav Gupta, and Martial Hebert

Aysun Koçak

Introduction

• Static images contain action and motion

information

• There are several approaches
• Common one is agent-centric

Activity forecasting, ECCV, 2012 Patch to the future: Unsupervised visual prediction, CVPR, 2014

Introduction

• Two disadvantages
• motion is modeled as a trajectory
• shown to perform in restrictive domains
• This paper proposes a generalized framework
• single or multiple agent
• indoor or outdoor environment

Related Work

• Non-parametric methods
• data-driven
• do not make any assumptions about the underlying

scene

A data-driven approach for event predictjon, ECCV, 2010

Related Work

• Parametric methods
• domain-specific approaches
• assumptions on what are the active elements

A hierarchical representatjon for future actjon predictjon, ECCV 2014

Related Work

• Hybrid methods

Patch to the Future: Unsupervised Visual Prediction, CVPR 2014

The Proposed Method

• Predict motion of each and every pixel in terms of optical

flow

• CNN model for motion prediction
• Agent-free
• Makes almost no assumptions about the underlying

scene

• Also makes long-range prediction

The Proposed Method

• Learn a mapping between the input RGB image and the
• utput space

Training Framework

• 1. Extract Optical Flow from Video Frames
• UCF-101 is action recognition data set consists of

13320 videos from 101 action categories

• HMDB-51 has 6849 videos from 51 action categories
• Model trained with over 350,000 frames from the

UCF-101 and over 150,000 frames from the HMDB- 51

• Labelled with DeepFlow
• Data augmentation

Training Framework

• 2. Assign Optical Flow Vectors to Clusters
• Regression as Classification
• motion estimation can be posed as a regression

problem

• but it has a drawback
• so reformulate as classification
• quantize optical flow vectors into 40 clusters by k-means

Training Framework

• 3. Train Convolutional Neural Network for a

Pixel Classification Problem

• Loss function

Experiments

• Test on
• UCF-101
• HMDB-51
• KTH contains 600 videos from 6 actions
• 3-fold cross-validation

Experiments

• Metrics
• Direction similarity
• Orientation similarity
• End-Point-Error

Experiments

• Metrics
• Top 5
• Top 10

Experiments

• UCF Dataset

Experiments

• HMDB Dataset

Experiments

Multi-Frame Prediction

Multi-Frame Prediction

Long-term Recurrent Convolutional Networks for Visual Recognition and Description

Thank You