Patch to the Future: Unsupervised Visual Prediction Jacob Walker, - - PowerPoint PPT Presentation

▶
patch to the future unsupervised visual prediction
SMART_READER_LITE
LIVE PREVIEW

Patch to the Future: Unsupervised Visual Prediction Jacob Walker, - - PowerPoint PPT Presentation

Feb 15, 2023 271 likes •884 views

Patch to the Future: Unsupervised Visual Prediction Jacob Walker, Abhinav Gupta, Martial Hebert The Robotics Institute Carnegie Mellon University Visual Prediction Goal Both the what and the how Goal Both the what and the how Goal Both the

slide-1
SLIDE 1

Patch to the Future: Unsupervised Visual Prediction

Jacob Walker, Abhinav Gupta, Martial Hebert The Robotics Institute Carnegie Mellon University

slide-2
SLIDE 2

Visual Prediction

slide-3
SLIDE 3

Goal

Both the what and the how

slide-4
SLIDE 4

Goal

Both the what and the how

slide-5
SLIDE 5

Goal

Both the what and the how

slide-6
SLIDE 6

Goal

Both the what and the how

slide-7
SLIDE 7

Goal

Both the what and the how

slide-8
SLIDE 8

Goal

Both the what and the how

slide-9
SLIDE 9

Background

Data-Driven

Yuen et al. 2010

slide-10
SLIDE 10

Background

Data-Driven

Yuen et al. 2010

slide-11
SLIDE 11

Background

Data-Driven

Yuen et al. 2010

slide-12
SLIDE 12

Background

Agent-Centric

Kitani et al. 2012, Koppula et al. 2013, etc.

slide-13
SLIDE 13

Background

Agent-Centric

Kitani et al. 2012, Koppula et al. 2013, etc.

slide-14
SLIDE 14

Our Approach

Data-Driven

slide-15
SLIDE 15

Our Approach

Data-Driven + Agent-Centric

slide-16
SLIDE 16

Our Approach

Unsupervised

slide-17
SLIDE 17

Limitations

Domain-Dependent

Train Test

slide-18
SLIDE 18

Limitations

Goal-Driven

slide-19
SLIDE 19

Limitations

No Inter-Element Prediction

slide-20
SLIDE 20

Overview

slide-21
SLIDE 21

Representation

Singh et al. 2012

slide-22
SLIDE 22

Action Space

slide-23
SLIDE 23

Scene Interaction

slide-24
SLIDE 24

Scene Interaction

slide-25
SLIDE 25

Scene Interaction

High Low

slide-26
SLIDE 26

Expected Reward

P(Transition) Reward(X,Y,C) E(Reward) = P(T) * R

slide-27
SLIDE 27

Planning

slide-28
SLIDE 28

Planning

slide-29
SLIDE 29

Planning

slide-30
SLIDE 30

Planning

slide-31
SLIDE 31

Planning

slide-32
SLIDE 32

Planning

slide-33
SLIDE 33

Planning

slide-34
SLIDE 34

Planning

slide-35
SLIDE 35

Planning

slide-36
SLIDE 36

Planning

slide-37
SLIDE 37

Planning

slide-38
SLIDE 38

Planning

slide-39
SLIDE 39

Planning

slide-40
SLIDE 40

Planning

slide-41
SLIDE 41

Training

Transitions Scene Interaction

slide-42
SLIDE 42

Training

Transitions Scene Interaction

slide-43
SLIDE 43

Training

Transitions

slide-44
SLIDE 44

Training

Patch Transitions

slide-45
SLIDE 45

Training

Transitions Scene Interaction

slide-46
SLIDE 46

Training

Scene Interaction

slide-47
SLIDE 47

Training

Scene Interaction

slide-48
SLIDE 48

Scene Interaction

Training

slide-49
SLIDE 49

Training

Scene Interaction

slide-50
SLIDE 50

Training

Scene Interaction

slide-51
SLIDE 51

Datasets

  • 183 Videos
  • 139 Training
  • 44 Testing
  • ~300 Minutes
slide-52
SLIDE 52

Qualitative Results

slide-53
SLIDE 53

Qualitative Results

slide-54
SLIDE 54

Quantitative Results

slide-55
SLIDE 55

Data-Driven Active Entity

Quantitative Results

Error (Top 6) NN + Sift-Flow Ours Mean 22.34 14.38 Median 16.68 10.91

slide-56
SLIDE 56

Human-Chosen Active Entity

Quantitative Results

Error (Top 1) NN+Sift-Flow Kitani et al. Ours

Mean 27.55 37.94 21.55 Median 23.77 30.23 14.98

slide-57
SLIDE 57

Second Dataset

VIRAT

slide-58
SLIDE 58
  • Unsupervised method for prediction
  • No explicit modeling of semantics
  • Models appearance changes
  • Code will be available!

Conclusion

slide-59
SLIDE 59

Thank You!