Patch to the Future: Unsupervised Visual Prediction Jacob Walker, - PowerPoint PPT Presentation

Feb 15, 2023 •271 likes •876 views

Patch to the Future: Unsupervised Visual Prediction Jacob Walker, Abhinav Gupta, Martial Hebert The Robotics Institute Carnegie Mellon University Visual Prediction Goal Both the what and the how Goal Both the what and the how Goal Both the

Patch to the Future: Unsupervised Visual Prediction Jacob Walker, Abhinav Gupta, Martial Hebert The Robotics Institute Carnegie Mellon University
Visual Prediction
Goal Both the what and the how
Goal Both the what and the how
Goal Both the what and the how
Goal Both the what and the how
Goal Both the what and the how
Goal Both the what and the how
Background Data-Driven Yuen et al. 2010
Background Data-Driven Yuen et al. 2010
Background Data-Driven Yuen et al. 2010
Background Agent-Centric Kitani et al. 2012, Koppula et al. 2013, etc.
Background Agent-Centric Kitani et al. 2012, Koppula et al. 2013, etc.
Our Approach Data-Driven
Our Approach Data-Driven + Agent-Centric
Our Approach Unsupervised
Limitations Domain-Dependent Train Test
Limitations Goal-Driven
Limitations No Inter-Element Prediction
Overview
Representation Singh et al. 2012
Action Space
Scene Interaction
Scene Interaction
Scene Interaction High Low
Expected Reward P(Transition) Reward(X,Y,C) E(Reward) = P(T) * R
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Planning
Training Transitions Scene Interaction
Training Transitions Scene Interaction
Training Transitions
Training Patch Transitions
Training Transitions Scene Interaction
Training Scene Interaction
Training Scene Interaction
Training Scene Interaction
Training Scene Interaction
Training Scene Interaction
Datasets • 183 Videos • 139 Training • 44 Testing • ~300 Minutes
Qualitative Results
Qualitative Results
Quantitative Results
Quantitative Results Data-Driven Active Entity Error (Top 6) NN + Sift-Flow Ours Mean 22.34 14.38 Median 16.68 10.91
Quantitative Results Human-Chosen Active Entity Error (Top 1) NN+Sift-Flow Kitani et al. Ours Mean 27.55 37.94 21.55 Median 23.77 30.23 14.98
VIRAT Second Dataset
Conclusion • Unsupervised method for prediction • No explicit modeling of semantics • Models appearance changes • Code will be available!
Thank You!

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Unsupervised Visual Representation Learning by Context Prediction Berkan Demirel Most slides in

Unsupervised Visual Representation Learning by Context Prediction Berkan Demirel Most slides in this representation are adopted from authors' original presentation at ICCV 2015 ImageNet + Deep Learning Beagle - Image Retrieval - Detection

2.18k views • 33 slides

Un Unsupervised Visu Visual Re Representation Le Learn rning by by Co Context Pr

Un Unsupervised Visu Visual Re Representation Le Learn rning by by Co Context Pr Prediction Carl Doersch, Abhinav Gupta, Alexei A. Efros Presenter: Yiming Pang Outline Motivation Approach Experiment Low-level

499 views • 34 slides

PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu,

PlaneMatch: Patch Coplanarity Prediction for Robust RGB-D Reconstruction Yifei Shi, Kai Xu, Matthias Niessner, Szymon Rusinkiewicz, Thomas Funkhouser Princeton University National University of Defense Technology Technical University of Munich

417 views • 38 slides

Towards Assumption-free Unsupervised Domain Adaptation for Visual recognition

Towards Assumption-free Unsupervised Domain Adaptation for Visual recognition Outline Background and Motivations Unsupervised Domain Adaptation - Estimating label information in

1.51k views • 47 slides

Transformation Equivariance vs. Invariance: Unsupervised Learning of Visual Representations

Transformation Equivariance vs. Invariance: Unsupervised Learning of Visual Representations Guo-Jun Qi guojunq@gmail.com Laboratory for MA chine P erception and LE arning ( MAPLE ) & Futurewei Technologies (Huawei Research USA) Laboratory

1.24k views • 58 slides

Overview Neural networks and visual processing So far we have discussed unsupervised learning up

Overview Neural networks and visual processing So far we have discussed unsupervised learning up to V1 For most technology applications (except perhaps compression), Mark van Rossum V1 description is not enough. Yet it is not clear how to

188 views • 18 slides

Unsupervised Learning of Visual Representation by Solving Jigsaw Puzzles, ECCV 16 2018/11/27

Unsupervised Learning of Visual Representation by Solving Jigsaw Puzzles, ECCV 16 2018/11/27 20173130 Jaeyoon Kim CS688 Paper Presentation Image Retrieval with Mixed initiative and Multimodal Feedback, BMVC 18 The system based on

336 views • 18 slides

Unsupervised learning of visual representations using videos X. Wang and A. Gupta ICCV 2015

Unsupervised learning of visual representations using videos X. Wang and A. Gupta ICCV 2015 Experiment presentation by Ashish Bora Motivation Supervised methods work very well But labels are expensive Lot of unlabeled data is

681 views • 32 slides

Fragment binding prediction using unsupervised learning of ligand substructure binding sites

Fragment binding prediction using unsupervised learning of ligand substructure binding sites Grace Tang Altman Lab July 20, 2013 Berlin, Germany 1 Structure Based Virtual Screening Virtual Databases End Goal Screening ZINC 1 21 million

491 views • 15 slides

Unsupervised Learning of Visual Structure Using Predictive Generative Networks William Lotter,

Unsupervised Learning of Visual Structure Using Predictive Generative Networks William Lotter, Gabriel Kreiman & David Cox Harvard University, Cambridge, USA Article overview by Ilya Kuzovkin Computational Neuroscience Seminar University

964 views • 45 slides

A Hybrid Deep Learning Approach For Chaotic Time Series Prediction Based On Unsupervised Feature

A Hybrid Deep Learning Approach For Chaotic Time Series Prediction Based On Unsupervised Feature Learning Norbert Ayine Agana Advisor: Abdollah Homaifar Autonomous Control & Information Technology Institute (ACIT), Departmenent of

379 views • 35 slides

Detecting and Characterizing Events Unsupervised Machine Learning for Social Science Allison

Detecting and Characterizing Events Unsupervised Machine Learning for Social Science Allison J.B. Chaney Princeton University supervised learning unsupervised learning methods input output prediction exploration goals 9 3 1 ? 5 6

920 views • 59 slides

An Analysis of Patch Plausibility and Correctness of

An Analysis of Patch Plausibility and Correctness of Generate-And- Validate Patch Genera8on System Zichao Qi , Fan Long, Sara Achour, and Mar8n Rinard MIT

323 views • 29 slides

Combining Visual Analytics and Machine Learning for Route Choice Prediction Application to Pre

Combining Visual Analytics and Machine Learning for Route Choice Prediction Application to Pre Tactical Traffic Forecast Rodrigo Marcos Data Scientist, Nommon Solutions and Technologies SESAR Innovation Days Beograd, 29 th November 2017 Scope

461 views • 22 slides

Summary of part I: prediction and RL Prediction is important for action selection The

Summary of part I: prediction and RL Prediction is important for action selection The problem: prediction of future reward The algorithm: temporal difference learning Neural implementation: dopamine dependent learning in BG A

939 views • 60 slides

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised

UNSUPERVISED LEARNING, CLUSTERING UNSUPERVISED LEARNING UNSUPERVISED LEARNING Supervised learning: X - y pairs, f(x) function approximation Unsupervised learning: only X, no y Exploring the space of X measurements,

440 views • 14 slides

Epoxy Rock Repairs Jim McAlister The Georgia Aquarium Inc. REPAIRING UNDERWATER ARTIFICIAL

Epoxy Rock Repairs Jim McAlister The Georgia Aquarium Inc. REPAIRING UNDERWATER ARTIFICIAL ROCKWORK Determining which product to use Mixing the epoxy Preparing the surface Placing the patch Finishing the patch ROCKWORK =

728 views • 24 slides

th An rs 4 th Ame Ameri rican Le Legion Riders Annual Sum Summit Merry-Go-Round Round

th An rs 4 th Ame Ameri rican Le Legion Riders Annual Sum Summit Merry-Go-Round Round Robin Unity Ride District Patch Sonny Decker & Jeff Hawk Merry-Go-Round & The Round Robin These month-long events are intended for our

841 views • 22 slides

NVIDIA TURF EFFECTS: MASSIVE GRASS RENDERING WITH DYNAMIC SIMULATION EVGENY MAKAROV , DEVTECH

NVIDIA TURF EFFECTS: MASSIVE GRASS RENDERING WITH DYNAMIC SIMULATION EVGENY MAKAROV , DEVTECH NVIDIA OUTLINE Real-time grass rendering Turf Effects: grass rendering Turf Effects: dynamic simulation Grass authoring Q&A REAL-TIME GRASS

854 views • 36 slides

How to install Patch Manager Plus at AWS Steps to install Patch Manager Plus at AWS 1. Login to

How to install Patch Manager Plus at AWS Steps to install Patch Manager Plus at AWS 1. Login to Aws console 2. Select launch instance 3. Select your OS 4. Choose instance type 5. Select VPC 6. Add storage 7. Tag your instance 8. Configure security

185 views • 14 slides

Indy Greenways INDY GREENWAYS Girl Scout Patch Program Welcome and Introductions INDY GREENWAYS

Indy Greenways INDY GREENWAYS Girl Scout Patch Program Welcome and Introductions INDY GREENWAYS Girl Scout Patch Program INDY GREENWAYS Girl Scout Patch Program Have you used Indy Greenways? INDY GREENWAYS Girl Scout Patch Program

459 views • 29 slides

Outline Texture Synthesis What is Texture? How to Synthesize? Graphcut Textures

Graphcut Textures: Image and Video Synthesis using Graph Cuts (Kwatra, et al) presented by Adeel Bhutta Outline Texture Synthesis What is Texture? How to Synthesize? Graphcut Textures Main Idea, Contribution Patch

588 views • 20 slides

Garbage Patch What is it and why is it occuring? -600,000 Square Miles (twice the size of Texas)

Great Pacific Garbage Patch What is it and why is it occuring? -600,000 Square Miles (twice the size of Texas) -1.8 Trillion pieces of microplastic (not biodegradable) -88,000 Tons -Comprised of the Western Garbage Patch (near Japan) and

388 views • 6 slides

Outline of remarks Habitat Corridors: what are they 1. Definitions of corridor and what are they

Outline of remarks Habitat Corridors: what are they 1. Definitions of corridor and what are they good for? a) Structural b) Functional 2. Effects of fragmentation 3. Advantages of corridors William Z. Lidicker, Jr. Museum of Vertebrate

540 views • 8 slides