learning how to move and where to look from unlabeled
play

Learning How to Move and Where to Look from Unlabeled Video Kristen - PowerPoint PPT Presentation

Feb 02, 2023 •150 likes •768 views

Learning How to Move and Where to Look from Unlabeled Video Kristen Grauman Department of Computer Science University of Texas at Austin Visual recognition Objects amusement park sky Activities Scenes Locations The Wicked Cedar Point

  1. Learning How to Move and Where to Look from Unlabeled Video Kristen Grauman Department of Computer Science University of Texas at Austin

  2. Visual recognition Objects amusement park sky Activities Scenes Locations The Wicked Cedar Point Text / writing Twister Faces Gestures Ferris ride Motions wheel ride Emotions… 12 E Lake Erie water ride tree tree people waiting in line people sitting on ride umbrellas tree maxair carousel deck bench tree pedestrians Kristen Grauman, UT Austin

  3. Visual recognition: applications Organizing visual content Science and medicine AI and autonomous robotics Personal photo/video collections Gaming, HCI, Augmented Reality Surveillance and security Kristen Grauman, UT Austin

  4. Significant recent progress in the field Big labeled Deep learning datasets ImageNet top-5 error (%) 30 25 20 GPU technology 15 10 5 0 1 2 3 4 5 6 Kristen Grauman, UT Austin

  5. Recognition benchmarks PASCAL (2007-12) BSD (2001) Caltech 101 (2004), Caltech 256 (2006) LabelMe (2007) ImageNet (2009) SUN (2010) Places (2014) MS COCO (2014) Visual Genome (2016) Kristen Grauman, UT Austin

  6. How do our systems learn about the visual world today? dog … Expensive and restrictive in scope … boat Kristen Grauman, UT Austin

  7. Big picture goal: Embodied visual learning Status quo : Learn from “disembodied” bag of labeled snapshots. Our goal: Visual learning in the context of acting and moving in the world. Inexpensive and unrestricted in scope Kristen Grauman, UT Austin

  8. Big picture goal: Embodied visual learning Status quo : Learn from “disembodied” bag of labeled snapshots. Our goal: Visual learning in the context of acting and moving in the world. Inexpensive and unrestricted in scope Kristen Grauman, UT Austin

  9. Talk overview Towards embodied visual learning 1. Learning representations tied to ego-motion 2. Learning representations from unlabeled video 3. Learning how to move and where to look Kristen Grauman, UT Austin

  10. The kitten carousel experiment [Held & Hein, 1963] passive kitten active kitten Key to perceptual development: self-generated motion + visual feedback Kristen Grauman, UT Austin

  11. Our idea: Ego-motion vision Goal: Teach computer vision system the connection: “how I move” “how my visual surroundings change” + Unlabeled video Ego-motion motor signals [Jayaraman & Grauman, ICCV 2015] Kristen Grauman, UT Austin

  12. Ego-motion vision: view prediction After moving: Kristen Grauman, UT Austin

  13. Ego-motion vision for recognition Learning this connection requires:  Depth, 3D geometry Also key to  Semantics recognition!  Context Can be learned without manual labels! Our approach: unsupervised feature learning using egocentric video + motor signals [Jayaraman & Grauman, ICCV 2015] Kristen Grauman, UT Austin

  14. Approach idea: Ego-motion equivariance Invariant features: unresponsive to some classes of transformations Simard et al, Tech Report, ’98 Wiskott et al, Neural Comp ’02 Hadsell et al, CVPR ’06 Mobahi et al, ICML ’09 Zou et al, NIPS ’12 Sohn et al, ICML ’12 Cadieu et al, Neural Comp ’12 Goroshin et al, ICCV ’15 Lies et al, PLoS computation biology ’14 … Kristen Grauman, UT Austin

  15. Approach idea: Ego-motion equivariance Invariant features: unresponsive to some classes of transformations Equivariant features: predictably responsive to some classes of transformations, through simple mappings (e.g., linear) “equivariance map” Invariance discards information; equivariance organizes it. Kristen Grauman, UT Austin

  16. Approach idea: Ego-motion equivariance Equivariant embedding Training data organized by ego-motions Unlabeled video + motor signals left turn right turn forward motor signal Learn Pairs of frames related by similar ego-motion should be related by same time feature transformation [Jayaraman & Grauman, ICCV 2015] Kristen Grauman, UT Austin

  17. Approach idea: Ego-motion equivariance Equivariant embedding Training data organized by ego-motions Unlabeled video + motor signals motor signal Learn time [Jayaraman & Grauman, ICCV 2015] Kristen Grauman, UT Austin

  18. Ego-motion equivariant feature learning Given: Desired : for all motions and all images , Unsupervised training � � � � � � Feature space � � (� � ) � � (�� � ) [Jayaraman & Grauman, ICCV 2015] Kristen Grauman, UT Austin

  19. Ego-motion equivariant feature learning Given: Desired : for all motions and all images , Unsupervised training � � � � � � Supervised training softmax loss � � � class , and jointly trained [Jayaraman & Grauman, ICCV 2015] Kristen Grauman, UT Austin

  20. Results: Recognition Learn from unlabeled car video (KITTI) Geiger et al, IJRR ’13 Exploit features for static scene classification (SUN, 397 classes) Xiao et al, CVPR ’10 Kristen Grauman, UT Austin

  21. Results: Recognition Ego-equivariance for unsupervised feature learning 9 SUN scenes: 397 multi-class accuracy 8 7 Accuracy (%) 6 Egomotion-equivariance induces the 5 strongest representations 4 3 2 1 0 1 2 3 4 5 Series1 Series2 Pre-trained models available Series3 Series4 + Hadsell, Chopra, LeCun, “Dimensionality Reduction by Learning an Invariant Mapping”, CVPR 2006 * Agrawal, Carreira, Malik, “Learning to see by moving”, ICCV 2015 Kristen Grauman, UT Austin

  22. Talk overview Towards embodied visual learning 1. Learning representations tied to ego-motion 2. Learning representations from unlabeled video 3. Learning how to move and where to look Kristen Grauman, UT Austin

  23. Learning from arbitrary unlabeled video? Unlabeled video Unlabeled video + ego-motion Kristen Grauman, UT Austin

  24. Learning from arbitrary unlabeled video? Unlabeled video Unlabeled video + ego-motion Kristen Grauman, UT Austin

  25. Background: Slow feature analysis [Wiskott & Sejnowski, 2002] Find functions g(x) that map quickly varying input slowly varying signal x( t ) features y( t ) Figure: Laurenz Wiskott, http://www.scholarpedia.org/article/File:SlowFeatureAnalysis-OptimizationProblem.png Kristen Grauman, UT Austin

  26. Background: Slow feature analysis [Wiskott & Sejnowski, 2002] Find functions g(x) that map quickly varying input slowly varying signal x( t ) features y( t ) Figure: Laurenz Wiskott, http://www.scholarpedia.org/article/File:SlowFeatureAnalysis-OptimizationProblem.png Kristen Grauman, UT Austin

  27. Prior work: Slow feature analysis Wiskott et al, 2002 Hadsell et al. 2006 Mobahi et al. 2009 Bergstra & Bengio 2009 Goroshin et al. 2013 Wang & Gupta 2015 … Learn feature map such that: (invariance) Kristen Grauman, UT Austin

  28. Our idea: Steady feature analysis Higher order temporal coherence Learn feature map such that: (invariance) (equivariance) [Jayaraman & Grauman, CVPR 2016] Kristen Grauman, UT Austin

  29. Our idea: Steady feature analysis Learn feature map such that: (invariance) (equivariance) [Jayaraman & Grauman, CVPR 2016] Kristen Grauman, UT Austin

  30. Datasets Unlabeled video Target task (few labels) Human Motion PASCAL 10 Actions Database (HMDB) SUN 397 Scenes KITTI Video NORB NORB 25 Objects 32 x 32 images or 96 x 96 images Kristen Grauman, UT Austin

  31. Results: Steady feature analysis * ** Multi-class recognition accuracy *Hadsell et al., Dimensionality Reduction by Learning an Invariant Mapping, CVPR’06 **Mobahi et al., Deep Learning from Temporal Coherence in Video, ICML’09 Kristen Grauman, UT Austin

  32. Pre-training a representation Supervised pre-training Labeled images Few labeled images from a related domain for target task Fine-tune Unsupervised “pre-training” Few labeled images Unlabeled video for target task Kristen Grauman, UT Austin

  33. Results: Can we learn more from unlabeled video than “related” labeled images? + HMDB (unlabeled video) CIFAR-100 PASCAL (labeled for other (few img labels) categories) Kristen Grauman, UT Austin

  34. Results: Can we learn more from unlabeled video than “related” labeled images? + HMDB (unlabeled video) Better even than providing 50,000 extra manual labels for auxiliary classification task! CIFAR-100 PASCAL (labeled for other (few img labels) categories) Kristen Grauman, UT Austin

  35. Talk overview Towards embodied visual learning 1. Learning representations tied to ego-motion 2. Learning representations from unlabeled video 3. Learning how to move and where to look Kristen Grauman, UT Austin

  36. Current recognition benchmarks Passive, disembodied snapshots at test time, too BSD (2001) PASCAL (2007-12) Caltech 101 (2004), Caltech 256 (2006) LabelMe (2007) ImageNet (2009) SUN (2010) Places (2014) MS COCO (2014) Visual Genome (2016) Kristen Grauman, UT Austin

  37. Current recognition benchmarks Passive, disembodied snapshots at test time, too Object recognition ? ? ? Scene recognition ? ? Kristen Grauman, UT Austin

  38. Moving to recognize Time to revisit active recognition in challenging settings! Bajcsy 1985, Aloimonos 1988, Ballard 1991, Wilkes 1992, Dickinson 1997, Schiele & Crowley 1998, Tsotsos 2001, Denzler 2002, Soatto 2009, Ramanathan 2011, Borotschnig 2011, … Kristen Grauman, UT Austin

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

10701 Semi supervised learning Can Unlabeled Data improve supervised learning? Important

10701 Semi supervised learning Can Unlabeled Data improve supervised learning? Important

10701 Semi supervised learning Can Unlabeled Data improve supervised learning? Important question! In many cases, unlabeled data is plentiful, labeled data expensive Image classification (x=images from the web, y=image type) Text

886 views • 37 slides
Ac#ve Learning Aarti Singh Machine Learning 10-601 Dec 6, 2011 Slides Courtesy: Burr

Ac#ve Learning Aarti Singh Machine Learning 10-601 Dec 6, 2011 Slides Courtesy: Burr

Ac#ve Learning Aarti Singh Machine Learning 10-601 Dec 6, 2011 Slides Courtesy: Burr Settles, Rui Castro, Rob Nowak 1 Learning from unlabeled data Semi-supervised learning: Design a predictor based on iid unlabeled

738 views • 45 slides
Learning from Unlabeled Data INFO-4604, Applied Machine Learning University of Colorado Boulder

Learning from Unlabeled Data INFO-4604, Applied Machine Learning University of Colorado Boulder

Learning from Unlabeled Data INFO-4604, Applied Machine Learning University of Colorado Boulder December 5-7, 2017 Prof. Michael Paul Types of Learning Recall the definitions of: Supervised learning Most of the semester has been

611 views • 50 slides
Learning on the move Hypertag project Learning on the move 18 month project 3 informal learning

Learning on the move Hypertag project Learning on the move 18 month project 3 informal learning

Learning on the move Hypertag project Learning on the move 18 month project 3 informal learning environments project partners Project aim - to develop interactive mobile multimedia interpretation tools - based on handheld devices -

670 views • 18 slides
Visual Learning with Unlabeled Video and Look-Around Policies Kristen Grauman Department of

Visual Learning with Unlabeled Video and Look-Around Policies Kristen Grauman Department of

Visual Learning with Unlabeled Video and Look-Around Policies Kristen Grauman Department of Computer Science University of Texas at Austin Visual recognition: significant recent progress Big labeled Deep learning datasets ImageNet top-5

804 views • 52 slides
Learning from Limited Labeled Data (but a lot of unlabeled data) NELL as a case study Tom M.

Learning from Limited Labeled Data (but a lot of unlabeled data) NELL as a case study Tom M.

Learning from Limited Labeled Data (but a lot of unlabeled data) NELL as a case study Tom M. Mitchell Carnegie Mellon University Thesis: We will never really understand learning until we build machines that learn many different things,

582 views • 43 slides
Clustering Aarti Singh Slides courtesy: Eric Xing Machine Learning 10-701/15-781 Oct 25, 2010

Clustering Aarti Singh Slides courtesy: Eric Xing Machine Learning 10-701/15-781 Oct 25, 2010

Clustering Aarti Singh Slides courtesy: Eric Xing Machine Learning 10-701/15-781 Oct 25, 2010 Unsupervised Learning Learning from unlabeled/ unannotated data (without supervision) Learning algorithm What can we predict from unlabeled

607 views • 32 slides
Principal Component Analysis 4/7/17 PCA: the setting Unsupervised learning Unlabeled data

Principal Component Analysis 4/7/17 PCA: the setting Unsupervised learning Unlabeled data

Principal Component Analysis 4/7/17 PCA: the setting Unsupervised learning Unlabeled data Dimensionality reduction Simplify the data representation What does the algorithm do? Performs an affine change of basis. Rotate and

426 views • 16 slides
UnTran: Recognizing Unseen Activities with Unlabeled data using Transfer Learning ACM/IEEE

UnTran: Recognizing Unseen Activities with Unlabeled data using Transfer Learning ACM/IEEE

UnTran: Recognizing Unseen Activities with Unlabeled data using Transfer Learning ACM/IEEE IoTDI'18 April 18 th , 2018 Md Abdullah Al Hafiz Khan, Nirmalya Roy Challenges in Scaling Activity Recognition Cross User Diversity Device

451 views • 20 slides
Active Learning with Active Learning with Model Selection Neil Rubens Sugiyama Lab / Tokyo

Active Learning with Active Learning with Model Selection Neil Rubens Sugiyama Lab / Tokyo

Active Learning with Active Learning with Model Selection Neil Rubens Sugiyama Lab / Tokyo Institute of Technology Active Learning (NLP Motivation) NLP (common scenario) Large amounts of unlabeled data Large amounts of unlabeled

706 views • 36 slides
Principal Component Ananalysis 4-8-2016 PCA: the setting Unsupervised learning Unlabeled

Principal Component Ananalysis 4-8-2016 PCA: the setting Unsupervised learning Unlabeled

Principal Component Ananalysis 4-8-2016 PCA: the setting Unsupervised learning Unlabeled data Dimensionality reduction Simplify the data representation Change of basis examples so far Support vector machines Data that's not

423 views • 15 slides
from unlabeled video Kristen Grauman Department of Computer Science The University of T exas at

from unlabeled video Kristen Grauman Department of Computer Science The University of T exas at

Learning image representations from unlabeled video Kristen Grauman Department of Computer Science The University of T exas at Austin Work with Dinesh Jayaraman Learning visual categories Recent major strides in category recognition

751 views • 51 slides
Learning to Extract Entities from Labeled and Unlabeled Text Rosie Jones Language Technologies

Learning to Extract Entities from Labeled and Unlabeled Text Rosie Jones Language Technologies

Learning to Extract Entities from Labeled and Unlabeled Text Rosie Jones Language Technologies Institute School of Computer Science Carnegie Mellon University May 5th, 2005 Extracting Information from Text Yesterday Rio de Janeiro was

1.05k views • 80 slides
K-Means Clustering 3/3/17 Unsupervised Learning We have a collection of unlabeled data

K-Means Clustering 3/3/17 Unsupervised Learning We have a collection of unlabeled data

K-Means Clustering 3/3/17 Unsupervised Learning We have a collection of unlabeled data points. We want to find underlying structure in the data. Examples: Identify groups of similar data points. Clustering Find a better basis

831 views • 14 slides
Decision Trees (Ch. 18.1-18.3) Learning We will (finally) move away from uncertainty (for a

Decision Trees (Ch. 18.1-18.3) Learning We will (finally) move away from uncertainty (for a

Decision Trees (Ch. 18.1-18.3) Learning We will (finally) move away from uncertainty (for a bit) and instead focus on learning Learning algorithms benefit from flexibility to solver a wide range of problems, especially: (1) Cannot explicitly

647 views • 40 slides
Mimicking Word Embeddings using Subword RNNs Yuval Pinter, Robert Guthrie, Jacob Eisenstein

Mimicking Word Embeddings using Subword RNNs Yuval Pinter, Robert Guthrie, Jacob Eisenstein

Mimicking Word Embeddings using Subword RNNs Yuval Pinter, Robert Guthrie, Jacob Eisenstein @yuvalpi Presented at EMNLP September 2017, Copenhagen The Word Embedding Pipeline Unlabeled Unlabeled corpus Unlabeled corpus Unlabeled corpus

1.27k views • 126 slides
Introduction to Virtual Reality Alberto Borghese Department of Computer Science University of

Introduction to Virtual Reality Alberto Borghese Department of Computer Science University of

Introduction to Virtual Reality Alberto Borghese Department of Computer Science University of Milano Laboratory of Motion Analysis & Virtual Reality, MAVR http://mavr.dsi.unimi.it Which is real, which is virtual? Laboratory of Motion

795 views • 34 slides
Methods and Applications Rashid Mehmood School of Engineering Swansea University

Methods and Applications Rashid Mehmood School of Engineering Swansea University

Computational Stochastic Modelling for Large-scale Systems: Methods and Applications Rashid Mehmood School of Engineering Swansea University R.Mehmood@swansea.ac.uk 17 June 2010 Modelling and Analysis of Distributed Systems, SICSA

895 views • 50 slides
CS 561: Artificial Intelligence Instructor: Prof Hadi Moradi Instructor: Prof. Hadi Moradi,

CS 561: Artificial Intelligence Instructor: Prof Hadi Moradi Instructor: Prof. Hadi Moradi,

CS 561: Artificial Intelligence Instructor: Prof Hadi Moradi Instructor: Prof. Hadi Moradi, moradi@usc.edu Lectures: M-Th 09:00-10:40, OHE136 Office hours: MW 2:30 4:00 pm, SAL310, Or by appointment O b i TAs:

672 views • 36 slides
model-based situational awareness P R E S E N T E D B Y T. Russell Gayle, Computer Scientist I

model-based situational awareness P R E S E N T E D B Y T. Russell Gayle, Computer Scientist I

REST -ful GIS services for simplified interoperability in blended LVC and model-based situational awareness P R E S E N T E D B Y T. Russell Gayle, Computer Scientist I n te ra c t i ve S yste m s , S i m u l a t i o n s , a n d A n a l

225 views • 11 slides
Computer Graphics SS2019 Christian Theobalt Mohamed Elgharib Vladislav Golyanik Graphics,

Computer Graphics SS2019 Christian Theobalt Mohamed Elgharib Vladislav Golyanik Graphics,

Computer Vision and Machine Learning for Computer Graphics SS2019 Christian Theobalt Mohamed Elgharib Vladislav Golyanik Graphics, Vision and Video Group, MPI Informatik Overview Organization Introduction Topics Summary

814 views • 63 slides
Faculty Lightning Introductions Scott Stevens Scott Stevens U.N.C.L.E The Benjamin

Faculty Lightning Introductions Scott Stevens Scott Stevens U.N.C.L.E The Benjamin

Faculty Lightning Introductions Scott Stevens Scott Stevens U.N.C.L.E The Benjamin Franklin Computer Game of Electric Charges and Fields Grand Prize Winner in the First International Computer EdGame Challenge, sponsored by Verbatim,

1.18k views • 86 slides
Large Scale Data Engineering Big Data Frameworks: Hadoop & Spark event.cwi.nl/lsde Key

Large Scale Data Engineering Big Data Frameworks: Hadoop & Spark event.cwi.nl/lsde Key

Large Scale Data Engineering Big Data Frameworks: Hadoop & Spark event.cwi.nl/lsde Key premise: divide and conquer work partition w 1 w 2 w 3 worker worker worker r 1 r 2 r 3 combine result event.cwi.nl/lsde Parallelisation

1.08k views • 65 slides
Prospects at ILC A gold place for QCD in the perturbative Regge limit Samuel Wallon 1 1

Prospects at ILC A gold place for QCD in the perturbative Regge limit Samuel Wallon 1 1

Prospects at ILC A gold place for QCD in the perturbative Regge limit Samuel Wallon 1 1 Laboratoire de Physique Thorique Universit Paris Sud Orsay Blois 2007, DESY, Hamburg 1 / 54 Outline QCD in the Regge limit: theoretical status 1 LL

900 views • 54 slides

More recommend