actbert learning global local video text representations
play

ActBERT: Learning Global-Local Video-Text Representations Linchao - PowerPoint PPT Presentation

Feb 10, 2024 •313 likes •571 views

ActBERT: Learning Global-Local Video-Text Representations Linchao Zhu Self-supervised pretraining Single modal pretraining Image Jigsaw, CPC, MoCO, SimCLR Video Shuffle and Learn, Video GAN Text Word2Vec, GPT, BERT

  1. ActBERT: Learning Global-Local Video-Text Representations Linchao Zhu

  2. Self-supervised pretraining • Single modal pretraining • Image • Jigsaw, CPC, MoCO, SimCLR • Video • Shuffle and Learn, Video GAN • Text • Word2Vec, GPT, BERT • Multi-modal pretraining • Image-text • Vilbert, LXMERT, VisualBERT, VLBERT, UNITER, Unified VLP • Video-text • HowTo100M, VideoBERT, CBT, MIL-NCE

  3. Video and text pre-training • Instructional videos • A natural source for video and text representation learning • Instructions are available from ASR • Diverse domains • Cooking • Assembling furniture

  4. Video and text pre-training • Howto100M Miech et al., HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips, ICCV 2019

  5. Video and text pre-training • VideoBERT Sun et al., VideoBERT: A Joint Model for Video and Language Representation Learning, ICCV 2019.

  6. ActBERT • Decouple verbs and nouns • Add spinach -> verb: “add”, noun: “spinach” • Verb label is extracted from the description. • Train a 3D ConvNet for verb classification • Object label can be produced from a pre- trained Faster R-CNN.

  7. Tangled transformer ActBERT with Tangled Joint embedding Transformer / BERT Action Language Clip-level Language Region feature model feature model feature Rotate Rotate shrimp balls shrimp balls

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Learning text representations from character-level data Grzegorz Chrupa la Department of

Learning text representations from character-level data Grzegorz Chrupa la Department of

Learning text representations from character-level data Grzegorz Chrupa la Department of Communication and Information Sciences Tilburg University CLIN 2013 Chrupa la (UvT) Text representations CLIN 2013 1 / 19 Text representations

221 views • 19 slides
7. Video databases Video data representations Video = time-ordered sequence of correlated

7. Video databases Video data representations Video = time-ordered sequence of correlated

7. Video databases Video data representations Video = time-ordered sequence of correlated images ( frames ) Video signal representations originate from TV technology; different standards in USA (NTSC) and Europe (PAL, SECAM) 25-30

571 views • 16 slides
Inductive Learning Algorithms and Representations for Text Categorization David Heckerman Susan

Inductive Learning Algorithms and Representations for Text Categorization David Heckerman Susan

Inductive Learning Algorithms and Representations for Text Categorization David Heckerman Susan Dumais John Platt Mehran Sahami Presenter: Haoran Hou Text Categorization real-time sorting emails/files topic identification structured search

282 views • 25 slides
Unsupervised Le Learning of Video Representations using LS LSTMs Srivastava et al. University

Unsupervised Le Learning of Video Representations using LS LSTMs Srivastava et al. University

Unsupervised Le Learning of Video Representations using LS LSTMs Srivastava et al. University of Toronto Presented by Shyam Tailor The Overall Idea Take a sequence of images and encode into a fixed size latent representation Decode

210 views • 20 slides
Learning Graph Representations for Video Understanding Xiaolong Wang Carnegie Mellon University

Learning Graph Representations for Video Understanding Xiaolong Wang Carnegie Mellon University

Learning Graph Representations for Video Understanding Xiaolong Wang Carnegie Mellon University Computer Vision Dog He et al. Mask R-CNN. ICCV 2017. Gler et al. DensePose: Dense Human Pose Estimation In The Wild. CVPR 2018. Deep Learning

941 views • 64 slides
Investigating the forensic applications of global and local temporal representations of speech for

Investigating the forensic applications of global and local temporal representations of speech for

Investigating the forensic applications of global and local temporal representations of speech for dialect discrimination Leah Bradshaw, Vincent Hughes, and Eleanor Chodroff Department of Language and Linguistic Science University of York

462 views • 29 slides
Privacy-preserving Neural Representations of Text Maximin Coavoux Shashi Narayan Shay B.

Privacy-preserving Neural Representations of Text Maximin Coavoux Shashi Narayan Shay B.

Privacy-preserving Neural Representations of Text Maximin Coavoux Shashi Narayan Shay B. Cohen University of Edinburgh ILCC EMNLP 2018 Brussels 1 / 15 Context: Privacy and Neural Networks Machine learning uses data (e.g. UGC)

481 views • 19 slides
Local Representations of Binding Randy Pollack LFCS, University of Edinburgh Joint work with

Local Representations of Binding Randy Pollack LFCS, University of Edinburgh Joint work with

Local Representations of Binding Local Representations of Binding Randy Pollack LFCS, University of Edinburgh Joint work with James McKinna, Christian Urban, Arthur Charguraud, Brian Aydemir, Benjamin Pierce, Stephanie Weirich Version of

754 views • 47 slides
Object Detection Deep ConvNets for Recognition for... Images (global) Objects (local) Video

Object Detection Deep ConvNets for Recognition for... Images (global) Objects (local) Video

Day 3 Lecture 4 Object Detection Deep ConvNets for Recognition for... Images (global) Objects (local) Video (2D+T) 2 Slide Credit: Xavier Gir Object Detection The task of assigning a label and a bounding box to all objects in the image

510 views • 31 slides
Presentation Video Retrieval using Automatically Recovered Slide and Spoken Text Matthew Cooper

Presentation Video Retrieval using Automatically Recovered Slide and Spoken Text Matthew Cooper

Presentation Video Retrieval using Automatically Recovered Slide and Spoken Text Matthew Cooper FX Palo Alto Laboratory Palo Alto, CA 94034 USA cooper@fxpal.com ABSTRACT Video is becoming a prevalent medium for e-learning. Lecture videos

394 views • 7 slides
Integrating Drawing Tablet and Video Capturing/Sharing to Facilitate Student Learning ACM Global

Integrating Drawing Tablet and Video Capturing/Sharing to Facilitate Student Learning ACM Global

Integrating Drawing Tablet and Video Capturing/Sharing to Facilitate Student Learning ACM Global Computing Education CompEd19 / May 18 / Chengdu, China Chen-Wei Wang York University, Toronto, Canada Challenges of Undergraduate Teaching 1.

643 views • 24 slides
Sequence to Sequence Video to Text Venugopalan et al. Given a variable-length sequence of

Sequence to Sequence Video to Text Venugopalan et al. Given a variable-length sequence of

Garrett Bingham Sequence to Sequence Video to Text Venugopalan et al. Given a variable-length sequence of video frames, Prev: Title Slide generate a variable-length natural language Problem description of the video. Next: Motivation

850 views • 15 slides
from unlabeled video Kristen Grauman Department of Computer Science The University of T exas at

from unlabeled video Kristen Grauman Department of Computer Science The University of T exas at

Learning image representations from unlabeled video Kristen Grauman Department of Computer Science The University of T exas at Austin Work with Dinesh Jayaraman Learning visual categories Recent major strides in category recognition

751 views • 51 slides
Global and local alignments Global vs. local alignments Global: align all nucleotides

Global and local alignments Global vs. local alignments Global: align all nucleotides

Global and local alignments Global vs. local alignments Global: align all nucleotides Local: align subsequences with best score Align these sequences: GCAT, GCT (match = 1, mismatch = -1, gap = -1) global alignment: local alignment: ?

506 views • 19 slides
Dense Encoding for Video-to-Text Matching Jianfeng Dong 1 , Xirong Li 2 , Chaoxi Xu 2 , Jing Cao 2

Dense Encoding for Video-to-Text Matching Jianfeng Dong 1 , Xirong Li 2 , Chaoxi Xu 2 , Jing Cao 2

Dense Encoding for Video-to-Text Matching Jianfeng Dong 1 , Xirong Li 2 , Chaoxi Xu 2 , Jing Cao 2 , Xun Wang 1 , Gang Yang 2 1 Zhejiang Gongshang University 2 AI & Media Computing Lab, Renmin University of China Video to Text (VTT) Task @

341 views • 18 slides
LOCAL REPRESENTATIONS OF BRAID GROUP AND ITS GENERALIZATIONS Yu. A. Mikhalchishina Knots,

LOCAL REPRESENTATIONS OF BRAID GROUP AND ITS GENERALIZATIONS Yu. A. Mikhalchishina Knots,

LOCAL REPRESENTATIONS OF BRAID GROUP AND ITS GENERALIZATIONS Yu. A. Mikhalchishina Knots, braids and automorphism groups NOVOSIBIRSK 2014 Yu. A. Mikhalchishina LOCAL REPRESENTATIONS OF BRAID GROUP AND ITS GENERALIZA The braid group.

273 views • 13 slides
Residual categories of Grassmannians Maxim Smirnov University of Augsburg October 1, 2020 based

Residual categories of Grassmannians Maxim Smirnov University of Augsburg October 1, 2020 based

Residual categories of Grassmannians Maxim Smirnov University of Augsburg October 1, 2020 based on joint work with Alexander Kuznetsov Exceptional collections X smooth projective variety over C D b ( X ) bounded derived category of

426 views • 19 slides
Decision Trees Administrative Homework goes out today, please contact Isaac Tian

Decision Trees Administrative Homework goes out today, please contact Isaac Tian

CSE 446: Week 2 Decision Trees Administrative Homework goes out today, please contact Isaac Tian (iytian@cs.washington.edu) if you have not been added to Gradescope Recap: Algorithm until Base Case 1 or Base Case 2 is reached: step over

239 views • 21 slides
Robust Attribution Regularization 2 , Yingyu Liang 1 , Jiefeng Chen *1 , Xi Wu *2 , Vaibhav

Robust Attribution Regularization 2 , Yingyu Liang 1 , Jiefeng Chen *1 , Xi Wu *2 , Vaibhav

Robust Attribution Regularization 2 , Yingyu Liang 1 , Jiefeng Chen *1 , Xi Wu *2 , Vaibhav Rastogi Somesh Jha 1,3 1 University of Wisconsin-Madison 2 Google 3 XaiPient NeurIPS2019 *Equal contribution Work done while at UW-Madison

457 views • 31 slides
On the K onigHall theorem for multidimensional matrices Anna Taranenko Sobolev Institute

On the K onigHall theorem for multidimensional matrices Anna Taranenko Sobolev Institute

On the K onigHall theorem for multidimensional matrices Anna Taranenko Sobolev Institute of Mathematics, Novosibirsk, Russia Groups and Graphs, Designs and Dynamics Yichang, China, 2019 Anna Taranenko Multidimensional K onigHall

867 views • 67 slides
Ramiro Sarabia Demetrius Cooper @ramsarabia on IG, Twtr, @thatsmycheese on IG LinkedIn

Ramiro Sarabia Demetrius Cooper @ramsarabia on IG, Twtr, @thatsmycheese on IG LinkedIn

Ramiro Sarabia Demetrius Cooper @ramsarabia on IG, Twtr, @thatsmycheese on IG LinkedIn Amy Lombardi Austin Music Foundation TuneCore Today well cover Considerations for music promo in 2019 Prototyping Create Your Own

527 views • 9 slides
Efficient Interpolant Generation in Satisfiability Modulo Linear Integer Arithmetic Alberto

Efficient Interpolant Generation in Satisfiability Modulo Linear Integer Arithmetic Alberto

Deduction at Scale Seminar 2011 Efficient Interpolant Generation in Satisfiability Modulo Linear Integer Arithmetic Alberto Griggio FBK-IRST, Trento joint work with Thi Thieu Hoa Le and Roberto Sebastiani, DISI - Univ. Trento Introduction

767 views • 39 slides
An Object-Oriented Dynamic Logic with Updates Andr Platzer University of Karlsruhe Andr

An Object-Oriented Dynamic Logic with Updates Andr Platzer University of Karlsruhe Andr

An Object-Oriented Dynamic Logic with Updates Andr Platzer University of Karlsruhe Andr Platzer. ODL Updates p.1/29 Motivation J AVA C ARD DL Andr Platzer. ODL Updates p.2/29 Motivation J AVA C ARD

882 views • 40 slides
Lecture #5: Higher-Order Functions Do You Understand the Machinery? (I) What is printed (0, 1, or

Lecture #5: Higher-Order Functions Do You Understand the Machinery? (I) What is printed (0, 1, or

Lecture #5: Higher-Order Functions Do You Understand the Machinery? (I) What is printed (0, 1, or error) and why? def f(): return 0 def g(): print(f()) def h(): def f(): return 1 g() h() Last modified: Sun Feb 19 14:55:31 2017 CS61A:

209 views • 4 slides

More recommend