self attention for generative models
play

Self-Attention For Generative Models Ashish Vaswani and Anna Huang - PowerPoint PPT Presentation

Dec 28, 2022 •137 likes •1.3k views

Self-Attention For Generative Models Ashish Vaswani and Anna Huang Joint work with: Noam Shazeer, Niki Parmar, Lukasz Kaiser, Illia Polosukhin, Llion Jones, Justin Gilmer, David Bieber, Jonathan Frankle, Jakob Uszkoreit, and others. Learning

  1. Continuations to given initial motif Given motif RNN-LSTM

  2. Continuations to given initial motif Given motif RNN-LSTM

  3. Continuations to given initial motif Given motif RNN-LSTM Transformer

  4. Continuations to given initial motif Given motif RNN-LSTM Transformer

  5. Continuations to given initial motif Given motif RNN-LSTM Transformer Music Transformer

  6. Continuations to given initial motif Given motif RNN-LSTM Transformer Music Transformer

  7. Self-Similarity in Music

  8. Sample from Music Transformer

  10. Attention: a weighted average TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90 TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90

  11. Attention: a weighted average TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90 TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90

  12. Convolution: Different linear transformations by relative position. TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90 TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90

  13. Relative attention (Shaw et al, 2018) Multihead attention + convolution? TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90 TimeShift100 TimeShift100 TimeShift30 NoteOn60 TimeShift20 NoteOn62 TimeShift90 NoteOff62 NoteOff60 TimeShift90

  14. Closer look at attention QE rT

  15. Closer look at relative attention Modulated by relative positions 0,0 0 0,1 1 0,2 2 1,0 -1 1,1 0 1,2 1 2,0 -2 2,1 -1 2,2 0 QE rT

  16. Machine Translation (Shaw et al, 2018) Model Position BLEU BLEU Representati En-De En-Fr on Transformer Big Absolute 27.9 41.3 Transformer Big Relative 29.2 41.5

  17. Previous work O(L 2 D): 8.5 GB per layer (Shaw et al, 2018) Per layer, L=2048, D=512 Relative embeddings Multiply by Q Relative distances 0 -1 0 -2 -1 0

  18. Our formulation O(LD): 4.2 MB per layer Per layer, L=2048, D=512 Absolute by absolute Absolute by relative Skew Pad Reshape Slice i q

  19. Goal of skewing procedure Indexed by absolute by absolute absolute by relative

  20. Skewing to reduce relative memory from O(L 2 D) to O(LD) Per layer, L=2048, D=512 0 -1 0 Previous work -2 -1 0 O(L 2 D): 8.5 GB Relative Multiply by Q i q embeddings E r Skew Our work O(LD): 4.2 MB QE rT S rel Reshape Slice Pad i q Directly multiply by Q Per layer, L=2048, D=512 O(L 2 D): 8.5 GB Q skew(QE T ) O(LD): 4.2 MB (ours) E T

  21. A Jazz sample from Music Transformer

  22. A Jazz sample from Music Transformer

  23. Convolutions and Translational Equivariance 0 32 0 32 0.5 0.5 32 32

  24. Relative positions Translational Equivariance 0 32 0 32 0.5 0.5 32 32

  25. Relative Attention And Graphs

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models

Sequence-to-sequence Models and Attention Graham Neubig Preliminaries: Language Models Language Models Language models are generative models of text s ~ P(x) The Malfoys! said Hermione. Harry was watching him. He looked like

1.18k views • 37 slides
Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com Nov. 2017 Figures

Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com Nov. 2017 Figures

Introduction to Generative Models (and GANs) Haoqiang Fan fhq@megvii.com Nov. 2017 Figures adapted from NIPS 2016 Tutorial Generative Adversarial Networks Generative Models: Learning the Distributions Discriminative: learns the likelihood

1.37k views • 77 slides
Attention Models Focus on parts of input Olof Mogren Improves NN performance on different

Attention Models Focus on parts of input Olof Mogren Improves NN performance on different

Attention Models Attention Models Focus on parts of input Olof Mogren Improves NN performance on different tasks Chalmers University of Technology IBM1 attention mechanism (1980s) Feb 2016 Attention Models Arxiv 2016

348 views • 6 slides
Probabilistic Models of Cognition: Generative models Table of Contents Chapter

Probabilistic Models of Cognition: Generative models Table of Contents Chapter

Probabilistic Models of Cognition: Generative models Table of Contents Chapter Content Generative Model Example: Plinko Machine Working Model can be used captures some for simulation structure of the world in useful way

825 views • 78 slides
Adversarial BoltzmannMachines Belief Nets Networks Variational NADE

Adversarial BoltzmannMachines Belief Nets Networks Variational NADE

Unsupervised Learning Non-probabilistic Models Probabilistic Sparse Coding (Generative) Models Autoencoders Others (e.g. k-means) Tractable Models Non-Tractable Models Generative Fully observed Adversarial

815 views • 58 slides
Classification with generative models 2 DSE 210 Classification with parametrized models

Classification with generative models 2 DSE 210 Classification with parametrized models

Classification with generative models 2 DSE 210 Classification with parametrized models Classifiers with a fixed number of parameters can represent a limited set of functions. Learning a model is about picking a good approximation. Typically the

427 views • 39 slides
Deep Generative models for Inverse Problems Alex Dimakis joint work with Ashish Bora, Dave Van

Deep Generative models for Inverse Problems Alex Dimakis joint work with Ashish Bora, Dave Van

Deep Generative models for Inverse Problems Alex Dimakis joint work with Ashish Bora, Dave Van Veen and Ajil Jalal, Sriram Vishwanath and Eric Price, UT Austin Outline Generative Models Using generative models for Inverse

1.17k views • 101 slides
Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin

Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin

Augmented Statistical Models: Exploiting Generative Models in Discriminative Classifiers Martin Layton & Mark Gales 9 December 2005 Cambridge University Engineering Department NIPS 2005 Augmented Statistical Models: Exploiting Generative

726 views • 25 slides
Invertible Generative Models for Inverse Problems Mitigating Representation Error and Dataset Bias

Invertible Generative Models for Inverse Problems Mitigating Representation Error and Dataset Bias

Invertible Generative Models for Inverse Problems Mitigating Representation Error and Dataset Bias M. Asim, M. Daniels, O. Leong, P . Hand, A. Ahmed Inverse Problems with Generative Models as Image Priors Inverse Problems with Generative Models

369 views • 22 slides
Modals, conditionals, and probabilistic generative models Topic 1: intro to probability &

Modals, conditionals, and probabilistic generative models Topic 1: intro to probability &

Modals, conditionals, and probabilistic generative models Topic 1: intro to probability & generative models; a bit on modality Dan Lassiter, Stanford Linguistics Universit de Paris VII, 25/11/19 4 lectures: The plan 1. probability,

694 views • 56 slides
LEARNING GENERATIVE MODELS ACROSS INCOMPARABLE SPACES Cha harlot otte Bunne unne , David

LEARNING GENERATIVE MODELS ACROSS INCOMPARABLE SPACES Cha harlot otte Bunne unne , David

LEARNING GENERATIVE MODELS ACROSS INCOMPARABLE SPACES Cha harlot otte Bunne unne , David Alvarez-Melis, Andreas Krause, Stefanie Jegelka Pos oster #173 173 Poster #173 173 & Charlotte Bunne Generative Modeling generative = network

250 views • 13 slides
Detecting Communities of Commuters: Graph Based Techniques vs Generative Models Ashish Dandekar,

Detecting Communities of Commuters: Graph Based Techniques vs Generative Models Ashish Dandekar,

Detecting Communities of Commuters: Graph Based Techniques vs Generative Models Ashish Dandekar, St ephane Bressan, Talel Abdessalem, Huayu Wu, Wee Siong Ng September 7, 2016 1 Introduction Related Work Generative Models Experiments

786 views • 44 slides
Outline Latent Variable Generative Models Cooperative Vector Quantizer Model Model

Outline Latent Variable Generative Models Cooperative Vector Quantizer Model Model

2/4/2020 CS 3750 Advanced Machine Learning Latent Variable Generative Models II Ahmad Diab AHD23@cs.pitt.edu Feb 4, 2020 Based on slides of Professor Milos Hauskrecht Outline Latent Variable Generative Models Cooperative Vector

435 views • 20 slides
Learning the Structure of Generative Models without Labeled Data Stephen Bach Bryan He

Learning the Structure of Generative Models without Labeled Data Stephen Bach Bryan He

Learning the Structure of Generative Models without Labeled Data Stephen Bach Bryan He Alex Ratner Chris R Stanford University This Talk We study structure learning for generative models in which a latent variable

599 views • 33 slides
Modals, conditionals, and probabilistic generative models Topic 2: Indicative conditionals

Modals, conditionals, and probabilistic generative models Topic 2: Indicative conditionals

Modals, conditionals, and probabilistic generative models Topic 2: Indicative conditionals Separating semantics from reasoning Dan Lassiter, Stanford Linguistics Universit de Paris VII, 2/12/19 Overall plan 1. probability, generative

997 views • 52 slides
Probability Functional Descent: A Unifying Perspective on GANs, VI, and RL Casey Chu <

Probability Functional Descent: A Unifying Perspective on GANs, VI, and RL Casey Chu <

Probability Functional Descent: A Unifying Perspective on GANs, VI, and RL Casey Chu < caseychu@stanford.edu > Jose Blanchet Peter Glynn Deep generative models Deep generative models Variational inference Deep generative models

275 views • 16 slides
Maximizing Skills in Office Bartholin duct and vulvar abscesses GYN Procedures Vaso-vagal

Maximizing Skills in Office Bartholin duct and vulvar abscesses GYN Procedures Vaso-vagal

Outline UCSF Essentials of Primary Care Conference Squaw Creek, CA Pain relief for office procedures August 8, 2019 Endometrial biopsy Vulvar biopsy Maximizing Skills in Office Bartholin duct and vulvar abscesses GYN Procedures

292 views • 25 slides
Health Impact Assessment (HIA) in context rainer.fehr @ uni-bielefeld.de, www.rfehr.eu [19-10] 1

Health Impact Assessment (HIA) in context rainer.fehr @ uni-bielefeld.de, www.rfehr.eu [19-10] 1

12th European Public Health Conference 2019 in Marseille, France: Building bridges for solidarity and public health Pre-conference : HIA institutionalization and multisectoral collaboration in Europe, 20 Nov 2019 Health Impact Assessment

429 views • 24 slides
What type(s) of ARC grant(s) did you apply for this year? Who prepared your ARC

What type(s) of ARC grant(s) did you apply for this year? Who prepared your ARC

G ETTING A H EAD S TART ON ARC G RANTS June 2016 C ENTER FOR C REATIVE L AND R ECYCLING Workshops Technical Assistance: EPA TAB grantee Policy & Research Consulting Online at www.cclr.org Sarah Sieloff

240 views • 23 slides
Vapor Intrusion: Regulatory Update and Advances in Assessment Tools January 14, 2016 SERDP and

Vapor Intrusion: Regulatory Update and Advances in Assessment Tools January 14, 2016 SERDP and

SERDP and ESTCP Webinar Series Vapor Intrusion: Regulatory Update and Advances in Assessment Tools January 14, 2016 SERDP and ESTCP Webinar Series (#25) SERDP and ESTCP Webinar Series Welcome and Introductions Rula A. Deeb, Ph.D. Webinar

839 views • 64 slides
CS224N/Ling284 Lecture 8: Machine Translation, Sequence-to-sequence and Attention Abigail See

CS224N/Ling284 Lecture 8: Machine Translation, Sequence-to-sequence and Attention Abigail See

Natural Language Processing with Deep Learning CS224N/Ling284 Lecture 8: Machine Translation, Sequence-to-sequence and Attention Abigail See Announcements We are taking attendance today Sign in with the TAs outside the auditorium

1.05k views • 79 slides
Attention-based Networks M. Malinowski Why attention? Long term memories - attending to

Attention-based Networks M. Malinowski Why attention? Long term memories - attending to

Attention-based Networks M. Malinowski Why attention? Long term memories - attending to memories Dealing with gradient vanishing problem Exceeding limitations of a global representation Attending/focusing to smaller parts of

1.42k views • 47 slides
Structured Attention Networks Yoon Kim Carl Denton Luong Hoang Alexander M. Rush

Structured Attention Networks Yoon Kim Carl Denton Luong Hoang Alexander M. Rush

Structured Attention Networks Yoon Kim Carl Denton Luong Hoang Alexander M. Rush HarvardNLP 1 Deep Neural Networks for Text Processing and Generation 2 Attention Networks 3 Structured Attention Networks Computational Challenges

1.33k views • 110 slides
A Neural Attention Model for Sentence Summarization Alexander M. Rush, Sumit Chopra, Jason

A Neural Attention Model for Sentence Summarization Alexander M. Rush, Sumit Chopra, Jason

A Neural Attention Model for Sentence Summarization Alexander M. Rush, Sumit Chopra, Jason Weston. EMNLP 2015 Presented by Peiyao Li, Spring 2020 Extractive vs. Abstractive Summarization Extractive Summarization: Abstractive Summarization:

782 views • 64 slides

More recommend