Attention, Transformer and BERT Prof. Kuan-Ting Lai 2020/6/16 - PowerPoint PPT Presentation

Apr 06, 2023 •397 likes •628 views

Attention, Transformer and BERT Prof. Kuan-Ting Lai 2020/6/16 Attention is All You Need! A. Waswani et al., NIPS , 2017 Google Brain & University of Toronto 2 Attention Visual attention and textual attention

Attention, Transformer and BERT Prof. Kuan-Ting Lai 2020/6/16
Attention is All You Need! A. Waswani et al., NIPS , 2017 Google Brain & University of Toronto 2
Attention • Visual attention and textual attention https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html 3
Seq2seq model • Language translation 4
Attention = Vector of Importance Weights 5
Transformer • http://jalammar.github.io/illustrated-transformer/ 6
Encoder and Decoder 7
8
Structure of the Encoder and Decoder • Self-attention • Encoder-decoder attention 9
10
Tensor2Tensor Notebook • https://colab.research.google.co m/github/tensorflow/tensor2ten sor/blob/master/tensor2tensor/ notebooks/hello_t2t.ipynb 11
Self-attention (query, key, value) 12 https://www.youtube.com/watch?v=ugWDIIOHtPA&t=1089s
Self-attention 13
14
Calculating 𝑐 2 15
Matrix Mutiplication 16
17
Adding Residual Connections 18
Layer Normalization 19
20
References 1. https://lilianweng.github.io/lil-log/2018/06/24/attention- attention.html 2. http://jalammar.github.io/illustrated-transformer/ 3. Hong-Yi Lee, Transformer, 2019 https://www.youtube.com/watch?v=ugWDIIOHtPA 21

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

CS480/680 Lecture 19: July 10, 2019 Attention and Transformer Networks [Vaswani et al.,

CS480/680 Lecture 19: July 10, 2019 Attention and Transformer Networks [Vaswani et al., Attention is All You Need, NeurIPS , 2017] University of Waterloo CS480/680 Spring 2019 Pascal Poupart 1 Attention Attention in Computer Vision

708 views • 13 slides

GPT-2 Language Models are Unsupervised Multi-Task Learners GPT-2 Fvrier 2019 Transformer XL

GPT-2 Language Models are Unsupervised Multi-Task Learners GPT-2 Fvrier 2019 Transformer XL 2019 Janvier 2019 BERT Octobre 2018 GPT Juin 2018 Transformer Decoder 2018 Janvier 2018 Attention is all you need 2017 Dcembre 2017

330 views • 20 slides

Machine Learning Lecture 11: Transformer and BERT Nevin L. Zhang lzhang@cse.ust.hk Department

Machine Learning Lecture 11: Transformer and BERT Nevin L. Zhang lzhang@cse.ust.hk Department of Computer Science and Engineering The Hong Kong University of Science and Technology This set of notes is based on internet resources and

624 views • 43 slides

The Journey from LSTM to BERT All slides are my own. Citations provided for borrowed images

The Journey from LSTM to BERT All slides are my own. Citations provided for borrowed images Kolluru Sai Keshav PhD Scholar Concepts Self-Attention Pooling Attention (Seq2Seq, Image Captioning) Structured Self-Attention

318 views • 30 slides

Attention is All You Need (Vaswani et. al. 2017) Slides and figures when not cited are from:

Attention is All You Need (Vaswani et. al. 2017) Slides and figures when not cited are from: Mausam, Jay Alammar The Illustrated Transformer Attention in seq2seq models (Bahdanau 2014) Multi-head attention Self-attention (single-head,

718 views • 48 slides

Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks Juho

Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks Juho Lee, Yoonho Lee, Jungtaek Kim , Adam R. Kosiorek, Seungjin Choi, and Yee Whye Teh Set-input problems and Deep Sets [Zaheer et al., 2017] Take

262 views • 10 slides

Transformer Sequence Models CSE354 - Spring 2020 Natural Language Processing Most NLP Tasks.

Transformer Sequence Models CSE354 - Spring 2020 Natural Language Processing Most NLP Tasks. E.g. Transformer Networks Sequence Tasks Transformers Language Modeling BERT Machine Translation Speech

1.26k views • 105 slides

Attention, Transformers, BERT, and ViLBERT Arjun Majumdar Georgia Tech Slide Credits: Andrej

Attention, Transformers, BERT, and ViLBERT Arjun Majumdar Georgia Tech Slide Credits: Andrej Karpathy, Justin Johnson, Dhruv Batra Recall: Recurrent Neural Networks Image Credit: Andrej Karpathy Sequence-to-Sequence with RNNs Input : Sequence

2.62k views • 88 slides

for Efficient Adaptation in Multi-Task Learning Asa Cooper Stickland and Iain Murray University

BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning Asa Cooper Stickland and Iain Murray University of Edinburgh Background: BERT Our model builds on BERT (Devlin et al., 2018), a powerful (and big) sentence

501 views • 9 slides

multi-hop attention and Transformers Outline Review of common (old fashioned) neural

multi-hop attention and Transformers Outline Review of common (old fashioned) neural architectures bags Attention Transformer Some (historically standard) neural architectures: Good (neural) models have existed for some data types for a

1.12k views • 67 slides

FASTER TRANSFORMER Bo Yang Hsueh, 2019/12/18 AGENDA What is Faster Transformer Introduce the

FASTER TRANSFORMER Bo Yang Hsueh, 2019/12/18 AGENDA What is Faster Transformer Introduce the Transformer and Faster Transformer 1.0 New Features in Faster Transformer 2.0 Introduce the Faster Transformer 2.0 Faster Transformer 2.0 performance

932 views • 55 slides

Transformer MT vs. Human translation 2 [https://www.eff.org/ai/metrics#Translation] Get rid of

Transformer MT vs. Human translation 2 [https://www.eff.org/ai/metrics#Translation] Get rid of RNNs in MT? RNNs are slow, because not parallelizable over timesteps Attention is parallelizable + have shorter gradient paths Sequence

961 views • 37 slides

Electronic DC Transformer Pavol Bauer Learning objectives What is an electronic DC

Electronic DC Transformer Pavol Bauer Learning objectives What is an electronic DC transformer link and how does it operate? How can a higher-power electronic transformer be built up in a modular way? Electronic DC Transformer

810 views • 13 slides

BERT 3.0 The New BERT Wheres Ernie????? Logging into Bert BERT now uses the same style logon as

BERT 3.0 The New BERT Wheres Ernie????? Logging into Bert BERT now uses the same style logon as the other SEIMS+ applications. BERT Main Screen Icons on Main Screen of BERT There are several icons on the main screen for BERT. These icons are

590 views • 16 slides

IEEE Transformer Committee PC57.167 Distribution Transformer Monitoring - User Mark Scarborough

IEEE Transformer Committee PC57.167 Distribution Transformer Monitoring - User Mark Scarborough Electrical Engineering Consultant DuPont January 9, 2019 Transformer Types in DuPont Control power Distribution (substation) dry-type

467 views • 21 slides

Transformer Program Trevor Foster Electrical Engineering Manager Calpine Transformer Program 1

Transformer Program Trevor Foster Electrical Engineering Manager Calpine Transformer Program 1 Calpine Transformer Program Fleet of combined-cycle and cogeneration plants Use both natural gas and steam to produce electricity

425 views • 20 slides

CVPR 2020 Video Pentathlon Challenge: Multi-modal Transformer for Video Retrieval Valentin

CVPR 2020 Video Pentathlon Challenge: Multi-modal Transformer for Video Retrieval Valentin Gabeur, Chen Sun, Karteek Alahari, Cordelia Schmid Video signal richness Video encoder Our cross-modal architecture Thank You

347 views • 5 slides

Midwinter Meeting February 29, 2020 The Who, Where, When, What, Why and How of Pharmacy

Midwinter Meeting February 29, 2020 The Who, Where, When, What, Why and How of Pharmacy Inspections Sharilee McIntyre Pharmacy Inspector Division of Occupational and Professional Licensing Utah Department of Commerce Division of Occupational

679 views • 25 slides

P Performance Analysis of f A l i f Ultra-Scaled InAs HEMTs Neerav Kharche 1 , Gerhard

P Performance Analysis of f A l i f Ultra-Scaled InAs HEMTs Neerav Kharche 1 , Gerhard Klimeck 1 , Dae-Hyun Kim 2,3 , Jess. A. del Alamo 2 , and Mathieu , , Luisier 1 1 Network for Computational Nanotechnology and 1 N t k f C t ti l N

450 views • 24 slides

Lecture 2.2 - Introduction to CUDA C Memory Allocation and Data Movement API Functions Objective

GPU Teaching Kit Accelerated Computing Lecture 2.2 - Introduction to CUDA C Memory Allocation and Data Movement API Functions Objective To learn the basic API functions in CUDA host code Device Memory Allocation Host-Device Data

246 views • 11 slides

neural networks making the world safe for Skynet David Levinthal Microsoft Azure Cloud Services

Performance analysis of deep neural networks making the world safe for Skynet David Levinthal Microsoft Azure Cloud Services Infrastructure Machine learning and Deep Neural Networks Machine learning works by building a network of simple

250 views • 20 slides

1 Mutation - selection equilibrium 1. Mutation pressure: ( ) ( ) = Let = the

Population Genetics 7: transient verses equilibrium polymorphism Mutation - selection equilibrium Mutation pressure and selection can operate in opposite directions as a force for change in allele frequencies in populations. Note that

359 views • 9 slides

A kinetic scheme for transient mixed flows in non uniform closed pipes: a global manner to upwind

Modelisation Kinetic approach Tests Conclusion A kinetic scheme for transient mixed flows in non uniform closed pipes: a global manner to upwind all the source terms C. B OURDARIAS M. E RSOY S. G ERBI LAMA-Universit de Savoie de Chambry,

811 views • 31 slides

Palomar Transient Factory and the Search for Progenitors Channels of SNe Ia Peter Nugent, Chris

Palomar Transient Factory and the Search for Progenitors Channels of SNe Ia Peter Nugent, Chris Frohmaier & Mark Sullivan Motivation SNe govern the heavy element production of the Universe and cycle matter in galaxies. - A SN rate provides

463 views • 27 slides