SLIDE 1
Confidential + Proprietary Confidential + Proprietary
Confidential + Proprietary
Text is just a sequence of words
["hi", "team", "the", "server", "appears", "to", "be", "dropping", "about", "10%", …]
Deep Learning for Language Understanding (at Google Scale) Anjuli - - PowerPoint PPT Presentation
Deep Learning for Language Understanding (at Google Scale) Anjuli - - PowerPoint PPT Presentation
Deep Learning for Language Understanding (at Google Scale) Anjuli Kannan Software Engineer, Google Brain Confidential + Proprietary Confidential + Proprietary Text is just a sequence of words ["hi", "team",
SLIDE 2
SLIDE 3
About me
- My team: Google Brain
○ "Make machines intelligent, improve people's lives." ○ Research + software + applications ○ g.co/brain
- My work is at boundary of research and applications
- Focus on natural language understanding
SLIDE 4
Neural network basics
SLIDE 5
Confidential + Proprietary
Neural network
Is a 4 Is a 5 ... ...
Image: Wikipedia
SLIDE 6
Confidential + Proprietary
Neural network
Neuron
Is a 4 Is a 5
SLIDE 7
Confidential + Proprietary
Basic building block is the neuron
Greg Corrado
SLIDE 8
Gradient descent
w’ = w - α ∂wL(w)
w w’
Learning Rate
Slide: Vincent Vanhoucke
SLIDE 9
Recurrent neural networks
SLIDE 10
Confidential + Proprietary
Recurrent neural networks can model sequences
SLIDE 11
Recurrent neural networks can model sequences
How Message
SLIDE 12
How are Message
Recurrent neural networks can model sequences
SLIDE 13
How are you Message
Recurrent neural networks can model sequences
SLIDE 14
How are you ? Message
Recurrent neural networks can model sequences
SLIDE 15
Internal state is a fixed length encoding of the message
How are you ? Message
Recurrent neural networks can model sequences
SLIDE 16
Sequence-to-sequence models
SLIDE 17
Suppose we want to generate email replies
Smartreply Incoming email Response email
SLIDE 18
Sequence-to-sequence model
Sutskever et al, NIPS 2014
SLIDE 19
Sequence-to-sequence model
encoder decoder
SLIDE 20
Sequence-to-sequence model
Ingests incoming message Generates reply message
SLIDE 21
Encoder ingests the incoming message
Internal state is a fixed length encoding of the message
How are you ? Message
SLIDE 22
Decoder is initialized with final state of encoder
How are you ? __ How are you ? Message
SLIDE 23
Decoder is initialized with final state of encoder
How are you ? __ How are you ? Message
SLIDE 24
How are you ? __ I Message Response
Decoder predicts next word
SLIDE 25
How are you ? __ I I am Message Response
Decoder predicts next word
SLIDE 26
How are you ? __ I am I am great Message Response
Decoder predicts next word
SLIDE 27
How are you ? __ I am great I am great ! Message Response
Vinyals & Le, ICML DL 2015 Kannan et al, KDD 2016
Decoder predicts next word
SLIDE 28
What the model can do
SLIDE 29
What the model can do
SLIDE 30
Summary
- Neural networks learn feature representations from raw data
- Recurrent neural networks have statefulness which allows them to model
sequences of data such as text
- The sequence-to-sequence model contains two recurrent neural networks: one
to encode an input sequence and one to generate an output sequence
SLIDE 31
Smartreply
SLIDE 32
Google Translate
SLIDE 33
Research: Speech recognition
SLIDE 34
Research: Electronic health records
SLIDE 35
What's next?
?
SLIDE 36
Resources
- All tensorflow tutorials:
https://www.tensorflow.org/versions/master/tutorials/index.html
- Sequence-to-sequence tutorial (machine translation):
https://www.tensorflow.org/versions/master/tutorials/seq2seq
- Chris Olah's blog: http://colah.github.io/
SLIDE 37