SLIDE 1 Plug and Play Language Model : A Simple Baseline for Controlled Language Generation
ICLR20
Xiachong Feng Sumanth Dathathri CMS, Caltech Eric Frank Uber AI Andrea Madotto HKUST Janice Lan Uber AI Jane Hung Uber AI Piero Molino Uber AI Jason Yosinski Uber AI Rosanne Liu Uber AI
SLIDE 2 Author
- 1. Sumanth Dathathri CMS, Caltech
- 2. Andrea Madotto HKUST
- 3. Janice Lan Uber AI
- 4. …… Uber AI
SLIDE 3 Background : Pre-trained LM
- GPT-2
- BERT
- Transformer XL
- ……..
SLIDE 4
Background : GPT-2
SLIDE 5 Background : GPT-2
https://gpt2.apps.allenai.org/?text=Joel%20is%20a
SLIDE 6 Background : Gradient
𝑧 = 𝑦$ 𝑧% = 2𝑦 𝑦 = 1 𝑧% = 2
SLIDE 7
Task : Controlled Generation
SLIDE 8
Overview : Plug and Play LM for controlled language generation
SLIDE 9
P(x) : Language Modeling With Transformers
SLIDE 10
P(x) : Language Modeling With Transformers
SLIDE 11
P(x) : Language Modeling With Transformers
SLIDE 12
P(x) : Language Modeling With Transformers
SLIDE 13
P(x) : Language Modeling With Transformers
SLIDE 14 P(a|x)
- Bag of Words (BoW)
- Discriminator
- Sentiment
SLIDE 15
- Suppose we want x && a==positive
- 1. Generate x à p(x)
- 2. Classifier à p(a|x)
- 3. If a==positive : Done
- 4. Else : Generate x……
SLIDE 16
Method
SLIDE 17
Method : Gradient based
SLIDE 18
Method : Gradient based
SLIDE 19
Method
SLIDE 20 Fluency
- Kullback–Leibler (KL) Divergence
- Post-norm Geometric Mean Fusion
SLIDE 21
Bag of Words (BoW)
SLIDE 22
Discriminator
SLIDE 23
Thanks!
