SLIDE 1
The reference game
2
Reasoning about pragma0cs with neural listeners and speakers Jacob - - PowerPoint PPT Presentation
Reasoning about pragma0cs with neural listeners and speakers Jacob - - PowerPoint PPT Presentation
Reasoning about pragma0cs with neural listeners and speakers Jacob Andreas and Dan Klein The reference game 2 The reference game 3 The reference game The one with the snake 4 The reference game Mike is holding a baseball bat 5 The
SLIDE 2
SLIDE 3
The reference game
3
SLIDE 4
The reference game
4
The one with the snake
SLIDE 5
The reference game
5
Mike is holding a baseball bat
SLIDE 6
The reference game
6
bat a is holding Mike baseball
SLIDE 7
The reference game
7
They are si4ng by a picnic table
SLIDE 8
The reference game
8
There is a bat
SLIDE 9
The reference game
9
There is a bat
SLIDE 10
The reference game
10
Why do we care about this game?
Don’t you think it’s a li:le cold in here? Do you know what <me it is? Some of the children played in the park.
SLIDE 11
Deriving pragma0cs from reasoning
11
Mike is holding a baseball bat
SLIDE 12
12
Jenny is running from the snake
Deriving pragma0cs from reasoning
SLIDE 13
13
Mike is holding a baseball bat
Deriving pragma0cs from reasoning
SLIDE 14
How to win
14
DERIVED STRATEGY:
Reason about listener beliefs
DIRECT STRATEGY:
Imitate successful human play
There is a snake There is a snake There is a snake ?
SLIDE 15
How to win
15
[Mao et al. 2015] [Kazemzadeh et al. 2014] [Fitzgerald et al., 2013] [Monroe and PoRs, 2015] [Smith et al. 2013] [Vogel et al. 2013] [Golland et al. 2010]
DERIVED STRATEGY:
Reason about listener beliefs
DIRECT STRATEGY:
Imitate successful human play
SLIDE 16
How to win
16
PRO: domain repr “for free” CON: past work needs
targeted data
PRO: pragma0cs “for free” CON: past work needs
hand-engineering
DERIVED STRATEGY:
Reason about listener beliefs
DIRECT STRATEGY:
Imitate successful human play
SLIDE 17
How to win
17
DERIVED STRATEGY:
Reason about listener beliefs
DIRECT STRATEGY:
Imitate successful human play Learn base models for interpreta0on & genera0on without pragma0c context Explicitly reason about base models to get novel behavior
SLIDE 18
Data
Abstract Scenes Dataset 1000 scenes 10k sentences Feature representa0ons
18
SLIDE 19
Approach
19
Literal speaker Literal listener Sampler Reasoning speaker
SLIDE 20
A literal speaker (S0)
20
Mike is holding a baseball bat
SLIDE 21
A literal speaker (S0)
21
Referent encoder Referent decoder
Mike is holding a baseball bat
SLIDE 22
Module architectures
22
ReLU FC Softmax FC referent wordn word<n wordn+1 FC ref features referent
Referent encoder Referent decoder
SLIDE 23
Training S0
23
Mike is holding a baseball bat
SLIDE 24
S0 A literal speaker (S0)
24
Mike is holding a baseball bat The sun is in the sky Jenny is standing next to Mike
SLIDE 25
A literal listener (L0)
25
Mike is holding a baseball bat
SLIDE 26
A literal listener (L0)
26
Descr. encoder Referent encoder Referent encoder Scorer
0.87
Mike is holding a baseball bat
0.13
SLIDE 27
Module architectures
27
Referent encoder Referent decoder
sentence
ReLU Sum FC Softmax choice referent desc FC ngram features desc
SLIDE 28
Training L0
28
Mike is holding a baseball bat (random distractor)
0.87
SLIDE 29
A literal listener (L0)
29
L0
Mike is holding a baseball bat
SLIDE 30
A reasoning speaker (S1)
30
Mike is holding a baseball bat
?
SLIDE 31
A reasoning speaker (S1)
31
Literal speaker
The sun is in the sky Jenny is standing next to Mike
Literal listener
0.9 0.5 0.7 Mike is a baseball bat
SLIDE 32
A reasoning speaker (S1)
32
Literal speaker
The sun is in the sky Jenny is standing next to Mike
Literal listener
0.9 0.5 0.7 Mike is a baseball bat 0.05 0.09 0.08
SLIDE 33
A reasoning speaker (S1)
33
Literal speaker
The sun is in the sky Jenny is standing next to Mike
Literal listener
0.91-λ 0.51-λ 0.71-λ Mike is a baseball bat 0.05 0.09 0.08 * 0.05λ * 0.09λ * 0.09λ
SLIDE 34
Experiments
34
SLIDE 35
Baselines
- Literal: the L0 model by itself
- ContrasIve: a condi0onal LM trained on both
the target image and a random distractor [Mao et al. 2015]
35
SLIDE 36
Results (test)
Literal
Contras0ve
Reasoning 64% 69% 81%
36
SLIDE 37
Accuracy and fluency
37
SLIDE 38
How many samples?
Accuracy 50 60 70 80 90 100 # Samples 1 10 100 1000
38
SLIDE 39
Examples
(a) the sun is in the sky [contrastive]
39
SLIDE 40
Examples
(c) the dog is standing beside jenny [contrastive]
40
SLIDE 41
Examples
(b) mike is wearing a chef’s hat [non-contrastive]
41
SLIDE 42
Conclusions
- Standard neural kit of parts for base models
- Probabilis0c reasoning for high-level goals
- A liRle bit of structure goes a long way!
42
SLIDE 43
Thank you!
SLIDE 44
“Compiling” the reasoning model
What if we train the contras0ve model on the
- utput of the reasoning model?
SLIDE 45