SLIDE 1
A game for humans
✔ ✔ ✔
everything but the blue shapes
- range square and non-squares
2 [FitzGerald et al. 2013]
Analogs of Linguistic Structure in Deep Representations Jacob - - PowerPoint PPT Presentation
Analogs of Linguistic Structure in Deep Representations Jacob - - PowerPoint PPT Presentation
Analogs of Linguistic Structure in Deep Representations Jacob Andreas and Dan Klein A game for humans everything but the blue shapes orange square and non-squares [FitzGerald et al. 2013] 2 A game for RNNs 1.0 2.3 -0.3 0.4
SLIDE 2
SLIDE 3
A game for RNNs
✔ ✔ ✔
3
1.0 2.3
- 0.3 0.4
- 1.2 1.1
[e.g. Lazaridou et al. 2016]
SLIDE 4
Questions
- 1. Does the RNN employ a human-like
communicative strategy?
4
everything but squares
= ?
1.0 2.3
- 0.3 0.4
- 1.2 1.1
SLIDE 5
Questions
- 2. Do RNN representations have interpretable
compositional structure?
5
1.0 2.3
- 0.3 0.4
- 1.2 1.1
∗ =
“not ” “red ”
?
SLIDE 6
not the red squares
Computing meaning representations
SLIDE 7
Computing meaning representations
λx.¬(sqr(x)∧red(x))
not the red squares
SLIDE 8
Computing meaning representations
λx.¬(sqr(x)∧red(x))
not the red squares not red or not square
λx.¬red(x)∨¬sqr(x)
SLIDE 9
Computing meaning representations
λx.¬(sqr(x)∧red(x))
not the red squares not red or not square
λx.¬red(x)∨¬sqr(x)
✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
SLIDE 10
Computing meaning representations
✔ ✔
λx.¬(sqr(x)∧red(x))
not the red squares
✔ ✔ ✔ ✔ ✔ ✔
not red or not square
λx.¬red(x)∨¬sqr(x)
SLIDE 11
✔ ✔
λx.¬(sqr(x)∧red(x))
not the red squares
✔ ✔
Computing meaning representations
- 0.1 1.3 0.5 -0.4
SLIDE 12
✔ ✔
λx.¬(sqr(x)∧red(x))
not the red squares
✔ ✔
Computing meaning representations
- 0.1 1.3 0.5 -0.4
✔ ✔ ✔ ✔
SLIDE 13
13
Computing meaning representations
- 0.1 1.3
0.5 -0.4 0.2 1.0
SLIDE 14
14
Computing meaning representations
- 0.1 1.3
0.5 -0.4 0.2 1.0
. . .
SLIDE 15
15
Computing meaning representations
✔ ✔ ✔ ✔ ✔ ✔
. . .
✔
- 0.1 1.3
0.5 -0.4 0.2 1.0
SLIDE 16
16
Computing meaning representations
✔ ✔ ✔ ✔ ✔ ✔
. . .
✔
everything but squares
✔ ✔ ✔ ✔ ✔
- 0.1 1.3
0.5 -0.4 0.2 1.0
SLIDE 17
17
Computing meaning representations
✔ ✔ ✔ ✔ ✔ ✔
. . .
✔
not the blue squares
✔ ✔ ✔ ✔ ✔ ✔ ✔
- 0.1 1.3
0.5 -0.4 0.2 1.0
SLIDE 18
Translating
By comparing denotations from logical forms and the decoder model, we can find utterances and vectors with the same meaning.
18 [A, Dragan & Klein 2013]
SLIDE 19
Questions
- 1. Does the RNN employ a human-like
communicative strategy?
- 2. Do RNN representations have interpretable
compositional structure?
19
SLIDE 20
Questions
- 1. Does the RNN employ a human-like
communicative strategy?
- 2. Do RNN representations have interpretable
compositional structure?
20
SLIDE 21
21
Comparing strategies
SLIDE 22
22
everything but squares
- 0.1 1.3
0.5 -0.4 0.2 1.0
Comparing strategies
SLIDE 23
23
everything but squares
Comparing strategies
✔ ✔ ✔ ✔
. . .
✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
- 0.1 1.3
0.5 -0.4 0.2 1.0
SLIDE 24
. . .
24
everything but squares
Comparing strategies
✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
?
- 0.1 1.3
0.5 -0.4 0.2 1.0
SLIDE 25
25
Evaluation: strategies
= ?
✔ ✔ ✔ ✔ ✔ ✔
92%
SLIDE 26
26
Evaluation: strategies
= ?
✔ ✔ ✔ ✔ ✔ ✔
92%
= ?
✔ ✔ ✔ ✔ ✔ ✔
50%
SLIDE 27
27
Evaluation: strategies
= ?
✔ ✔ ✔ ✔ ✔ ✔
92%
= ?
✔ ✔ ✔ ✔ ✔ ✔
74%
SLIDE 28
Experiments
- 1. Does the RNN employ a human-like
communicative strategy?
- 2. Do RNN representations have interpretable
compositional structure?
28
SLIDE 29
Collecting translation data
29
all the red shapes blue objects everything but red green squares not green squares
SLIDE 30
Collecting translation data
30
λx.red(x) λx.blu(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x))
SLIDE 31
Collecting translation data
31
0.1 -0.3 0.5 1.1
- 0.3 0.2 0.1 0.1
1.4 -0.3 -0.5 0.8 0.2 -0.2 0.5 -0.1 0.3 -1.3 -1.5 0.1 λx.red(x) λx.blu(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x))
SLIDE 32
Extracting related pairs
32
λx.red(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x)) 0.1 -0.3 0.5 1.1 1.4 -0.3 -0.5 0.8 0.2 -0.2 0.5 -0.1 0.3 -1.3 -1.5 0.1
SLIDE 33
Extracting related pairs
33
λx.red(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x)) 0.1 -0.3 0.5 1.1 1.4 -0.3 -0.5 0.8 0.2 -0.2 0.5 -0.1 0.3 -1.3 -1.5 0.1
SLIDE 34
Learning compositional operators
34
argmin
2
f(x) ¬f(x)
SLIDE 35
Evaluating learned operators
λx.red(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x)) 0.1 -0.3 0.5 1.1 1.4 -0.3 -0.5 0.8 0.2 -0.2 0.5 -0.1 0.3 -1.3 -1.5 0.1 λx.f(x) 0.2 -0.2 0.5 -0.1
SLIDE 36
Evaluating learned operators
λx.red(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x)) 0.1 -0.3 0.5 1.1 1.4 -0.3 -0.5 0.8 0.2 -0.2 0.5 -0.1 0.3 -1.3 -1.5 0.1 λx.f(x) 0.2 -0.2 0.5 -0.1
- 0.2 0.4 -0.3 0.0
SLIDE 37
Evaluating learned operators
λx.red(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x)) 0.1 -0.3 0.5 1.1 1.4 -0.3 -0.5 0.8 0.2 -0.2 0.5 -0.1 0.3 -1.3 -1.5 0.1 λx.f(x) 0.2 -0.2 0.5 -0.1 ???
- 0.2 0.4 -0.3 0.0
SLIDE 38
38
Evaluation: negation
= ?
✔ ✔ ✔ ✔ ✔ ✔
97%
¬f(x)
SLIDE 39
39
Evaluation: negation
= ?
✔ ✔ ✔ ✔ ✔ ✔
97%
= ?
✔ ✔ ✔ ✔ ✔ ✔
50%
¬f(x)
SLIDE 40
40
all the toys that are not red every thing that is red
- nly the blue and
green objects all items that are not blue or green
Input Predicted True
Visualizing negation
SLIDE 41
41
Input Predicted True
all of the red objects the blue and red items the blue objects the blue and yellow items all the yellow toys all yellow or red items
Visualizing disjunction
SLIDE 42
- Under the right conditions, RNN reprs exhibit
interpretable pragmatics & compositional structure
- Not just communication games—language might be a
good general-purpose tool for interpreting deep reprs.
Conclusions
42
SLIDE 43
- Under the right conditions, RNN reprs exhibit
interpretable pragmatics & compositional structure
- Not just communication games—language might be a
good general-purpose tool for interpreting deep reprs.
Conclusions
43
SLIDE 44
1.0 2.3
- 0.3 0.4
- 1.2 1.1