SLIDE 1
Learning from Language
Jacob Andreas
Doing things with language
2
Learning from Language Jacob Andreas Doing things with language 2 - - PowerPoint PPT Presentation
Learning from Language Jacob Andreas Doing things with language 2 - - PowerPoint PPT Presentation
Learning from Language Jacob Andreas Doing things with language 2 Doing things with language Who is left of Go up, then go left. the truck? The hooded oriole is a large bird with black wings. A man with a white shirt and
SLIDE 2
SLIDE 3
Doing things with language
Who is left of the truck? Go up, then go left.
✔
A man with a white shirt and black pants.
3
The hooded oriole is a large bird with black wings.
SLIDE 4
The hooded oriole is a large bird with black wings.
Doing things with language
Go up, then go left.
✔
A man with a white shirt and black pants.
4
Who is left of the truck?
SLIDE 5
Doing things with language
Go up, then go left.
✔
A man with a white shirt and black pants.
5
The hooded oriole is a large bird with black wings. Who is left of the truck?
SLIDE 6
Doing things with language
Go up, then go left. The hooded oriole is a large bird with black wings.
✔
A man with a white shirt and black pants.
6
Who is left of the truck?
SLIDE 7
Words and primitives
left color black white
7
SLIDE 8
Syntax and composition
8
[Who is [left of the truck]]]? [Turn [left] [at the black hallway]].
[Does the [blue cylinder] have the [same material as the [big block [on the right side of [the red metallic thing]]]]]?
black left turn left who blue cylinder … truck
SLIDE 9
Learning reusable abstractions
9
black left turn man left color blue cylinder …
SLIDE 10
LANGUAGE & REASONING
What does the truck
- n the left sell?
ice cream
SLIDE 11
LANGUAGE & LEARNING
Go up, then go left.
SLIDE 12
LANGUAGE & BELIEF
large bird, black wings
SLIDE 13
A et al. Neural Module Networks. CVPR 16. A et al. Learning to Compose Neural Networks for Question Answering. NAACL 16. Hu, Rohrbach, A et al. Modeling Relationships in Referential Expressions […]. CVPR 17. Hu, A et al. Learning to Reason: End-to-End Module Networks […]. ICCV 17.
LANGUAGE & REASONING LEARNING BELIEF
SLIDE 14
Answering questions
14
yellow What color is the necktie?
SLIDE 15
Answering questions
15
name type coastal Columbia city no Cooper river yes Charleston city yes
Cooper What rivers are in South Carolina?
SLIDE 16
Answering questions
16
What rivers are in South Carolina?
name type coastal Columbia city no Cooper river yes Charleston city yes
λx. river(x) ∧ in(x, SC) λx. river(x) ∧ in(x, SC)
prolog
[Tang & Mooney 01, Artzi & Zettlemoyer 13]
Cooper
SLIDE 17
Answering questions
17
???
λx.∃y. color-of(x, y) ∧ necktie(y)
What color is the necktie?
yellow
SLIDE 18
Answering questions
18
What color is the necktie?
yellow
SLIDE 19
Answering questions
19
λx.∃y. color-of(x, y) ∧ necktie(y)
What color is the necktie?
yellow
SLIDE 20
Neural module networks
20
Is there a red shape above a circle? yes
↦ ↦ ↦
red exists above
SLIDE 21
Neural module networks
21
Is there a red shape above a circle? yes
↦ ↦ ↦
red exists above
SLIDE 22
Neural module networks
22
Is there a red shape above a circle? yes
↦ ↦ ↦
red exists above
yes
SLIDE 23
Perceptual primitives
23
What color is the necktie?
SLIDE 24
Perceptual primitives
24
Is there a red shape above a circle?
SLIDE 25
Perceptual primitives
25
Is there a red shape above a circle?
SLIDE 26
Perceptual primitives
26
Is there a red shape above a circle?
SLIDE 27
Perceptual primitives
27
Is there a red shape above a circle?
SLIDE 28
Perceptual primitives
28
Is there a red shape above a circle?
SLIDE 29
Perceptual primitives
29
Is there a red shape above a circle?
SLIDE 30
Meanings are computations
30
exists and red above circle
Is there a red shape above a circle?
[Montague 70]
SLIDE 31
Meanings are computations
31
exists and red above circle
yes
↦ ↦ ↦
red exists above
[e.g. Liang et al. 11]
SLIDE 32
Learning compositional operators
32
exists and red above circle
shapes.where(_.color == “red”)
d => d.nonEmpty ? true : false d => d.map(_.neighborAbove)
SLIDE 33
33
exists and red above circle
0.0 0.9 1.0
yes
↦ ↦ ↦
red exists above
[Beltagy et al. 13, Lewis & Steedman 13, Malinowski & Fritz 14]
Learning compositional operators
SLIDE 34
34
exists and red above circle
yes
↦ ↦ ↦
red exists above
[Bottou et al. 97, Socher et al. 2011]
Learning compositional operators
yes
SLIDE 35
35
yes
↦ ↦ ↦
red exists above red above circle exists and
Composing neural networks
SLIDE 36
36
yes
↦ ↦ ↦
above circle exists and red exists above
Composing neural networks
red
SLIDE 37
37
yes
↦ ↦ ↦
Composing neural networks
SLIDE 38
38
yes
↦ ↦ ↦
red exists above red above circle exists and
Composing neural networks
SLIDE 39
Anatomy of a module: Types
39
SLIDE 40
Anatomy of a module: Types
40
above
(entities) (entities)
SLIDE 41
Anatomy of a module: Types
41
circle
(entities) ()
SLIDE 42
Anatomy of a module: Types
42
red
color
(entities) (labels)
SLIDE 43
Anatomy of a module: Parameters
43
true
any
SLIDE 44
Anatomy of a module
44
tie city
name type coastal Columbia city no Cooper river yes Myrtle Beach city yes
Columbia Cooper Myrtle Beach
0.9 0.8 0.1
SLIDE 45
Simple predicates
45
Columbia
0.9
red
SLIDE 46
Simple predicates
46
red
red
SLIDE 47
red
Simple predicates
47
0.9
red
SLIDE 48
red
Simple predicates
48
0.9
red
SLIDE 49
red
Simple predicates
49
0.1
red
SLIDE 50
Learning
50
Is there a red shape above a circle? What color is the shape right of a circle?
yes blue
red above circle exists and color right circle
SLIDE 51
Learning
Is there a red shape above a circle? What color is the shape right of a circle?
yes blue
51
SLIDE 52
Parameter tying
Is there a red shape above a circle? What color is the shape right of a circle?
yes blue
circle circle
52
SLIDE 53
Parameter tying
Is there a red shape above a circle? What color is the shape right of a circle?
yes blue
circle circle
53
SLIDE 54
EXTREME parameter tying
54
blue above circle exists and color right circle blue above square exists and right shape right square
SLIDE 55
EXTREME parameter tying
55
blue above circle exists and color right circle blue above square exists and right shape right square
Σ log p( | ; W)
yes
,
W
arg max
SLIDE 56
Learning with fixed layouts is easy!
56
Module specialization is driven entirely by context!
true
circle above any
SLIDE 57
circle above any
Learning with fixed layouts is easy!
57
Module specialization is driven entirely by context!
true
SLIDE 58
58 58
Where do network structures come from?
Is there any red shape above a circle?
SLIDE 59
59 59
Where do network structures come from?
SQ NP NP is there any red shape NP PP above a circle NP
Is there any red shape above a circle?
SLIDE 60
60 60
Where do network structures come from?
SQ NP NP is there any red shape NP PP above a circle NP
Is there any red shape above a circle?
SLIDE 61
61 61
Where do network structures come from?
SQ NP NP is there any red shape NP PP above a circle NP
Is there any red shape above a circle?
SLIDE 62
62 62
Where do network structures come from?
SQ NP NP is there any red shape NP PP above a circle NP
Is there any red shape above a circle?
SLIDE 63
Is there any red shape above a circle?
63 63
Where do network structures come from?
SQ NP NP is there any red shape NP PP above a circle NP
SLIDE 64
hard choice of layouts and answers
64
Where do network structures come from?
SLIDE 65
hard choice of layouts and answers
65
Where do network structures come from?
arg max log p( y | , )
𝔽
θ
ans ques
SLIDE 66
hard choice of layouts and answers
66
Where do network structures come from?
hybrid supervised / policy gradient
∇𝔽r ≈ ∇ log p( y | z , w) + λ (∇ log p(z | x )) · log p( y | z , w)
ans ans ques
arg max log p( y | , )
𝔽
θ
ans ques
SLIDE 67
67
Does the blue cylinder have the same material as the big block on the right side of the red metallic thing?
67 [Johnson et al. 17]
Experimental evaluation
SLIDE 68
68
What is in the sheep’s ear?
68 [Agrawal et al. 15]
Experimental evaluation
SLIDE 69
Experimental evaluation [ARDK16a, HARDS17]
69
50 75 100
83.7
NMN NMN [JH+17]
69 [Agrawal et al. 15, Johnson et al. 17, Fukui et al. 16]
50 60 70
62.5 64.7 64.9
CNN + RNN MCB NMN
How many other things are the same size as the cylinder? What color is she wearing?
CNN + RNN
96.9 68.5
SLIDE 70
70 70
Does the blue cylinder have the same material as the big block on the right side
- f the red metallic thing?
blue cylinder right side same material red metallic big block
yes
Experimental evaluation [ARDK16a, HARDS17]
SLIDE 71
71
What is behind the foot
- f the bed?
71
.what. bed behind
desk
Experimental evaluation [ARDK16a, HARDS17]
SLIDE 72
72
NMNs and strong generalization [ARDK16a]
Is there anything left of a circle? Is there anything above a circle? Is there anything above and left
- f a circle?
TRAIN TEST
SLIDE 73
73
Is there anything left of a circle? Is there anything above a circle? Is there anything above and left
- f a circle?
TRAIN TEST
50 63 75 88 100
76.5 90.6
NMN
CNN + RNN
NMNs and strong generalization [ARDK16a]
SLIDE 74
NMNS for other tasks
Is Key Largo an island?
name type coastal island Columbia city no no Cooper river yes no Charleston city yes noThere is exactly one black triangle not touching any edge. [A, Rohrbach, Darrell, Klein; 16b] [Suhr, Lewis, Artzi; 17] man in sunglasses walking towards two men [Hu, Rohrbach, A, Darrell, Saenko; 17] [Cirik, Berg-Kirkpatrick, Morency; 18]
74
[Yu, Lin, Shen, Yang, Lu, Bansal, Berg; 18]
SLIDE 75
75
Lessons
yes
↦ ↦
red above circle exists and
Linguistic structure lets us learn composable neural modules from weak supervision. These modules allow us to more accurately interpret new statements, questions and references.
SLIDE 76
A, Klein & Levine. Modular Multitask Reinforcement Learning […]. ICML 17.
REASONING LANGUAGE & LEARNING BELIEF
SLIDE 77
Learning classifiers
77
Is there a red shape above a circle? What color is the shape right of a circle?
yes blue
red above circle exists and color right circle
SLIDE 78
Learning behaviors
78
Make planks: get wood, then use a saw. Make sticks: get wood, then use an axe.
use saw use axe get wood get wood
SLIDE 79
Learning from intermediate rewards
79
r r
[Kearns & Singh 02, Kulkarni et al. 16]
SLIDE 80
Learning from demonstrations
80
[Stolle & Precup 02, Fox & Krishnan et al. 16]
SLIDE 81
Learning from intermediate rewards
81
[e.g. SacerdoE 75, Hauskrecht et al. 98]
+has(wood)
- has(wood)
+has(plank) +at(saw)
SLIDE 82
Learning from sketches
82
Ï
use saw get wood
SLIDE 83
Learning from sketches
83
Make planks:
use saw get wood
SLIDE 84
Learning from sketches
84
Make sticks:
get wood use axe
SLIDE 85
Learning from sketches
85
π1 π2 π1 π3
use saw get wood get wood use axe STOP STOP STOP
SLIDE 86
Experiments: crafting game
86
SLIDE 87
Experiments: crafting game
87
SLIDE 88
Experiments: crafting game
88
x 106 episodes 1 2 3 Reward Unsupervised Sketches / Modular Instruction following
SLIDE 89
Experiments: crafting game
89
x 106 episodes 1 2 3 Reward Unsupervised Sketches / Modular Instruction following
SLIDE 90
Experiments: locomotion
90
SLIDE 91
Experiments: locomotion
91
x 108 timesteps 1 2 3 log Reward Unsupervised Sketches / Modular Instruction following
SLIDE 92
Fast adaptation
92
What if I don’t get a sketch at test time?
???
SLIDE 93
Fast adaptation
93
What if I don’t get a sketch at test time?
SLIDE 94
Fast adaptation
94 get iron use axe
What if I don’t get a sketch at test time?
SLIDE 95
Fast adaptation
95 use saw get iron
What if I don’t get a sketch at test time?
SLIDE 96
Fast adaptation
96
25 50 75 100
76 42 1
Ordinary RL
What if I don’t get a sketch at test time?
- Avg. Reward
- Unsup. /
Modular Sketches / Modular
SLIDE 97
97
Lessons
We can also learn modular behaviors from ungrounded “sketches” of abstract plans. We can use these modules to help reinforcement learning even when sketches are not available.
use axe use saw get wood use axe get iron
SLIDE 98
Beyond “tasks”
98
Man in glasses near two men.
LOCALIZATION Q&A POLICY SEARCH
How many men? go near the corner
SLIDE 99
Toward a model of everything
99
Man in glasses near two men.
LANGUAGE LEARNING
How many men? go near the corner
SLIDE 100
A & Klein. Reasoning about Pragmatics with Neural Listeners and Speakers. EMNLP 16. A, Drăgan & Klein. Translating Neuralese. ACL 17. A & Klein. Analogs of Linguistic Structure in Deep Representations. EMNLP 17. Fried, A & Klein. Unified Pragmatic Models for Generating and Following […]. NAACL 18.
REASONING LEARNING LANGUAGE & BELIEF
SLIDE 101
Interpreting language
101
What kind of bird is this? What are you going to do?
SLIDE 102
Generating “language”
102
- riole
What kind of bird is this? What are you going to do? [GO NORTH, GO WEST]
SLIDE 103
Generating “language”
103
What kind of bird is this? What are you going to do? [GO NORTH, GO WEST] ???
SLIDE 104
Generating informative language
104
What kind of bird is this? What are you going to do? Reach the end of the blue path. large bird, black wings
SLIDE 105
Explaining behaviors
105 [MacMahon et al. 06, Daniele et al. 17]
SLIDE 106
Explaining behaviors
106
SLIDE 107
Image captioning
107
A group of young men playing a game of soccer.
[Donahue et al. 15]
SLIDE 108
Image captioning
108
I will make a turn.
SLIDE 109
Reasoning about outcomes
109
I will make a turn.
Captioner Interpreter
SLIDE 110
Reasoning about outcomes
110
Speaker Listener I will make a turn.
SLIDE 111
Reasoning about outcomes
111
Listener I will make a turn.
SLIDE 112
Reasoning about outcomes
112
Listener I will go straight through.
SLIDE 113
Reasoning about outcomes
113
Listener I will turn left at the brick intersection.
SLIDE 114
Reasoning about outcomes
114
Listener Speaker I will turn left at the brick intersection.
SLIDE 115
Reasoning about belief
115 [Frank & Goodman 12]
I will turn left at the brick intersection.
SLIDE 116
Experimental results [FAK18]
116
seq-to-seq Belief
25 50 75 100
Navigation Alchemy Scene Tangrams
Human accuracy at predicting behavior
SLIDE 117
Experimental results [FAK18]
117
25 50 75 100
Navigation Alchemy Scene Tangrams
Human seq-to-seq Belief
Human accuracy at predicting behavior
SLIDE 118
Explaining behaviors
118
SLIDE 119
Explaining models
119
SLIDE 120
Multi-agent communication
120
SLIDE 121
Neuralese
121
1.0 2.3
- 0.3 0.4
- 1.2 1.1
SLIDE 122
Communication and behavior
122
SLIDE 123
Translating neuralese
123
1.0 2.3
- 0.3 0.4
- 1.2 1.1
all clear
SLIDE 124
Translating via belief
124
1.0 2.3
- 0.3 0.4
- 1.2 1.1
SLIDE 125
Translating via belief
125
1.0 2.3
- 0.3 0.4
- 1.2 1.1
in the intersection
SLIDE 126
Translating via belief
126
1.0 2.3
- 0.3 0.4
- 1.2 1.1
I’m going north
SLIDE 127
Translating via belief
127
1.0 2.3
- 0.3 0.4
- 1.2 1.1
I’m going north
argmin
a KL( || )
p(·| ) p(·| )
a
SLIDE 128
Example translations
128
- wn
at goal done left to top you first following going down going in intersection proceed going
SLIDE 129
Translating deep representations
129
1
1.0 2.3
- 0.3 0.4
- 1.2 1.1
SLIDE 130
Translating deep representations
130
1
1.0 2.3
- 0.3 0.4
- 1.2 1.1
large bird, black wings
SLIDE 131
Translation games [ADK17]
131
PROPOSITION: For agents cooperating via an approximately belief-
preserving translation layer, we can bound loss relative to agents with a common language.
SLIDE 132
Translation games [ADK17]
132
50 75
Neuralese → English English → Neuralese
seq-to-seq Belief
57 55 75 60 50
PROPOSITION: For agents cooperating via an approximately belief-
preserving translation layer, we can bound loss relative to agents with a common language.
SLIDE 133
133
Explaining classifiers
Learned classifier Interpreted language
blue and orange squares
square and blue
- range
- r
SLIDE 134
Learning compositional operators
134
0.1 -0.3 0.5 1.4 -0.3 -0.5
not
???
not
=
SLIDE 135
Learning negation [AK17]
135
all the toys that are not red every thing that is red
- nly the blue and
green objects all items that are not blue or green
Input Predicted True
SLIDE 136
Lessons
136
Language lets us find interpretable compositional operators in black- box deep models.
λx.f(x) λx.¬f(x)
not
Explicitly modeling listener beliefs helps us build informative models for language generation.
SLIDE 137
Safe exploration
137
I will sprint 300 meters forward. I will wiggle my front left leg.
SLIDE 138
Demonstrating competence
138
Defuse the bomb. I will cut open the box and snip the blue wire while avoiding the red
- ne.
SLIDE 139
139
Explaining limitations
All western tanagers have yellow heads. I can’t tell the difgerence between ravens and crows.
1
SLIDE 140
Last lessons
140
red above circle exists and
The structure of language helps us design models that reflect the compositional structure of the world. These models provide more accurate and interpretable learning for language processing and more.
SLIDE 141
Ronghang Hu Marcus Rohrbach Trevor Darrell Anca Drăgan Dan Klein Sergey Levine Kate Saenko Daniel Fried
SLIDE 142
you thank
SLIDE 143
Bonus parse tree [AK14, SAK17]
She enjoys playing tennis S NP VP
SLIDE 144
Learning with latent language [AKL18]
0.0 0.9 0.8
true true
true
true
evaluation
there is a green square a gray square is above a square a red cross is below a square
0.2
a red cross is below a square
concept learning language learning
true
there is a pink pentagon