Structure and Interpretation of Neural Codes Jacob Andreas - - PowerPoint PPT Presentation

Structure and Interpretation


• f Neural Codes

Jacob Andreas

Translating Neuralese

Jacob Andreas, Anca Dragan and Dan Klein

Learning to Communicate

3 3 [Wagner et al. 03, Sukhbaatar et al. 16, Foerster et al. 16]

Learning to Communicate

4 4

Neuralese

5 5

1.0 2.3

• 0.3 0.4
• 1.2 1.1

Translating neuralese

1.0 2.3

• 0.3 0.4
• 1.2 1.1

all clear

6

• Interoperate with 


autonomous systems

• Diagnose errors
• Learn from solutions

Translating neuralese

1.0 2.3

• 0.3 0.4
• 1.2 1.1

all clear

[Lazaridou et al. 16] 7

Outline

Natural language & neuralese Statistical machine translation Semantic machine translation Implementation details Evaluation

8

Outline

Natural language & neuralese Statistical machine translation Semantic machine translation Implementation details Evaluation

9

Outline

Natural language & neuralese Statistical machine translation Semantic machine translation Implementation details Evaluation

10

Outline

Natural language & neuralese Statistical machine translation Semantic machine translation Implementation details Evaluation

11 11

Outline

Natural language & neuralese Statistical machine translation Semantic machine translation Implementation details Evaluation

12

Outline

Natural language & neuralese Statistical machine translation Semantic machine translation Implementation details Evaluation

13

A statistical MT problem

14

max p( | ) p( )

a a

a

1.0 2.3

• 0.3 0.4
• 1.2 1.1

all clear

[e.g. Koehn 10]

A statistical MT problem

15

How do we induce a translation model?

A statistical MT problem

16

max p( | ) p( )

a a

a

max Σ p( | ) p( | ) p( )

a

a

∝

Strategy mismatch

17

ζ(s) = 1 Γ(s) ∞ 1 ex − 1xs dx x

Strategy mismatch

18

not sure

ζ(s) = 1 Γ(s) ∞ 1 ex − 1xs dx x

Strategy mismatch

19

not sure dunno

Strategy mismatch

20

not sure dunno yes

Strategy mismatch

21

not sure dunno yes yes no yes

Strategy mismatch

22

not sure yes

Σ p( , | not sure ) p( not sure )

Stat MT criterion doesn’t capture meaning

23

moving
 (0,3)→(1,4) In the intersection

Outline

Natural language & neuralese Statistical machine translation Semantic machine translation Implementation details Evaluation

✘

24

A “semantic MT” problem

25

I’m going north

The meaning of an utterance is given by its truth conditions

[Davidson 67]

A “semantic MT” problem

26

✔ ✔ ✘

I’m going north

The meaning of an utterance is given by its truth conditions

[Davidson 67]

A “semantic MT” problem

27

(loc (goal blue) north) I’m going north

The meaning of an utterance is given by its truth conditions

A “semantic MT” problem

28

0.4 0.2 0.001

I’m going north

The meaning of an utterance is given by its truth conditions the distribution over states in which it is uttered

[Beltagy et al. 14]

A “semantic MT” problem

29

0.4 0.2 0.001

I’m going north

The meaning of an utterance is given by its truth conditions the distribution over states in which it is uttered

• r equivalently, the belief it induces in listeners

[Frank et al. 09, A & Klein 16]

Representing meaning

30

The meaning of an utterance is given by its truth conditions the distribution over states in which it is uttered

• r equivalently, the belief it induces in listeners

Representing meaning

31

The meaning of an utterance is given by its truth conditions the distribution over states in which it is uttered

• r equivalently, the belief it induces in listeners

This distribution is well-defined even if the “utterance” is a vector rather than a sequence of tokens.

Translating with meaning

32

1.0 2.3

• 0.3 0.4
• 1.2 1.1

Translating with meaning

33

1.0 2.3

• 0.3 0.4
• 1.2 1.1

In the intersection

Translating with meaning

34

1.0 2.3

• 0.3 0.4
• 1.2 1.1

I’m going north

Translating with meaning

35

1.0 2.3

• 0.3 0.4
• 1.2 1.1

I’m going north

p( | ) p( | )

a

Translating with meaning

36

1.0 2.3

• 0.3 0.4
• 1.2 1.1

I’m going north

β( ) β( )

a

Interlingua!

37

source text target text

β( ) β( )

a

KL( || )

Translation criterion

argmin

β( ) β( )

a

a

38

KL( || )

Translation criterion

argmin

β( ) β( )

a

a

39

KL( || )

Translation criterion

argmin

β( ) β( )

a

a

40

KL( || )

Translation criterion

argmin

β( ) β( )

a

a

41

Computing representations

argmin

a KL( || )

β( ) β( )

a

42

Computing representations: sparsity

argmin

a KL( || )

β( ) β( )

a

p( | )

a

p( | )

43

Computing representations: smoothing

argmin

a KL( || )

β( ) β( )

a

actions & messages agent
 policy

44

argmin

a KL( || )

β( ) β( )

a

actions & messages agent
 policy agent
 model

45

Computing representations: smoothing

argmin

a KL( || )

β( ) β( )

a

actions & messages human

46

Computing representations: smoothing

argmin

a KL( || )

β( ) β( )

a

actions & messages human
 policy human
 model

47

Computing representations: smoothing

argmin

a KL( || )

β( ) β( )

a

0.10 0.05 0.13 0.08 0.01 0.22

a

48

Computing representations: smoothing

Computing KL

argmin

a KL( || )

β( ) β( )

a

49

Computing KL

argmin

a KL( || )

β( ) β( )

a

KL(p || q) = E p( ) q( )

50

p

Computing KL: sampling

argmin

a KL( || )

β( ) β( )

a

KL(p || q) = Σ p( ) log p( ) q( )

51

i i i i

Finding translations

argmin

a KL( || )

β( ) β( )

a

52

Finding translations: brute force

argmin

a KL( || )

β( ) β( )

a

going north crossing the intersection I’m done after you 0.5 2.3 0.2 9.7

53

argmin

a KL( || )

β( ) β( )

a

going north crossing the intersection I’m done after you 0.5 2.3 0.2 9.7

54

Finding translations: brute force

KL( || )

Finding translations

argmin

β( ) β( )

a

a

55

Outline

Natural language & neuralese Statistical machine translation Semantic machine translation Implementation details Evaluation

56

Referring expression games

57

1.0 2.3

• 0.3 0.4
• 1.2 1.1
• range bird

with black face

Evaluation: translator-in-the-loop

58

1.0 2.3

• 0.3 0.4
• 1.2 1.1
• range bird

with black face

59

1.0 2.3

• 0.3 0.4
• 1.2 1.1
• range bird

with black face

Evaluation: translator-in-the-loop

Experiment: color references

60

Experiment: color references

0.50 1.00 Neuralese → Neuralese English → English* 0.83

61

0.50 1.00 Neuralese → English* English → Neuralese Neuralese → Neuralese English → English* Statistical MT 0.83

0.72 0.70

62

Experiment: color references

0.50 1.00 Neuralese → English* English → Neuralese Neuralese → Neuralese English → English* Statistical MT Semantic MT

0.72 0.70 0.86 0.73

0.83

63

Experiment: color references

magenta, hot, rose magenta, hot, violet

• live, puke, pea

pinkish, grey, dull

64

Experiment: color references

magenta, hot, rose magenta, hot, violet

• live, puke, pea

pinkish, grey, dull

65

Experiment: color references

magenta, hot, rose magenta, hot, violet

• live, puke, pea

pinkish, grey, dull

66

Experiment: color references

magenta, hot, rose magenta, hot, violet

• live, puke, pea

pinkish, grey, dull

67

Experiment: color references

magenta, hot, rose magenta, hot, violet

• live, puke, pea

pinkish, grey, dull

68

Experiment: color references

magenta, hot, rose magenta, hot, violet

• live, puke, pea

pinkish, grey, dull

69

Experiment: color references

Experiment: image references

50 95 Neuralese → English* English → Neuralese Neuralese → Neuralese English → English* Statistical MT Semantic MT 77

57 55 75 60

70

large bird, black wings, black crown

large bird, black wings, black crown small brown, light brown, dark brown

71

Experiment: image references

Experiment: driving game

1.35 1.93 Neuralese ↔ English* Neuralese → Neuralese Statistical MT Semantic MT

1.49 1.54

72

How to translate

• wn

at goal
 done
 left to top going in intersection
 proceed
 going you first
 following
 going down

• Classical notions of “meaning” apply even to


un-language-like things (e.g. RNN states)

• These meanings can be compactly represented

without logical forms if we have access to world states

• Communicating policies “say” interpretable things!

Conclusions so far

74

• Classical notions of “meaning” apply even to


non-language-like things (e.g. RNN states)

• These meanings can be compactly represented

without logical forms if we have access to world states

• Communicating policies “say” interpretable things!

75

Conclusions so far

• Classical notions of “meaning” apply even to


non-language-like things (e.g. RNN states)

• These meanings can be compactly represented

without logical forms if we have access to world states

• Communicating policies “say” interpretable things!

76

Conclusions so far

Limitations

argmin

a KL( || )

β( ) β( )

a

KL(p || q) = Σ p( ) log p( ) q( )

77

i i i i

but what about compositionality?

Analogs of linguistic structure in deep representations

Jacob Andreas and Dan Klein

“Flat” semantics

• wn

at goal
 done going in intersection
 proceed
 going you first
 following

80

Compositional semantics

81

Compositional semantics

82

Compositional semantics

✔ ✔ ✔

83

message

Compositional semantics

✔ ✔ ✔

everything but the blue shapes

• range square and non-squares

84 [FitzGerald et al. 2013]

Compositional semantics

✔ ✔ ✔

lambda x: not(blue(x)) lambda x: or(orange(x), not(square(x))

85 [FitzGerald et al. 2013]

Compositional semantics

✔ ✔ ✔

86

???

Model architecture

87

Model architecture

88

Model architecture

✔ ✔ ✔

89

Model architecture

✔ ✔ ✔

90

1.0 2.3

• 0.3 0.4
• 1.2 1.1

Computing meaning representations

91

1.0 2.3

• 0.3 0.4
• 1.2 1.1
• n the left

92

• 0.1 1.3 


0.5 -0.4 
 0.2 1.0

✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

everything but squares

Computing meaning representations

everything but squares

93

• 0.1 1.3 


0.5 -0.4 
 0.2 1.0

✔ ✔ ✔ ✔ ✔ ✔ ✔

Computing meaning representations

everything but squares

94

• 0.1 1.3 


0.5 -0.4 
 0.2 1.0

✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Computing meaning representations

everything but squares

95

• 0.1 1.3 


0.5 -0.4 
 0.2 1.0

✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

Computing meaning representations

96

• 0.1 1.3 


0.5 -0.4 
 0.2 1.0

✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

everything but squares

Computing meaning representations

97

• 0.1 1.3 


0.5 -0.4 
 0.2 1.0

✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

lambda x: not(square(x))

Computing meaning representations

98

• 0.1 1.3 


0.5 -0.4 
 0.2 1.0

✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

lambda x: not(square(x))

Computing meaning representations

q( , ) =

a

KL( || ) β( ) β( )

a

0.10 0.05 0.13 0.08 0.01 0.22

99

a

Translation criterion

q( , ) =

a

β( ) E[ ] β( )

a a

100

✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

=

Translation criterion

Experiments

“High-level” communicative behavior “Low-level” message structure

101

Experiments

“High-level” communicative behavior “Low-level” message structure

102

103

Comparing strategies

104

everything but squares

• 0.1 1.3 


0.5 -0.4 
 0.2 1.0

Comparing strategies

105

everything but squares

✔ ✔ ✔ ✔ ✔ ✔ ✔

• 0.1 1.3 


0.5 -0.4 
 0.2 1.0

✔ ✔ ✔ ✔ ✔ ✔ ✔

Comparing strategies

106

everything but squares

✔ ✔ ✔ ✔ ✔ ✔ ✔

=

?

• 0.1 1.3 


0.5 -0.4 
 0.2 1.0

✔ ✔ ✔ ✔ ✔ ✔ ✔

Comparing strategies

107

✔ ✔ ✔ ✔ ✔ ✔

=

?

• 0.1 1.3 


0.5 -0.4 
 0.2 1.0

✔ ✔ ✔ ✔ ✔ ✔ ✔

Theories of model behavior: random

108

• 0.1 1.3 


0.5 -0.4 
 0.2 1.0

✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔

=

?

Theories of model behavior: literal

109

0.00 0.50 1.00

50 63 27 Literal Human

Evaluation: high-level scene agreement

110

0.00 0.50 1.00

72 50 92 74 Random Literal Human

Evaluation: high-level object agreement

Experiments

“High-level” communicative behavior “Low-level” message structure

111

Collecting translation data

112

all the red shapes blue objects everything but red green squares not green squares

Collecting translation data

113

λx.red(x) λx.blu(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x))

Collecting translation data

114

λx.red(x) λx.blu(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x)) 0.1 -0.3 0.5 1.1

• 0.3 0.2 0.1 0.1

1.4 -0.3 -0.5 0.8 0.2 -0.2 0.5 -0.1 0.3 -1.3 -1.5 0.1

Extracting related pairs

115

λx.red(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x)) 0.1 -0.3 0.5 1.1 1.4 -0.3 -0.5 0.8 0.2 -0.2 0.5 -0.1 0.3 -1.3 -1.5 0.1

Extracting related pairs

116

λx.red(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x)) 0.1 -0.3 0.5 1.1 1.4 -0.3 -0.5 0.8 0.2 -0.2 0.5 -0.1 0.3 -1.3 -1.5 0.1

Learning compositional operators

117

argmin

2

Evaluating learned operators

λx.red(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x)) 0.1 -0.3 0.5 1.1 1.4 -0.3 -0.5 0.8 0.2 -0.2 0.5 -0.1 0.3 -1.3 -1.5 0.1 λx.f(x) 0.2 -0.2 0.5 -0.1

Evaluating learned operators

λx.red(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x)) 0.1 -0.3 0.5 1.1 1.4 -0.3 -0.5 0.8 0.2 -0.2 0.5 -0.1 0.3 -1.3 -1.5 0.1 λx.f(x) 0.2 -0.2 0.5 -0.1

• 0.2 0.4 -0.3 0.0

Evaluating learned operators

λx.red(x) λx.¬red(x) λx.grn(x)∧sqr(x) λx.¬(grn(x)∧sqr(x)) 0.1 -0.3 0.5 1.1 1.4 -0.3 -0.5 0.8 0.2 -0.2 0.5 -0.1 0.3 -1.3 -1.5 0.1 λx.f(x) 0.2 -0.2 0.5 -0.1 ???

• 0.2 0.4 -0.3 0.0

121

Evaluation: scene agreement for negation

0.00 0.50 1.00

50 81 12

122

all the toys that are not red every thing that is red

• nly the blue and

green objects all items that are not blue or green

Input Predicted True

Visualizing negation

123

0.00 0.50 1.00

50 54 9

Evaluation: scene agreement for disjunction

124

Input Predicted True

all of the red objects the blue and red items the blue objects the blue and yellow items all the yellow toys all yellow or red items

Visualizing disjunction

• We can translate between neuralese and natural lang.

by grounding in distributions over world states

• Under the right conditions, neuralese exhibits

interpretable pragmatics & compositional structure

• Not just communication games—language might be a

good general-purpose tool for interpreting deep reprs.

Conclusions

125

• We can translate between neuralese and natural lang.

by grounding in distributions over world states

• Under the right conditions, neuralese exhibits

interpretable pragmatics & compositional structure

• Not just communication games—language might be a

good general-purpose tool for interpreting deep reprs.

Conclusions

126

• We can translate between neuralese and natural lang.

by grounding in distributions over world states

• Under the right conditions, neuralese exhibits

interpretable pragmatics & compositional structure

• Not just communication games—language might be a

good general-purpose tool for interpreting deep reprs.

Conclusions

127

Conclusions

128

not sure dunno yes yes no yes

1.0 2.3

• 0.3 0.4
• 1.2 1.1

Thank you!

http://github.com/jacobandreas/{neuralese,rnn-syn}