Structure and Interpretation of Neural Codes Jacob Andreas - PowerPoint PPT Presentation

Conclusions so far • Classical notions of “meaning” apply even to 
 un-language-like things (e.g. RNN states) • These meanings can be compactly represented without logical forms if we have access to world states • • Communicating policies “say” interpretable things! 74

Conclusions so far • Classical notions of “meaning” apply even to 
 non-language-like things (e.g. RNN states) • These meanings can be compactly represented without logical forms if we have access to world states • • Communicating policies “say” interpretable things! 75

Conclusions so far • Classical notions of “meaning” apply even to 
 non-language-like things (e.g. RNN states) • These meanings can be compactly represented without logical forms if we have access to world states • • Communicating policies “say” interpretable things! 76

Limitations a KL( || ) β ( ) β ( ) argmin 0 a p( ) i KL(p || q) = Σ p( ) log q( ) i i i 77

but what about compositionality?

Analogs of linguistic structure in deep representations Jacob Andreas and Dan Klein

“Flat” semantics at goal 
 you first 
 done following going in intersection 
 proceed 
 going 80 own

Compositional semantics 81

Compositional semantics 82

Compositional semantics message ✔ ✔ ✔ 83

Compositional semantics everything but the blue shapes orange square and non-squares ✔ ✔ ✔ [FitzGerald et al. 2013] 84

Compositional semantics lambda x: not(blue(x)) lambda x: or(orange(x), not(square(x)) ✔ ✔ ✔ [FitzGerald et al. 2013] 85

Compositional semantics ??? ✔ ✔ ✔ 86

Model architecture 87

Model architecture 88

Model architecture ✔ ✔ ✔ 89

Model architecture ✔ ✔ ✔ 1.0 2.3 -0.3 0.4 -1.2 1.1 90

Computing meaning representations 1.0 2.3 on the left -0.3 0.4 -1.2 1.1 91

Computing meaning representations -0.1 1.3 
 everything but 0.5 -0.4 
 squares 0.2 1.0 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 92

Computing meaning representations -0.1 1.3 
 everything but 0.5 -0.4 
 squares 0.2 1.0 ✔ ✔ ✔ ✔ ✔ ✔ ✔ 93

Computing meaning representations -0.1 1.3 
 everything but 0.5 -0.4 
 squares 0.2 1.0 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 94

Computing meaning representations -0.1 1.3 
 everything but 0.5 -0.4 
 squares 0.2 1.0 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 95

Computing meaning representations -0.1 1.3 
 everything but 0.5 -0.4 
 squares 0.2 1.0 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 96

Computing meaning representations -0.1 1.3 
 lambda x: 0.5 -0.4 
 not(square(x)) 0.2 1.0 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 97

Computing meaning representations -0.1 1.3 
 lambda x: 0.5 -0.4 
 not(square(x)) 0.2 1.0 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 98

Translation criterion KL( || ) β ( ) β ( ) q( , ) = 0 a 0 a a 0 0.10 0.08 0.05 0.01 0.13 0.22 99

Translation criterion E [ ] β ( ) β ( ) = q( , ) = 0 a 0 a a 0 ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ 100