BV LC Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein - PowerPoint PPT Presentation

Neural Module Networks BV LC Jacob Andreas, Marcus Rohrbach, 
 Trevor Darrell, Dan Klein

Visual ques>on answering What color is 
 yellow the neck/e? 2

Visual ques>on answering Is there a red yes shape above 
 a circle? 3

Neural module networks Is there a red shape 
 above a circle? ↦ red ↦ true exists ↦ above 4

Neural module networks Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 5

Neural module networks yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 6

Nearest neighbors Structured neural models • [Socher et al. 2011, BoIou et al. 1997, Mnih et al. 2014] Probabilis7c formal seman7cs / predicate learning • [Beltagy et al. 2013, Lewis & Steedman 2013, 
 Malinowski & Fritz 2014] 7

Represen>ng meaning Is there a red shape above a circle? 8

Represen>ng meaning Is there a red shape above a circle? 9

AIen>ons encode meaning Is there a red shape above a circle? 10

AIen>on transforma>ons encode meaning Is there a red shape above a circle? 11

Set transforma>ons encode meaning Is there a red shape above a circle? 12

Sentence meanings are computa>ons Is there a red shape above a circle? exists and red above circle 13

Sentence meanings are computa>ons Is there a red shape above a circle? exists and red above circle 14

Composi>ons of vector func>ons are neural nets 15

Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 16

Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 17

Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 18

The [find] module red 19

The [find] module necktie 20

The [find] module red 21

The [find] module red 22

The [find] module red red 23

The [find] module 0.9 red red 24

The [find] module 0.9 red red 25

The [find] module 0.1 red red 26

The [describe] module red color 27

The [describe] module neck/e what 28

The [describe] module red color 29

The [describe] module red . . . color 30

The [describe] module red . . . color 31

Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 32

Where do layouts come from? Is there a red shape above a circle? be and shape there shape red above any red above circle circle a [Reddy et al. 2016] 33

Learning yes blue exists color and right_of red above circle circle Is there a red shape above a circle? What color is the shape right of a circle? 34

Learning yes blue Is there a red shape above a circle? What color is the shape right of a circle? 35

Parameter tying yes blue circle circle Is there a red shape above a circle? What color is the shape right of a circle? 36

Parameter tying yes blue circle circle Is there a red shape above a circle? What color is the shape right of a circle? 37

Extreme parameter tying exists exists color shape and and right_of right_of red above above red circle square circle left_of circle 38

Learning with fixed layouts is easy! Σ p( | ; W ) arg max , yes W (where every root module outputs a distribu>on over answers and W is the set of all module parameters) 39

Learning module behaviors Module specializa>on is driven en>rely 
 right_of by context! square 40

Experiments 41

Experiments: VQA dataset What color 
 yellow is the neck/e? What is in the tag sheep’s ear? [Antol et al. 2015] 42

Experiments: VQA dataset 60.00 59.4 58.9 56.67 57.4 55.9 53.33 Noh Yang Zhou Ours (2015) (2015) (2015) 50.00 43

Experiments: VQA dataset 60.00 59.4 58.9 56.67 57.4 55.9 53.33 Noh Yang Zhou Ours (2015) (2015) (2015) 50.00 44

Experiments: VQA dataset What color is 
 color white she wearing? and wear 45

Experiments: VQA Dataset What color is 
 color white she wearing? and wear 46

Experiments: VQA Dataset what What is in the tag and and sheep’s ear? sheep ear 47

Experiments: VQA Dataset what What is in the tag and and sheep’s ear? sheep ear 48

Experiments: VQA Dataset what What is in the tag and and sheep’s ear? sheep ear 49

Experiments: SHAPES dataset Is there a red yes shape above 
 a circle? Is a green shape no above le< of a red shape? c.f. [Xu and Saenko, 2016] 50

Experiments: SHAPES dataset 100.00 87.50 90.6 75.00 76.5 62.50 65.3 *Zhou *Yang Ours 50.00 51

Experiments: SHAPES dataset find[red] attend[red] yes combine[and] exists combine[and] measure[is] relate[above] find[circle] attend[circle] re-attend[above] 52

Neural module networks yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 53

Neural module networks Linguis7c structure dynamically generates model structure Combines advantages of: exists and • Representa>on learning (like a neural net) red above circle • Composi>onality (like a seman>c parser) 54

thank you Download our code at hIp://github.com/jacobandreas/nmn2