Learning to compose neural networks for ques5on answering Jacob - PowerPoint PPT Presentation

Learning to compose neural networks for ques5on answering Jacob Andreas, Marcus Rohrbach, 
 Trevor Darrell, Dan Klein

Grounded ques5on answering What color is 
 yellow the neck/e? 2

Grounded ques5on answering name type coastal What rivers city no Columbia Cooper are in South river yes Cooper Carolina? city yes Charleston 3

Grounded ques5on answering Is there a red yes shape above 
 a circle? 4

Neural nets learn lexical groundings Is there a red yes shape above 
 a circle? [Iyyer et al. 2014, Bordes et al. 2014, Yang et al. 2015, Malinowski et al., 2015] 5

Seman5c parsers learn composi5on Is there a red yes shape above 
 a circle? [Wong & Mooney 2007, Kwiatkowski et al. 2010, Liang et al. 2011, A et al. 2013] 6

Neural module networks learn both! and and red Is there a red yes shape above 
 a circle? 7

Neural module networks Is there a red shape 
 above a circle? ↦ red ↦ true exists ↦ above 8

Neural module networks Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 9

Neural module networks yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 10

Nearest neighbors Structured neural models • [Socher et al. 2011, BoXou et al. 1997, Mnih et al. 2014] Probabilis<c formal seman<cs / predicate learning • [Beltagy et al. 2013, Lewis & Steedman 2013, 
 Malinowski & Fritz 2014] 11

Represen5ng meaning Is there a red shape above a circle? 12

Represen5ng meaning Is there a red shape above a circle? 13

Sets encode meaning Is there a red shape above a circle? 14

Sets encode meaning Is there a red shape above a circle? 15

Set transforma5ons encode meaning Is there a red shape above a circle? 16

Set transforma5ons encode meaning Is there a red shape above a circle? 17

Sentence meanings are computa5ons Is there a red shape above a circle? exists and red above circle 18

Sentence meanings are computa5ons Is there a red shape above a circle? exists and red above circle 19

Computa5ons are built from set func5ons exists ↦ red and ↦ true exists red above ↦ above circle 20

Modules can be manually specified… exists shapes.where(_.color == “red”) and d => d.nonEmpty ? true : false red above d => d.map(_.neighborAbove) circle [Liang et al. 2011] 21

…or learned as classifiers… exists x ↦ sign( a ⊤ ɸ( x )) and x ↦ sign( b ⊤ ɸ( x )) red above x ↦ sign( c ⊤ ɸ( x )) circle [Krishnamurthy & Kollar 2013] 22

…or relaxed to real-valued vectors exists ↦ red and ↦ true exists red above ↦ above circle 23

…or relaxed to real-valued vectors 0.0 0.9 1.0 exists ↦ red and ↦ true exists red above ↦ above circle 24

Composing vector func5ons exists ↦ red and ↦ true exists red above ↦ above circle 25

Composing vector func5ons exists ↦ red and ↦ true exists red above ↦ above circle 26

Composing vector func5ons ↦ exists red and ↦ true red above exists circle ↦ above 27

Composi5ons of vector func5ons are neural nets ↦ ↦ true ↦ 28

Composi5ons of vector func5ons are neural nets ↦ exists red and ↦ true red above exists circle ↦ above 29

Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 30

Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 31

Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 32

Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 33

Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 34

Anatomy of a module above 35

Anatomy of a module red color 36

What modules do we need? Is there a red shape above a circle? What color is the triangle? Who is running in the grass? What ci/es are south of San Diego? 37

A module for predicates Is there a red shape above a circle? [find] What color is the triangle? Who is running in the grass? What ci/es are south of San Diego? 38

A module for rela5ons Is there a red shape above a circle? [relate] What color is the triangle? Who is running in the grass? What ci/es are south of San Diego? [find] 39

A module for quan5fiers true Is there a red shape above a circle? [exists] What color is the triangle? Who is running in the grass? What ci/es are south of San Diego? [find] [relate] 40

A module for aXributes circle Is there a red shape above a circle? [describe] What color is the triangle? Who is running in the grass? What ci/es are south of San Diego? [exists] [find] [relate] 41

A module for logic Is there a red shape above a circle? [and] What color is the triangle? Who is running in the grass? [describe] What ci/es are south of San Diego? [exists] [find] [relate] 42

Module inventory Is there a red shape above a circle? What color is the triangle? Who is running in the grass? [and] [describe] What ci/es are south of San Diego? [exists] [find] [relate] 43

The [find] module red 44

The [find] module necktie [Xu et al. 2015] 45

The [find] module 0.9 Columbia 0.1 Cooper 0.8 Myrtle Beach city name type coastal city no Columbia river yes Cooper Myrtle city yes Beach 46

The [find] module red 47

The [find] module red 48

The [find] module red red 49

The [find] module 0.9 red red 50

The [find] module 0.9 red red 51

The [find] module 0.1 red red 52

The [describe] module red color 53

The [describe] module neck/e what 54

The [describe] module red color 55

The [describe] module red . . . color 56

The [describe] module red . . . color 57

Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 58

Learning yes blue exists color and right_of red above circle circle Is there a red shape above a circle? What color is the shape right of a circle? 59

Learning yes blue Is there a red shape above a circle? What color is the shape right of a circle? 60

Parameter tying yes blue circle circle Is there a red shape above a circle? What color is the shape right of a circle? 61

Parameter tying yes blue circle circle Is there a red shape above a circle? What color is the shape right of a circle? 62

Extreme parameter tying exists exists color shape and and right_of right_of red above above red circle square circle left_of circle 63

Learning with fixed layouts is easy! Σ p( | ; W ) arg max , yes W (where every root module outputs a distribu5on over answers and W is the set of all module parameters) 64

Maximum likelihood es5ma5on 65

Proper5es of learning Module specializa5on is driven en5rely 
 right_of by context square “Lexicon learning” is a con5nuous green op5miza5on problem chartreuse 66

Outline yes Is there a red shape 
 above a circle? exists and red above ↦ circle red ↦ true exists ↦ above 67

Where do layouts come from? Is there a red shape above a circle? be shape there any red above circle a [Reddy et al. 2016] 68

Where do layouts come from? Is there a red shape above a circle? be shape red above circle 69

Where do layouts come from? Is there a red shape above a circle? be shape above red circle 70

Where do layouts come from? Is there a red shape above a circle? shape above red circle 71

Where do layouts come from? Is there a red shape above a circle? and shape red above circle 72

Where do layouts come from? Is there a red shape above a circle? exists and and . . . and red above shape red above red above circle circle circle 73

Choosing among layouts exists exists and and and red above red above red above red circle circle circle chooser Is there a red shape above a circle? 74

Learning to choose layouts exists exists and and red above red above red circle circle chooser Is there a red shape above a circle? 75

Learning with unknown layouts uses RL exists exists and and red above red above red circle circle chooser Is there a red shape above a circle? [Williams 1992] 76

Experiments name type coastal city no Columbia river yes Cooper city yes Charleston 77