[PPT] - Learning to compose neural networks for ques5on answering Jacob PowerPoint Presentation

SLIDE 1

Learning to compose neural networks for ques5on answering

Jacob Andreas, Marcus Rohrbach,   Trevor Darrell, Dan Klein

SLIDE 2

Grounded ques5on answering

What color is  the neck/e? yellow

2

SLIDE 3

Grounded ques5on answering

name type coastal Columbia city no Cooper river yes Charleston city yes

What rivers are in South Carolina?

Cooper

3

SLIDE 4

Grounded ques5on answering

yes

4

Is there a red shape above   a circle?

SLIDE 5

Neural nets learn lexical groundings

yes

5

[Iyyer et al. 2014, Bordes et al. 2014, Yang et al. 2015, Malinowski et al., 2015]

Is there a red shape above   a circle?

SLIDE 6

Seman5c parsers learn composi5on

yes

6

[Wong & Mooney 2007, Kwiatkowski et al. 2010, Liang et al. 2011, A et al. 2013]

Is there a red shape above   a circle?

SLIDE 7

Neural module networks learn both!

yes Is there a red shape above   a circle?

red and and

7

SLIDE 8

Neural module networks

Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

8

SLIDE 9

Neural module networks

Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

9

SLIDE 10

Neural module networks

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

10

SLIDE 11

Nearest neighbors

Structured neural models

[Socher et al. 2011, BoXou et al. 1997, Mnih et al. 2014]

Probabilis<c formal seman<cs / predicate learning

[Beltagy et al. 2013, Lewis & Steedman 2013,

Malinowski & Fritz 2014]

11

SLIDE 12

Represen5ng meaning

Is there a red shape above a circle?

12

SLIDE 13

Represen5ng meaning

Is there a red shape above a circle?

13

SLIDE 14

Sets encode meaning

Is there a red shape above a circle?

14

SLIDE 15

Sets encode meaning

Is there a red shape above a circle?

15

SLIDE 16

Set transforma5ons encode meaning

Is there a red shape above a circle?

16

SLIDE 17

Set transforma5ons encode meaning

Is there a red shape above a circle?

17

SLIDE 18

Is there a red shape above a circle?

exists and red above circle

Sentence meanings are computa5ons

18

SLIDE 19

Is there a red shape above a circle?

exists and red above circle

Sentence meanings are computa5ons

19

SLIDE 20

exists and red above circle red exists

true

↦ ↦

above

↦

Computa5ons are built from set func5ons

20

SLIDE 21

exists and red above circle

shapes.where(_.color == “red”)

d => d.nonEmpty ? true : false d => d.map(_.neighborAbove)

Modules can be manually specified…

21

[Liang et al. 2011]

SLIDE 22

exists and red above circle

x ↦ sign(a⊤ɸ(x)) x ↦ sign(b⊤ɸ(x)) x ↦ sign(c⊤ɸ(x))

…or learned as classifiers…

22

[Krishnamurthy & Kollar 2013]

SLIDE 23

…or relaxed to real-valued vectors

exists and red above circle red exists

true

↦ ↦

above

↦

23

SLIDE 24

…or relaxed to real-valued vectors

exists and red above circle red exists

true

↦ ↦

above

↦

24

0.0 0.9 1.0

SLIDE 25

Composing vector func5ons

exists and red above circle red exists

true

↦ ↦

above

↦

25

SLIDE 26

Composing vector func5ons

exists and red above circle red exists

true

↦ ↦

above

↦

26

SLIDE 27

Composing vector func5ons

circle red above and exists red exists

true

↦ ↦

above

↦

27

SLIDE 28

Composi5ons of vector func5ons are neural nets

true

↦ ↦ ↦

28

SLIDE 29

circle red above and exists red exists

true

↦ ↦

above

↦

Composi5ons of vector func5ons are neural nets

29

SLIDE 30

Outline

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

30

SLIDE 31

Outline

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

31

SLIDE 32

Outline

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

32

SLIDE 33

Outline

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

33

SLIDE 34

Outline

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

34

SLIDE 35

Anatomy of a module

35

above

SLIDE 36

Anatomy of a module

36

color

red

SLIDE 37

What modules do we need?

Is there a red shape above a circle? Who is running in the grass? What ci/es are south of San Diego? What color is the triangle?

37

SLIDE 38

A module for predicates

Is there a red shape above a circle? Who is running in the grass? What ci/es are south of San Diego? What color is the triangle?

[find]

38

SLIDE 39

A module for rela5ons

Is there a red shape above a circle? Who is running in the grass? What ci/es are south of San Diego? What color is the triangle?

[find] [relate]

39

SLIDE 40

A module for quan5fiers

Is there a red shape above a circle? Who is running in the grass? What ci/es are south of San Diego? What color is the triangle?

[exists] [find] [relate]

40

true

SLIDE 41

A module for aXributes

Is there a red shape above a circle? Who is running in the grass? What ci/es are south of San Diego? What color is the triangle?

[describe]

[find] [relate] [exists]

41

circle

SLIDE 42

A module for logic

Is there a red shape above a circle? Who is running in the grass? What ci/es are south of San Diego? What color is the triangle?

[and] [find] [relate] [exists]

[describe]

42

SLIDE 43

Module inventory

Is there a red shape above a circle? Who is running in the grass? What ci/es are south of San Diego? What color is the triangle?

[find] [relate] [exists]

[describe]

[and]

43

SLIDE 44

The [find] module

red

44

SLIDE 45

The [find] module

necktie

45

[Xu et al. 2015]

SLIDE 46

The [find] module

city

name type coastal Columbia city no Cooper river yes Myrtle Beach city yes

Columbia Cooper Myrtle Beach

0.9 0.8 0.1

46

SLIDE 47

The [find] module

red

47

SLIDE 48

The [find] module

red

48

SLIDE 49

The [find] module

red

49

red

SLIDE 50

red

The [find] module

red

0.9

50

SLIDE 51

red

The [find] module

red

0.9

51

SLIDE 52

red

The [find] module

red

0.1

52

SLIDE 53

The [describe] module

color

red

53

SLIDE 54

The [describe] module

what

neck/e

54

SLIDE 55

The [describe] module

color

red

55

SLIDE 56

The [describe] module

color

red

. . .

56

SLIDE 57

The [describe] module

color

red

. . .

57

SLIDE 58

Outline

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

58

SLIDE 59

Learning

Is there a red shape above a circle? What color is the shape right of a circle?

circle red above exists and circle right_of color

59

yes blue

SLIDE 60

Learning

60

yes blue

Is there a red shape above a circle? What color is the shape right of a circle?

SLIDE 61

Parameter tying

circle circle

61

yes blue

Is there a red shape above a circle? What color is the shape right of a circle?

SLIDE 62

Parameter tying

circle circle

62

yes blue

Is there a red shape above a circle? What color is the shape right of a circle?

SLIDE 63

Extreme parameter tying

circle red above exists and circle right_of color square right_of shape circle above red exists and

left_of

63

SLIDE 64

Learning with fixed layouts is easy!

Σ p( | ; W)

yes

,

W

arg max

(where every root module outputs a distribu5on over answers and W is the set of all module parameters)

64

SLIDE 65

Maximum likelihood es5ma5on

65

SLIDE 66

Proper5es of learning

Module specializa5on is driven en5rely   by context “Lexicon learning” is a con5nuous

p5miza5on problem

square right_of green chartreuse

66

SLIDE 67

Outline

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

67

SLIDE 68

Where do layouts come from?

Is there a red shape above a circle?

be red shape there any circle above a

68

[Reddy et al. 2016]

SLIDE 69

Is there a red shape above a circle?

be red shape circle above

Where do layouts come from?

69

SLIDE 70

Is there a red shape above a circle?

be

circle red above shape

Where do layouts come from?

70

SLIDE 71

Is there a red shape above a circle?

circle red above shape

Where do layouts come from?

71

SLIDE 72

Is there a red shape above a circle?

circle red above shape and

Where do layouts come from?

72

SLIDE 73

Is there a red shape above a circle?

circle red above and circle red above and exists

. . .

Where do layouts come from?

73

circle red above shape and

SLIDE 74

Choosing among layouts

Is there a red shape above a circle?

circle red above exists and circle red above and circle red above exists and red

chooser

74

SLIDE 75

Learning to choose layouts

circle red above exists and circle red above exists and red

Is there a red shape above a circle? chooser

75

SLIDE 76

Learning with unknown layouts uses RL

circle red above exists and circle red above exists and red

Is there a red shape above a circle? chooser

76

[Williams 1992]

SLIDE 77

Experiments

77

name type coastal Columbia city no Cooper river yes Charleston city yes

SLIDE 78

Experiments: VQA dataset

What is in the sheep’s ear? tag What color   is the neck/e? yellow

78

[Antol et al. 2015]

SLIDE 79

Experiments: VQA dataset

50.00 53.33 56.67 60.00

55.9 57.4 58.9 59.4

Zhou (2015) Noh (2015) Yang (2015)

Ours

79

SLIDE 80

Experiments: VQA dataset

50.00 53.33 56.67 60.00

55.9 57.4 58.9 59.4

Zhou (2015) Noh (2015) Yang (2015)

Ours

80

SLIDE 81

50.00 62.50 75.00 87.50 100.00

65.3 76.5 90.6

*Zhou

Ours

Experiments: SHAPES dataset

81

*Yang

SLIDE 82

Experiments: VQA dataset

What color is   she wearing?

and color wear

white

82

SLIDE 83

Experiments: VQA Dataset

What color is   she wearing?

and color wear

white

83

SLIDE 84

Experiments: VQA Dataset

sheep ear and what and

What is in the sheep’s ear? tag

84

SLIDE 85

Experiments: VQA Dataset

sheep ear and what and

What is in the sheep’s ear? tag

85

SLIDE 86

Experiments: VQA Dataset

sheep ear and what and

What is in the sheep’s ear? tag

86

SLIDE 87

Experiments: GeoQA dataset

name type coastal Columbia city no Cooper river yes Charleston city yes

What are some beaches in Florida? Daytona Beach Is Key Largo   an island? Yes

name type coastal Miami city no Daytona Beach city yes Everglades park no

87

[Krishnamurthy & Kollar 2013]

SLIDE 88

Experiments: GeoQA dataset

45.00 50.00 55.00

48 51 54.3

UBL LSP-W Ours

88

[Kwiatkowski et al. 2010, Krishnamurthy & Kollar 2013]

SLIDE 89

Experiments: GeoQA dataset

45.00 50.00 55.00

48 51 54.3

UBL LSP-W Ours

89

[Kwiatkowski et al. 2010, Krishnamurthy & Kollar 2013]

SLIDE 90

Experiments: GeoQA dataset

Is Key Largo an island? yes

Key Largo island and exists and

90

SLIDE 91

Experiments: GeoQA dataset

What are some beaches in Florida? yes

exists beach Florida in and

(wrong parse)

91

SLIDE 92

Experiments: GeoQA dataset

What beach city is there in Florida? {}

beach Florida in and

(wrong module behavior)

city

92

SLIDE 93

Neural module networks

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

93

SLIDE 94

Neural module networks

Combines advantages of:

Representa5on learning (like a neural net)
Composi5onality (like a seman5c parser)

94

circle red above exists and

Linguis<c structure dynamically generates model structure

SLIDE 95

thank you

Download our code at hXp://github.com/jacobandreas/nmn2