[PPT] - BV LC Jacob Andreas, Marcus Rohrbach, Trevor Darrell, Dan Klein PowerPoint Presentation

SLIDE 1

Neural Module Networks

Jacob Andreas, Marcus Rohrbach,   Trevor Darrell, Dan Klein

BV LC

SLIDE 2

Visual ques>on answering

What color is  the neck/e? yellow

2

SLIDE 3

Visual ques>on answering

yes

3

Is there a red shape above   a circle?

SLIDE 4

Neural module networks

Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

4

SLIDE 5

Neural module networks

Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

5

SLIDE 6

Neural module networks

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

6

SLIDE 7

Nearest neighbors

Structured neural models

[Socher et al. 2011, BoIou et al. 1997, Mnih et al. 2014]

Probabilis7c formal seman7cs / predicate learning

[Beltagy et al. 2013, Lewis & Steedman 2013,

Malinowski & Fritz 2014]

7

SLIDE 8

Represen>ng meaning

Is there a red shape above a circle?

8

SLIDE 9

Represen>ng meaning

Is there a red shape above a circle?

9

SLIDE 10

AIen>ons encode meaning

Is there a red shape above a circle?

10

SLIDE 11

AIen>on transforma>ons encode meaning

Is there a red shape above a circle?

11

SLIDE 12

Set transforma>ons encode meaning

Is there a red shape above a circle?

12

SLIDE 13

Is there a red shape above a circle?

exists and red above circle

Sentence meanings are computa>ons

13

SLIDE 14

Is there a red shape above a circle?

exists and red above circle

Sentence meanings are computa>ons

14

SLIDE 15

Composi>ons of vector func>ons are neural nets

15

SLIDE 16

Outline

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

16

SLIDE 17

Outline

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

17

SLIDE 18

Outline

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

18

SLIDE 19

The [find] module

red

19

SLIDE 20

The [find] module

necktie

20

SLIDE 21

The [find] module

red

21

SLIDE 22

The [find] module

red

22

SLIDE 23

The [find] module

red

23

red

SLIDE 24

red

The [find] module

red

0.9

24

SLIDE 25

red

The [find] module

red

0.9

25

SLIDE 26

red

The [find] module

red

0.1

26

SLIDE 27

The [describe] module

color

red

27

SLIDE 28

The [describe] module

what

neck/e

28

SLIDE 29

The [describe] module

color

red

29

SLIDE 30

The [describe] module

color

red

. . .

30

SLIDE 31

The [describe] module

color

red

. . .

31

SLIDE 32

Outline

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

32

SLIDE 33

Where do layouts come from?

Is there a red shape above a circle?

be red shape there any circle above a

33

[Reddy et al. 2016]

circle red above shape and

SLIDE 34

Learning

Is there a red shape above a circle? What color is the shape right of a circle?

circle red above exists and circle right_of color

34

yes blue

SLIDE 35

Learning

35

yes blue

Is there a red shape above a circle? What color is the shape right of a circle?

SLIDE 36

Parameter tying

circle circle

36

yes blue

Is there a red shape above a circle? What color is the shape right of a circle?

SLIDE 37

Parameter tying

circle circle

37

yes blue

Is there a red shape above a circle? What color is the shape right of a circle?

SLIDE 38

Extreme parameter tying

circle red above exists and circle right_of color square right_of shape circle above red exists and

left_of

38

SLIDE 39

Learning with fixed layouts is easy!

Σ p( | ; W)

yes

,

W

arg max

(where every root module outputs a distribu>on over answers and W is the set of all module parameters)

39

SLIDE 40

Learning module behaviors

Module specializa>on is driven en>rely   by context!

square right_of

40

SLIDE 41

Experiments

41

SLIDE 42

Experiments: VQA dataset

What is in the sheep’s ear? tag What color   is the neck/e? yellow

42

[Antol et al. 2015]

SLIDE 43

Experiments: VQA dataset

50.00 53.33 56.67 60.00

55.9 57.4 58.9 59.4

Zhou (2015) Noh (2015) Yang (2015)

Ours

43

SLIDE 44

Experiments: VQA dataset

50.00 53.33 56.67 60.00

55.9 57.4 58.9 59.4

Zhou (2015) Noh (2015) Yang (2015)

Ours

44

SLIDE 45

Experiments: VQA dataset

What color is   she wearing?

and color wear

white

45

SLIDE 46

Experiments: VQA Dataset

What color is   she wearing?

and color wear

white

46

SLIDE 47

Experiments: VQA Dataset

sheep ear and what and

What is in the sheep’s ear? tag

47

SLIDE 48

Experiments: VQA Dataset

sheep ear and what and

What is in the sheep’s ear? tag

48

SLIDE 49

Experiments: VQA Dataset

sheep ear and what and

What is in the sheep’s ear? tag

49

SLIDE 50

Experiments: SHAPES dataset

Is a green shape above le< of a red shape? no Is there a red shape above   a circle? yes

50

c.f. [Xu and Saenko, 2016]

SLIDE 51

Experiments: SHAPES dataset

51 50.00 62.50 75.00 87.50 100.00

65.3 76.5 90.6

*Zhou

Ours

*Yang

SLIDE 52

Experiments: SHAPES dataset

52

re-attend[above]

yes

attend[circle] combine[and] attend[red] measure[is]

find[red] find[circle] relate[above]

combine[and] exists

SLIDE 53

Neural module networks

yes Is there a red shape   above a circle?

red exists

true

↦ ↦

above

↦

circle red above exists and

53

SLIDE 54

Neural module networks

Combines advantages of:

Representa>on learning (like a neural net)
Composi>onality (like a seman>c parser)

54

circle red above exists and

Linguis7c structure dynamically generates model structure

SLIDE 55

thank you

Download our code at hIp://github.com/jacobandreas/nmn2