Show, Attend and Tell: Neural Image Caption Generation with Visual - - PowerPoint PPT Presentation

Show, Attend and Tell:

Neural Image Caption Generation with Visual Attention

Kelvin Xu*, Jimmy Ba†, Ryan Kiros†, Kyunghyun Cho*, Aaron Courville*, Ruslan Salakhutdinov†, Richard Zemel†, Yoshua Bengio*

Universit´ e de Montr´ eal*/ University of Toronto†

(some figures from Hugo Larochelle)

July 8, 2015

1 / 46

Caption generation is another building block

Level of Task

Low Level High Level

feature and descriptors texture shape tracking actions activity understanding segmentation Detection Scene Understanding

Human Time

causality goals and intentions

90 ms 150 ms 1 sec

situation functionality social roles

Object Scene Activity

Figure: adapted from a figure from Feifei Li

2 / 46

What our model does:

Figure: A bird flying over a body of water .

3 / 46

Overview

Recent work in image caption generation

Encoder-decoder models from a number of groups: (Berkeley, Google, Stanford, Toronto, others) Motivating our architecture from human attention Our proposed attention model Model Description Quantitative/Qualitative Results

4 / 46

This talk:

Recent work in image caption generation

Encoder-decoder models from a number of groups: (Berkeley, Google, Stanford, Toronto, others) Motivating our architecture from human attention Our proposed attention model Model Description Quantitative/Qualitative Results

5 / 46

Recent surge of interest in image captioning

◮ Submissions on this topic at CVPR 2015

(from groups at Google, Berkeley, Stanford, Microsoft.. etc)

◮ Inspired by some successes in machine translation

(Kalchbrenner et al. 2013, Sutskever et al. 2014, Cho et al. 2014)

6 / 46

Theme: Use a convnet to condition

Figure: from Karpathy et al. (2015)

7 / 46

Figure: Vinyal et al. (2015) model is quite similar

8 / 46

This talk:

Recent work in image caption generation

Encoder-decoder models from a number of groups: (Berkeley, Google, Stanford, Toronto, others) Motivating our architecture from human attention Our proposed attention model Model Description Quantitative/Qualitative Results

9 / 46

What are some things we know about human attention?

10 / 46

(1) human vision is foveated & sequential

◮ Particular parts of an image come to the forefront

1 2 3 1 2 3

◮ It is a sequential decision process (“saccades”, glimpses)

11 / 46

(2) bottom-up input influences

Figure: from Borji and Itti. (2013) [2]

12 / 46

(3) top-down task level control

mechanisms ¡at ¡work…

Figure: from Yarbus (1967)

13 / 46

Summary: useful aspects of attention

◮ foveated visual field (spatial focus) ◮ sequential decision making (temporal dynamics) ◮ bottom-up input influence ◮ top-down modulation of specific task

14 / 46

This talk:

Recent work in image caption generation

Encoder-decoder models from a number of groups: (Berkeley, Google, Stanford, Toronto, others) Motivating our architecture from human attention Our proposed attention model Model Description Quantitative/Qualitative Results

15 / 46

Our proposed attention model

◮ ”Low Level” convolutional feature extraction:

a = {a1, a2, .., aL}

◮ Compute the importance of each of these regions

α = {α1, α2, .., αL}

◮ Combine α and a to represent the image (context: ˆ

zi)

16 / 46

A little bit more specific

• utput = ( a, man, is, jumping, into, a, lake, . )

17 / 46

Convolutional feature extraction

Annotation Vectors

• utput = ( a, man, is, jumping, into, a, lake, . )

aj

Convolutional Neural Network 18 / 46

Given a initial hidden state (predicted from image)..

Annotation Vectors Recurrent State

hi

• utput = ( a, man, is, jumping, into, a, lake, . )

aj

Convolutional Neural Network 19 / 46

Predict the “importance” of each region

Annotation Vectors Recurrent State

hi

• utput = ( a, man, is, jumping, into, a, lake, . )

aj

Attention Mechanism

αj αj Σ =1

Convolutional Neural Network 20 / 46

Combine with annotation vectors..

Annotation Vectors Recurrent State

hi

• utput = ( a, man, is, jumping, into, a, lake, . )

+ aj

Attention Mechanism

α

Attention weight

j

αj Σ =1

Convolutional Neural Network 21 / 46

Feed into next hidden state and predict the next word

Annotation Vectors Word Sample

y i

Recurrent State

hi

• utput = ( a, man, is, jumping, into, a, lake, . )

+ aj

Attention Mechanism

α

Attention weight

j

αj Σ =1

Convolutional Neural Network 22 / 46

In the next step, we use the new hidden state

Annotation Vectors Word Sample

y i

Recurrent State

hi

• utput = ( a, man, is, jumping, into, a, lake, . )

+ aj

Attention Mechanism

α

Attention weight

j

αj Σ =1

Convolutional Neural Network 23 / 46

Continue until end of sequence

Annotation Vectors Word Sample

y i

Recurrent State

hi

• utput = ( a, man, is, jumping, into, a, lake, . )

+ aj

Attention Mechanism

α

Attention weight

j

αj Σ =1

Convolutional Neural Network 24 / 46

The attention is driven by the recurrent state + image

◮ At every time step, compute the importance of each region

depending on the top-down + bottom-up signals eti =fatt(ai, ht−1) αti = exp(eti) L

k=1 exp(etk) ◮ We use a softmax to constrain that these weights sum to 1 ◮ We explore two different ways use the above distribution to

compute a meaningful image representation

25 / 46

Stochastic or Deterministic?

Annotation Vectors Word Sample

y i

Recurrent State

hi

• utput = ( a, man, is, jumping, into, a, lake, . )

+ aj

Attention Mechanism

α

Attention weight

j

αj Σ =1

Convolutional Neural Network Stochastic

• r

Deterministic 26 / 46

Quick note on our decoder: LSTM (Hochreiter et al. 1997)

f c

• i

ht ht-1 zt Eyt-1 ht-1 zt Eyt-1 ht-1 zt Eyt-1 ht-1 zt Eyt-1

input gate

• utput gate

memory cell forget gate input modulator

27 / 46

Deterministic (Soft) Attention

◮ Feed in a attention weighted image input:

ˆ zt =

L

• i=1

αt,iai

◮ This is what A. Graves (2013)/D. Bahdanau et al (2015) did

in handwriting recognition/machine translation

28 / 46

Alternatively: Stochastic (Hard) Attention

◮ Sample α stochastically at every time step ◮ In RL terms, think of softmax α as a Boltzmann Policy:

Ls =

• s

p(s | a) log p(y | s, a) ≤ log p(y | a) ∂Ls ∂W ≈ 1 N

N

• n=1

∂ log p(y | ˜ sn, a) ∂W + log p(y | ˜ sn, a)∂ log p(˜ sn | a) ∂W

• By Williams 1992, and re-popularized recently by Mnih et al. 2014,

Ba et al. 2015

29 / 46

Quantitative Results

30 / 46

A footnote on these metrics

31 / 46

Under automatic metrics, humans are not great :(

32 / 46

But human evaluation (mechanical turks) is quite different

33 / 46

Stochastic or Deterministic?

Annotation Vectors Word Sample

y i

Recurrent State

hi

• utput = ( a, man, is, jumping, into, a, lake, . )

+ aj

Attention Mechanism

α

Attention weight

j

αj Σ =1

Convolutional Neural Network Stochastic

• r

Deterministic 34 / 46

Visualizing our learned attention: the good

35 / 46

Visualizing the our learned attention: the bad

36 / 46

Other fun things you can do:

37 / 46

A soccer ball ..

38 / 46

Two cakes on a plate..

39 / 46

Important previous work

40 / 46

attention in machine translation

• (4)

ap- distinct com-

Figure: also from UdeM lab (Bahdanau et al. 2014) [1]

41 / 46

attention mechanism in handwritten character generation

• Figure: from (Graves et al. 2013) [3]

42 / 46

Recently, many more..

43 / 46

Thanks for attending!

44 / 46

Thanks for attending! Code: https://github.com/kelvinxu/arctic-captions

45 / 46

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014. Ali Borji and Laurent Itti. State-of-the-art in visual attention modeling. Pattern Analysis and Machine Intelligence, IEEE Transactions

• n, 35(1):185–207, 2013.

Alex Graves. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850, 2013.

46 / 46