better action retrieval in images Inkyu An Content 1. Background - - PowerPoint PPT Presentation

Learning semantic relationships for better action retrieval in images

Inkyu An

2

• 1. Background
• 2. Motivation
• 3. Related Work
• 4. Approach
• 5. Result

Content

3

Background | Semantic ?

What comes to mind when you see below picture ? There are many parked vehicles on either side of the road.

4

Background | Semantic labeling

http://rodrigob.github.io/are_we_there_yet/build/semantic_l abeling_datasets_results.html#4d5352432d3231

5

Background | Semantic labeling

More complex - A wide variety of classes

6

Background | Semantic labeling

Poodle Yorkshire Terrier Collie Samoyed Great Dane Labrador Retriever Pomeranian Retriever Vizsla Bull Terrier

More complex - A wide variety of classes

7

Background | More and more complex

She is stretching her right leg over listening a music

8

Motivation | Action retrieval in images

Person interacting with panda Query image Image Search ???

9

Motivation | Action retrieval in images

Person interacting with panda Query image Image Search Result of Prior work

False Positive

10

Motivation | Action retrieval in images

Person interacting with panda

Query image

Person feeding panda Person holding animals Person feeding calf

Implied-by Type-of Mutual-exclusive Result images

11

Motivation | Action retrieval in images

Three kinds of relations

• 1. Implied-by
• 2. Type-of
• 3. Mutual-exclusive

HEX-graph

Large-scale object classification using label relation graphs [ECCV 2014]

12

Motivation | Action retrieval in images

“Person interacting with panda” is represented by a weight vector 𝓧𝑩

Skip-grams

Distributed Representations of Words and Phrases and their Compositionality [NIPS 2013]

13

Motivation | Action retrieval in images

They needed to get a score of relationship of sentences pair.

Neural Tensor Network

Reasoning With Neural Tensor Networks for Knowledge Base Completion [NIPS 2013]

14

Related Work |

• 1. HEX-graph
• Three kinds of relations
• 2. Skip-grams
• Weight vectors of actions(Sentence)
• 3. Neural Tensor Network
• Scores of relationship of pairs of actions

15

Bengal cat Russian Blue Siberian Husky Poodle Bulldog

Classifier

Related Work | HEX-graph _ Motivation

Dog Cat

16

Classifier

Related Work | HEX-graph _ Motivation

Siberian Husky Puppy Dog Cat Subsumption Exclusion

HEX-graph

17

Related Work | HEX-graph _ Problem Definition

<HEX-graph>

subsumption

exclusion

Dog Puppy Husky Cat

𝑂𝑝𝑒𝑓𝑡 𝑊 ∶

Dog Cat Puppy Husky

𝐼𝑗𝑓𝑠𝑏𝑠𝑑ℎ𝑧 𝑓𝑒𝑕𝑓 𝐹ℎ ∶ 𝐹𝑦𝑑𝑚𝑣𝑡𝑗𝑝𝑜 𝑓𝑒𝑕𝑓 𝐹𝑓 ∶

subsumption exclusion Relations : Dog Puppy Dog Cat Husky Puppy : subsumption : exclusion : overlap

21

Related Work | skip-grams

Nearby words

• The training objective is to learn word vector representations

that are good at predicting the nearby words The average log probability Input sentence  Training

23

Related Work | Neural Tensor Networks (NTN)

• The model returns a high score if they are in that

relationship and a low on otherwise

24

Approach | Problem setup

Action : Person riding bike A set of actions 𝒝

• Person riding bike
• Person riding horse
• Person preparing food
• Chef cooking pasta

Two SVO structure : 1. <subject, verb, object>

• 2. <subject, verb, prepositional object>
• Person walking with a horse

Related images

25

Approach | Problem setup _ three kinds of relations

• 1. Implied-by :
• 2. Type-of :
• 3. Mutually exclusive :

Person preparing food Chef cooking pasta Person doing football Man playing soccer Person riding horse Man riding camel

26

Approach | Full model

𝐷 = 𝐷𝑏𝑑 + 𝛽𝑠𝐷𝑠𝑓𝑑 + 𝛽𝑜𝐷𝑜𝑚𝑞 + 𝛽𝑑𝐷𝑑𝑝𝑜𝑡 + 𝜇 𝑋 2

2

Full model : Basic action retrieval model

[Image + Action]

Language prior [only Action] Visual objective [Image + Action] Consistency

• bjective

[only Action] The weights in the model

27

Approach | Full model

𝐷 = 𝐷𝑏𝑑 + 𝛽𝑠𝐷𝑠𝑓𝑑 + 𝛽𝑜𝐷𝑜𝑚𝑞 + 𝛽𝑑𝐷𝑑𝑝𝑜𝑡 + 𝜇 𝑋 2

2

Full model : Basic action retrieval model

[Image + Action]

Language prior [only Action] Visual objective [Image + Action] Consistency

• bjective

[only Action] The weights in the model

𝑋 = 𝑋

𝑗𝑛, 𝐵∈𝒝

𝑥𝐵 , 𝑋

𝑠𝑓𝑚

𝑏𝑠𝑓 𝑚2 𝑠𝑓𝑕𝑣𝑚𝑏𝑠𝑗𝑨𝑓𝑒 𝑥𝑗𝑢ℎ 𝑏 𝑠𝑓𝑕𝑣𝑚𝑏𝑠𝑗𝑨𝑏𝑢𝑗𝑝𝑜 𝑑𝑝𝑓𝑔𝑔𝑗𝑑𝑗𝑓𝑜𝑢 𝜇

28

Approach | Full model

𝐷 = 𝐷𝑏𝑑 + 𝛽𝑠𝐷𝑠𝑓𝑑 + 𝛽𝑜𝐷𝑜𝑚𝑞 + 𝛽𝑑𝐷𝑑𝑝𝑜𝑡 + 𝜇 𝑋 2

2

Full model : Basic action retrieval model

[Image + Action]

Language prior [only Action] Visual objective [Image + Action] Consistency

• bjective

[only Action] The weights in the model

29

Approach | Basic action retrieval model

𝑔

𝐽 = 𝑋 𝑗𝑛𝐷𝑂𝑂 𝐽 + 𝑐𝑗𝑛

𝐷𝑏𝑑 =

𝐵 𝐽+∈𝒰

𝐵

𝐽−∈𝒰

𝐵

max 0,1 + 𝑥𝐵

𝑈(𝑔 𝐽− − 𝑔 𝐽+)

CNN 𝑋

𝑗𝑛

𝑋

𝑗𝑛

𝑐𝑗𝑛 𝑐𝑗𝑛 Person riding bike 𝐽𝐵 + I − 𝐵𝑑𝑢𝑗𝑝𝑜 𝐵 Skip-grams 𝒙𝑩 𝒈𝑩 + 𝒈 − 𝒙𝑩𝒈𝑩 + 𝒙𝑩𝒈 − Action prediction loss

𝒰

𝐵 : a set of positive images of A

𝒰

𝐵 : a set of negative images of A

Skip-grams CNN

30

Approach | Full model

𝐷 = 𝐷𝑏𝑑 + 𝛽𝑠𝐷𝑠𝑓𝑑 + 𝛽𝑜𝐷𝑜𝑚𝑞 + 𝛽𝑑𝐷𝑑𝑝𝑜𝑡 + 𝜇 𝑋 2

2

Full model : Basic action retrieval model

[Image + Action]

Language prior [only Action] Visual objective [Image + Action] Consistency

• bjective

[only Action] The weights in the model

31

Approach | Relationship prediction

Goal : Denote the relationship by a vector 𝑠

𝐵𝐶

= 𝑠

𝐵𝐶 𝑗 , 𝑠 𝐵𝐶 𝑢 , 𝑠 𝐵𝐶 𝑛 ∈ 0,1 3

Implied by, type-of and mutually exclusive

Person riding bike 𝐵𝑑𝑢𝑗𝑝𝑜 𝐵 Skip-grams Person riding camel 𝐵𝑑𝑢𝑗𝑝𝑜 𝐶 Neural Tensor Network

𝑥𝐵, 𝑥𝐶 𝑋

𝑠𝑓𝑚 1:3

Softmax

𝒔𝑩𝑪

𝑠

𝐵𝐶 = 𝑡𝑝𝑔𝑢𝑛𝑏𝑦𝛾 𝑥𝐵⨂𝑋 𝑠𝑓𝑚 1:3 ⨂𝑥𝐶 + 𝑐𝑠𝑓𝑚

Skip-grams Neural Tensor Network

32

Approach | Language prior for relationship

• 1. Implied-by :
• 2. Type-of :
• 3. Mutually exclusive :

Person preparing food Chef cooking pasta Man eating fish Person feeding a fish Person riding horse Man riding camel

Wrong

• NLP prior

33

Approach | Language prior for relationship

𝐷𝑜𝑚𝑞 =

𝐵 𝐶∈ℛ𝐵

𝑠

𝐵𝐶 −

𝑠

𝐵𝐶

NLP prior Relationship prediction

𝑫𝒐𝒎𝒒: 𝒔𝑩𝑪: 𝒔𝑩𝑪:

The loss function of language-based relationship

• NLP priors are not always

accurate

• They treated NLP priors as a

noisy prior

34

Approach | Full model

𝐷 = 𝐷𝑏𝑑 + 𝛽𝑠𝐷𝑠𝑓𝑑 + 𝛽𝑜𝐷𝑜𝑚𝑞 + 𝛽𝑑𝐷𝑑𝑝𝑜𝑡 + 𝜇 𝑋 2

2

Full model : Basic action retrieval model

[Image + Action]

Language prior [only Action] Visual objective [Image + Action] Consistency

• bjective

[only Action] The weights in the model

35

Approach | Action retrieval with relationship

• Visual objective

→ 𝐷𝐵𝐶

𝑗 = 𝐽𝑐∈𝒰

𝐶

𝐽−∈𝒰

𝐵

max 0,1 + 𝑥𝐵

𝑈 𝑔 𝐽− − 𝑔𝐽𝑐

→ 𝐷𝐵𝐶

𝑗 = 𝐽𝑏∈𝒰

𝐵

𝐽−∈𝒰

𝐶

max 0,1 + 𝑥𝐶

𝑈 𝑔 𝐽− − 𝑔𝐽𝑏

→ 𝐷𝐵𝐶

𝑗 = 𝐽𝑏∈𝒰

𝐵

𝐽𝑐∈𝒰

𝐶

max 0,1 + 𝑥𝐵

𝑈 𝑔𝐽𝑐 − 𝑔𝐽𝑏

A is implied-by B : A is Type-of B : A is Mutually : exclusive of B

𝒰

𝐶 : a set of positive images of B

𝒰

𝐵 : a set of negative images of A

𝒰

𝐵 : a set of positive images of A

𝒰

𝐶 : a set of negative images of B

𝒰

𝐵 : a set of positive images of A

𝒰

𝐶 : a set of positive images of B

Rank the positive images of B higher than the negatives

• f A

Rank the positive images of A higher than negatives of B Rank the positive images of A higher than the positives

• f B

36

Approach | Action retrieval with relationship

• Visual objective

𝑃𝑐𝑘𝑓𝑑𝑢𝑗𝑤𝑓: 𝐷𝑠𝑓𝑑 =

𝐵∈𝒝 𝐶∈ℛ𝐵

𝑠

𝐵𝐶 𝑗 ⋅ 𝐷 𝐵𝐶 𝑗

+ 𝑠

𝐵𝐶 𝑢 ⋅ 𝐷 𝐵𝐶 𝑢 + 𝑠 𝐵𝐶 𝑛 ⋅ 𝐷𝐵𝐶 𝑛

Relationship prediction 𝑠

𝐵𝐶 = {𝑠 𝐵𝐶 𝑗 , 𝑠 𝐵𝐶 𝑢 , 𝑠 𝐵𝐶 𝑛 }

 Summarize costs(𝐷

𝐵𝐶 𝑗 , 𝐷𝐵𝐶 𝑢 , 𝐷 𝐵𝐶 𝑛 ) of each relations, when

relationship prediction({𝑠

𝐵𝐶 𝑗 , 𝑠 𝐵𝐶 𝑢 , 𝑠 𝐵𝐶 𝑛 }) is ‘1’.

37

Approach | Full model

𝐷 = 𝐷𝑏𝑑 + 𝛽𝑠𝐷𝑠𝑓𝑑 + 𝛽𝑜𝐷𝑜𝑚𝑞 + 𝛽𝑑𝐷𝑑𝑝𝑜𝑡 + 𝜇 𝑋 2

2

Full model : Basic action retrieval model

[Image + Action]

Language prior [only Action] Visual objective [Image + Action] Consistency

• bjective

[only Action] The weights in the model

38

Approach | Action retrieval with relationship

• Consistency

𝐷𝑑𝑝𝑜𝑡 =

𝐵 𝐶∈ℛ𝐵 𝐷∈ℛ𝐶 𝑒∈ 𝑞,𝑢,𝑛 3

𝑠

𝐵𝐶 𝑒1 ⋅ 𝑠 𝐶𝐷 𝑒2 ⋅ 𝑠 𝐷𝐵 𝑒3

Constrain the relationship assignment between actions Ex) : A is implied-by B B is implied-by C A is mutually exclusive of C Inconsistent relationships They wanted to avoid those kind of problems  A is implied-by C

39

Approach | Full model

𝐷 = 𝐷𝑏𝑑 + 𝛽𝑠𝐷𝑠𝑓𝑑 + 𝛽𝑜𝐷𝑜𝑚𝑞 + 𝛽𝑑𝐷𝑑𝑝𝑜𝑡 + 𝜇 𝑋 2

2

Full model : Basic action retrieval model

[Image + Action]

Language prior [only Action] Visual objective [Image + Action] Consistency

• bjective

[only Action] The weights in the model

• The full objective is minimized through downpour stochastic

gradient descent.

• Hyper-parameters of models : 𝛾, 𝜇, 𝛽𝑠, 𝛽𝑜, 𝛽𝑑

40

Result |

41

• Thank you.

Q & A |