Jointly Learning Word Representations and Composition Functions - - PowerPoint PPT Presentation

10/28/2014 EMNLP 2014 in Doha, Qatar

Jointly Learning Word Representations and Composition Functions Using Predicate-Argument Structures

Kazuma Hashimoto (UT) Pontus Stenetorp (UT) Makoto Miwa (TTI) Yoshimasa Tsuruoka (UT) University of Tokyo (UT) Toyota Technological Institute (TTI)

• Neural networks + large unlabeled corpora

Neural Word Vector Representations

10/28/2014 EMNLP 2014 in Doha, Qatar

• Neural networks + large unlabeled corpora

– Learn word (i.e. single token) representations

• e.g.) word2vec

(Mikolov+ 2013; Mnih and Kavukcuoglu 2013; inter alia)

Neural Word Vector Representations

10/28/2014 EMNLP 2014 in Doha, Qatar

• Neural networks + large unlabeled corpora

– Learn word (i.e. single token) representations

• e.g.) word2vec

(Mikolov+ 2013; Mnih and Kavukcuoglu 2013; inter alia)

– Learn composed vector representations

• e.g.) compositional neural language models

for verb-object vectors (Tsubaki+ 2013)

Neural Word Vector Representations

10/28/2014 EMNLP 2014 in Doha, Qatar

Relation to Previous Work

10/28/2014 EMNLP 2014 in Doha, Qatar

word2vec Compositional neural language models Our model single token representations

✓ ✓ ✓

recursive structures

• f syntactic relations

x x ✓

pre-training

✓ x ✓

composition

x ✓ ✓

Relation to Previous Work

10/28/2014 EMNLP 2014 in Doha, Qatar

word2vec Compositional neural language models Our model single token representations

✓ ✓ ✓

recursive structures

• f syntactic relations

x x ✓

pre-training

✓ x ✓

composition

x ✓ ✓

Relation to Previous Work

10/28/2014 EMNLP 2014 in Doha, Qatar

word2vec Compositional neural language models Our model single token representations

✓ ✓ ✓

recursive structures

• f syntactic relations

x x ✓

pre-training

✓ x ✓

composition

x ✓ ✓

Relation to Previous Work

10/28/2014 EMNLP 2014 in Doha, Qatar

word2vec Compositional neural language models Our model single token representations

✓ ✓ ✓

recursive structures

• f syntactic relations

x x ✓

pre-training

✓ x ✓

composition

x ✓ ✓

• Learning word and composed representations

A Joint Learning Model

10/28/2014 EMNLP 2014 in Doha, Qatar

• Learning word and composed representations

– using syntactic structures of unlabeled corpora d vectors

A Joint Learning Model

10/28/2014 EMNLP 2014 in Doha, Qatar

• Learning word and composed representations

– using syntactic structures of unlabeled corpora – without pre-trained word vectors

A Joint Learning Model

10/28/2014 EMNLP 2014 in Doha, Qatar

• Learning word and composed representations

– using syntactic structures of unlabeled corpora – without pre-trained word vectors

A Joint Learning Model

10/28/2014 EMNLP 2014 in Doha, Qatar

storm downpour pay solve

• vercome

• Learning word and composed representations

– using syntactic structures of unlabeled corpora – without pre-trained word vectors

A Joint Learning Model

10/28/2014 EMNLP 2014 in Doha, Qatar

storm downpour heavy rain make payment pay solve problem achieve objective bridge gap solve

• vercome

• Learning word and composed representations

– using syntactic structures of unlabeled corpora – without pre-trained word vectors

A Joint Learning Model

10/28/2014 EMNLP 2014 in Doha, Qatar

storm downpour heavy rain make payment pay solve problem achieve objective bridge gap solve

• vercome

State-of-the-art scores for phrase similarity tasks with transitive verbs

• 1. Learning word representations

using predicate-argument structures

• 2. Jointly learning word representations and

composition functions

• 3. Evaluation on phrase similarity tasks
• 4. Conclusion

Overview

10/28/2014 EMNLP 2014 in Doha, Qatar

• 1. Learning word representations

using predicate-argument structures

• 2. Jointly learning word representations and

composition functions

• 3. Evaluation on phrase similarity tasks
• 4. Conclusion

Overview

10/28/2014 EMNLP 2014 in Doha, Qatar

• Standard dependency structures

– Relations between heads and modifiers

Predicate-Argument Structures (PASs)

10/28/2014 EMNLP 2014 in Doha, Qatar

the heavy rain caused the car accidents

• Standard dependency structures

– Relations between heads and modifiers

Predicate-Argument Structures (PASs)

10/28/2014 EMNLP 2014 in Doha, Qatar

the heavy rain caused the car accidents nn det det amod nsubj dobj root

• Standard dependency structures

– Relations between heads and modifiers

• Predicate-Argument Structures (PASs)

– Relations between predicates and arguments

Predicate-Argument Structures (PASs)

10/28/2014 EMNLP 2014 in Doha, Qatar

the heavy rain caused the car accidents nn det det amod nsubj dobj root the heavy rain caused the car accidents

• Each predicate in a sentence has

Predicate-Argument Structures (PASs)

10/28/2014 EMNLP 2014 in Doha, Qatar

(Enju parser (Miyao and Tsujii 2008))

the heavy rain caused the car accidents

• Each predicate in a sentence has

– a specific category

Predicate-Argument Structures (PASs)

10/28/2014 EMNLP 2014 in Doha, Qatar

(Enju parser (Miyao and Tsujii 2008))

the heavy rain caused the car accidents

• Each predicate in a sentence has

– a specific category – zero or more arguments

Predicate-Argument Structures (PASs)

10/28/2014 EMNLP 2014 in Doha, Qatar

(Enju parser (Miyao and Tsujii 2008))

the heavy rain caused the car accidents

• Each predicate in a sentence has

– a specific category – zero or more arguments

Predicate-Argument Structures (PASs)

10/28/2014 EMNLP 2014 in Doha, Qatar

(Enju parser (Miyao and Tsujii 2008))

the heavy rain caused the car accidents adjective argument 1

• Each predicate in a sentence has

– a specific category – zero or more arguments

Predicate-Argument Structures (PASs)

10/28/2014 EMNLP 2014 in Doha, Qatar

(Enju parser (Miyao and Tsujii 2008))

the heavy rain caused the car accidents verb adjective argument 1 argument 1 argument 2

• Each predicate in a sentence has

– a specific category – zero or more arguments

Predicate-Argument Structures (PASs)

10/28/2014 EMNLP 2014 in Doha, Qatar

(Enju parser (Miyao and Tsujii 2008))

the heavy rain caused the car accidents verb adjective noun argument 1 argument 1 argument 1 argument 2

• Given a PAS, discriminating between

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

• Given a PAS, discriminating between

– a word in the specific PAS

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

• Given a PAS, discriminating between

– a word in the specific PAS and – a word drawn from a noise distribution

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

• Given a PAS, discriminating between

– a word in the specific PAS and – a word drawn from a noise distribution

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

rain cause accident verb argument 1 argument 2

• Given a PAS, discriminating between

– a word in the specific PAS and – a word drawn from a noise distribution

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

a target word: cause rain cause accident verb argument 1 argument 2

• Given a PAS, discriminating between

– a word in the specific PAS and – a word drawn from a noise distribution

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

a target word: cause

a noise distribution (scaled unigram distribution in (Mikolov+, 2013))

rain cause accident verb argument 1 argument 2

• Given a PAS, discriminating between

– a word in the specific PAS and – a word drawn from a noise distribution

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

a target word: cause vs a drawn word: eat

a noise distribution (scaled unigram distribution in (Mikolov+, 2013))

rain eat accident verb argument 1 argument 2

• Given a PAS, discriminating between

– a word in the specific PAS and – a word drawn from a noise distribution

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

a target word: cause vs a drawn word: eat

a noise distribution (scaled unigram distribution in (Mikolov+, 2013))

rain eat accident verb argument 1 argument 2

context information

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

rain cause accident verb argument 1 argument 2

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

word vectors

rain cause accident verb argument 1 argument 2

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

argument 2 word vectors

rain cause accident verb argument 1 argument 2 𝑞 cause = tanh(ℎ𝑏𝑠𝑕1

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(rain) +

ℎ𝑏𝑠𝑕2

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12∗ 𝑤(accident))

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

argument 2

𝑞 cause = tanh(ℎ𝑏𝑠𝑕1

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(rain) +

ℎ𝑏𝑠𝑕2

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12∗ 𝑤(accident))

word vectors

rain cause accident verb argument 1 argument 2

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

argument 2

𝑞 cause = tanh(ℎ𝑏𝑠𝑕1

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(rain) +

ℎ𝑏𝑠𝑕2

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12∗ 𝑤(accident))

word vectors

rain cause accident verb argument 1 argument 2

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

argument 2

𝑞 cause = tanh(ℎ𝑏𝑠𝑕1

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(rain) +

ℎ𝑏𝑠𝑕2

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12∗ 𝑤(accident))

word vectors

rain cause accident verb argument 1 argument 2

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

cause

𝑡

argument 2

𝑞 cause = tanh(ℎ𝑏𝑠𝑕1

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(rain) +

ℎ𝑏𝑠𝑕2

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12∗ 𝑤(accident))

𝑡 = 𝑤 cause ∙ 𝑞(cause) 𝑡′ = 𝑤 eat ∙ 𝑞 cause

word vectors

rain cause accident verb argument 1 argument 2

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

cause eat

𝑡

argument 2

𝑡′

𝑞 cause = tanh(ℎ𝑏𝑠𝑕1

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(rain) +

ℎ𝑏𝑠𝑕2

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12∗ 𝑤(accident))

𝑡 = 𝑤 cause ∙ 𝑞(cause) 𝑡′ = 𝑤 eat ∙ 𝑞 cause

word vectors

rain cause accident verb argument 1 argument 2

A Word Prediction Model Using PASs

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

cause eat

𝑡

argument 2

𝑡′ 𝐝𝐩𝐭𝐮: 𝐧𝐛𝐲(𝟏, 𝟐 − 𝒕 + 𝒕′)

𝑞 cause = tanh(ℎ𝑏𝑠𝑕1

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(rain) +

ℎ𝑏𝑠𝑕2

𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12∗ 𝑤(accident))

𝑡 = 𝑤 cause ∙ 𝑞(cause) 𝑡′ = 𝑤 eat ∙ 𝑞 cause

word vectors

rain cause accident verb argument 1 argument 2

• Learning word representations based on

What We Expect from the Model

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

cause eat

𝑡

argument 2

𝑡′

• Learning word representations based on

– specific PAS categories

What We Expect from the Model

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

cause eat

𝑡

argument 2

𝑡′

• Learning word representations based on

– specific PAS categories – selectional preferences

What We Expect from the Model

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

cause eat

𝑡

argument 2

𝑡′

• Learning word representations based on

– specific PAS categories – selectional preferences

What We Expect from the Model

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

cause eat

𝑡

argument 2

𝑡′

• Learning word representations based on

– specific PAS categories – selectional preferences

What We Expect from the Model

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

cause eat

𝑡

argument 2

𝑡′

• Learning word representations based on

– specific PAS categories – selectional preferences

What We Expect from the Model

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

cause eat

𝑡

argument 2

𝑡′

• Learning word representations based on

– specific PAS categories – selectional preferences

What We Expect from the Model

10/28/2014 EMNLP 2014 in Doha, Qatar

``rain’’ can be

• a subject of ``cause’’

(not ``eat’’) 𝑤 rain 𝑤 accident

argument 1

+

cause eat

𝑡

argument 2

𝑡′

• Learning word representations based on

– specific PAS categories – selectional preferences

What We Expect from the Model

10/28/2014 EMNLP 2014 in Doha, Qatar

``rain’’ can be

• a subject of ``cause’’

(not ``eat’’)

• a cause of ``accident’’

𝑤 rain 𝑤 accident

argument 1

+

cause eat

𝑡

argument 2

𝑡′

Examples

10/28/2014 EMNLP 2014 in Doha, Qatar

eat at restaurant preposition argument 1 argument 2 heavy rain adjective argument 1

Examples

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 eat 𝑤 a𝑢

argument 1

+

predicate

eat at restaurant preposition argument 1 argument 2 heavy rain adjective argument 1

Examples

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 eat 𝑤 a𝑢

argument 1

+

restaurant cupboard

𝑡

predicate

𝑡′

eat at restaurant preposition argument 1 argument 2 heavy rain adjective argument 1

Examples

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 eat 𝑤 a𝑢

argument 1

+

restaurant cupboard

𝑡

predicate

𝑡′

𝑤 rain

argument 1

+

heavy delicious

𝑡 𝑡′

eat at restaurant preposition argument 1 argument 2 heavy rain adjective argument 1

• Providing additional context information

Adding Bag-of-Words Contexts

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

cause eat

𝑡

argument 2

𝑡′

• Providing additional context information

– Nouns and Verbs in the same sentences

Adding Bag-of-Words Contexts

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident

argument 1

+

cause eat

𝑡

argument 2

𝑡′

• Providing additional context information

– Nouns and Verbs in the same sentences

Adding Bag-of-Words Contexts

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident 𝑤 road 𝑤 injure

argument 1

+

cause eat

𝑡

argument 2

𝑡′

+

• Providing additional context information

– Nouns and Verbs in the same sentences

Adding Bag-of-Words Contexts

10/28/2014 EMNLP 2014 in Doha, Qatar

𝑤 rain 𝑤 accident 𝑤 road 𝑤 injure

argument 1

+

cause eat

𝑡

argument 2

𝑡′

+

BoW

• Learning representations composed by

Beyond Single Word Representations

10/28/2014 EMNLP 2014 in Doha, Qatar

• Learning representations composed by

– multiple words and

Beyond Single Word Representations

10/28/2014 EMNLP 2014 in Doha, Qatar

• Learning representations composed by

– multiple words and – specific relation categories

Beyond Single Word Representations

10/28/2014 EMNLP 2014 in Doha, Qatar

• Learning representations composed by

– multiple words and – specific relation categories

Beyond Single Word Representations

10/28/2014 EMNLP 2014 in Doha, Qatar

storm downpour

• Learning representations composed by

– multiple words and – specific relation categories

Beyond Single Word Representations

10/28/2014 EMNLP 2014 in Doha, Qatar

storm downpour

heavy rain adjective argument 1

• Learning representations composed by

– multiple words and – specific relation categories

Beyond Single Word Representations

10/28/2014 EMNLP 2014 in Doha, Qatar

storm downpour heavy rain

heavy rain adjective argument 1

• Using connections on graphs of PASs

A Specific PAS as a Single Token

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

𝑤 rain 𝑤 accident rain cause accident verb argument 1 argument 2

• Using connections on graphs of PASs

A Specific PAS as a Single Token

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

rain cause accident verb argument 1 argument 2 heavy adjective argument 1 car noun argument 1 𝑤 rain 𝑤 accident

• Using connections on graphs of PASs

A Specific PAS as a Single Token

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

𝑤 heavy__rain 𝑤 car__accident rain cause accident verb argument 1 argument 2 heavy adjective argument 1 car noun argument 1

parameterization

• Using connections on graphs of PASs

A Specific PAS as a Single Token

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

Same as Previously! rain cause accident verb argument 1 argument 2 heavy adjective argument 1 car noun argument 1 𝑤 heavy__rain 𝑤 car__accident

parameterization

• Similar tokens for each PAS representation

in terms of cosine similarity

Learned PAS Representations

10/28/2014 EMNLP 2014 in Doha, Qatar

heavy_rain chief_executive world_war rain thunderstorm downpour blizzard much_rain general_manager vice_president executive_director project_manager managing_director second_war plane_crash riot last_war great_war

• Similar tokens for each PAS representation

in terms of cosine similarity

Learned PAS Representations

10/28/2014 EMNLP 2014 in Doha, Qatar

make_payment solve_problem meeting_take_place make_order carry_survey pay_tax pay impose_tax achieve_objective bridge_gap improve_quality deliver_information encourage_development hold_meeting event_take_place end_season discussion_take_place do_work

• 1. Learning word representations

using predicate-argument structures

• 2. Jointly learning word representations and

composition functions

• 3. Evaluation on phrase similarity tasks
• 4. Conclusion

Overview

10/28/2014 EMNLP 2014 in Doha, Qatar

Why Composition?

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

𝑤 heavy__rain 𝑤 car__accident

Why Composition?

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

fully parameterized PAS representations

𝑤 heavy__rain 𝑤 car__accident

Why Composition?

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

fully parameterized PAS representations

• Very large number of combinations of words

𝑤 heavy__rain 𝑤 car__accident

Why Composition?

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

fully parameterized PAS representations

• Very large number of combinations of words

 Data sparseness 𝑤 heavy__rain 𝑤 car__accident

Why Composition?

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

fully parameterized PAS representations

• Very large number of combinations of words

 Data sparseness

• Ignoring information from individual words

𝑤 heavy__rain 𝑤 car__accident

Incorporating Composed Vectors

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

𝑤 heavy rain 𝑤 car accident

Incorporating Composed Vectors

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

𝑤 heavy rain 𝑤 car accident 𝑤 heavy 𝑤 rain 𝑤 car 𝑤 accident

word vectors

Incorporating Composed Vectors

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

𝑤 heavy rain 𝑤 car accident 𝑤 heavy 𝑤 rain 𝑤 car 𝑤 accident 𝒉𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 𝒉𝒐𝒑𝒗𝒐_𝒃𝒔𝒉𝟐

composition functions word vectors

Incorporating Composed Vectors

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

𝑤 heavy rain 𝑤 car accident 𝑤 heavy 𝑤 rain 𝑤 car 𝑤 accident 𝒉𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 𝒉𝒐𝒑𝒗𝒐_𝒃𝒔𝒉𝟐

composition functions composed vectors word vectors

Incorporating Composed Vectors

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

𝑤 heavy rain 𝑤 car accident 𝑤 heavy 𝑤 rain 𝑤 car 𝑤 accident 𝒉𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 𝒉𝒐𝒑𝒗𝒐_𝒃𝒔𝒉𝟐

composition functions composed vectors

Same as Previously!

word vectors

Incorporating Composed Vectors

10/28/2014 EMNLP 2014 in Doha, Qatar

argument 1

+

cause eat

𝑡

argument 2

𝑡′

𝑤 heavy 𝑤 rain 𝑤 car 𝑤 accident 𝒉𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 𝒉𝒐𝒑𝒗𝒐_𝒃𝒔𝒉𝟐

composition functions

𝑤 heavy rain 𝑤 car accident

• Simple element-wise composition functions

with and without tanh

Composition Functions in this Work

10/28/2014 EMNLP 2014 in Doha, Qatar

• Simple element-wise composition functions

with and without tanh – e.g.)

Composition Functions in this Work

10/28/2014 EMNLP 2014 in Doha, Qatar

Composition Function 𝒉𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 𝑤 heavy rain = 𝒉𝒃𝒆𝒌_𝒃𝒔𝒉𝟐(𝑤 heavy , 𝑤 rain )

• Simple element-wise composition functions

with and without tanh – e.g.)

Composition Functions in this Work

10/28/2014 EMNLP 2014 in Doha, Qatar

Composition Function 𝒉𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 Add𝑚 𝑤 heavy + 𝑤 rain Add𝑜𝑚 tanh(𝑤 heavy + 𝑤 rain ) 𝑤 heavy rain = 𝒉𝒃𝒆𝒌_𝒃𝒔𝒉𝟐(𝑤 heavy , 𝑤 rain )

• Simple element-wise composition functions

with and without tanh – e.g.)

Composition Functions in this Work

10/28/2014 EMNLP 2014 in Doha, Qatar

Composition Function 𝒉𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 Add𝑚 𝑤 heavy + 𝑤 rain Add𝑜𝑚 tanh(𝑤 heavy + 𝑤 rain ) WAdd𝑚 𝑛𝑞𝑠𝑓𝑒

𝑏𝑒𝑘_𝑏𝑠𝑕1 ∗ 𝑤 heavy + 𝑛𝑏𝑠𝑕1 𝑏𝑒𝑘_𝑏𝑠𝑕1 ∗ 𝑤 rain

WAdd𝑜𝑚 tanh(𝑛𝑞𝑠𝑓𝑒

𝑏𝑒𝑘_𝑏𝑠𝑕1 ∗ 𝑤 heavy + 𝑛𝑏𝑠𝑕1 𝑏𝑒𝑘_𝑏𝑠𝑕1 ∗ 𝑤 rain )

𝑤 heavy rain = 𝒉𝒃𝒆𝒌_𝒃𝒔𝒉𝟐(𝑤 heavy , 𝑤 rain )

Learned Composed Vectors

10/28/2014 EMNLP 2014 in Doha, Qatar

make payment solve problem run company make repayment make money make indemnity make saving make sum solve dilemma solve task solve difficulty solve trouble solve contradiction run firm run industry run corporation run enterprise run club

• Similar composed representations in terms of

cosine similarity

Learned Composed Vectors

10/28/2014 EMNLP 2014 in Doha, Qatar

people kill animal animal kill people meeting take place anyone kill animal man kill animal person kill animal people kill bird predator kill animal creature kill people effusion kill people elephant kill people tiger kill people people kill people briefing take place party take place session take place conference take place investiture take place

• Similar composed representations in terms of

cosine similarity

• L2-norms of the weight vectors of WAdd𝑜𝑚

Learned Composition Weights

10/28/2014 EMNLP 2014 in Doha, Qatar

Category Predicate Argument 1 Argument 2 adj_arg1 2.38 6.55

• noun_arg1

3.37 5.60

• verb_arg12

6.78 2.57 2.18

• L2-norms of the weight vectors of WAdd𝑜𝑚

– Clearly emphasizing head words

Learned Composition Weights

10/28/2014 EMNLP 2014 in Doha, Qatar

Category Predicate Argument 1 Argument 2 adj_arg1 2.38 6.55

• noun_arg1

3.37 5.60

• verb_arg12

6.78 2.57 2.18 nouns verbs

• 1. Learning word representations

using predicate-argument structures

• 2. Jointly learning word representations and

composition functions

• 3. Evaluation on phrase similarity tasks
• 4. Conclusion

Overview

10/28/2014 EMNLP 2014 in Doha, Qatar

• Training data

– PASs from BNC (~6 million sentences)

• adjective-noun, noun-noun
• prepositions and verbs with 2 arguments

Experimental Settings

10/28/2014 EMNLP 2014 in Doha, Qatar

• Training data

– PASs from BNC (~6 million sentences)

• adjective-noun, noun-noun
• prepositions and verbs with 2 arguments
• Dimensionality

– 50 and 1,000

Experimental Settings

10/28/2014 EMNLP 2014 in Doha, Qatar

• Training data

– PASs from BNC (~6 million sentences)

• adjective-noun, noun-noun
• prepositions and verbs with 2 arguments
• Dimensionality

– 50 and 1,000

• Optimization

– AdaGrad (Duchi+ 2011)

• learning rate: 0.05, mini-batch size: 32

Experimental Settings

10/28/2014 EMNLP 2014 in Doha, Qatar

• Measuring the semantic similarity between

Datasets for Evaluation

10/28/2014 EMNLP 2014 in Doha, Qatar

• Measuring the semantic similarity between

– Adjective-Noun phrases (AN) – Noun-Noun phrases (NN) – Verb-Object phrases (VO)

Datasets for Evaluation

10/28/2014 EMNLP 2014 in Doha, Qatar

(Mitchell and Lapata 2010)

• Measuring the semantic similarity between

– Adjective-Noun phrases (AN) – Noun-Noun phrases (NN) – Verb-Object phrases (VO) – Subject-Verb-Object phrases (SVO)

Datasets for Evaluation

10/28/2014 EMNLP 2014 in Doha, Qatar

(Mitchell and Lapata 2010) (Grefenstette and Sadrzadeh 2011)

• Measuring the semantic similarity between

– Adjective-Noun phrases (AN) – Noun-Noun phrases (NN) – Verb-Object phrases (VO) – Subject-Verb-Object phrases (SVO)

Datasets for Evaluation

10/28/2014 EMNLP 2014 in Doha, Qatar

(Mitchell and Lapata 2010) (Grefenstette and Sadrzadeh 2011) p1: vast amount p2: large quantity

AN dataset

• Measuring the semantic similarity between

– Adjective-Noun phrases (AN) – Noun-Noun phrases (NN) – Verb-Object phrases (VO) – Subject-Verb-Object phrases (SVO)

Datasets for Evaluation

10/28/2014 EMNLP 2014 in Doha, Qatar

(Mitchell and Lapata 2010) (Grefenstette and Sadrzadeh 2011) p1: vast amount p2: large quantity

AN dataset

human annotator similarity score 7

• Measuring the semantic similarity between

– Adjective-Noun phrases (AN) – Noun-Noun phrases (NN) – Verb-Object phrases (VO) – Subject-Verb-Object phrases (SVO)

Datasets for Evaluation

10/28/2014 EMNLP 2014 in Doha, Qatar

(Mitchell and Lapata 2010) (Grefenstette and Sadrzadeh 2011) p1: vast amount p2: large quantity

AN dataset

human annotator

cos 𝑤 𝑞1 , 𝑤 𝑞2 = 0.85

similarity score 7

• Measuring the semantic similarity between

– Adjective-Noun phrases (AN) – Noun-Noun phrases (NN) – Verb-Object phrases (VO) – Subject-Verb-Object phrases (SVO)

Datasets for Evaluation

10/28/2014 EMNLP 2014 in Doha, Qatar

(Mitchell and Lapata 2010) (Grefenstette and Sadrzadeh 2011) p1: vast amount p2: large quantity

AN dataset

human annotator

 Spearman’s rank correlation cos 𝑤 𝑞1 , 𝑤 𝑞2 = 0.85

similarity score 7

• Examples of phrase pairs for noun phrase tasks

Examples of Phrase Pairs

10/28/2014 EMNLP 2014 in Doha, Qatar

AN phrase pair score vast amount large quantity 7 important part significant role 7 efficient use little room 1 early stage dark eye 1 NN phrase pair score wage increase tax rate 7 education course training programme 6

• ffice worker

kitchen door 2 study group news agency 1

• Examples of phrase pairs for verb phrase tasks

Examples of Phrase Pairs

10/28/2014 EMNLP 2014 in Doha, Qatar

VO phrase pair score start work begin career 7 pour tea drink water 6 shut door close eye 1 wave hand start work 1 SVO phrase pair score student write name student spell name 7 child show sign child express sign 6 river meet sea river visit sea 1 system meet criterion system visit criterion 1

• Strong baselines produced by word2vec

Main Results (50dim)

10/28/2014 EMNLP 2014 in Doha, Qatar

0.1 0.2 0.3 0.4 0.5 0.6 0.7 AN NN VO SVO Correlation Score Add_l Add_nl Wadd_l Wadd_nl word2vec Human

• Strong baselines produced by word2vec

Main Results (50dim)

10/28/2014 EMNLP 2014 in Doha, Qatar

0.1 0.2 0.3 0.4 0.5 0.6 0.7 AN NN VO SVO Correlation Score Add_l Add_nl Wadd_l Wadd_nl word2vec Human

• Strong baselines produced by word2vec

Main Results (50dim)

10/28/2014 EMNLP 2014 in Doha, Qatar

0.1 0.2 0.3 0.4 0.5 0.6 0.7 AN NN VO SVO Correlation Score Add_l Add_nl Wadd_l Wadd_nl word2vec Human

• Strong baselines produced by word2vec
• Nice scores for verb phrase tasks

Main Results (50dim)

10/28/2014 EMNLP 2014 in Doha, Qatar

0.1 0.2 0.3 0.4 0.5 0.6 0.7 AN NN VO SVO Correlation Score Add_l Add_nl Wadd_l Wadd_nl word2vec Human

• Nice scores for verb phrase tasks
• Consistently outperforming 50 dimensional vectors

Main Results (1,000 dim)

10/28/2014 EMNLP 2014 in Doha, Qatar

0.1 0.2 0.3 0.4 0.5 0.6 0.7 AN NN VO SVO Correlation Score Add_l Add_nl Wadd_l Wadd_nl word2vec Human

• The AN, NN, and VO tasks

– BL: element-wise multiplications

(Blacoe and Lapata 2012)

– HB: recursive neural networks with CCGs

(Hermann and Blunsom 2013)

– KS: tensor-based composition models

(Kartsaklis and Sadrzadeh 2013)

• The SVO task

– GS, VC: tensor-based composition models

(Grefenstette and Sadrzadeh 2011), (Van de Cruys+ 2013)

Comparison with Previous Work

10/28/2014 EMNLP 2014 in Doha, Qatar

The AN, NN, and VO Tasks

10/28/2014 EMNLP 2014 in Doha, Qatar

0.1 0.2 0.3 0.4 0.5 0.6 0.7 AN NN VO Correlation Score Add_nl Wadd_nl BL HB KS Human

• 50 dim

– Comparable to state-of-the-art scores

The AN, NN, and VO Tasks

10/28/2014 EMNLP 2014 in Doha, Qatar

0.1 0.2 0.3 0.4 0.5 0.6 0.7 AN NN VO Correlation Score Add_nl Wadd_nl BL HB KS Human

• 1,000 dim

– New state-of-the-art score for the VO task

The AN, NN, and VO Tasks

10/28/2014 EMNLP 2014 in Doha, Qatar

0.1 0.2 0.3 0.4 0.5 0.6 0.7 AN NN VO Correlation Score Add_nl Wadd_nl BL HB KS Human

The SVO Task

10/28/2014 EMNLP 2014 in Doha, Qatar

0.1 0.2 0.3 0.4 0.5 0.6 0.7 SVO Correlation Score Wadd_nl GS VC Human

BNC ukWaC

• State-of-the-art models use large corpora

– e.g.) ukWaC corpus (~ 2B words)

• Achieving the state-of-the-art score using

a much smaller corpus – BNC (~ 0.1B words) vs ukWaC (~ 2B words)

The SVO Task

10/28/2014 EMNLP 2014 in Doha, Qatar

0.1 0.2 0.3 0.4 0.5 0.6 0.7 SVO Correlation Score Wadd_nl GS VC Human

BNC BNC ukWaC

• BoW contexts are helpful for the verb phrase tasks

– The results might be dependent on how to construct BoW contexts

Effects of BoW Contexts

10/28/2014 EMNLP 2014 in Doha, Qatar

0.1 0.2 0.3 0.4 0.5 0.6 0.7 AN NN VO SVO Correlation Score Wadd_nl w/o BoW Wadd_nl w/ BoW Human

• 1. Learning word representations

using predicate-argument structures

• 2. Jointly learning word representations and

composition functions

• 3. Evaluation on phrase similarity tasks
• 4. Conclusion

Overview

10/28/2014 EMNLP 2014 in Doha, Qatar

• Jointly learning composition functions

– with syntactic structures – without any pre-trained word vectors

• State-of-the-art scores for verb phrase similarity

tasks

Conclusion

10/28/2014 EMNLP 2014 in Doha, Qatar

• Incorporating more sophisticated composition

functions to improve verb phrase representations

• Learning full phrase representations rather than
• nly 2 or 3 word phrases

Future Work

10/28/2014 EMNLP 2014 in Doha, Qatar

• Any questions?

Thank You Very Much!

10/28/2014 EMNLP 2014 in Doha, Qatar