Jointly Learning Word Representations and Composition Functions - PowerPoint PPT Presentation

A Word Prediction Model Using PASs rain cause accident verb argument 2 argument 1 + 𝑞 cause = 𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(rain) + tanh(ℎ 𝑏𝑠𝑕1 argument argument 𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(accident)) ℎ 𝑏𝑠𝑕2 1 2 word vectors 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

A Word Prediction Model Using PASs rain cause accident verb argument 2 argument 1 + 𝑞 cause = 𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(rain) + tanh(ℎ 𝑏𝑠𝑕1 argument argument 𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(accident)) ℎ 𝑏𝑠𝑕2 1 2 word vectors 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

A Word Prediction Model Using PASs rain cause accident verb argument 2 argument 1 𝑡 𝑡 = 𝑤 cause ∙ 𝑞(cause) 𝑡 ′ = 𝑤 eat ∙ 𝑞 cause cause + 𝑞 cause = 𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(rain) + tanh(ℎ 𝑏𝑠𝑕1 argument argument 𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(accident)) ℎ 𝑏𝑠𝑕2 1 2 word vectors 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

A Word Prediction Model Using PASs rain cause accident verb argument 2 argument 1 𝑡 𝑡′ 𝑡 = 𝑤 cause ∙ 𝑞(cause) 𝑡 ′ = 𝑤 eat ∙ 𝑞 cause cause eat + 𝑞 cause = 𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(rain) + tanh(ℎ 𝑏𝑠𝑕1 argument argument 𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(accident)) ℎ 𝑏𝑠𝑕2 1 2 word vectors 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

A Word Prediction Model Using PASs rain cause accident verb 𝐝𝐩𝐭𝐮: 𝐧𝐛𝐲(𝟏, 𝟐 − 𝒕 + 𝒕′) argument 2 argument 1 𝑡 𝑡′ 𝑡 = 𝑤 cause ∙ 𝑞(cause) 𝑡 ′ = 𝑤 eat ∙ 𝑞 cause cause eat + 𝑞 cause = 𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(rain) + tanh(ℎ 𝑏𝑠𝑕1 argument argument 𝑤𝑓𝑠𝑐_𝑏𝑠𝑕12 ∗ 𝑤(accident)) ℎ 𝑏𝑠𝑕2 1 2 word vectors 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

What We Expect from the Model • Learning word representations based on 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

What We Expect from the Model • Learning word representations based on – specific PAS categories 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

What We Expect from the Model • Learning word representations based on – specific PAS categories – selectional preferences 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

What We Expect from the Model • Learning word representations based on – specific PAS categories – selectional preferences 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

What We Expect from the Model • Learning word representations based on – specific PAS categories – selectional preferences 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

What We Expect from the Model • Learning word representations based on – specific PAS categories – selectional preferences 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

What We Expect from the Model • Learning word representations based on – specific PAS categories – selectional preferences 𝑡 𝑡′ cause eat ``rain’’ can be + a subject of `` cause ’’ • (not `` eat ’’) argument argument 1 2 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

What We Expect from the Model • Learning word representations based on – specific PAS categories – selectional preferences 𝑡 𝑡′ cause eat ``rain’’ can be + a subject of `` cause ’’ • (not `` eat ’’) argument argument a cause of `` accident ’’ • 1 2 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

Examples heavy rain eat at restaurant preposition adjective argument 1 argument 2 argument 1 10/28/2014 EMNLP 2014 in Doha, Qatar

Examples heavy rain eat at restaurant preposition adjective argument 1 argument 2 argument 1 + argument predicate 1 𝑤 eat 𝑤 a𝑢 10/28/2014 EMNLP 2014 in Doha, Qatar

Examples heavy rain eat at restaurant preposition adjective argument 1 argument 2 argument 1 𝑡 𝑡′ restaurant cupboard + argument predicate 1 𝑤 eat 𝑤 a𝑢 10/28/2014 EMNLP 2014 in Doha, Qatar

Examples heavy rain eat at restaurant preposition adjective argument 1 argument 2 argument 1 𝑡 𝑡 𝑡′ 𝑡′ restaurant cupboard heavy delicious + + argument argument predicate 1 1 𝑤 eat 𝑤 a𝑢 𝑤 rain 10/28/2014 EMNLP 2014 in Doha, Qatar

Adding Bag-of-Words Contexts • Providing additional context information 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

Adding Bag-of-Words Contexts • Providing additional context information – Nouns and Verbs in the same sentences 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

Adding Bag-of-Words Contexts • Providing additional context information – Nouns and Verbs in the same sentences 𝑡 𝑡′ cause eat + argument argument 1 2 + 𝑤 rain 𝑤 accident 𝑤 road 𝑤 injure 10/28/2014 EMNLP 2014 in Doha, Qatar

Adding Bag-of-Words Contexts • Providing additional context information – Nouns and Verbs in the same sentences 𝑡 𝑡′ cause eat + argument argument BoW 1 2 + 𝑤 rain 𝑤 accident 𝑤 road 𝑤 injure 10/28/2014 EMNLP 2014 in Doha, Qatar

Beyond Single Word Representations • Learning representations composed by 10/28/2014 EMNLP 2014 in Doha, Qatar

Beyond Single Word Representations • Learning representations composed by – multiple words and 10/28/2014 EMNLP 2014 in Doha, Qatar

Beyond Single Word Representations • Learning representations composed by – multiple words and – specific relation categories 10/28/2014 EMNLP 2014 in Doha, Qatar

Beyond Single Word Representations • Learning representations composed by – multiple words and – specific relation categories storm downpour 10/28/2014 EMNLP 2014 in Doha, Qatar

Beyond Single Word Representations • Learning representations composed by – multiple words and – specific relation categories heavy rain adjective argument 1 storm downpour 10/28/2014 EMNLP 2014 in Doha, Qatar

Beyond Single Word Representations • Learning representations composed by – multiple words and – specific relation categories heavy rain adjective argument 1 storm heavy rain downpour 10/28/2014 EMNLP 2014 in Doha, Qatar

A Specific PAS as a Single Token • Using connections on graphs of PASs 𝑡 𝑡′ cause eat + rain cause accident argument argument verb 1 2 argument 2 argument 1 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

A Specific PAS as a Single Token • Using connections on graphs of PASs argument 1 argument 1 𝑡 𝑡′ cause eat heavy car adjective noun + rain cause accident argument argument verb 1 2 argument 2 argument 1 𝑤 rain 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

A Specific PAS as a Single Token • Using connections on graphs of PASs argument 1 argument 1 𝑡 𝑡′ cause eat heavy car adjective noun + rain cause accident argument argument verb 1 2 argument 2 argument 1 𝑤 heavy__rain 𝑤 car__accident parameterization 10/28/2014 EMNLP 2014 in Doha, Qatar

A Specific PAS as a Single Token • Using connections on graphs of PASs Same as Previously! argument 1 argument 1 𝑡 𝑡′ cause eat heavy car adjective noun + rain cause accident argument argument verb 1 2 argument 2 argument 1 𝑤 heavy__rain 𝑤 car__accident parameterization 10/28/2014 EMNLP 2014 in Doha, Qatar

Learned PAS Representations • Similar tokens for each PAS representation in terms of cosine similarity heavy_rain chief_executive world_war rain general_manager second_war thunderstorm vice_president plane_crash downpour executive_director riot blizzard project_manager last_war much_rain managing_director great_war 10/28/2014 EMNLP 2014 in Doha, Qatar

Learned PAS Representations • Similar tokens for each PAS representation in terms of cosine similarity make_payment solve_problem meeting_take_place make_order achieve_objective hold_meeting carry_survey bridge_gap event_take_place pay_tax improve_quality end_season pay deliver_information discussion_take_place impose_tax encourage_development do_work 10/28/2014 EMNLP 2014 in Doha, Qatar

Overview 1. Learning word representations using predicate-argument structures 2. Jointly learning word representations and composition functions 3. Evaluation on phrase similarity tasks 4. Conclusion 10/28/2014 EMNLP 2014 in Doha, Qatar

Why Composition? 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 heavy__rain 𝑤 car__accident 10/28/2014 EMNLP 2014 in Doha, Qatar

Why Composition? 𝑡 𝑡′ cause eat + argument argument 1 2 fully parameterized 𝑤 heavy__rain 𝑤 car__accident PAS representations 10/28/2014 EMNLP 2014 in Doha, Qatar

Why Composition? • Very large number of combinations of words 𝑡 𝑡′ cause eat + argument argument 1 2 fully parameterized 𝑤 heavy__rain 𝑤 car__accident PAS representations 10/28/2014 EMNLP 2014 in Doha, Qatar

Why Composition? • Very large number of combinations of words  Data sparseness 𝑡 𝑡′ cause eat + argument argument 1 2 fully parameterized 𝑤 heavy__rain 𝑤 car__accident PAS representations 10/28/2014 EMNLP 2014 in Doha, Qatar

Why Composition? • Very large number of combinations of words  Data sparseness • Ignoring information from individual words 𝑡 𝑡′ cause eat + argument argument 1 2 fully parameterized 𝑤 heavy__rain 𝑤 car__accident PAS representations 10/28/2014 EMNLP 2014 in Doha, Qatar

Incorporating Composed Vectors 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 heavy rain 𝑤 car accident 10/28/2014 EMNLP 2014 in Doha, Qatar

Incorporating Composed Vectors 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 heavy rain 𝑤 car accident word vectors 𝑤 heavy 𝑤 rain 𝑤 car 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

Incorporating Composed Vectors 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 heavy rain 𝑤 car accident 𝒉 𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 composition functions 𝒉 𝒐𝒑𝒗𝒐_𝒃𝒔𝒉𝟐 word vectors 𝑤 heavy 𝑤 rain 𝑤 car 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

Incorporating Composed Vectors 𝑡 𝑡′ cause eat + argument argument 1 2 composed vectors 𝑤 heavy rain 𝑤 car accident 𝒉 𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 composition functions 𝒉 𝒐𝒑𝒗𝒐_𝒃𝒔𝒉𝟐 word vectors 𝑤 heavy 𝑤 rain 𝑤 car 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

Incorporating Composed Vectors 𝑡 𝑡′ cause eat Same as Previously! + argument argument 1 2 composed vectors 𝑤 heavy rain 𝑤 car accident 𝒉 𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 composition functions 𝒉 𝒐𝒑𝒗𝒐_𝒃𝒔𝒉𝟐 word vectors 𝑤 heavy 𝑤 rain 𝑤 car 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

Incorporating Composed Vectors 𝑡 𝑡′ cause eat + argument argument 1 2 𝑤 heavy rain 𝑤 car accident 𝒉 𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 composition functions 𝒉 𝒐𝒑𝒗𝒐_𝒃𝒔𝒉𝟐 𝑤 heavy 𝑤 rain 𝑤 car 𝑤 accident 10/28/2014 EMNLP 2014 in Doha, Qatar

Composition Functions in this Work • Simple element-wise composition functions with and without tanh 10/28/2014 EMNLP 2014 in Doha, Qatar

Composition Functions in this Work • Simple element-wise composition functions with and without tanh – e.g.) 𝑤 heavy rain = 𝒉 𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 (𝑤 heavy , 𝑤 rain ) Composition Function 𝒉 𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 10/28/2014 EMNLP 2014 in Doha, Qatar

Composition Functions in this Work • Simple element-wise composition functions with and without tanh – e.g.) 𝑤 heavy rain = 𝒉 𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 (𝑤 heavy , 𝑤 rain ) Composition Function 𝒉 𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 Add 𝑚 𝑤 heavy + 𝑤 rain Add 𝑜𝑚 tanh(𝑤 heavy + 𝑤 rain ) 10/28/2014 EMNLP 2014 in Doha, Qatar

Composition Functions in this Work • Simple element-wise composition functions with and without tanh – e.g.) 𝑤 heavy rain = 𝒉 𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 (𝑤 heavy , 𝑤 rain ) Composition Function 𝒉 𝒃𝒆𝒌_𝒃𝒔𝒉𝟐 Add 𝑚 𝑤 heavy + 𝑤 rain Add 𝑜𝑚 tanh(𝑤 heavy + 𝑤 rain ) 𝑏𝑒𝑘_𝑏𝑠𝑕1 ∗ 𝑤 heavy + 𝑛 𝑏𝑠𝑕1 𝑏𝑒𝑘_𝑏𝑠𝑕1 ∗ 𝑤 rain WAdd 𝑚 𝑛 𝑞𝑠𝑓𝑒 𝑏𝑒𝑘_𝑏𝑠𝑕1 ∗ 𝑤 heavy + 𝑛 𝑏𝑠𝑕1 𝑏𝑒𝑘_𝑏𝑠𝑕1 ∗ 𝑤 rain ) WAdd 𝑜𝑚 tanh(𝑛 𝑞𝑠𝑓𝑒 10/28/2014 EMNLP 2014 in Doha, Qatar

Learned Composed Vectors • Similar composed representations in terms of cosine similarity make payment solve problem run company make repayment solve dilemma run firm make money solve task run industry make indemnity solve difficulty run corporation make saving solve trouble run enterprise make sum solve contradiction run club 10/28/2014 EMNLP 2014 in Doha, Qatar

Learned Composed Vectors • Similar composed representations in terms of cosine similarity people kill animal animal kill people meeting take place anyone kill animal creature kill people briefing take place man kill animal effusion kill people party take place person kill animal elephant kill people session take place people kill bird tiger kill people conference take place predator kill animal people kill people investiture take place 10/28/2014 EMNLP 2014 in Doha, Qatar

Learned Composition Weights • L2-norms of the weight vectors of WAdd 𝑜𝑚 Category Predicate Argument 1 Argument 2 adj_arg1 2.38 6.55 - noun_arg1 3.37 5.60 - verb_arg12 6.78 2.57 2.18 10/28/2014 EMNLP 2014 in Doha, Qatar

Learned Composition Weights • L2-norms of the weight vectors of WAdd 𝑜𝑚 – Clearly emphasizing head words nouns Category Predicate Argument 1 Argument 2 adj_arg1 2.38 6.55 - noun_arg1 3.37 5.60 - verb_arg12 6.78 2.57 2.18 verbs 10/28/2014 EMNLP 2014 in Doha, Qatar

Overview 1. Learning word representations using predicate-argument structures 2. Jointly learning word representations and composition functions 3. Evaluation on phrase similarity tasks 4. Conclusion 10/28/2014 EMNLP 2014 in Doha, Qatar

Experimental Settings • Training data – PASs from BNC ( ~ 6 million sentences) • adjective-noun, noun-noun • prepositions and verbs with 2 arguments 10/28/2014 EMNLP 2014 in Doha, Qatar

Experimental Settings • Training data – PASs from BNC ( ~ 6 million sentences) • adjective-noun, noun-noun • prepositions and verbs with 2 arguments • Dimensionality – 50 and 1,000 10/28/2014 EMNLP 2014 in Doha, Qatar

Experimental Settings • Training data – PASs from BNC ( ~ 6 million sentences) • adjective-noun, noun-noun • prepositions and verbs with 2 arguments • Dimensionality – 50 and 1,000 • Optimization – AdaGrad (Duchi+ 2011) • learning rate: 0.05, mini-batch size: 32 10/28/2014 EMNLP 2014 in Doha, Qatar

Datasets for Evaluation • Measuring the semantic similarity between 10/28/2014 EMNLP 2014 in Doha, Qatar

Datasets for Evaluation • Measuring the semantic similarity between – A djective- N oun phrases ( AN ) – N oun- N oun phrases ( NN ) (Mitchell and Lapata 2010) – V erb- O bject phrases ( VO ) 10/28/2014 EMNLP 2014 in Doha, Qatar

Datasets for Evaluation • Measuring the semantic similarity between – A djective- N oun phrases ( AN ) – N oun- N oun phrases ( NN ) (Mitchell and Lapata 2010) – V erb- O bject phrases ( VO ) – S ubject- V erb- O bject phrases ( SVO ) (Grefenstette and Sadrzadeh 2011) 10/28/2014 EMNLP 2014 in Doha, Qatar

Datasets for Evaluation • Measuring the semantic similarity between – A djective- N oun phrases ( AN ) – N oun- N oun phrases ( NN ) (Mitchell and Lapata 2010) – V erb- O bject phrases ( VO ) – S ubject- V erb- O bject phrases ( SVO ) (Grefenstette and Sadrzadeh 2011) p1: vast amount p2: large quantity AN dataset 10/28/2014 EMNLP 2014 in Doha, Qatar

Datasets for Evaluation • Measuring the semantic similarity between – A djective- N oun phrases ( AN ) – N oun- N oun phrases ( NN ) (Mitchell and Lapata 2010) – V erb- O bject phrases ( VO ) – S ubject- V erb- O bject phrases ( SVO ) (Grefenstette and Sadrzadeh 2011) human similarity score p1: vast amount annotator 7 p2: large quantity AN dataset 10/28/2014 EMNLP 2014 in Doha, Qatar

Datasets for Evaluation • Measuring the semantic similarity between – A djective- N oun phrases ( AN ) – N oun- N oun phrases ( NN ) (Mitchell and Lapata 2010) – V erb- O bject phrases ( VO ) – S ubject- V erb- O bject phrases ( SVO ) (Grefenstette and Sadrzadeh 2011) human similarity score p1: vast amount annotator 7 p2: large quantity cos 𝑤 𝑞1 , 𝑤 𝑞2 = 0.85 AN dataset 10/28/2014 EMNLP 2014 in Doha, Qatar