Scott Wen-tau Yih # , What are the names of Obamas daughters? Q: - - PowerPoint PPT Presentation

Scott Wen-tau Yih#,

“What are the names of Obama’s daughters?”

Q: Who won the best actor Oscar in 1973? S1: Jack Lemmon was awarded the Best Actor Oscar for Save the Tiger (1973). S2: Academy award winner Kevin Spacey said that Jack Lemmon is remembered

as always making time for others.

• Question distribution
• Candidate selection bias
• Excluding questions that have no correct answers

Q: How did Seminole war end? A: Ultimately, the Spanish Crown ceded the colony to United States rule.

• Question distribution
• Candidate selection
• Including questions that have no correct answers

• Introduction
• WikiQA Dataset
• Data construction and annotation
• Data statistics

“Who wrote second Corinthians?”

• Step 1: Does the short paragraph answer the question?

Question: Who wrote second Corinthians? Second Epistle to the Corinthians The Second Epistle to the Corinthians, often referred to as Second Corinthians (and written as 2 Corinthians), is the eighth book of the New Testament of the Bible. Paul the Apostle and “Timothy our brother” wrote this epistle to “the church of God which is at Corinth, with all the saints which are in all Achaia”.

• Step 2: Check all the sentences that can answer the question

in isolation (assuming coreferences and pronouns have been resolved correctly)

Question: Who wrote second Corinthians? Second Epistle to the Corinthians q The Second Epistle to the Corinthians, often referred to as Second Corinthians (and written as 2 Corinthians), is the eighth book of the New Testament of the Bible. q Paul the Apostle and “Timothy our brother” wrote this epistle to “the church of God which is at Corinth, with all the saints which are in all Achaia”.

227 3,047

• 500

1,000 1,500 2,000 2,500 3,000 3,500

# of questions

QASent WikiQA

13.4x

227 8,478 3,047 29,258

• 5,000

10,000 15,000 20,000 25,000 30,000 35,000

# of questions # of sentences

QASent WikiQA

3.5x

Desc. 7% Human 29% Numeri c 31% Locatio n 16% Abbr. 1% Entity 16%

QASent

Desc. 35% Human 16% Numeri c 22% Locatio n 12% Abbr. 1% Entity 14%

WikiQA

• Introduction
• WikiQA Dataset
• Experiments
• Baseline systems
• Evaluation on answer sentence selection
• Evaluation metric & results on answer triggering

• Word matching count (Wd-Cnt)
• # non-stopwords in Q that also occur in S
• Latent word alignment (Wd-Algn) [Yih+ ACL-13]
• Convolutional NN (CNN) [Yu+ DLWorkshop-14]
• Convolutional NN & Wd-Cnt (CNN-Cnt)

𝑟: What is the fastest car in the world? 𝑡: The Jaguar XJ220 is the dearest, fastest and most sought after car on the planet. 𝒊 𝑔(𝑟,𝑡)=​𝜄↑𝑈 Φ(ℎ)

0.6662 0.7617 0.623 0.7633

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

QASent

Mean Reciprocal Rank (MRR) Wd-Cnt Wd-Algn CNN CNN-Cnt

0.6662 0.4924 0.7617 0.6086 0.623 0.6281 0.7633 0.6652

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9

QASent WikiQA*

Mean Reciprocal Rank (MRR) Wd-Cnt Wd-Algn CNN CNN-Cnt

• Data – All questions in WikiQA are included in this

task

30.61

28 28.5 29 29.5 30 30.5 31 31.5 32 32.5 33

Question-level F1 score CNN-Cnt +Q-Length +S-Length +Q-Class +All

30.61 32.17

28 28.5 29 29.5 30 30.5 31 31.5 32 32.5 33

Question-level F1 score CNN-Cnt +Q-Length +S-Length +Q-Class +All

30.61 32.17 30.92 30.34 31.64

28 28.5 29 29.5 30 30.5 31 31.5 32 32.5 33

Question-level F1 score CNN-Cnt +Q-Length +S-Length +Q-Class +All

http://aka.ms/WikiQA

• Includes “answer phrases” labeled by authors