Question Answering and AnswerFinder Diego Moll a Centre for - - PowerPoint PPT Presentation

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Question Answering and AnswerFinder

Diego Moll´ a

Centre for Language Technology Department of Computer Science Macquarie University

Athens, 4 July 2007

Diego Moll´ a 1/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Programme

1

AnswerFinder Question Answering

2

Stages in Question Answering

3

Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Diego Moll´ a 2/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Programme

1

AnswerFinder Question Answering

2

Stages in Question Answering

3

Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Diego Moll´ a 3/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Who Are We? - Centre for Language Technology

Centre for Language Technology http://www.clt.mq.edu.au Located at Macquarie University, Sydney Members:

Director: Robert Dale Core academic staff: Steve Cassidy, Mark Dras, Diego Moll´ a, Debbie Richards, Rolf Schwitter, Paul Watters Honorary associate members: Dominique Estival, Mark Lauer, C´ ecile Paris 4 Research Assistants 11 PhD Students

Diego Moll´ a 4/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Who Are We? - AnswerFinder

http://www.ics.mq.edu.au/˜diego/answerfinder/ Funded by the Australian Research Council (ARC Discovery Grant) Members:

1

Diego Moll´ a (Project Director, Chief Investigator)

Design, Sentence Representation, QA Rules

2

Robert Dale (Chief Investigator)

Generation, Summarisation

3

Menno van Zaanen (Research Associate)

Implementation, Machine Learning

4

Luiz Augusto Pizzato (PhD student)

Document Retrieval, Web

5

Daniel Smith (Research Programmer)

Named Entity Recognition

Diego Moll´ a 5/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Programme

1

AnswerFinder Question Answering

2

Stages in Question Answering

3

Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Diego Moll´ a 6/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

What is Text-based Question Answering (QA)?

Steps in Text-based Question Answering

1 Accepts any English question 2 Searches a collection of text documents 3 Returns an answer to the user question

What Questions can we Answer? Factoid

Text Cross-lingual Speech transcripts

Query-driven summaries

Diego Moll´ a 7/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Issues in Question Answering

Size of the corpus Types of questions Types of expected answers Paraphrases of questions and answers Locating the exact answer

Diego Moll´ a 8/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Issues in Question Answering

Size of the corpus Types of questions Types of expected answers Paraphrases of questions and answers Locating the exact answer

Diego Moll´ a 8/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Issues in Question Answering

Size of the corpus Types of questions Types of expected answers Paraphrases of questions and answers Locating the exact answer

Diego Moll´ a 8/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Issues in Question Answering

Size of the corpus Types of questions Types of expected answers Paraphrases of questions and answers Locating the exact answer

Diego Moll´ a 8/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Issues in Question Answering

Size of the corpus Types of questions Types of expected answers Paraphrases of questions and answers Locating the exact answer

Diego Moll´ a 8/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Architecture

Off-line

Documents Document Preprocessing Document Image

On-line

Document Preselection Question Question Analysis Candidate Answer Selection Answer Extraction Answer Generation Answer Diego Moll´ a 9/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Architecture

Off-line

Documents Document Preprocessing Document Image

On-line

Document Preselection Question Question Analysis Candidate Answer Selection Answer Extraction Answer Generation Answer Diego Moll´ a 9/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Main Points of AnswerFinder I

Engineering

1 Build an environment for the development of QA systems 2 Emphasis on:

Flexibility Easy to modify and adapt to diverse applications Configurability Easy to integrate new algorithms and to try different parameters

3 Integration of third-party modules whenever possible Diego Moll´ a 10/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Question Answering

Main Points of AnswerFinder II

Theory

1 Focus on the selection of the answer sentence and extraction

• f the answer

But we are experimenting with the implementation of other modules as well

2 Use abstract (logical) representations of que question and text

sentences

3 Integrate machine learning techniques

Learning of logical form patterns Learning of weights in logical forms and patterns Question classification (in the future) Named entity recognition

Diego Moll´ a 11/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Programme

1

AnswerFinder Question Answering

2

Stages in Question Answering

3

Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Diego Moll´ a 12/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Architecture

Off-line

Documents Document Preprocessing Document Image

On-line

Document Preselection Question Question Analysis Candidate Answer Selection Answer Extraction Answer Generation Answer Diego Moll´ a 13/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Document Preprocessing (Indexing)

Goals

1 Build a document image that can be used efficiently 2 Do as much as possible off-line

What we are Doing Here

1 Experimenting with using syntactic information and semantic

labeling (named entities)

Diego Moll´ a 14/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Document Preprocessing (Indexing)

Goals

1 Build a document image that can be used efficiently 2 Do as much as possible off-line

What we are Doing Here

1 Experimenting with using syntactic information and semantic

labeling (named entities)

Diego Moll´ a 14/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Architecture

Off-line

Documents Document Preprocessing Document Image

On-line

Document Preselection Question Question Analysis Candidate Answer Selection Answer Extraction Answer Generation Answer Diego Moll´ a 15/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Document Preselection

Goal Find the documents (or text fragments) with highest likelihood to contain the answer Typical Methods Use an information retrieval system

As a blackbox, or Adapted to the task, or Designed specifically for the task

Diego Moll´ a 16/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Document Preselection

Goal Find the documents (or text fragments) with highest likelihood to contain the answer Our Current Method Use an information retrieval system

As a blackbox, or Adapted to the task, or Designed specifically for the task

Diego Moll´ a 16/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Document Preselection

Goal Find the documents (or text fragments) with highest likelihood to contain the answer Our Future Method Use an information retrieval system

As a blackbox, or Adapted to the task, or Designed specifically for the task

Diego Moll´ a 16/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Architecture

Off-line

Documents Document Preprocessing Document Image

On-line

Document Preselection Question Question Analysis Candidate Answer Selection Answer Extraction Answer Generation Answer Diego Moll´ a 17/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Question Analysis

Goal Determine all the important information about the question

1 Question type 2 Expected answer type 3 Additional information (e.g. question focus) 4 Question representation

Keywords Syntactic information Semantic information

Diego Moll´ a 18/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Our Method to Question Analysis

Current Method Use surface patterns of regular expressions Coming Soon Learning by structure induction

Diego Moll´ a 19/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Our Method to Question Analysis

Current Method Use surface patterns of regular expressions Coming Soon Learning by structure induction

Diego Moll´ a 19/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Architecture

Off-line

Documents Document Preprocessing Document Image

On-line

Document Preselection Question Question Analysis Candidate Answer Selection Answer Extraction Answer Generation Answer Diego Moll´ a 20/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Candidate Answer Selection

Goal Find all sentences (or text fragments) that contain the answer Typical Method

1 Ignore all text zones that can’t contain the answer

Use shallow methods

2 Find (and score) the sentences (fragments) containing the

answer

Combination of shallow and deep methods

Diego Moll´ a 21/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Candidate Answer Selection

Goal Find all sentences (or text fragments) that contain the answer Our Current Method

1 Ignore all sentences that do not contain a string of the

expected answer type

Use a named entity recogniser

2 Score remaining sentences using a combination of lexical,

syntactic, and semantic features

Diego Moll´ a 21/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Our Sentence Scoring Method

Lexical Information Word overlap Syntactical Information Grammatical relation overlap Semantic Information Logical forms, logical graphs

Diego Moll´ a 22/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Architecture

Off-line

Documents Document Preprocessing Document Image

On-line

Document Preselection Question Question Analysis Candidate Answer Selection Answer Extraction Answer Generation Answer Diego Moll´ a 23/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Answer Extraction

Goal Find the exact answer Elliminate noise Typical Methods Use of lexical, syntactic, semantic information Use of patterns Answer validation

“Can I prove that this answer is not impossible?” “Is there enough evidence that this is the answer?”

Diego Moll´ a 24/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Architecture

Off-line

Documents Document Preprocessing Document Image

On-line

Document Preselection Question Question Analysis Candidate Answer Selection Answer Extraction Answer Generation Answer Diego Moll´ a 25/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules

Answer Generation

Goal Present the answer to the user Methods Present the exact answer Include links to the source Merge answers Adapt the answers to the user model

Diego Moll´ a 26/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Programme

1

AnswerFinder Question Answering

2

Stages in Question Answering

3

Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Diego Moll´ a 27/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Background Info

Rule-based Approaches to QA Find answers by applying rules

example

Need to write a large number of rules

The problem of paraphrasing

Problems with portability So let’s learn the QA rules

Diego Moll´ a 28/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Background Info

Rule-based Approaches to QA Find answers by applying rules

example

Need to write a large number of rules

The problem of paraphrasing

Problems with portability So let’s learn the QA rules

Diego Moll´ a 28/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Example of a Rule

Question Pattern Who is the <X>position of <Y>country Answer Pattern Y’s X ANSWERtwo capitalised words The Rule Applied Q Who is the <Prime Minister>position of <Australia>country? A <Australia>’s <Prime Minister> <John Howard>ANSWER

Diego Moll´ a 29/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Graph Rule from a Question Answering Pair

Q: What book did Rachel Carson write in 1962?

v ch loc do in subj pcomp carson 1962 attr rachel write

• bj

book det what

A: In 1962 Carson wrote “Silent Spring”

subj tmp carson in attr 1962 write

• bj

spring attr silent Diego Moll´ a 30/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Graph Rule from a Question Answering Pair

Q: What book did Rachel Carson write in 1962?

v ch loc do in subj pcomp carson 1962 attr rachel write

• bj

book det what

A: In 1962 Carson wrote “Silent Spring”

subj tmp carson in attr 1962 write

• bj

spring attr silent Diego Moll´ a 30/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Graph Rule from a Question Answering Pair

Q: What book did Rachel Carson write in 1962?

v ch loc do in subj pcomp carson 1962 attr rachel write

• bj

book det what

A: In 1962 Carson wrote “Silent Spring”

subj tmp carson in attr 1962 write

• bj

spring attr silent

The Rule

X

• bj

Y det what ANSWER Diego Moll´ a 30/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Applying the Rule to the Question

Q: What book did Rachel Carson write in 1962?

v ch loc do in subj pcomp carson 1962 attr rachel write

• bj

book det what

The Rule

X

• bj

Y det what ANSWER Diego Moll´ a 31/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Applying the Rule to the Question

Q: What book did Rachel Carson write in 1962?

v ch loc do in subj pcomp carson 1962 attr rachel write

• bj

book det what

The Rule

X

• bj

Y det what ANSWER Diego Moll´ a 31/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Applying the Rule to the Question

Q: What book did Rachel Carson write in 1962?

v ch loc do in subj pcomp carson 1962 attr rachel write

• bj

book det what ANSWER

The Rule

X

• bj

Y det what ANSWER Diego Moll´ a 31/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Question Answering

Q: What book did Michael Ende write in 1984?

write

• bj

v ch loc do in subj pcomp ende 1984 attr michael book det what

A: In 1984 Michael Ende wrote the novel titled “The Neverending Story”

novel write v ch

• bj

loc do in subj pcomp ende 1984 attr michael det mod the title mod story det attr the neverending

The Rule

X

• bj

Y det what ANSWER Diego Moll´ a 32/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Question Answering

Q: What book did Michael Ende write in 1984?

write

• bj

v ch loc do in subj pcomp ende 1984 attr michael book det what

A: In 1984 Michael Ende wrote the novel titled “The Neverending Story”

novel write v ch

• bj

loc do in subj pcomp ende 1984 attr michael det mod the title mod story det attr the neverending

The Rule

X

• bj

Y det what ANSWER Diego Moll´ a 32/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Question Answering

Q: What book did Michael Ende write in 1984?

write

• bj

v ch loc do in subj pcomp ende 1984 attr michael book det what ANSWER

A: In 1984 Michael Ende wrote the novel titled “The Neverending Story”

novel write v ch

• bj

loc do in subj pcomp ende 1984 attr michael det mod the title mod story det attr the neverending

The Rule

X

• bj

Y det what ANSWER Diego Moll´ a 32/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Question Answering

Q: What book did Michael Ende write in 1984?

write

• bj

v ch loc do in subj pcomp ende 1984 attr michael book det what ANSWER

A: In 1984 Michael Ende wrote the novel titled “The Neverending Story”

novel write v ch

• bj

loc do in subj pcomp ende 1984 attr michael det mod the title mod story det attr the neverending

The Rule

X

• bj

Y det what ANSWER Diego Moll´ a 32/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Question Answering

Q: What book did Michael Ende write in 1984?

write

• bj

v ch loc do in subj pcomp ende 1984 attr michael book det what ANSWER

A: In 1984 Michael Ende wrote the novel titled “The Neverending Story”

novel write v ch

• bj

loc do in subj pcomp ende 1984 attr michael novel det mod the title mod story det attr the neverending

The Rule

X

• bj

Y det what ANSWER Diego Moll´ a 32/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Programme

1

AnswerFinder Question Answering

2

Stages in Question Answering

3

Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Diego Moll´ a 33/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

The Method

Finding the Question Pattern Question pattern: overlap between training question and answer sentence

Thus, question-only terms are ignored

Graph Theory: Find the maximum common subgraph Finding the Extension Graph Extension graph: path from the MCS to the answer:

The subgraph of the answer sentence that connects the MCS with the answer

Diego Moll´ a 34/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

The Method

Finding the Question Pattern Question pattern: overlap between training question and answer sentence

Thus, question-only terms are ignored

Graph Theory: Find the maximum common subgraph Finding the Extension Graph Extension graph: path from the MCS to the answer:

The subgraph of the answer sentence that connects the MCS with the answer

Diego Moll´ a 34/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Example

Q: What book did Rachel Carson write in 1962?

book det what v ch loc do in subj pcomp carson 1962 attr rachel write

• bj

A: In 1962 Carson wrote “Silent Spring”

subj tmp carson in attr 1962 write

• bj

spring attr silent

The Rule

Diego Moll´ a 35/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Example

Q: What book did Rachel Carson write in 1962?

book det what v ch loc do in subj pcomp carson 1962 attr rachel write

• bj

A: In 1962 Carson wrote “Silent Spring”

subj tmp carson in attr 1962 write

• bj

spring attr silent

The Rule

write

• bj

Diego Moll´ a 35/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Example

Q: What book did Rachel Carson write in 1962?

book det what v ch loc do in subj pcomp carson 1962 attr rachel write

• bj

A: In 1962 Carson wrote “Silent Spring”

subj tmp carson in attr 1962 write

• bj

spring attr silent

The Rule

write

• bj

ANSWER Diego Moll´ a 35/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Generalising and Weighting the Rule

Generalising The rule needs to be generalised to make it apply to unknown data Weighting Some rules are better than others; need to weight them Weight based on rule precision on training data: W(r) = # correct answers found # answers found

Diego Moll´ a 36/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Programme

1

AnswerFinder Question Answering

2

Stages in Question Answering

3

Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Diego Moll´ a 37/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

A Logical Graph

Features Format inspired on Sowa’s Conceptual Graphs Directed, bipartite graphs

Concepts Relations

Built from the output of Connexor FDG parser Tom believes that Mary wants to marry a sailor

tom 1 believe 2 want 1 mary 2 marry 1 2 sailor

Diego Moll´ a 38/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

QA with Logical Graphs I

Generalising the Graph Rule Relations are left unmodified Concepts are converted into variables, except for a list of “stop concepts” and, or, not, nor, if, otherwise, have, be, become, do, make

Diego Moll´ a 39/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

QA with Logical Graphs II

Ranking the Answer Answer score: overlap size × rule weight Overlap size: weighted sum of concepts and relations

Concept and relation weights inspired on IDF (Inverse document frequency)

Answer duplicates merge by adding up their individual scores Answers occurring in the question are ignored

Diego Moll´ a 40/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Further Work

Implement more accurate generalisations

Combine WordNet with a NE recogniser Use ML techniques (grammar induction) to learn types of vertices

Introduce question-only words in the question pattern Pay further attention to paraphrases

Synonyms Nominalisations Hyponyms

Diego Moll´ a 41/42

AnswerFinder Stages in Question Answering Learning Question Answering Rules Learning of Graph Rules Application: QA with Logical Graphs

Thank You!

Any questions...?

Diego Moll´ a 42/42