[PPT] - Question Generation with Minimal Recursion Semantics Xuchen Yao 1 PowerPoint Presentation

SLIDE 1

Introduction Background System Architecture Evaluation Conclusion References

Question Generation with Minimal Recursion Semantics

Xuchen Yao1 and Yi Zhang2

1European Masters in Language and Communication Technologies

University of Groningen & Saarland University

2Saarland University

German Research Center for Artificial Intelligence

18 June, QG/QGSTEC/2010

SLIDE 2

Introduction Background System Architecture Evaluation Conclusion References

Outline

Introduction Template/Syntax/Semantics-based Approaches Why Semantics-based? Background MRS/ERG/PET/LKB System Architecture Overview MRS Transformation for Simple Sentences MRS Decomposition for Complex Sentences Language Independence and Domain Adaptability Evaluation

SLIDE 3

Introduction Background System Architecture Evaluation Conclusion References

Approaches

Template-based (Mostow and Chen (2009))
What did <character> <verb>?
Syntax-based (Wyse and Piwek (2009), Heilman and Smith

(2009))

John plays football. (S NP (VP (V NP)))
John plays what? (S NP (VP (V WHNP)))
John does play what? (S NP (VP (Aux-V V WHNP)))
Does John play what? (S Aux-V NP (VP (V WHNP)))
What does John play? (S WHNP Aux-V NP (VP (V)))
Semantics-based
play(John, football)
play(John, what)
play(who, football)

SLIDE 4

Introduction Background System Architecture Evaluation Conclusion References

Approaches

Template-based (Mostow and Chen (2009))
What did <character> <verb>?
Syntax-based (Wyse and Piwek (2009), Heilman and Smith

(2009))

John plays football. (S NP (VP (V NP)))
John plays what? (S NP (VP (V WHNP)))
John does play what? (S NP (VP (Aux-V V WHNP)))
Does John play what? (S Aux-V NP (VP (V WHNP)))
What does John play? (S WHNP Aux-V NP (VP (V)))
Semantics-based
play(John, football)
play(John, what)
play(who, football)

SLIDE 5

Introduction Background System Architecture Evaluation Conclusion References

Outline

Introduction Template/Syntax/Semantics-based Approaches Why Semantics-based? Background MRS/ERG/PET/LKB System Architecture Overview MRS Transformation for Simple Sentences MRS Decomposition for Complex Sentences Language Independence and Domain Adaptability Evaluation

SLIDE 6

Introduction Background System Architecture Evaluation Conclusion References

Why Semantics-based?

Something different than template/syntax-based.
More intuitive?
More language independent (universal)?
Make use of the generation function of the English Resource

Grammar

Sag, I. A. & Flickinger, D. Generating Questions with Deep

Reversible Grammars. In Proceedings of the First Workshop on the Question Generation Shared Task and Evaluation

Challenge. 2008.
Deeper is better?

SLIDE 7

Introduction Background System Architecture Evaluation Conclusion References

Why Semantics-based?

Something different than template/syntax-based.
More intuitive?
More language independent (universal)?
Make use of the generation function of the English Resource

Grammar

Sag, I. A. & Flickinger, D. Generating Questions with Deep

Reversible Grammars. In Proceedings of the First Workshop on the Question Generation Shared Task and Evaluation

Challenge. 2008.
Deeper is better?

SLIDE 8

Introduction Background System Architecture Evaluation Conclusion References

Outline

Introduction Template/Syntax/Semantics-based Approaches Why Semantics-based? Background MRS/ERG/PET/LKB System Architecture Overview MRS Transformation for Simple Sentences MRS Decomposition for Complex Sentences Language Independence and Domain Adaptability Evaluation

SLIDE 9

Introduction Background System Architecture Evaluation Conclusion References

DELPH-IN (MRS/ERG/PET/LKB)

Deep Linguistic Processing with HPSG: http://www.delph-in.net/ INDEX: e2 RELS: < [ PROPER_Q_REL<0:4> LBL: h3 ARG0: x6 RSTR: h5 BODY: h4 ] [ _like_v_1_rel<5:10> LBL: h8 ARG0: e2 [ e SF: PROP TENSE: PRES ] ARG1: x6 ARG2: x9 [ PROPER_Q_REL<11:17> LBL: h10 ARG0: x9 RSTR: h12 BODY: h11 ] > HCONS: < h5 qeq h7 h12 qeq h13 > [ NAMED_REL<0:4> LBL: h7 ARG0: x6 (PERS: 3 NUM: SG) CARG: "John" ] [ NAMED_REL<11:17> LBL: h13 ARG0: x9 (PERS: 3 NUM: SG) CARG: "Mary" ]

John likes Mary. like(John, Mary)

Parsing with PET Generation with LKB

John likes Mary.

Minimal Recursion Semantics

English Resource Grammar

SLIDE 10

Introduction Background System Architecture Evaluation Conclusion References

Details

(THEORY)MRS: Minimal Recursion Semantics

a meta-level language for describing semantic structures in some underlying object language.

(GRAMMAR)ERG: English Resource Grammar

a general-purpose broad-coverage grammar implementation under the HPSG framework.

(TOOL)LKB: Linguistic Knowledge Builder

a grammar development environment for grammars in typed feature structures and unification-based formalisms.

(TOOL)PET: a platform for experimentation with efficient HPSG processing techniques

a two-stage parsing model with HPSG rules and PCFG models, balancing between precise linguistic interpretation and robust probabilistic coverage.

SLIDE 11

Introduction Background System Architecture Evaluation Conclusion References

Details

(THEORY)MRS: Minimal Recursion Semantics

a meta-level language for describing semantic structures in some underlying object language.

(GRAMMAR)ERG: English Resource Grammar

a general-purpose broad-coverage grammar implementation under the HPSG framework.

(TOOL)LKB: Linguistic Knowledge Builder

a grammar development environment for grammars in typed feature structures and unification-based formalisms.

(TOOL)PET: a platform for experimentation with efficient HPSG processing techniques

a two-stage parsing model with HPSG rules and PCFG models, balancing between precise linguistic interpretation and robust probabilistic coverage.

SLIDE 12

Introduction Background System Architecture Evaluation Conclusion References

Outline

Introduction Template/Syntax/Semantics-based Approaches Why Semantics-based? Background MRS/ERG/PET/LKB System Architecture Overview MRS Transformation for Simple Sentences MRS Decomposition for Complex Sentences Language Independence and Domain Adaptability Evaluation

SLIDE 13

Introduction Background System Architecture Evaluation Conclusion References

MrsQG (Task B)

http://code.google.com/p/mrsqg/

SLIDE 14

Introduction Background System Architecture Evaluation Conclusion References

Term Extraction

MRS XML

Plain text Term extraction FSC construction Parsing with PET 2 3 1 Generation with LKB Output selection 5 6 7 MRS

Transformation

MRS Decomposition 8 Output to console/XML

FSC XML

Apposition Decomposer Coordination Decomposer Subclause Decomposer Subordinate Decomposer Why Decomposer

MRS XML

4

Stanford Named Entity Recognizer
a regular expression NE tagger
an Ontology NE tagger

Jackson was born on August 29, 1958 in Gary, Indiana.

NEperson NEusPresident NEcapital NEmonth

NEday

NEyear NEfirstName NElocation NEprovince NEstate NElocation

NEperson NElocation NEdate

SLIDE 15

Introduction Background System Architecture Evaluation Conclusion References

Term Extraction

MRS XML

Plain text Term extraction FSC construction Parsing with PET 2 3 1 Generation with LKB Output selection 5 6 7 MRS

Transformation

MRS Decomposition 8 Output to console/XML

FSC XML

Apposition Decomposer Coordination Decomposer Subclause Decomposer Subordinate Decomposer Why Decomposer

MRS XML

4

Stanford Named Entity Recognizer
a regular expression NE tagger
an Ontology NE tagger

Jackson was born on August 29, 1958 in Gary, Indiana.

who which day where which location NEperson NElocation NEdate when

SLIDE 16

Introduction Background System Architecture Evaluation Conclusion References

Outline

Introduction Template/Syntax/Semantics-based Approaches Why Semantics-based? Background MRS/ERG/PET/LKB System Architecture Overview MRS Transformation for Simple Sentences MRS Decomposition for Complex Sentences Language Independence and Domain Adaptability Evaluation

SLIDE 17

Introduction Background System Architecture Evaluation Conclusion References

MRS Transformation

MRS XML

Plain text Term extraction FSC construction Parsing with PET 2 3 1 Generation with LKB Output selection 5 6 7 MRS

Transformation

MRS Decomposition 8 Output to console/XML

FSC XML

Apposition Decomposer Coordination Decomposer Subclause Decomposer Subordinate Decomposer Why Decomposer

MRS XML

4

SLIDE 18

Introduction Background System Architecture Evaluation Conclusion References

WHO

proper_q(x) named(x,"John") proper_q(y) named(y,"Mary") like(e,x,y) which_q(x) person(x) proper_q(y) named(y,"Mary") like(e,x,y)

Figure: “John likes Mary” → “Who likes Mary?”

SLIDE 19

Introduction Background System Architecture Evaluation Conclusion References

WHERE

proper_q(x) named(x,"Broadway") proper_q(y) named(y,"Mary") sing(e,y),

n_p(x)

which_q(x) place(x) proper_q(y) named(y,"Mary") sing(e,y), loc_nonsp(x)

Figure: “Mary sings on Broadway.” → “Where does Mary sing?”

SLIDE 20

Introduction Background System Architecture Evaluation Conclusion References

WHEN

def_implict_q(x) numbered_hour(x,"10") proper_q(y) named(y,"Mary") sing(e,y), at_p_temp(x) which_q(x) time(x) proper_q(y) named(y,"Mary") sing(e,y), loc_nonsp(x)

Figure: “Mary sings at 10.” → “When does Mary sing?”

SLIDE 21

Introduction Background System Architecture Evaluation Conclusion References

WHY

proper_q(x) named(x,"Mary") proper_q(y) named(y,"John") fight(e,y), for_p(x) which_q(x) reason(x) proper_q(y) named(y,"John") fight(e,y), for_p(x)

Figure: “John fights for Mary.” → “Why does John fight?”

SLIDE 22

Introduction Background System Architecture Evaluation Conclusion References

Outline

Introduction Template/Syntax/Semantics-based Approaches Why Semantics-based? Background MRS/ERG/PET/LKB System Architecture Overview MRS Transformation for Simple Sentences MRS Decomposition for Complex Sentences Language Independence and Domain Adaptability Evaluation

SLIDE 23

Introduction Background System Architecture Evaluation Conclusion References

MRS Decomposition

Complex Sentences -> Simple Sentences

MRS XML

Plain text Term extraction FSC construction Parsing with PET 2 3 1 Generation with LKB Output selection 5 6 7 MRS

Transformation

MRS Decomposition 8 Output to console/XML

FSC XML

Apposition Decomposer Coordination Decomposer Subclause Decomposer Subordinate Decomposer Why Decomposer

MRS XML

4

SLIDE 24

Introduction Background System Architecture Evaluation Conclusion References

Subclause Decomposer

identifies the verb, extracts its arguments and reconstructs MRS

proper_q(x) named(x,"Bart") the_q(y) the_q(z) dog(z) cat(y),chase(e2,y,z) be(e1,x,y) proper_q(x) named(x,"Bart") the_q(y) cat(y) be(e1,x,y)

Figure: “Bart is the cat that chases the dog.” → “Bart is the cat.”

SLIDE 25

Introduction Background System Architecture Evaluation Conclusion References

Subclause Decomposer

identifies the verb, extracts its arguments and reconstructs MRS

proper_q(x) named(x,"Bart") the_q(y) the_q(z) dog(z) cat(y),chase(e2,y,z) be(e1,x,y) the_q(y) cat(y) the_q(z) dog(z) chase(e2,y,z)

Figure: “Bart is the cat that chases the dog.” → “The cat chases the dog.”

SLIDE 26

Introduction Background System Architecture Evaluation Conclusion References

MRS Decomposition

Complex Sentences -> Simple Sentences

MRS XML

Plain text Term extraction FSC construction Parsing with PET 2 3 1 Generation with LKB Output selection 5 6 7 MRS

Transformation

MRS Decomposition 8 Output to console/XML

FSC XML

Apposition Decomposer Coordination Decomposer Subclause Decomposer Subordinate Decomposer Why Decomposer

MRS XML

4

SLIDE 27

Introduction Background System Architecture Evaluation Conclusion References

Outline

Introduction Template/Syntax/Semantics-based Approaches Why Semantics-based? Background MRS/ERG/PET/LKB System Architecture Overview MRS Transformation for Simple Sentences MRS Decomposition for Complex Sentences Language Independence and Domain Adaptability Evaluation

SLIDE 28

Introduction Background System Architecture Evaluation Conclusion References

Language Independence

MrsQG aims to stay language-neutral based on a semantics transformation of sentences.

In Principle

It needs little modification to adapt to other languages.

In Practice

It is difficult to guarantee absolute language independence.

SLIDE 29

Introduction Background System Architecture Evaluation Conclusion References

Domain Adaptability

MRS XML

Plain text Term extraction FSC construction Parsing with PET 2 3 1 Generation with LKB Output selection 5 6 7 MRS

Transformation

MRS Decomposition 8 Output to console/XML

FSC XML

Apposition Decomposer Coordination Decomposer Subclause Decomposer Subordinate Decomposer Why Decomposer

MRS XML

4 Needs to re-train or modify:

Stanford Named Entity Recognizer
a regular expression NE tagger
an Ontology NE tagger

PET Parser:

re-train with an HPSG treebank.

HPSG grammars:

Hand-written
Generalize well
Steady performance

SLIDE 30

Introduction Background System Architecture Evaluation Conclusion References

QGSTEC2010

The Question Generation Shared Task and Evaluation Challenge (QGSTEC) 2010

Task B: QG from Sentences.

Participants are given one complete sentence from which their system must generate questions.

1. Relevance. Questions should be relevant to the input

sentence.

2. Question type. Questions should be of the specified target

question type.

3. Syntactic correctness and fluency. The syntactic

correctness is rated to ensure systems can generate sensible

utput.
4. Ambiguity. The question should make sense when asked

more or less out of the blue.

5. Variety. Pairs of questions in answer to a single input are

evaluated on how different they are from each other.

SLIDE 31

Introduction Background System Architecture Evaluation Conclusion References

Examples

TEXT: Alexander Graham Bell, who had risen to prominence through his invention of the telephone, took a great interest in recording sounds, even suggesting to Edison that they might collaborate. WHO: Who took a great interest in recording sounds? WHO: Who is Alexander Graham Bell? WHAT: A great interest in what did Alexander Graham Bell take? WHAT: What did Alexander Graham Bell take a great interest in? WHY: Why Alexander Graham Bell took a great interest in cording sounds? WHY: Why do they collaborate?

SLIDE 32

Introduction Background System Architecture Evaluation Conclusion References

Outline

Introduction Template/Syntax/Semantics-based Approaches Why Semantics-based? Background MRS/ERG/PET/LKB System Architecture Overview MRS Transformation for Simple Sentences MRS Decomposition for Complex Sentences Language Independence and Domain Adaptability Evaluation

SLIDE 33

Introduction Background System Architecture Evaluation Conclusion References

Conclusion

semantics-based (easy in theory, difficult in practice)
multi-linguality
cross-domain
deep grammar (worry less, wait more)
generation <-> grammaticality
heavy machinery

SLIDE 34

Introduction Background System Architecture Evaluation Conclusion References

Conclusion

semantics-based (easy in theory, difficult in practice)
multi-linguality
cross-domain
deep grammar (worry less, wait more)
generation <-> grammaticality
heavy machinery

SLIDE 35

Introduction Background System Architecture Evaluation Conclusion References

References

M. Heilman and N. A. Smith. Question Generation via

Overgenerating Transformations and Ranking. Technical report, Language Technologies Institute, Carnegie Mellon University Technical Report CMU-LTI-09-013, 2009. Jack Mostow and Wei Chen. Generating instruction automatically for the reading strategy of self-questioning. In Proceeding of the 2009 conference on Artificial Intelligence in Education, Amsterdam, The Netherlands, 2009. ISBN 978-1-60750-028-5. Brendan Wyse and Paul Piwek. Generating Questions from OpenLearn study units. In Proceedings of the 2nd Workshop on Question Generation In Craig, S.D. & Dicheva, S. (Eds.) (2009) AIED 2009: 14th International Conference on Artificial Intelligence in Education: Workshops Proceedings, 2009.