The Generation of Referring Expressions: The Generation of Referring - - PowerPoint PPT Presentation

The Generation of Referring Expressions: The Generation of Referring Expressions: Where We've Been, How We Got Here, and Where Where We've Been, How We Got Here, and Where We We're Going re Going We We re Going re Going

Robert Dale Robert Dale Robert Dale Robert Dale Robert.Dale@mq.edu.au Robert.Dale@mq.edu.au

Athens 2008-05-21 1

The Aims of This Talk The Aims of This Talk

• To outl

To outline ine what referring expressi hat referring expression generation is about

• n generation is about
• To characterise

To characterise the current state the current state of the art and developments in

• f the art and developments in

the field the field

• To outl

To outline an ine an agenda agenda for future work in the area for future work in the area

Athens 2008-05-21 2

Outline Outline

• The Context: Natural Language Generation

The Context: Natural Language Generation

• The Story So Far:

The Story So Far: Algorithm Development to Empiricism Algorithm Development to Empiricism

• Challenges for the Future

Challenges for the Future

Athens 2008-05-21 3

The Context The Context

• Natural Language Generation

Natural Language Generation is concerned with generati is concerned with generating lingui ng linguistic tic material from some non material from some non ling ling istic base stic base material from some non material from some non-ling linguistic base istic base

• Why is this important?

Why is this important? – Applications Applications: : – any situation where it is not pr any situation where it is not practical to construct the full actical to construct the full f f i d h d f f i range range of f requ require red d outpu

• utputs ahea

ead d of f time me – Theory Theory: : – understanding what drives choice-making in language understanding what drives choice-making in language

Athens 2008-05-21 4

Natural Language Generation Applications Natural Language Generation Applications

• Generating text from large data sets:

Generating text from large data sets: – Weather reports, stock market reports Weather reports, stock market reports

• Information personalisation:

Information personalisation: – Tailored web pages that take account of what you know Tailored web pages that take account of what you know

• Context-sensitive

Context-sensitive generation: eneration: g – Dynamic utterance construction in dialog systems Dynamic utterance construction in dialog systems

• Multilingual generation:

Multilingual generation:

• Multilingual generation:

Multilingual generation: – Multiple languages from a Multiple languages from a common knowledge source common knowledge source

Athens 2008-05-21 5

NL Understanding vs NL Understanding vs NL Generation NL Generation

• The view from Natural Language Understanding:

The view from Natural Language Understanding: – Deriving meaning from text means throwing away Deriving meaning from text means throwing away or

• r

ignoring irrelevant detail ignoring irrelevant detail

• The view from Natural Language Generation:

The view from Natural Language Generation: – Very Very few, if any, few, if any, surface variations are surface variations are meaningless; we meaningless; we d d l i h h i f i if if d d h h nee need d to to exp explain their f funct unction

• n if

if we we are are to to un understan erstand d them em properly properly

Athens 2008-05-21 6

Mapping Between Representations: NLU Mapping Between Representations: NLU

[e,x,y]kissing(e) [e,x,y]kissing(e) ∧ past(e) past(e) ∧ name(x name(x John) hn) ∧ name(y name(yMar Mary) ) ∧ name(x name(x,Jo John) ∧ name(y name(y,Mar Mary) ) ∧ agent(e,x) ent(e,x) ∧ patient(e,y) patient(e,y) John kissed Mar hn kissed Mar Mar Mar as kissed b as kissed b J John hn John kissed Mar hn kissed Mary. y. Mar Mary was kissed b as kissed by Jo John.

Athens 2008-05-21 7

Mapping Between Representations: NLG Mapping Between Representations: NLG

[e,x,y]kissing(e) [e,x,y]kissing(e) ∧ past(e) past(e) ∧ name(x name(x John) hn) ∧ name(y name(yMar Mary) ) ∧ name(x name(x,Jo John) ∧ name(y name(y,Mar Mary) ) ∧ agent(e,x) ent(e,x) ∧ patient(e,y) patient(e,y) ... ... ∧ fo focus(y) ... ... ∧ fo focus(x) John kissed Mar hn kissed Mar Mar Mar as kissed b as kissed b J John hn John kissed Mar hn kissed Mary. y. Mar Mary was kissed b as kissed by Jo John.

Athens 2008-05-21 8

The NLGer’s The NLGer’s Position

• sition
• If we understand how and why te

If we understand how and why texts are xts are put together the way put together the way the the are are e e ill be in a ill be in a better position to take them apart better position to take them apart the they are are, we e will be in a ill be in a better position to take them apart better position to take them apart

• Generation provides insights that should improve

Generation provides insights that should improve – Information extraction: working out what parts of a Information extraction: working out what parts of a text are text are important important T i T i i ki h l i l – Text ext summar summarisat sation:

• n: wor

worki king ng out

• ut h

how

• w to

to rep replace ace i incomp ncomplete ete references in extracted material references in extracted material M h M hi t l ti ti ki ki h i th th t t i t t t – Mac achi hine ne trans ranslati tion:

• n: ma

maki king ng choices ces th that t are are appropr appropriate t to context context

Athens 2008-05-21 9

An Architecture for Generation An Architecture for Generation

Document Document Pl Pl i

Content Determinati Content Determination T S T S i

Pl Plann anning ng

Text ext S Structur tructuring ng Le Lexicali xicalisati sation

• n

Micr Micro

• Plannin

Planning

Le Lexicali xicalisati sation

• n

Ag Aggr gregation ation R f R f i E E i G G i

g S f S f

Refer erring ng E Expr xpress ssion

• n G

Gener enerat ation

• n

Syntax m morphology

• gy

Sur urface ace Realization alization

Syntax Syntax, m morphology

• gy,
• r
• rthogr

thograph phy and pr y and prosod

• sody

Athens 2008-05-21 10

Referring Expression Generation Referring Expression Generation

D i D i M M d l Input pr Input propositions:

• positions:
• wns
• wns(m,

m, j1), wear , wears(m, m, j1, d1 1, d1) Doma

• main M

Model: What What Ther ere is In T e is In The W e World ( j ( j ) ( j ) Discour Discourse Model: e Model: What Has Been T What Has Been Talked lked About About Re Referring Expr Expressio ession Gener Generator tor User Mode User Model: l: What the What the Hear Hearer Knows About er Knows About NP semantics: NP semantics: isa(j isa(j1, jac , jacket et) ) ∧ colour(j1, w colour(j1, white) ite) What the What the Hear Hearer Knows About er Knows About

Athens 2008-05-21 11

The Effect of Discourse Context on Reference The Effect of Discourse Context on Reference

• Example 1:

Example 1: ( j1) j1) M M hi j j k – owns

• wns(m,

m, j1) j1) → Matt att owns

• wns a whi

hite te j jac acket. et. – wears(m, j1, wears(m, j1, d) d) → He wears it He wears it on Sundays.

• n Sundays.

E l E l Dif Differ erent ent

• Examp

xample 2: 2: – owns(m, [j1+c1])

• wns(m, [j1+c1]) → Matt owns a

Matt owns a white jacket and a white jacket and a white coat. white coat. – wears(m, wears(m, j1, d) 1, d) → He wears the He wears the jacket acket on Sundays.

• n Sundays.
• Example 3:

Example 3: Same Same – owns

• wns(m, [

(m, [j1+ 1+j2]) 2]) →Matt owns Matt owns a a white white jacket and a acket and a blue blue jacket. acket. – wears(m, j1, wears(m, j1, d) d) → He wears the white one He wears the white one on Sundays.

• n Sundays.

Athens 2008-05-21 12

Outline Outline

• The Context: Natural Language Generation

The Context: Natural Language Generation

• The Story So Far:

The Story So Far: Algorithm Development to Empiricism Algorithm Development to Empiricism

• Challenges for the Future

Challenges for the Future

Athens 2008-05-21 13

The Consensus Problem Statement The Consensus Problem Statement

The goal: The goal: G G di di i i hi hi d d i i Generate enerate a di dist stingu nguishi hing ng d descr escript ption

• n

Given: Given: i d i d d f f

• an

an inten ntended d re referent erent;

• a

a knowledge base of entiti knowledge base of entities es characterised characterised by properties y properties expressed as expressed as attribute attribute value pairs value pairs; and ; and expressed as expressed as attribute attribute–value pairs value pairs; and ; and

• a context

a context consis consistin ting of other enti

• f other entities that are

ties that are salient; salient; Then: Then: Then: Then:

• choose a

choose a set of attribute–value pairs that uniquely identi set of attribute–value pairs that uniquely identify the fy the intended referent intended referent

Athens 2008-05-21 14

intended referent intended referent

Computing Distinguishing Descriptions Computing Distinguishing Descriptions

Three steps which are Three steps which are repeated until a repeated until a successful successful description has description has been constr been constr cted cted been constr been constructed cted: Start with a Start with a null description. null description. 1. 1. Check whether the description co Check whether the description constructed so far is successful nstructed so far is successful in picking out the intended refe in picking out the intended referent from the context set. rent from the context set. If If so so quit quit so so, quit quit. 2. 2. If it's not sufficient, choose a If it's not sufficient, choose a property that will contribute to property that will contribute to the description the description the description the description. 3. 3. Extend the descripti Extend the description with n with this property, and reduce the this property, and reduce the context set accordi context set accordingly gly Go to Step 1 Go to Step 1

Athens 2008-05-21 15

context set accordi context set accordingly

• gly. Go to Step 1

Go to Step 1.

Com Computin uting Distin Distinguishin uishing Descri Descriptions: tions: p g p g g g p The Greedy Algorithm The Greedy Algorithm

Init Initial Co ial Cond nditions: itions: Cr = = 〈all all entiti ntities〉; P ; Pr = = 〈all all prop properti rties true of r es true of r〉; ; Lr = {} = {} 1. 1. Chec Check Success k Success if |C if |Cr| | = 1 then = 1 then return return Lr as a a d dist stinguishing inguishing description scription elsei elseif P = 0 then = 0 then return return L as a non as a non-dd dd elsei elseif Pr = 0 then = 0 then return return Lr as a non as a non-dd dd else goto else goto Step 2. tep 2. 2. 2. Choose Property Choose Property for each p for each pi ∈ Pr do: C do: Cri ← Cr ∩ {x | {x | pi(x)} (x)} Chosen prop Chosen property is y is pj, where C , where Crj is smallest is smallest set. set. goto goto Step 3. Step 3. goto goto Step 3. Step 3. 3. 3. Extend Descri Extend Description (wrt n (wrt the the chosen p chosen pj) Lr ← Lr ∪ {p {pj}; C }; Cr ← Crj; P ; Pr ← Pr ⎯ {p {pj}; goto }; goto Step 1. tep 1. [Dal [Dale 1987] e 1987]

Athens 2008-05-21 16

An Example An Example

• Suppose x1 is the intended referent:

Suppose x1 is the intended referent:

Entity Type Size State x1 dog small mangy x1 dog small mangy x2 dog large scurvy x3 cat small mangy x3 cat small mangy

Athens 2008-05-21 17

An Example An Example

• Choose ‘mangy’ to rule out x2:

Choose ‘mangy’ to rule out x2:

Entity Type Size State x1 dog small mangy x1 dog small mangy x2 dog large scurvy x3 cat small mangy x3 cat small mangy

Athens 2008-05-21 18

An Example An Example

• Choose ‘mangy’ to rule out x2:

Choose ‘mangy’ to rule out x2:

Entity Type Size State x1 dog small mangy x1 dog small mangy x2 dog large scurvy x3 cat small mangy x3 cat small mangy

Athens 2008-05-21 19

An Example An Example

• Choose ‘dog’ to rule out x3:

Choose ‘dog’ to rule out x3:

Entity Type Size State x1 dog small mangy x1 dog small mangy x2 dog large scurvy x3 cat small mangy x3 cat small mangy

Athens 2008-05-21 20

An Example An Example

• Choose ‘dog’ to rule out x3:

Choose ‘dog’ to rule out x3:

Entity Type Size State x1 dog small mangy x1 dog small mangy x2 dog large scurvy x3 cat small mangy

• The result is

The result is ‘the the mangy mangy dog dog’

x3 cat small mangy

• The result is

The result is the the mangy mangy dog dog

• ‘The small dog’ is also a

‘The small dog’ is also a disti distingui guishi hing description. ng description.

Athens 2008-05-21 21

Problem #1: Problem #1: Computational Complexity Computational Complexity

• The algorithm does not guarantee to find a

The algorithm does not guarantee to find a minimal minimal disting disting ishing description [Reiter 1990] shing description [Reiter 1990] disti distinguishing description [Reiter 1990] ishing description [Reiter 1990]

Athens 2008-05-21 22

Problem #2: Problem #2: No User Model No User Model

• The algorithm assumes that all pr

The algorithm assumes that all properties are

• perties are equal: it is only

equal: it is only the relati the relati e discrimi e discriminator nator po po er er and nothi and nothing else g else that ca that ca ses ses the relati the relative discriminator e discriminatory po power er, and nothing else and nothing else, that ca that causes ses a particul a particular property to be selected. ar property to be selected.

• Some properties are

Some properties are more useful more useful than other properties which than other properties which

• Some properties are

Some properties are more useful more useful than other properties which than other properties which have the same discriminatory power. have the same discriminatory power. A A talks to B talks to B on the tram:

• n the tram:

A Whi Whi h t d d I I t f t f th th i ? A: Whi Which h stop

• p d

do I I wan want f t for

• r th

the cinema nema? B: B: You should take the stop before mine. You should take the stop before mine.

Athens 2008-05-21 23

Problem #3: Problem #3: It’s Not What People Do It’s Not What People Do

• Context Set = b1, c1, c2

Context Set = b1, c1, c2

• Intended Referent =

Intended Referent = b1 b1

• Domain Model:

Domain Model: – bird(b1), white(b1) bird(b1), white(b1) – cu cup( p(c1 c1), ), black black(c1 c1) p( p( ), ), ( ) – cup(c2), white(c2) cup(c2), white(c2)

• Typical descri

Typical descripti ption: n: ‘the white bird the white bird’

• Typical descri

Typical descripti ption: n: the white bird the white bird

Athens 2008-05-21 24

A Res A Response:

• nse:

p The Incremental Algorithm The Incremental Algorithm

Init Initial Co ial Cond nditions: itions: – Cr = = 〈all entiti all entities es〉; P ; P = = 〈preferr preferred ed attributes attributes〉; ; Lr = {} = {} 1. 1. Chec Check Success k Success – if |C if |Cr| | = 1 then = 1 then return return Lr as a a d dist stinguishing inguishing description scription – elsei elseif P = P = 0 then return 0 then return L as a non as a non-dd dd elsei elseif P = P = 0 then return 0 then return Lr as a non as a non-dd dd – else goto else goto Step 2. tep 2. 2. 2. Evaluate Next Property Evaluate Next Property – get next p get next pi ∈ P such that userknows(p P such that userknows(pi(r)) (r)) – if |{x if |{x ∈ Cr | p | pi(x)} (x)}| < | < |C |Cr| then goto | then goto Step 3 tep 3 – else else goto goto Step 2. Step 2. else else goto goto Step 2. Step 2. 3. 3. Extend Descri Extend Description (wrt n (wrt the the chosen p chosen pj) – Lr ← Lr ∪ {p {pj}; C }; Cr ← Crj

rj;

; goto goto Step 1. tep 1. [Rei [Reiter and ter and Dale 1992] Dale 1992]

Athens 2008-05-21 25

The Ke The Key Pro Propert erty of

• f

y p y p y the Incremental Algorithm the Incremental Algorithm

• Principle distinction between:

Principle distinction between: – the way the way choices are choices are made (domain independent) made (domain independent) – the choices available (domain dependent) the choices available (domain dependent)

Athens 2008-05-21 26

Extensions to the Basic Al Extensions to the Basic Algorithms:

• rithms:

g Relations Relations

• What happens if you need to ment

What happens if you need to mention another entity in order to ion another entity in order to identif identif the intended referent? the intended referent? identif identify the intended referent? the intended referent? – ‘the dog next to the small cat’ ‘the dog next to the small cat’

• Extensions to incorporate relations:

Extensions to incorporate relations: – constraint-based extension for relational properties [Dale constraint-based extension for relational properties [Dale d H d H dd k ] ] an and H d Hadd ddoc

• ck

k 1991 1991] ] – referring to parts of hierarchically structured objects referring to parts of hierarchically structured objects [H k H k 2006] 2006] [Horace

• racek 2006]

2006]

Athens 2008-05-21 27

Extensions to the Basic Al Extensions to the Basic Algorithms:

• rithms:

g Disjunction and Negation of Properties Disjunction and Negation of Properties

• What happens if there are

What happens if there are mult multiple entities instead of one? iple entities instead of one? – ‘the two dogs’ ‘the two dogs’ – ‘the dog and the cat’ ‘the dog and the cat’

• What happens if a

What happens if a distinguishing distinguishing characteristic is that the characteristic is that the intended referent lacks intended referent lacks some property? some property? – ‘the dog that isn’t a ‘the dog that isn’t a poodle’ poodle’

• Extensions:

Extensions: – Sets [Stone 2000] Sets [Stone 2000] – Negation and Disju Negation and Disjuncti ction [van n [van Deemter Deemter 2002]: 2002]:

Athens 2008-05-21 28

Negation and Disju Negation and Disjuncti ction [van n [van Deemter Deemter 2002]: 2002]:

More Al More Algorithm Develo

• rithm Development:

ment: g p g p A Selection A Selection

• Integration of

Integration of linguistic reference and pointing [Reithinger 1987] linguistic reference and pointing [Reithinger 1987] G ti tifi tifi [C [C 1996] 1996]

• Genera

enerati ting ng quan quantifi tifiers ers [C [Creaney reaney 1996] 1996]

• Integration of

Integration of constraint-based an constraint-based and incremental approaches [Horacek d incremental approaches [Horacek 1996] 96] 1996] 96]

• Incorporation of

Incorporation of linguistic constraints to linguistic constraints to ensure expressibility [Horacek ensure expressibility [Horacek 1997] 97]

• Simultaneous semantic and syntactic

Simultaneous semantic and syntactic construction [Stone and construction [Stone and Webber Webber 1998] 98]

• Incorporation of

Incorporation of a treatment of a treatment of salience [Krahmer and Theune 2002] salience [Krahmer and Theune 2002]

• Incorporation of

Incorporation of a treatment of a treatment of salience [Krahmer and Theune 2002] salience [Krahmer and Theune 2002]

• Extension to

Extension to sets [Gatt 2007] sets [Gatt 2007]

Athens 2008-05-21 29

Consolidation and Dissent: Consolidation and Dissent: Unifying Frameworks Unifying Frameworks

• Reconceptualisation as

Reconceptualisation as subgraph subgraph construction [Krahmer et al construction [Krahmer et al 2001 2001 2002] 2002] 2001 2001, 2002] 2002]

• Reconceptualisation as

Reconceptualisation as parameterised search [Bohnet and Dale parameterised search [Bohnet and Dale 2005] 2005] 2005] 2005]

Athens 2008-05-21 30

Current Preoccu Current Preoccupations in The Field: ations in The Field: p Empiricism and Evaluation Empiricism and Evaluation

• How do our algorith

How do our algorithms compare with what people do? ms compare with what people do?

• How do our algorith

How do our algorithms compare against each other? ms compare against each other?

• Not covered here:

Not covered here: Anja Anja Belz’s Belz’s work on Shared Task work on Shared Task Evaluation Campaigns Evaluation Campaigns (see (see http://www.itri.brighton.ac.uk/research/reg08/) http://www.itri.brighton.ac.uk/research/reg08/)

Athens 2008-05-21 31

What Do People Do? What Do People Do?

• The HCRC Map Task Corpus [Varges

The HCRC Map Task Corpus [Varges 2005] 2005]

• The Macquarie Drawers Corpus [Viethen and Dale 2006]

The Macquarie Drawers Corpus [Viethen and Dale 2006]

• The TUNA Corpus

The TUNA Corpus [van Deemter [van Deemter et al 2006] t al 2006]

• The Macquarie Blocks Corpus [Viethen and Dale 2008]

The Macquarie Blocks Corpus [Viethen and Dale 2008]

Athens 2008-05-21 32

Ex Experiment #1: eriment #1: p The Macquarie Drawers Corpus The Macquarie Drawers Corpus

The Drawers Domain [Viethen + The Drawers Domain [Viethen + Dale 2006]: Dale 2006]: id id f f 4 4 4 fili 4 fili bi bi t t d

• a gr

grid id of 4 f 4 × 4 fili 4 filing ng ca cabi bine net d t drawers rawers

• each has a number

each has a number in the range in the range 1–16 16 1 16 16

• four

four drawers each are blue, yellow, drawers each are blue, yellow, pink and orange pink and orange Task: Task:

• Given

Given the number of a drawer, describe it to an onlooker without the number of a drawer, describe it to an onlooker without mentioning any of mentioning any of the numbers the numbers mentioning any of mentioning any of the numbers the numbers

• 20 participants

20 participants → 140 descriptions (between 3 140 descriptions (between 3 and and 12 per 12 per drawer) drawer)

Athens 2008-05-21 33

Some Human-generated Descriptions Some Human-generated Descriptions

• D3: the top drawer second from the

D3: the top drawer second from the right right right right

• D9: the orange drawer on the left

D9: the orange drawer on the left

• D12: the orange drawer between two

D12: the orange drawer between two pink ones pink ones D D h b b l l f d d

• D16:

16: the b bottom

• ttom l

left d drawer rawer

Athens 2008-05-21 34

Characteristics of the Data Set Characteristics of the Data Set

• Peopl

People don’t always produce minimal descriptions don’t always produce minimal descriptions: – Minimal Descriptions: 75.4% Minimal Descriptions: 75.4% (89) (89) – Redundant Descriptions: 24.6% Redundant Descriptions: 24.6% (29) (29)

• Peopl

People rarely rarely use relational descriptions: use relational descriptions: – One- One-plac lace P Pred edicates O icates Only: 87.3 : 87.3% (103 103) p y p y % ( ) – Relational Descriptions: 12.7% Relational Descriptions: 12.7% (15) (15)

Athens 2008-05-21 35

Redundant Descriptions Redundant Descriptions

• D6: the yellow drawer in the third

D6: the yellow drawer in the third col col mn from the left second from n from the left second from col column from the left second from mn from the left second from the top the top

• D1: the blue drawer in the top left

D1: the blue drawer in the top left

• D1: the blue drawer in the top left

D1: the blue drawer in the top left corner corner

• D14: the orange drawer below the

D14: the orange drawer below the

• D14: the orange drawer below the

D14: the orange drawer below the two yellow drawers two yellow drawers

Athens 2008-05-21 36

How Do Our Algorithms Fare? How Do Our Algorithms Fare?

Minimal Minimal Redundant dundant Re Relational Over Overall all Desc Descri ription type type Al Al ith 100% 100% Minimal Minimal 31.0% 31.0% Redundant dundant

• 79.6%

79.6% Gr Greed eedy [Dale y [Dale Re Relational Over Overall all Al Algo gorith ithm 100% 100% 82.8% 82.8%

• 95.1%

95.1% Incr Incremental [Dale emental [Dale + R + Reiter iter 1995 1995] 1989] 1989] 0% 0% 0% 0% 0% 0% 0% 0% Relational [Dale + lational [Dale + Haddo Haddock 1991] 1991] ]

Athens 2008-05-21 37

The Problem with Relations The Problem with Relations

• The Dale and Haddock algorithm

The Dale and Haddock algorithm prefers relations over other prefers relations over other potential elements to incl potential elements to incl de de potential elements to incl potential elements to include de: – the drawer above the drawer above the drawer above the the drawer above the drawer above the drawer above the pink drawer pink drawer pink drawer pink drawer

Athens 2008-05-21 38

Ex Experiment #2: eriment #2: p The Macquarie Blocks Corpus The Macquarie Blocks Corpus

• Question:

Question: Do people use relations only when they are Do people use relations only when they are absol absol tel el necessar necessar ? absol absolutel tely necessar necessary?

• Materials: 20

Materials: 20 different simple different simple blocksworld blocksworld scenes containing cenes containing three objects three objects split into two trials; each subject sees 10 split into two trials; each subject sees 10 scenes scenes three objects three objects, split into two trials; each subject sees 10 split into two trials; each subject sees 10 scenes scenes

• Task: subj

Task: subject has to provide a ect has to provide a distin distinguis guishin hing descripti description in n in each scene for one of the object each scene for one of the objects; scenes constructed so that s; scenes constructed so that each scene for one of the object each scene for one of the objects; scenes constructed so that s; scenes constructed so that relations are relations are never necessary never necessary

• Subjects: 74

Subjects: 74 participants recruited via the Internet participants recruited via the Internet

• Subjects: 74

Subjects: 74 participants recruited via the Internet participants recruited via the Internet

Athens 2008-05-21 39

The Macquarie Blocks Corpus The Macquarie Blocks Corpus

Athens 2008-05-21 40

The Data The Data

• 740 descriptions

740 descriptions

• Data for 11

Data for 11 sub subjects removed: ects removed: – 1on participant’s request 1on participant’s request – 1because subject was colour blind 1because subject was colour blind – 9 9 because of a because of app pparent misunderstandin arent misunderstanding of the task

• f the task

pp pp g

• Final set =

Final set = 630 630 descriptions descriptions

Athens 2008-05-21 41

Some Results Some Results

• Over a

Over a third (231 or 36.6%) of third (231 or 36.6%) of the descriptions use spatial the descriptions use spatial relations relations relations relations

• 40

40 (63.5%) of the 63 (63.5%) of the 63 participants used relations participants used relations

• 23

23 (36.5%) of the participants never used relations (36.5%) of the participants never used relations

• 11 (over

11 (over 25%) of 25%) of the the relation-usin relation-using participants did so in all g participants did so in all f i f i i h d li d 10 10 re referr erring ng express expressions

• ns they

ey d deli livere vered

Athens 2008-05-21 42

Variation Across Duration of Trial Variation Across Duration of Trial

Athens 2008-05-21 43

Interim Conclusions Interim Conclusions

• Spatial relations are

Spatial relations are used even when unnecessary used even when unnecessary

• There is a

There is a training effect: people become more confident in not training effect: people become more confident in not using relations using relations

• Landmark salience encourages use of relations

Landmark salience encourages use of relations

Athens 2008-05-21 44

Consequences for Algorithm Development Consequences for Algorithm Development

• Need to incorporate scope for in

Need to incorporate scope for individual variation: perhaps a dividual variation: perhaps a ‘risk ‘risk ’ ers ers s ‘ca ‘ca tio io s’ parameter? [ s’ parameter? [Carletta Carletta 1992] 1992] ‘risk ‘risky’ ’ vers ersus ‘ca s ‘cautio tious’ parameter? [ s’ parameter? [Carletta Carletta 1992] 1992]

• Need finer-grained account of ch

Need finer-grained account of characteristics of properties in aracteristics of properties in the domain the domain: the domain the domain: – the ease with which a the ease with which a potential landmark can be potential landmark can be distinguished distinguished and its visual salience and its visual salience distinguished stinguished, and its visual salience and its visual salience – the type of spatial relation between the target and a the type of spatial relation between the target and a potential landmark potential landmark potential landmark potential landmark – the ease with which the target the ease with which the target can be described without the can be described without the use of spatial relations use of spatial relations use of spatial relations use of spatial relations

Athens 2008-05-21 45

Outline Outline

• The Context: Natural Language Generation

The Context: Natural Language Generation

• The Story So Far:

The Story So Far: Algorithm Development to Empiricism Algorithm Development to Empiricism Challenges for the Future Challenges for the Future

Athens 2008-05-21 46

So Where Are We At Now? So Where Are We At Now?

• A

A number of base algorithms within the standard framework number of base algorithms within the standard framework

• Some tentative explorations into

Some tentative explorations into other ways of think

• ther ways of thinking about

ng about the probl the problem; extensi em; extensions ns to accommodate sets, negation, to accommodate sets, negation, disjun disjunction ction bridgin bridging reference reference salience salience poin pointing linguis linguisti tic c disjun disjunction ction, bridging reference bridging reference, salience salience, poin pointing, linguistic linguistic constraints, quantifiers constraints, quantifiers ... ... lots of pieces that haven’t yet lots of pieces that haven’t yet been been glued to lued together ether g g g g

• An evolving understanding of

An evolving understanding of the role of evaluation and the role of evaluation and em empirical data irical data gatherin athering p g p g g

Athens 2008-05-21 47

Challenges for the Future Challenges for the Future

1. 1. Consolidation Consolidation 2. 2. Use in Applications Use in Applications 3. 3. Broadening the Story: Broadening the Story: Other Uses of Reference Other Uses of Reference

Athens 2008-05-21 48

Challen Challenge #1: e #1: g Consolidation Consolidation

• We have many piecemeal algorith

We have many piecemeal algorithms for different aspects of ms for different aspects of referring e referring e pressi ression generation

• n generation

referring e referring expressi pression generation

• n generation
• Nobody so far

Nobody so far has glued them altogether has glued them altogether

Athens 2008-05-21 49

A Skeletal Algorithm A Skeletal Algorithm

Given an Given an intended referent x: intended referent x: begin begin if x if x is in focus is in focus then use a then use a pronoun pronoun elseif x elseif x has been mentioned alread has been mentioned already then build a then build a defin definite noun phrase te noun phrase else build an initial indefinite reference else build an initial indefinite reference else build an initial indefinite reference else build an initial indefinite reference end end

Athens 2008-05-21 50

But What About Pronouns? But What About Pronouns?

Given an Given an intended referent x: intended referent x: begin begin if x if x is in focus is in focus then use a then use a pronoun pronoun elseif x elseif x has been mentioned alread has been mentioned already then build a then build a defin definite noun phrase te noun phrase else build an initial indefinite reference else build an initial indefinite reference else build an initial indefinite reference else build an initial indefinite reference end end

Athens 2008-05-21 51

And What About Initial Reference? And What About Initial Reference?

Given an Given an intended referent x: intended referent x: begin begin if x if x is in focus is in focus then use a then use a pronoun pronoun elseif x elseif x has been mentioned alread has been mentioned already then build a then build a defin definite noun phrase te noun phrase else build an initial indefinite reference else build an initial indefinite reference else build an initial indefinite reference else build an initial indefinite reference end end

Athens 2008-05-21 52

Consolidation Challenges Consolidation Challenges

• Covering the ‘Identification Space’

Covering the ‘Identification Space’ – Pronomi Pronominal Reference al Reference – Initi Initial Reference l Reference

• Scaling up syntactic and semantic coverage

Scaling up syntactic and semantic coverage

• Inte

Integration of ex ration of experimental findin erimental findings g p g p g

Athens 2008-05-21 53

Challen Challenge #2: e #2: g Use in Applications Use in Applications

• Referring Expression Generati

Referring Expression Generation is still a

• n is still a theory-bound

theory-bound enterprise enterprise enterprise enterprise

• But there is real scope fo

But there is real scope for practical appli r practical applications ations: – Entity description in tailored instructions Entity description in tailored instructions – Landmarks and directi Landmarks and directions in route descripti ns in route descriptions ns – Entity references in automaticall Entity references in automatically-generated summaries y-generated summaries

Athens 2008-05-21 54

Instructions Instructions

1.

• 1. Remove

Remove the modem card the modem card from from its packaging its packaging. . 2.

• 2. Align

Align the card the card to to the the matching matching ISA or PCI slot ISA or PCI slot. 3.

• 3. Remove

Remove the slot cover the slot cover to allow to allow the modem ports the modem ports to be to be accessi accessible from le from the outside of the computer the outside of the computer. . 4.

• 4. Carefull

Carefully insert y insert the card the card into into the slot the slot and push firmly into and push firmly into l S l S h d h d i h i h i i h l h l b place.

• ace. S

Secure ecure the car card with h a screw screw in the meta metal l ta tab. 5.

• 5. Replace

Replace the cover the cover, plug in , plug in the power cord the power cord, and turn on , and turn on the the t compu computer er.

Athens 2008-05-21 55

Route Descriptions Route Descriptions

• A

A couple of kilometers couple of kilometers after after the M2 the M2 turn off turn off is is Herring Road Herring Road, at , at the top of a the top of a hill hill the top of a the top of a hill hill.

• You

You'll pass through 'll pass through a a built up suburb with lots of shops called built up suburb with lots of shops called St Ives St Ives; then ; then you you'll ll go under go under the Pacific Highway the Pacific Highway at which point at which point St Ives St Ives; then ; then you you ll ll go under go under the Pacific Highway the Pacific Highway, at which point at which point the road the road changes changes its name its name to to Ryde Road Ryde Road. .

• After going downhi

After going downhill ll and up again and up again you you'll start going down hill ll start going down hill

• After going downhi

After going downhill ll and up again and up again, you you ll start going down hill ll start going down hill into into a a valley through which the Lane Cover River runs valley through which the Lane Cover River runs; ; the the road road's called 's called Lane Cove Road Lane Cove Road at at this this point

• int.

. p

• Turn left at

Turn left at the first set of lights the first set of lights, which will take you , which will take you into into the the universi university.

Athens 2008-05-21 56

y

Entity Reference in News Stories Entity Reference in News Stories

Morgan Stockbrokin Morgan Stockbroking Ltd Ltd said it was recommending said it was recommending newly-li newly-listed sted eq eq ipment hire gro ipment hire gro p Coates Hire Ltd Coates Hire Ltd as a b as a b reflecting reflecting good good eq equipment hire gro ipment hire group Coates Hire Ltd Coates Hire Ltd as a b as a buy, uy, reflecting good reflecting good growth prospects. growth prospects. “The company The company is attractively priced based on is attractively priced based on 1997 fundamentals 1997 fundamentals,” analyst analyst John Clifford John Clifford said in a said in a report report. . Coates Coates 1997 fundamentals 1997 fundamentals, , analyst analyst John Clifford John Clifford said in a said in a report report. . Coates Coates listed this month after the sale of listed this month after the sale of Australi Australian National Industries an National Industries Ltd Ltd's 100 's 100 percent holding had a percent holding had a balance sheet “comfortably balance sheet “comfortably d” d d i f i i h geare geared” at at 46 46 percent percent an and i d interest nterest cover cover f forecast

• recast to

to rise se to to eight times in the year times in the year ended June 30, ended June 30, 1997 1997 from 6.7 from 6.7 times in times in 1995/96 1995/96 Clifford Clifford said said 1995/96 1995/96, Clifford Clifford said said.

Athens 2008-05-21 57

Challen Challenge #3: e #3: g The Discourse Functions of Reference The Discourse Functions of Reference

• There is more to reference than attribute selection for

There is more to reference than attribute selection for discrimin discrimination tion discrimin discrimination tion

• The role

The role that a that a noun phrase plays in a noun phrase plays in a discourse impacts on the discourse impacts on the attribu attribute selecti e selection process n process attribu attribute selecti e selection process n process – Maintaining focus Maintaining focus S i S i h h f b f – Sett etting ng the stage stage f for

• r su

subsequent sequent re reference erence – Contrasting one entity with another Contrasting one entity with another – Highlighting specific properties Highlighting specific properties

Athens 2008-05-21 58

Addin Adding Discourse Pur Discourse Purpose to Referrin

• se to Referring

g p g p g Expression Generation Expression Generation

• We already have theory of discourse structure that has been

We already have theory of discourse structure that has been ell ell e plored in NLG lored in NLG Rhetorical Str Rhetorical Str ct re Theor e Theor well ell-explored in NLG plored in NLG: Rhetorical Rhetorical Str Struct cture Theor re Theory

• Consider each element of a

Consider each element of a nomin nominal expression l expression as as being being licenced licenced by some rhetorical functi by some rhetorical function or purpos

• n or purpose

licenced licenced by some rhetorical functi by some rhetorical function or purpos

• n or purpose

– Distingu Distinguish ishing from potenti ng from potential distractors al distractors is just one functi s just one function

• n

Th Th h h ll l h i h i f h i l f f i

• Th

The chall llenge: enge: to to cata catalog

• g the i

inventory nventory of f rhetor etorica cal f l funct unctions

• ns

that surface in nominal expressions that surface in nominal expressions Lik Lik l t t b b d d i d d ifi ifi

• Lik

Likely t to b be d doma

• main-

n- an and d genre-spec genre-specifi ific

Athens 2008-05-21 59

Conclusions Conclusions

• Referring Expression Generation

Referring Expression Generation is the most well-defined and is the most well-defined and de de eloped s eloped s bfield of NLG field of NLG b t b t e’ e’ e onl e onl j j st got started t got started de developed s eloped subfield of NLG bfield of NLG ... ... b but t we’ e’ve onl e only j just got started st got started

• There are

There are real real near-term practica near-term practical applications that can l applications that can benefit: benefit: – Instruction Manuals and Technical Support Instruction Manuals and Technical Support – Route Description Route Description – Entity Reference in Document Summarisation Entity Reference in Document Summarisation

• Natural lan

Natural langua uage e generation remains the best theoretical eneration remains the best theoretical g g g g g perspective for understandi perspective for understanding how language really works ng how language really works

Athens 2008-05-21 60