N EU G EN Text Generation from Meaning Representations Yannis - - PowerPoint PPT Presentation

NEUGEN

• Text Generation from Meaning Representations

Yannis Konstas

• Joint work with

Mark Yatskar, Luke Zettlemoyer and Yejin Choi (UW) ~ Yonatan Bisk and Daniel Marcu (ISI)

Motivation

Motivation

Machine-generated Representation

Motivation

Machine-generated Representation

Motivation

Machine-generated Representation

Motivation

Machine-generated Representation

source block: hk target block: ms pos RP: W scale: small

Motivation

Machine-generated Representation

source block: hk target block: ms pos RP: W scale: small

Place the heineken block west of the mercedes block.

Motivation

Machine-generated Representation

source block: hk target block: ups pos RP: W scale: big

Motivation

Machine-generated Representation

source block: hk target block: ups pos RP: W scale: big

Place the heineken block in the first

• pen space to the left of ups block.

Motivation

Motivation

Machine-generated Representation

Motivation

Machine-generated Representation Sentence Compression

Motivation

Machine-generated Representation Sentence Compression Sentence Fusion

Motivation

Machine-generated Representation Sentence Compression Sentence Fusion Paraphrasing

NLG Pipeline

Content Selection Document Planning

Surface Realisation Sentence Planning

Text

Communicative Goal Content Planning

A A A A m

Input

Reordering/Linearization Splitting/Aggregation Lexicalization

Framework

NLG Pipeline

Content Selection Document Planning

Surface Realisation Sentence Planning

Text

Communicative Goal Content Planning

A A A A m

Input

Reordering/Linearization Splitting/Aggregation Lexicalization

Framework

NLG Pipeline

Content Selection Document Planning

Surface Realisation Sentence Planning

Text

Communicative Goal Content Planning

A A A A m

Input

Reordering/Linearization Splitting/Aggregation Lexicalization

• Neural Encoder
• RNN Decoder
• Graph Alignment

Framework

Joint Generator

Temperature Time Min Mean Max 06:00-21:00 9 15 21 Wind Speed Time Min Mean Max 06:00-21:00 15 20 30 Cloud Sky Cover Time Percent (%) 06:00-09:00 25-50 09:00-12:00 50-75 Wind Direction Time Mode 06:00-21:00 S

...

...

...

...

...

? ? ?

Cloudy, with temperatures between 10 and 20 degrees. South wind around 20 mph.

Training S → R(start) R(ri.t)→FS(rj, start)R(rj.t) R(ri.t)→FS(rj, start) FS(r, r.fi)→F(r, r.fj)FS(r, r.fj) FS(r, r.fi)→F(r, r.fj) F(r, r.f)→W(r, r.f)F(r, r.f) F(r, r.f)→W(r, r.f) W(r, r.f)→α W(r, r.f)→g(f.v) FS0,1(temp1,start) FS0,2(temp1,start) F0,1(temp1,min) F0,1(temp1,max) F0,2(temp1,min) F0,2(temp1,max) FS1,2(temp1,start) (2) Hypergraph Representation

FS0,5(skyCover1.t,start)     mostly cloudy ? the morning mostly cloudy ? after 11am mostly cloudy ? then becoming · · ·     F0,2(skyCover1.t,%)     mostly cloudy mostly clouds cloudy , · · ·     W4,5(skyCover1.t,time)     morning 11am after · · ·     W0,1(skyCover1.t,%)     mostly cloudy sunny · · ·     W1,2(skyCover1.t,%)     mostly cloudy sunny · · ·    

(3) k-best decoding via integration Testing (1) PCFG Grammar

Konstas and Lapata, 2012, 2013

Joint Generator

Temperature Time Min Mean Max 06:00-21:00 9 15 21 Wind Speed Time Min Mean Max 06:00-21:00 15 20 30 Cloud Sky Cover Time Percent (%) 06:00-09:00 25-50 09:00-12:00 50-75 Wind Direction Time Mode 06:00-21:00 S

...

...

...

...

...

? ? ?

Cloudy, with temperatures between 10 and 20 degrees. South wind around 20 mph.

Training S → R(start) R(ri.t)→FS(rj, start)R(rj.t) R(ri.t)→FS(rj, start) FS(r, r.fi)→F(r, r.fj)FS(r, r.fj) FS(r, r.fi)→F(r, r.fj) F(r, r.f)→W(r, r.f)F(r, r.f) F(r, r.f)→W(r, r.f) W(r, r.f)→α W(r, r.f)→g(f.v) FS0,1(temp1,start) FS0,2(temp1,start) F0,1(temp1,min) F0,1(temp1,max) F0,2(temp1,min) F0,2(temp1,max) FS1,2(temp1,start) (2) Hypergraph Representation

FS0,5(skyCover1.t,start)     mostly cloudy ? the morning mostly cloudy ? after 11am mostly cloudy ? then becoming · · ·     F0,2(skyCover1.t,%)     mostly cloudy mostly clouds cloudy , · · ·     W4,5(skyCover1.t,time)     morning 11am after · · ·     W0,1(skyCover1.t,%)     mostly cloudy sunny · · ·     W1,2(skyCover1.t,%)     mostly cloudy sunny · · ·    

(3) k-best decoding via integration Testing (1) PCFG Grammar

Konstas and Lapata, 2012, 2013

Joint Generator

Temperature Time Min Mean Max 06:00-21:00 9 15 21 Wind Speed Time Min Mean Max 06:00-21:00 15 20 30 Cloud Sky Cover Time Percent (%) 06:00-09:00 25-50 09:00-12:00 50-75 Wind Direction Time Mode 06:00-21:00 S

...

...

...

...

...

? ? ?

Cloudy, with temperatures between 10 and 20 degrees. South wind around 20 mph.

Training S → R(start) R(ri.t)→FS(rj, start)R(rj.t) R(ri.t)→FS(rj, start) FS(r, r.fi)→F(r, r.fj)FS(r, r.fj) FS(r, r.fi)→F(r, r.fj) F(r, r.f)→W(r, r.f)F(r, r.f) F(r, r.f)→W(r, r.f) W(r, r.f)→α W(r, r.f)→g(f.v) FS0,1(temp1,start) FS0,2(temp1,start) F0,1(temp1,min) F0,1(temp1,max) F0,2(temp1,min) F0,2(temp1,max) FS1,2(temp1,start) (2) Hypergraph Representation

FS0,5(skyCover1.t,start)     mostly cloudy ? the morning mostly cloudy ? after 11am mostly cloudy ? then becoming · · ·     F0,2(skyCover1.t,%)     mostly cloudy mostly clouds cloudy , · · ·     W4,5(skyCover1.t,time)     morning 11am after · · ·     W0,1(skyCover1.t,%)     mostly cloudy sunny · · ·     W1,2(skyCover1.t,%)     mostly cloudy sunny · · ·    

(3) k-best decoding via integration Testing (1) PCFG Grammar

Konstas and Lapata, 2012, 2013

More Complex Structure

More Complex Structure

know-01 I planet lazy

ARG0 ARG1

inhabit-01

man

ARG1-of ARG0 mod

Input (Graph or Tree)

More Complex Structure

know-01 I planet lazy

ARG0 ARG1

inhabit-01

man

ARG1-of ARG0 mod

Input (Graph or Tree)

AMR DAG

More Complex Structure

know-01 I planet lazy

ARG0 ARG1

inhabit-01

man

ARG1-of ARG0 mod

Input (Graph or Tree)

AMR DAG λ-calculus Expression Tree

More Complex Structure

know-01 I planet lazy

ARG0 ARG1

inhabit-01

man

ARG1-of ARG0 mod

Input (Graph or Tree)

AMR DAG λ-calculus Expression Tree ECIs Tree

More Complex Structure

know-01 I planet lazy

ARG0 ARG1

inhabit-01

man

ARG1-of ARG0 mod

Input (Graph or Tree)

I know a planet that is inhabited by a lazy man.

Output (Text)

AMR DAG λ-calculus Expression Tree ECIs Tree

More Complex Structure

I knew a planet that was inhabited by a lazy

• man. (lpp_1943.249)

I have known a planet that was inhabited by a lazy man.

know-01 I planet lazy

ARG0 ARG1

inhabit-01

man

ARG1-of ARG0 mod

Input (Graph or Tree)

I know a planet that is inhabited by a lazy man.

Output (Text)

AMR DAG λ-calculus Expression Tree ECIs Tree

More Complex Structure

I knew a planet that was inhabited by a lazy

• man. (lpp_1943.249)

I have known a planet that was inhabited by a lazy man. I know about a planet. It is inhabited by a lazy man.

know-01 I planet lazy

ARG0 ARG1

inhabit-01

man

ARG1-of ARG0 mod

Input (Graph or Tree)

I know a planet that is inhabited by a lazy man.

Output (Text)

AMR DAG λ-calculus Expression Tree ECIs Tree

NEUGEN Framework

ARG1-of

NEUGEN Framework

ARG1-of

Encoder Decoder

G w

NEUGEN Framework

Encoder

• Bag of Words: Concepts and roles
• Attention mechanism

ARG1-of

Encoder Decoder

G w

NEUGEN Framework

Encoder

• Bag of Words: Concepts and roles
• Attention mechanism

Decoder

• Left-to-right word LSTM
• Greedy beam search

ARG1-of

Encoder Decoder

G w

NEUGEN Framework

Encoder

• Bag of Words: Concepts and roles
• Attention mechanism

Decoder

• Left-to-right word LSTM
• Greedy beam search

ARG1-of

Encoder Decoder

G w

Reranker

NEUGEN Framework

Encoder

• Bag of Words: Concepts and roles
• Attention mechanism

Decoder

• Left-to-right word LSTM
• Greedy beam search

Reranker

ARG1-of

Encoder Decoder

G w

Reranker

NEUGEN Framework

Encoder

• Bag of Words: Concepts and roles
• Attention mechanism

Decoder

• Left-to-right word LSTM
• Greedy beam search

Reranker

• CCG parser -> semantics

ARG1-of

Encoder Decoder

G w

Reranker

NEUGEN Framework

Encoder

• Bag of Words: Concepts and roles
• Attention mechanism

Decoder

• Left-to-right word LSTM
• Greedy beam search

Reranker

• CCG parser -> semantics
• Partial propositions graph Eincr

ARG1-of

Encoder Decoder

G w

Reranker

NEUGEN Framework

Encoder

• Bag of Words: Concepts and roles
• Attention mechanism

Decoder

• Left-to-right word LSTM
• Greedy beam search

Reranker

• CCG parser -> semantics
• Partial propositions graph Eincr

ARG1-of

Aligner Encoder Decoder

G w Eincr G

Aligner

• Feed-forward NN with attention

Reranker

NEUGEN Framework

Encoder

• Bag of Words: Concepts and roles
• Attention mechanism

Decoder

• Left-to-right word LSTM
• Greedy beam search

Reranker

• CCG parser -> semantics
• Partial propositions graph Eincr

ARG1-of

Aligner Encoder Decoder

G w Eincr G salign

Aligner

• Feed-forward NN with attention
• salign = fit(G, Eincr)

Reranker

Alignment

Alignment

w: A man inhabited a planet

Alignment

w: A man inhabited a planet

man inhabit

agent

planet

patient

Eincr

Alignment

know-01 I planet lazy

ARG0 ARG1

inhabit-01

man

ARG1-of ARG0 mod

G

w: A man inhabited a planet

man inhabit

agent

planet

patient

Eincr

Alignment

know-01 I planet lazy

ARG0 ARG1

inhabit-01

man

ARG1-of ARG0 mod

G

w: A man inhabited a planet

man inhabit

agent

planet

patient

Eincr

fit(G, Eincr)

Alignment

know-01 I planet lazy

ARG0 ARG1

inhabit-01

man

ARG1-of ARG0 mod

G

w: A man inhabited a planet

man inhabit

agent

planet

patient

Eincr

fit(G, Eincr) <inhabit, man, agent> <inhabit, planet, patient> <know-01, I, A0> <know-01, planet, A1> <planet, inhabit-01, A1-of> <inhabit-01, man, A0> <man, lazy, mod>

Alignment

<know, I, A0>

• <know, planet, A1>
• <planet, inhabit, A1-of>
• <inhabit, man, A0>
• <man, lazy, mod>

G

[0..1]

fitness score

<inhabit, man, agent>

• <inhabit, planet, patient>

Eincr E1: .

attention weights

+

max

Alignment

<inhabit, planet, patient>

• <inhabit, man, agent>

<know, I, A0>

• <know, planet, A1>
• <planet, inhabit, A1-of>
• <inhabit, man, A0>
• <man, lazy, mod>

Eincr G E2:

fit(G, Eincr) = 1 n

n

X

i=0

fit(G, Ei)

[0..1]

fitness score

attention weights

+

max

.

Decoding

encoder(G)

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

G

Decoding

encoder(G)

∅

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

G

Decoding

encoder(G)

∅

I The A …

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

G

Decoding

encoder(G)

∅

I The A …

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

…

w11: I

The

w12:

Man

w13:

Tree

w14:

G

Decoding

encoder(G)

∅

I The A …

re-score:

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

…

w11: I

The

w12:

Man

w13:

Tree

w14:

G

sLSTM

Decoding

encoder(G)

∅

I The A … know knew planet …

re-score:

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

…

w11: I

The

w12:

Man

w13:

Tree

w14:

G

sLSTM

Decoding

encoder(G)

∅

I The A … know knew planet …

re-score:

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

…

w11: I

The

w12:

Man

w13:

Tree

w14:

… I know

agent

w21:

I knew

agent

w22:

The planet

w23:

Man planet

w24:

G

sLSTM + salign

Decoding

encoder(G)

∅

I The A … know knew planet …

re-score:

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

…

w11: I

The

w12:

Man

w13:

Tree

w14:

… I know

agent

w21:

I knew

agent

w22:

The planet

w23:

Man planet

w24:

G

sLSTM + salign

Decoding

encoder(G)

∅

I The A … know knew planet … a planet man …

re-score:

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

…

…

w11: I

The

w12:

Man

w13:

Tree

w14:

… I know

agent

w21:

I knew

agent

w22:

The planet

w23:

Man planet

w24:

G

sLSTM + salign

Decoding

encoder(G)

∅

I The A … know knew planet … a planet man …

re-score:

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

…

…

w11: I

The

w12:

Man

w13:

Tree

w14:

… I know

agent

w21:

I knew

agent

w22:

The planet

w23:

Man planet

w24: w41:

I know

agent

a planet

patient

w42:

I knew

agent

planets that

patient

w43: The planet I

knew

agent patient

w44: Man know

agent

a planet

patient

that …

G

sLSTM + salign

Decoding

encoder(G)

∅

I The A … know knew planet … a planet man …

re-score:

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

…

…

w11: I

The

w12:

Man

w13:

Tree

w14:

… I know

agent

w21:

I knew

agent

w22:

The planet

w23:

Man planet

w24: w41:

I know

agent

a planet

patient

w42:

I knew

agent

planets that

patient

w43: The planet I

knew

agent patient

w44: Man know

agent

a planet

patient

that …

G

sLSTM + salign

?

Decoding

encoder(G)

∅

I The A … know knew planet … a planet man …

re-score:

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

…

…

w11: I

The

w12:

Man

w13:

Tree

w14:

… I know

agent

w21:

I knew

agent

w22:

The planet

w23:

Man planet

w24: w41:

I know

agent

a planet

patient

w42:

I knew

agent

planets that

patient

w43: The planet I

knew

agent patient

w44: Man know

agent

a planet

patient

that …

G

sLSTM + salign

?

Decoding

encoder(G)

∅

I The A … know knew planet … a planet man …

re-score:

know I planet lazy

ARG0 ARG1

inhabit man

ARG1-of ARG0 mod

…

…

w11: I

The

w12:

Man

w13:

Tree

w14:

… I know

agent

w21:

I knew

agent

w22:

The planet

w23:

Man planet

w24: w41:

I know

agent

a planet

patient

w42:

I knew

agent

planets that

patient

w43: The planet I

knew

agent patient

w44: Man know

agent

a planet

patient

that …

inhabit inhabited was …

G

sLSTM + salign

Preliminary Results

Preliminary Results

Experimental Setup

Preliminary Results

Experimental Setup

• Baseline RNN w/o alignment
• NeuGen

Preliminary Results

Experimental Setup

• Baseline RNN w/o alignment
• NeuGen

Dataset

Preliminary Results

Experimental Setup

• Baseline RNN w/o alignment
• NeuGen

Dataset

• AMR LDC2015E86
• Train: 15k, Dev: 1.2k, Test: 1.2k

Preliminary Results

Experimental Setup

• Baseline RNN w/o alignment
• NeuGen

Dataset

• AMR LDC2015E86
• Train: 15k, Dev: 1.2k, Test: 1.2k

5.5 11 16.5 22 BLEU-4 METEOR

Baseline NeuGen

Example Output

Example Output

person Gallardo found-01

domain ARG0-of

Tijuana Drug Cartel

ARG1

G

flee-01

ARG0-of

Example Output

Gold: Gallardo is the fugitive founder of Tijuana Drug Cartel.

• Baseline: Gallardo is the founder of Tijuana Drug Cartel.
• NeuGen: Tijunana Drug Cartel is the founder of Gallardo.

person Gallardo found-01

domain ARG0-of

Tijuana Drug Cartel

ARG1

G

flee-01

ARG0-of

Example Output

Gold: Gallardo is the fugitive founder of Tijuana Drug Cartel.

• Baseline: Gallardo is the founder of Tijuana Drug Cartel.
• NeuGen: Tijunana Drug Cartel is the founder of Gallardo.

person Gallardo found-01

domain ARG0-of

Tijuana Drug Cartel

ARG1

G

flee-01

ARG0-of Gallardo

• Tijuana Drug Cartel
• State
• Country
• …

Example Output

Gold: Gallardo is the fugitive founder of Tijuana Drug Cartel.

• Baseline: Gallardo is the founder of Tijuana Drug Cartel.
• NeuGen: Tijunana Drug Cartel is the founder of Gallardo.

person Gallardo found-01

domain ARG0-of

Tijuana Drug Cartel

ARG1

G

flee-01

ARG0-of Gallardo

• Tijuana Drug Cartel
• State
• Country
• …

Gallardo is the founder of

• Gallardo ’s Tijuana Drug Cartel is the
• …

patient agent

• 19.82
• 17.79

Tijuana Drug Cartel is the founder of

• Tijuana Drug Cartel was the founder of
• …

patient patient

• 18.67
• 20.24

Example Output

Gold: Gallardo is the fugitive founder of Tijuana Drug Cartel.

• Baseline: Gallardo is the founder of Tijuana Drug Cartel.
• NeuGen: Tijunana Drug Cartel is the founder of Gallardo.

person Gallardo found-01

domain ARG0-of

Tijuana Drug Cartel

ARG1

G

flee-01

ARG0-of Gallardo

• Tijuana Drug Cartel
• State
• Country
• …

Gallardo is the founder of

• Gallardo ’s Tijuana Drug Cartel is the
• …

patient agent

• 19.82
• 17.79

Tijuana Drug Cartel is the founder of

• Tijuana Drug Cartel was the founder of
• …

patient patient

• 18.67
• 20.24

Gallardo is the founder of Tijuana Drug Cartel.

agent patient

• 19.59

Tijuana Drug Cartel is the founder of Gallardo.

agent patient

• 17.84

Next Steps

Next Steps

• “Machine-generated” representations
• λ-calculus (ATIS)
• Blocks World complex descriptions (from ISI)

Next Steps

• “Machine-generated” representations
• λ-calculus (ATIS)
• Blocks World complex descriptions (from ISI)

Next Steps

• “Machine-generated” representations
• λ-calculus (ATIS)
• Blocks World complex descriptions (from ISI)
• Downstream Text-to-Text Applications

THANK You