NNLG Neural Natural Language Generation Yannis Konstas Joint work - - PowerPoint PPT Presentation

▶
nnlg
SMART_READER_LITE
LIVE PREVIEW

NNLG Neural Natural Language Generation Yannis Konstas Joint work - - PowerPoint PPT Presentation

Dec 16, 2022 358 likes •1.28k views

NNLG Neural Natural Language Generation Yannis Konstas Joint work with Srinivasan Iyer, Mark Yatskar, Rik Koncel-Kedziorski, Li Zilles, Luke Zettlemoyer, Yejin Choi, Hannaneh Hajishirzi NLG Pipeline Communicative Goal Input Content Planning

slide-1
SLIDE 1

NNLG

Neural Natural Language Generation

Yannis Konstas Joint work with Srinivasan Iyer, Mark Yatskar, Rik Koncel-Kedziorski, Li Zilles, Luke Zettlemoyer, Yejin Choi, Hannaneh Hajishirzi

slide-2
SLIDE 2

NLG Pipeline

Content Selection Document Planning

Surface Realisation Sentence Planning

Text

Communicative Goal Content Planning

Reordering/Linearization Splitting/Aggregation Lexicalization

Input

slide-3
SLIDE 3

NLG Pipeline

Content Selection Document Planning

Surface Realisation Sentence Planning

Text

Communicative Goal Content Planning

Reordering/Linearization Splitting/Aggregation Lexicalization

Input

?

slide-4
SLIDE 4

NLG Pipeline

Content Selection Document Planning

Surface Realisation Sentence Planning

Text

Communicative Goal Content Planning

Reordering/Linearization Splitting/Aggregation Lexicalization

Input

?

  • Records / Fields / Values
  • Source Code
  • Predicate-Argument Structure
  • Algebra equation
  • Text / Script
  • Multiple Sources
slide-5
SLIDE 5

NLG Pipeline

Content Selection Document Planning

Surface Realisation Sentence Planning

Text

Communicative Goal Content Planning

Reordering/Linearization Splitting/Aggregation Lexicalization

Input

?

  • Records / Fields / Values
  • Source Code
  • Predicate-Argument Structure
  • Algebra equation
  • Text / Script
  • Multiple Sources
  • Single utterance
  • Single (complex) sentence
  • Multiple sentences
  • Multiple paragraphs
slide-6
SLIDE 6

NLG Pipeline

Content Selection Document Planning

Surface Realisation Sentence Planning

Text

Communicative Goal Content Planning

Reordering/Linearization Splitting/Aggregation Lexicalization

Input

?

  • Records / Fields / Values
  • Source Code
  • Predicate-Argument Structure
  • Algebra equation
  • Text / Script
  • Multiple Sources
  • Single utterance
  • Single (complex) sentence
  • Multiple sentences
  • Multiple paragraphs

?

slide-7
SLIDE 7

NLG Pipeline

Content Selection Document Planning

Surface Realisation Sentence Planning

Text

Communicative Goal Content Planning

Reordering/Linearization Splitting/Aggregation Lexicalization

Input

?

  • Records / Fields / Values
  • Source Code
  • Predicate-Argument Structure
  • Algebra equation
  • Text / Script
  • Multiple Sources
  • Single utterance
  • Single (complex) sentence
  • Multiple sentences
  • Multiple paragraphs

? ?

slide-8
SLIDE 8

NLG is everywhere

(A Global Model for Concept-to-Text Generation. Konstas and Lapata, JAIR 2013.)

slide-9
SLIDE 9

NLG is everywhere

Concept-to-Text Generation Input: Machine-generated Representation

(A Global Model for Concept-to-Text Generation. Konstas and Lapata, JAIR 2013.)

slide-10
SLIDE 10

NLG is everywhere

Concept-to-Text Generation Input: Machine-generated Representation

(A Global Model for Concept-to-Text Generation. Konstas and Lapata, JAIR 2013.)

slide-11
SLIDE 11

NLG is everywhere

Concept-to-Text Generation Input: Machine-generated Representation

(A Global Model for Concept-to-Text Generation. Konstas and Lapata, JAIR 2013.)

slide-12
SLIDE 12

NLG is everywhere

Concept-to-Text Generation Input: Machine-generated Representation

source block: hk target block: ms pos RP: W scale: small

(A Global Model for Concept-to-Text Generation. Konstas and Lapata, JAIR 2013.)

slide-13
SLIDE 13

NLG is everywhere

Concept-to-Text Generation Input: Machine-generated Representation

source block: hk target block: ms pos RP: W scale: small

Place the heineken block west of the mercedes block.

(A Global Model for Concept-to-Text Generation. Konstas and Lapata, JAIR 2013.)

slide-14
SLIDE 14

NLG is everywhere

(Summarizing Source Code using a Neural Attention Model. Iyer, Konstas, Cheung, Zettlemoyer. ACL 2016.)

slide-15
SLIDE 15

NLG is everywhere

Code-to-Text Generation Input: Source Code

(Summarizing Source Code using a Neural Attention Model. Iyer, Konstas, Cheung, Zettlemoyer. ACL 2016.)

slide-16
SLIDE 16

NLG is everywhere

Code-to-Text Generation Input: Source Code

(Summarizing Source Code using a Neural Attention Model. Iyer, Konstas, Cheung, Zettlemoyer. ACL 2016.)

CODE-NN

slide-17
SLIDE 17

NLG is everywhere

Code-to-Text Generation Input: Source Code

(Summarizing Source Code using a Neural Attention Model. Iyer, Konstas, Cheung, Zettlemoyer. ACL 2016.)

CODE-NN

public int TextWidth (string text) { TextBlock t = new TextBlock(); t.Text = text; return (int) Math.Ceiling(t.ActualWidth); }

slide-18
SLIDE 18

NLG is everywhere

Code-to-Text Generation Input: Source Code

(Summarizing Source Code using a Neural Attention Model. Iyer, Konstas, Cheung, Zettlemoyer. ACL 2016.)

CODE-NN

public int TextWidth (string text) { TextBlock t = new TextBlock(); t.Text = text; return (int) Math.Ceiling(t.ActualWidth); }

Get rendered width of string rounded up to the nearest integer.

slide-19
SLIDE 19

NLG is everywhere

(Flanigan et al, NAACL 2016, Pourdamaghani and Knight, INLG 2016, Song et al, EMNLP 2016.)

Meaning Representation Generation Input: Predicate - Argument Structure

know I planet lazy ARG0 ARG1 inhabit man ARG1-of ARG0 mod

slide-20
SLIDE 20

NLG is everywhere

(Flanigan et al, NAACL 2016, Pourdamaghani and Knight, INLG 2016, Song et al, EMNLP 2016.)

Meaning Representation Generation Input: Predicate - Argument Structure

know I planet lazy ARG0 ARG1 inhabit man ARG1-of ARG0 mod

I knew a planet that was inhabited by a lazy man.

slide-21
SLIDE 21

NLG is everywhere

(Flanigan et al, NAACL 2016, Pourdamaghani and Knight, INLG 2016, Song et al, EMNLP 2016.)

Meaning Representation Generation Input: Predicate - Argument Structure

know I planet lazy ARG0 ARG1 inhabit man ARG1-of ARG0 mod

I knew a planet that was inhabited by a lazy man. I have known a planet that was inhabited by a lazy man.

slide-22
SLIDE 22

NLG is everywhere

(Flanigan et al, NAACL 2016, Pourdamaghani and Knight, INLG 2016, Song et al, EMNLP 2016.)

Meaning Representation Generation Input: Predicate - Argument Structure

know I planet lazy ARG0 ARG1 inhabit man ARG1-of ARG0 mod

I knew a planet that was inhabited by a lazy man. I have known a planet that was inhabited by a lazy man. There is a lazy man who inhabited a planet I know about.

know I planet lazy ARG0 ARG1 inhabit man ARG0 ARG1-of mod

slide-23
SLIDE 23

NLG is everywhere

Instructional Text Generation Input: Goal Cue - Bag of concepts

slide-24
SLIDE 24

NLG is everywhere

(Globally Coherent Text Generation with Neural Checklist Models. Kiddon et al, EMNLP 2016.)

Instructional Text Generation Input: Goal Cue - Bag of concepts

Spanakopita (Greek Spinach Pie)

Ingredients

3 tbsp olive oil 1 large onion, chopped 1 bunch green onions, chopped 2 cloves garlic, minced 2 pounds spinach 1/2 cup chopped fresh parsley 2 eggs 1/2 cup ricotta cheese 1 cup feta cheese 8 sheets filo dough 1/4 cup olive oil

slide-25
SLIDE 25

NLG is everywhere

(Globally Coherent Text Generation with Neural Checklist Models. Kiddon et al, EMNLP 2016.)

Instructional Text Generation Input: Goal Cue - Bag of concepts

Spanakopita (Greek Spinach Pie)

Ingredients

3 tbsp olive oil 1 large onion, chopped 1 bunch green onions, chopped 2 cloves garlic, minced 2 pounds spinach 1/2 cup chopped fresh parsley 2 eggs 1/2 cup ricotta cheese 1 cup feta cheese 8 sheets filo dough 1/4 cup olive oil

Preheat oven to 350 degrees F (175 degrees C). Lightly oil a 9x9 inch square baking pan. Heat 3 tablespoons olive oil in a large skillet over medium heat. Saute

  • nion, green onions and garlic, until soft and lightly browned. Stir in

spinach and parsley, and continue to saute until spinach is limp, about 2 minutes. Remove from heat and set aside to cool. In a medium bowl, mix together eggs, ricotta, and feta. Stir in spinach

  • mixture. Lay 1 sheet of phyllo dough in prepared baking pan, and

brush lightly with olive oil. Lay another sheet of phyllo dough on top, brush with olive oil, and repeat process with two more sheets of phyllo. The sheets will overlap the pan. Spread spinach and cheese mixture into pan and fold overhanging dough over filling. Brush with oil, then layer remaining 4 sheets of phyllo dough, brushing each with oil. Tuck

  • verhanging dough into pan to seal filling.

Bake in preheated oven for 30 to 40 minutes, until golden brown. Cut into squares and serve while hot.

slide-26
SLIDE 26

NLG is everywhere

Storytelling Generation Input: Script - Text - N/A

slide-27
SLIDE 27

NLG is everywhere

Storytelling Generation Input: Script - Text - N/A Jim was obsessed with super heroes.

His sister told him if he tied a sheet on his back he could fly. She convinced Jim to climb the ladder to the roof and jump off. When he got up there he felt like he was superman.

slide-28
SLIDE 28

NLG is everywhere

Storytelling Generation Input: Script - Text - N/A Jim was obsessed with super heroes.

His sister told him if he tied a sheet on his back he could fly. She convinced Jim to climb the ladder to the roof and jump off. When he got up there he felt like he was superman.

He ended up having a great time!

slide-29
SLIDE 29

NLG is everywhere

Storytelling Generation Input: Script - Text - N/A Jim was obsessed with super heroes.

His sister told him if he tied a sheet on his back he could fly. She convinced Jim to climb the ladder to the roof and jump off. When he got up there he felt like he was superman.

He ended up having a great time! Jim broke his arm and his sister was grounded for a year.

slide-30
SLIDE 30

NLG is everywhere

Storytelling Generation Input: Equation + Theme

(Koncel-Kedziorski, Konstas, Zettlemoyer, Hajishirzi. A Theme-Rewriting Approach for Generating Algebra Word Problems. EMNLP 2016.)

slide-31
SLIDE 31

NLG is everywhere

Storytelling Generation Input: Equation + Theme

504 + x = 639 +

(Koncel-Kedziorski, Konstas, Zettlemoyer, Hajishirzi. A Theme-Rewriting Approach for Generating Algebra Word Problems. EMNLP 2016.)

slide-32
SLIDE 32

NLG is everywhere

Storytelling Generation Input: Equation + Theme

Luke Skywalker has 639 blasters. Leia has 504

  • blasters. How many more blasters does Luke

Skywalker have than Leia?

504 + x = 639 +

(Koncel-Kedziorski, Konstas, Zettlemoyer, Hajishirzi. A Theme-Rewriting Approach for Generating Algebra Word Problems. EMNLP 2016.)

slide-33
SLIDE 33

NNLG Framework

input

slide-34
SLIDE 34

NNLG Framework

Encoder

input

slide-35
SLIDE 35

NNLG Framework

Encoder Decoder

input

  • utput

I The A … know knew planet … a planet man …

…

inhabit inhabited was …

slide-36
SLIDE 36

NNLG Framework

Attention Encoder Decoder

input

  • utput

think :arg0 you :arg1 quest what do you think ? </s>

I The A … know knew planet … a planet man …

…

inhabit inhabited was …

slide-37
SLIDE 37

Encoding

slide-38
SLIDE 38

Encoding

Bag of Words

(Summarizing Source Code using a Neural Attention Model. Iyer, Konstas, Cheung, Zettlemoyer. ACL 2016.)

CODE-NN

SELECT max(marks) FROM stud_records WHERE marks < (SELECT max(marks) FROM stud_records);

slide-39
SLIDE 39

Encoding

Bag of Words

(Summarizing Source Code using a Neural Attention Model. Iyer, Konstas, Cheung, Zettlemoyer. ACL 2016.)

CODE-NN

SELECT max(marks) FROM stud_records WHERE marks < (SELECT max(marks) FROM stud_records); SELECT max(col0) FROM tab0 WHERE col0 < (SELECT max(col1) FROM tab1);

anonymization

slide-40
SLIDE 40

Encoding

Bag of Words

(Summarizing Source Code using a Neural Attention Model. Iyer, Konstas, Cheung, Zettlemoyer. ACL 2016.)

CODE-NN

SELECT max(marks) FROM stud_records WHERE marks < (SELECT max(marks) FROM stud_records); SELECT max(col0) FROM tab0 WHERE col0 < (SELECT max(col1) FROM tab1);

anonymization

SELECT max col0 FROM tab0

+

h(s)

slide-41
SLIDE 41

Encoding

Linearize —> RNN encoding AMR Generation

know I planet lazy ARG0 ARG1 inhabit man ARG1-of ARG0 mod

slide-42
SLIDE 42

Encoding

Linearize —> RNN encoding AMR Generation

know I planet lazy ARG0 ARG1 inhabit man ARG1-of ARG0 mod

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

slide-43
SLIDE 43

Encoding

Linearize —> RNN encoding AMR Generation

know I planet lazy ARG0 ARG1 inhabit man ARG1-of ARG0 mod

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

know ARG0 I ARG1 planet

slide-44
SLIDE 44

Encoding

Linearize —> RNN encoding AMR Generation

know I planet lazy ARG0 ARG1 inhabit man ARG1-of ARG0 mod

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

know ARG0 I ARG1 planet

h1(s) h2(s) h3(s) h4(s) h5(s)

slide-45
SLIDE 45

Encoding

Linearize —> RNN encoding AMR Generation

know I planet lazy ARG0 ARG1 inhabit man ARG1-of ARG0 mod

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

know ARG0 I ARG1 planet

h1(s) h2(s) h3(s) h4(s) h5(s)

slide-46
SLIDE 46

Encoding

Linearize —> RNN encoding AMR Generation

know I planet lazy ARG0 ARG1 inhabit man ARG1-of ARG0 mod

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

know ARG0 I ARG1 planet

slide-47
SLIDE 47

Encoding

Linearize —> RNN encoding AMR Generation

know I planet lazy ARG0 ARG1 inhabit man ARG1-of ARG0 mod

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s)

slide-48
SLIDE 48

Encoding

Linearize —> RNN encoding AMR Generation

know I planet lazy ARG0 ARG1 inhabit man ARG1-of ARG0 mod

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s)

slide-49
SLIDE 49

Encoding

Hierarchical RNN encoding Storytelling Generation

slide-50
SLIDE 50

Encoding

Hierarchical RNN encoding Storytelling Generation

Jim was obsessed with superheroes. His sister told him if he tied a sheet on his back he could fly. She convinced Jim to climb the ladder to the roof and jump off. When he got up there he felt like he was superman.

slide-51
SLIDE 51

Encoding

Hierarchical RNN encoding Storytelling Generation

Jim was obsessed with superheroes. His sister told him if he tied a sheet on his back he could fly. She convinced Jim to climb the ladder to the roof and jump off. When he got up there he felt like he was superman.

Jim was

  • bsessed

with superheroes

slide-52
SLIDE 52

Encoding

Hierarchical RNN encoding Storytelling Generation

Jim was obsessed with superheroes. His sister told him if he tied a sheet on his back he could fly. She convinced Jim to climb the ladder to the roof and jump off. When he got up there he felt like he was superman.

Jim was

  • bsessed

with superheroes

His

sister told him if

slide-53
SLIDE 53

Encoding

Hierarchical RNN encoding Storytelling Generation

Jim was obsessed with superheroes. His sister told him if he tied a sheet on his back he could fly. She convinced Jim to climb the ladder to the roof and jump off. When he got up there he felt like he was superman.

Jim was

  • bsessed

with superheroes

His

sister told him if

She

convinced Jim to climb

slide-54
SLIDE 54

Encoding

Hierarchical RNN encoding Storytelling Generation

Jim was obsessed with superheroes. His sister told him if he tied a sheet on his back he could fly. She convinced Jim to climb the ladder to the roof and jump off. When he got up there he felt like he was superman.

Jim was

  • bsessed

with superheroes

His

sister told him if

She

convinced Jim to climb

h1(s) h2(s) h3(s)

slide-55
SLIDE 55

Encoding

Hierarchical RNN encoding Storytelling Generation

Jim was obsessed with superheroes. His sister told him if he tied a sheet on his back he could fly. She convinced Jim to climb the ladder to the roof and jump off. When he got up there he felt like he was superman.

Jim was

  • bsessed

with superheroes

His

sister told him if

She

convinced Jim to climb

h1(s) h2(s) h3(s)

slide-56
SLIDE 56

Decoding

hN(s)

Beam search (Left-to-Right)

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s)

slide-57
SLIDE 57

Decoding

hN(s)

∅

Beam search (Left-to-Right)

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s)

slide-58
SLIDE 58

Decoding

hN(s)

∅

I The A …

Beam search (Left-to-Right)

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s)

slide-59
SLIDE 59

Decoding

hN(s)

∅

I The A … know knew planet …

…

w11: I

The

w12:

Man

w13:

Tree

w14:

Beam search (Left-to-Right)

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s)

slide-60
SLIDE 60

Decoding

hN(s)

∅

I The A … know knew planet …

…

w11: I

The

w12:

Man

w13:

Tree

w14:

… I know

w21:

I knew

w22:

The planet

w23:

Man planet

w24:

Beam search (Left-to-Right)

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s)

slide-61
SLIDE 61

Decoding

hN(s)

∅

I The A … know knew planet … a planet man …

…

…

w11: I

The

w12:

Man

w13:

Tree

w14:

… I know

w21:

I knew

w22:

The planet

w23:

Man planet

w24:

Beam search (Left-to-Right)

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s)

slide-62
SLIDE 62

Decoding

hN(s)

∅

I The A … know knew planet … a planet man …

…

…

w11: I

The

w12:

Man

w13:

Tree

w14:

… I know

w21:

I knew

w22:

The planet

w23:

Man planet

w24: w41:

I know a planet

w42:

I knew planets that

w43: The planet I

knew

w44: Man know a planet that

…

Beam search (Left-to-Right)

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s)

slide-63
SLIDE 63

Decoding

hN(s)

∅

I The A … know knew planet … a planet man …

…

…

w11: I

The

w12:

Man

w13:

Tree

w14:

… I know

w21:

I knew

w22:

The planet

w23:

Man planet

w24: w41:

I know a planet

w42:

I knew planets that

w43: The planet I

knew

w44: Man know a planet that

…

inhabit inhabited was …

Beam search (Left-to-Right)

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s)

slide-64
SLIDE 64

Attention

h2(t) h3(t)

the planet man …

w2: know

slide-65
SLIDE 65

Attention

h2(t) h3(t)

the planet man …

w2: know

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s) c2

slide-66
SLIDE 66

Attention

h2(t) h3(t)

the planet man …

w2: know

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s) c2

ai = softmax

  • fi(h(s), h(t)

i−1)

slide-67
SLIDE 67

Attention

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

<s> I know the planet a lazy man inhabits </s>

h2(t) h3(t)

the planet man …

w2: know

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s) c2

ai = softmax

  • fi(h(s), h(t)

i−1)

slide-68
SLIDE 68

Attention

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

<s> I know the planet a lazy man inhabits </s>

h2(t) h3(t)

the planet man …

w2: know

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s) c2

ai = softmax

  • fi(h(s), h(t)

i−1)

slide-69
SLIDE 69

Attention

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

<s> I know the planet a lazy man inhabits </s>

h2(t) h3(t)

the planet man …

w2: know

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s) c2

ai = softmax

  • fi(h(s), h(t)

i−1)

slide-70
SLIDE 70

Attention

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

<s> I know the planet a lazy man inhabits </s>

h2(t) h3(t)

the planet man …

w2: know

know ARG0 I ARG1 planet

[ ] [ ] [ ] [ ] [ ]

h1(s) h2(s) h3(s) h4(s) h5(s) c2

ai = softmax

  • fi(h(s), h(t)

i−1)

slide-71
SLIDE 71

Issues to Address

Max-probability search w∗ = arg max

w

sc

  • t(s), w
  • sc
  • t(s), w
  • = p
  • w|t(s)

where

slide-72
SLIDE 72

Issues to Address

Max-probability search w∗ = arg max

w

sc

  • t(s), w
  • sc
  • t(s), w
  • = p
  • w|t(s)

where Issues

  • short / similar outputs
  • no guarantee that input is covered
slide-73
SLIDE 73

Issues to Address

slide-74
SLIDE 74

Issues to Address

Max-probability search

  • Length Penalty

sc

  • t(s), w
  • = log
  • p
  • w|t(s)

|t(s)|α

slide-75
SLIDE 75

Issues to Address

Max-probability search

  • Length Penalty

sc

  • t(s), w
  • = log
  • p
  • w|t(s)

|t(s)|α cp

  • t(s), w
  • = β ∗

|t(s)|

X

i=1

log

  • min

|w| X

j=1

ai,j, 1.0

  • Coverage Penalty
slide-76
SLIDE 76

Issues to Address

Max-probability search

  • Length Penalty

sc

  • t(s), w
  • = log
  • p
  • w|t(s)

|t(s)|α cp

  • t(s), w
  • = β ∗

|t(s)|

X

i=1

log

  • min

|w| X

j=1

ai,j, 1.0

  • Coverage Penalty
  • Integrating in model
  • Neural Checklist Model (Kiddon et al, EMNLP 2016)
  • Coverage Model (Tu et al, ACL 2016)
  • Structural Biases (Cohn et al, NAACL 2016)
  • Fertility, HMM bias
slide-77
SLIDE 77

Issues to Address

slide-78
SLIDE 78

Issues to Address

Sparsity

  • Anonymize NE tokens
slide-79
SLIDE 79

Issues to Address

Sparsity

  • Anonymize NE tokens

state ARG0 person_name_0 ARG1 keep ARG0 country_name_1 …

President Obama stated that UK should keep … person_name_0 stated that country_name_1 should keep …

slide-80
SLIDE 80

Issues to Address

Sparsity

  • Anonymize NE tokens

state ARG0 person_name_0 ARG1 keep ARG0 country_name_1 …

President Obama stated that UK should keep … person_name_0 stated that country_name_1 should keep …

  • Copy from input
slide-81
SLIDE 81

Issues to Address

Sparsity

  • Anonymize NE tokens

state ARG0 person_name_0 ARG1 keep ARG0 country_name_1 …

President Obama stated that UK should keep … person_name_0 stated that country_name_1 should keep …

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

<s> I know the planet a lazy man UNK </s>

  • Copy from input
slide-82
SLIDE 82

Issues to Address

Sparsity

  • Anonymize NE tokens

state ARG0 person_name_0 ARG1 keep ARG0 country_name_1 …

President Obama stated that UK should keep … person_name_0 stated that country_name_1 should keep …

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

<s> I know the planet a lazy man UNK </s>

  • Copy from input
slide-83
SLIDE 83

Issues to Address

Sparsity

  • Anonymize NE tokens

state ARG0 person_name_0 ARG1 keep ARG0 country_name_1 …

President Obama stated that UK should keep … person_name_0 stated that country_name_1 should keep …

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

<s> I know the planet a lazy man UNK </s>

  • Copy from input
slide-84
SLIDE 84

Issues to Address

Sparsity

  • Anonymize NE tokens

state ARG0 person_name_0 ARG1 keep ARG0 country_name_1 …

President Obama stated that UK should keep … person_name_0 stated that country_name_1 should keep …

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

<s> I know the planet a lazy man UNK </s>

input

  • utput

prob inhabit inhabits 0.6 inhabit 0.2 inhabiting 0.1 … …

  • Copy from input
slide-85
SLIDE 85

Issues to Address

state ARG0 person_name_0 ARG1 keep ARG0 country_name_1 …

President Obama stated that UK should keep … person_name_0 stated that country_name_1 should keep …

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

<s> I know the planet a lazy man inhabits </s>

input

  • utput

prob inhabit inhabits 0.6 inhabit 0.2 inhabiting 0.1 … …

Sparsity

  • Anonymize NE tokens
  • Copy from input
slide-86
SLIDE 86

Issues to Address

state ARG0 person_name_0 ARG1 keep ARG0 country_name_1 …

President Obama stated that UK should keep … person_name_0 stated that country_name_1 should keep …

know ARG0 I ARG1 planet ARG1-of inhabit ARG0 man mod lazy

<s> I know the planet a lazy man inhabits </s>

input

  • utput

prob inhabit inhabits 0.6 inhabit 0.2 inhabiting 0.1 … …

Sparsity

  • Anonymize NE tokens
  • Copy from input
  • Data Augmentation (Sennrich et al, ACL 2016)
slide-87
SLIDE 87

Open Questions

slide-88
SLIDE 88

Open Questions

Representations

  • Probably shouldn’t treat all inputs as strings…
slide-89
SLIDE 89

Open Questions

Representations

  • Probably shouldn’t treat all inputs as strings…

Loss on some intermediate / latent goal

  • Don’t want just good-looking string of [X_language]…
slide-90
SLIDE 90

Open Questions

Representations

  • Probably shouldn’t treat all inputs as strings…

Loss on some intermediate / latent goal

  • Don’t want just good-looking string of [X_language]…

Document Plans

  • Maybe shouldn’t treat output as stream of strings…
slide-91
SLIDE 91

THANK You