Statistical NLP Spring 2011 Lecture 25: Summarization Dan Klein - - PDF document

▶
statistical nlp
SMART_READER_LITE
LIVE PREVIEW

Statistical NLP Spring 2011 Lecture 25: Summarization Dan Klein - - PDF document

Mar 12, 2023 228 likes •480 views

Statistical NLP Spring 2011 Lecture 25: Summarization Dan Klein UC Berkeley Document Summarization 1 Multi-document Summarization 27,000+ more Extractive Summarization 2 Selection mid-90s Maximum Marginal Relevance Greedy

slide-1
SLIDE 1

1

Statistical NLP

Spring 2011

Lecture 25: Summarization

Dan Klein – UC Berkeley

Document Summarization

slide-2
SLIDE 2

2

Multi-document Summarization

… 27,000+ more

Extractive Summarization

slide-3
SLIDE 3

3

Selection

  • Maximum Marginal Relevance

mid-‘90s present

Maximize similarity to the query Minimize redundancy

[Carbonell and Goldstein, 1998] s1 s3 s s2 s s4

Q

Greedy search over sentences

  • Maximum Marginal Relevance
  • Graph algorithms [Mihalcea 05++]

mid-‘90s present

Selection

slide-4
SLIDE 4

4

  • Maximum Marginal Relevance
  • Graph algorithms

mid-‘90s present

s1 s3 s2 s4

Nodes are sentences

Selection

  • Maximum Marginal Relevance
  • Graph algorithms

mid-‘90s present

s1 s3 s2 s4

Nodes are sentences Edges are similarities

Selection

slide-5
SLIDE 5

5

  • Maximum Marginal Relevance
  • Graph algorithms

mid-‘90s present

s1 s3 s2 s s4

Nodes are sentences Edges are similarities Stationary distribution represents node centrality

Selection

  • Maximum Marginal Relevance
  • Graph algorithms
  • Word distribution models

mid-‘90s present

Input document distribution Summary distribution

~

w P PA(w) (w) Obama ? speech ? health ? Montana ? w PD(w) (w) Obama 0.017 speech 0.024 health 0.009 Montana 0.002

Selection

slide-6
SLIDE 6

6

  • Maximum Marginal Relevance
  • Graph algorithms
  • Word distribution models

mid-‘90s present

SumBasic [Nenkova and Vanderwende, 2005]

Value(wi) = PD(wi) Value(si) = sum of its word values Choose si with largest value Adjust PD(w) Repeat until length constraint

Selection

  • Maximum Marginal Relevance
  • Graph algorithms
  • Word distribution models
  • Regression models

mid-‘90s present

s1

1

s2

2

s3

3

word values word values position position length length 12 1 24 4 2 14 6 3 18

s s2 s s3 s s1

F(x)

frequency is just one of many features

Selection

slide-7
SLIDE 7

7

  • Maximum Marginal Relevance
  • Graph algorithms
  • Word distribution models
  • Regression models
  • Topic model-based

[Haghighi and Vanderwende, 2009]

mid-‘90s present

Selection

slide-8
SLIDE 8

8

slide-9
SLIDE 9

9

slide-10
SLIDE 10

10

slide-11
SLIDE 11

11

H & V 09 PYTHY

slide-12
SLIDE 12

12

  • Maximum Marginal Relevance
  • Graph algorithms
  • Word distribution models
  • Regression models
  • Topic models
  • Globally optimal search

mid-‘90s present

[McDonald, 2007]

s1 s3 s s2 s s4

Q Optimal search using MMR Integer Linear Program

Selection

slide-13
SLIDE 13

13

[Gillick and Favre, 2008]

Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration.

s1 s2 s3 s4

concept concept value value

Selection

[Gillick and Favre, 2008]

Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration.

concept concept value value

  • bama

3

s1 s2 s3 s4

Selection

slide-14
SLIDE 14

14

[Gillick and Favre, 2008]

Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration.

concept concept value value

  • bama

3 health 2

s1 s2 s3 s4

Selection

[Gillick and Favre, 2008]

Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration.

concept concept value value

  • bama

3 health 2 house 1

s1 s2 s3 s4

Selection

slide-15
SLIDE 15

15

[Gillick and Favre, 2008]

Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm.

concept concept value value

  • bama

3 health 2 house 1

s1 s2 s3 s4

The health care bill is a major test for the Obama administration.

summary summary length length value value {s1, s3} 17 5 {s2, s3, s4} 17 6

Length limit: 18 words greedy

  • ptimal

Selection Maximize Concept Coverage

[Gillick and Favre 09]

Optimization problem: Set Coverage

Value of concept c Set of concepts present in summary s Set of extractive summaries

  • f document set D

Results

2009 Baseline

Bigram Recall

2009 Baseline

Pyramid

23.5 35.0 4.00 6.85

slide-16
SLIDE 16

16

[Gillick, Riedhammer, Favre, Hakkani-Tur, 2008]

total concept value summary length limit maintain consistency between selected sentences and concepts

Integer Linear Program for the maximum coverage model

Selection

[Gillick and Favre, 2009] This ILP is tractable for reasonable problems

Selection

slide-17
SLIDE 17

17

  • 52 submissions
  • 27 teams
  • 44 topics
  • 10 input docs
  • 100 word summaries

Gillick & Favre

  • Rating scale: 1-10
  • Humans in [8.3, 9.3]
  • Rating scale: 1-10
  • Humans in [8.5, 9.3]
  • Rating scale: 0-1
  • Humans in [0.62, 0.77]
  • Rating scale: 0-1
  • Humans in [0.11, 0.15]

Results [G & F, 2009]

[Gillick and Favre, 2008]

Error Breakdown?

slide-18
SLIDE 18

18

How to include sentence position?

First sentences are unique

Selection

Some interesting work on sentence ordering

[Barzilay et. al., 1997; 2002]

But choosing independent sentences is easier

  • First sentences usually stand alone well
  • Sentences without unresolved pronouns
  • Classifier trained on OntoNotes: <10% error rate

Baseline ordering module (chronological) is not

  • bviously worse than anything fancier

Selection

slide-19
SLIDE 19

19

Problems with Extraction

It is therefore unsurprising that Lindsay pleaded not guilty yesterday afternoon to the charges filed against her, according to her publicist.

What would a human do?

Problems with Extraction

It is therefore unsurprising that Lindsay pleaded not guilty yesterday afternoon to the charges filed against her, according to her publicist.

What would a human do?

slide-20
SLIDE 20

20

Sentence Rewriting

[Berg-Kirkpatrick, Gillick, and Klein 11]

Sentence Rewriting

[Berg-Kirkpatrick, Gillick, and Klein 11]

slide-21
SLIDE 21

21

Sentence Rewriting

[Berg-Kirkpatrick, Gillick, and Klein 11]

Sentence Rewriting

[Berg-Kirkpatrick, Gillick, and Klein 11]

New Optimization problem: Safe Deletions

Set branch cut deletions made in creating summary s Value of deletion d

How do we know how much a given deletion costs?

slide-22
SLIDE 22

22

Learning

[Berg-Kirkpatrick, Gillick, and Klein 11]

Features: Embed ILP in cutting plane algorithm. Results

7.75 6.85 4.00

Now 2009 Baseline

Bigram Recall

35.0 23.5

Now 2009 Baseline

Pyramid

41.3

Sentence extraction is limiting ... and boring! But abstractive summaries are much harder to generate…

in 25 words?

Beyond Extraction / Compression?

slide-23
SLIDE 23

23

http://www.rinkworks.com/bookaminute/