Document Summarization Statistical NLP Spring 2011 Lecture 25: - - PDF document

▶

Aug 26, 2022 159 likes •252 views

Document Summarization Statistical NLP Spring 2011 Lecture 25: Summarization Dan Klein UC Berkeley Multi-document Summarization Extractive Summarization 27,000+ more Selection Selection mid-90s mid-90s Maximum Marginal

SLIDE 1

1 Statistical NLP

Spring 2011

Lecture 25: Summarization

Dan Klein – UC Berkeley

Document Summarization Multi-document Summarization

… 27,000+ more

Extractive Summarization

Selection

Maximum Marginal Relevance

mid-‘90s present

Maximize similarity to the query Minimize redundancy [Carbonell and Goldstein, 1998] s1 s3 s2 s4 Q Q

Greedy search over sentences

Maximum Marginal Relevance
Graph algorithms [Mihalcea 05++]

mid-‘90s present

Selection

SLIDE 2

2

Maximum Marginal Relevance
Graph algorithms

mid-‘90s present

s1 s3 s2 s4 Nodes are sentences

Selection

Maximum Marginal Relevance
Graph algorithms

mid-‘90s present

s1 s3 s2 s4 Nodes are sentences Edges are similarities

Selection

Maximum Marginal Relevance
Graph algorithms

mid-‘90s present

s3 s2 s4 Nodes are sentences Edges are similarities Stationary distribution represents node centrality

Selection

Maximum Marginal Relevance
Graph algorithms
Word distribution models

mid-‘90s present

Input document distribution Summary distribution

~

w w PA(w) (w) Obama ? speech ? health ? Montana ? w PD(w) (w) Obama 0.017 speech 0.024 health 0.009 Montana 0.002

Selection

Maximum Marginal Relevance
Graph algorithms
Word distribution models

mid-‘90s present

SumBasic [Nenkova and Vanderwende, 2005] Value(wi) = PD(wi) Value(si) = sum of its word values Choose si with largest value Adjust PD(w) Repeat until length constraint

Selection

Maximum Marginal Relevance
Graph algorithms
Word distribution models
Regression models

mid-‘90s present

s1 s2 s3 word values word values position position length length 12 1 24 4 2 14 6 3 18 s2 s3 s1

F(x)

frequency is just one of many features

Selection

SLIDE 3

3

Maximum Marginal Relevance
Graph algorithms
Word distribution models
Regression models
Topic model-based

[Haghighi and Vanderwende, 2009]

mid-‘90s present

Selection

SLIDE 4

4

H & V 09 PYTHY

Maximum Marginal Relevance
Graph algorithms
Word distribution models
Regression models
Topic models
Globally optimal search

mid-‘90s present

[McDonald, 2007] s1 s3 s2 s4 Q Q Optimal search using MMR Integer Linear Program

Selection

SLIDE 5

5

[Gillick and Favre, 2008]

Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration.

s1 s2 s3 s4

concept concept value value

Selection

[Gillick and Favre, 2008]

Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration.

concept concept value value

bama

s1 s2 s3 s4

Selection

[Gillick and Favre, 2008]

Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration.

concept concept value value

bama

3 health 2

s1 s2 s3 s4

Selection

[Gillick and Favre, 2008]

Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm. The health care bill is a major test for the Obama administration.

concept concept value value

bama

3 health 2 house 1

s1 s2 s3 s4

Selection

[Gillick and Favre, 2008]

Universal health care is a divisive issue. Obama addressed the House on Tuesday. President Obama remained calm.

concept concept value value

bama

3 health 2 house 1

s1 s2 s3 s4

The health care bill is a major test for the Obama administration.

summary summary length length value value {s1, s3} 17 5 {s2, s3, s4} 17 6 Length limit: 18 words greedy

ptimal

Selection Maximize Concept Coverage

[Gillick and Favre 09]

Optimization problem: Set Coverage

Value of concept c Set of concepts present in summary s Set of extractive summaries

f document set D

Results

2009 Baseline

Bigram Recall

2009 Baseline

Pyramid

23.5 35.0 4.00 6.85

SLIDE 6

6

[Gillick, Riedhammer, Favre, Hakkani-Tur, 2008] total concept value summary length limit maintain consistency between selected sentences and concepts

Integer Linear Program for the maximum coverage model

Selection

[Gillick and Favre, 2009] This ILP is tractable for reasonable problems

Selection

52 submissions
27 teams
44 topics
10 input docs
100 word summaries

Gillick & Favre

Rating scale: 1-10
Humans in [8.3, 9.3]
Rating scale: 1-10
Humans in [8.5, 9.3]
Rating scale: 0-1
Humans in [0.62, 0.77]
Rating scale: 0-1
Humans in [0.11, 0.15]

Results [G & F, 2009]

[Gillick and Favre, 2008]

Error Breakdown?

How to include sentence position?

First sentences are unique

Selection

Some interesting work on sentence ordering

[Barzilay et. al., 1997; 2002]

But choosing independent sentences is easier

First sentences usually stand alone well
Sentences without unresolved pronouns
Classifier trained on OntoNotes: <10% error rate

Baseline ordering module (chronological) is not

bviously worse than anything fancier

Selection

SLIDE 7

7 Problems with Extraction

It is therefore unsurprising that Lindsay pleaded not guilty yesterday afternoon to the charges filed against her, according to her publicist.

What would a human do?