[PPT] - Automatic Summarization (and other stuff) Taylor Berg-Kirkpatrick PowerPoint Presentation

SLIDE 1

Automatic Summarization (and other stuff)

Taylor Berg-Kirkpatrick CS 288 UC Berkeley

Summarization Summarization Multidocument Summarization Sentence Extraction

Lindsay Lohan pleaded not guilty Wednesday to felony grand theft

f a 2,500 necklace, a case that

could return the troubled starlet to jail rather than the big screen.

Sentence Extraction

Lindsay Lohan pleaded not guilty Wednesday to felony grand theft

f a 2,500 necklace, a case that

could return the troubled starlet to jail rather than the big screen. Saying it appeared that Lohan had violated her probation in a 2007 drunken driving case the judge set bail at $40,000 and warned that if Lohan was accused of breaking the law while free he would have her held without bail.

SLIDE 2

Sentence Extraction

Lindsay Lohan pleaded not guilty Wednesday to felony grand theft

f a 2,500 necklace, a case that

could return the troubled starlet to jail rather than the big screen. Saying it appeared that Lohan had violated her probation in a 2007 drunken driving case the judge set bail at $40,000 and warned that if Lohan was accused of breaking the law while free he would have her held without bail. The Mean Girls star is due back in court on Feb. 23 an important hearing in which Lohan could opt to end the case early.

Maximum Marginal Relevance Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

S1: She traveled to France on Friday.

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

S1: She traveled to France on Friday.

SLIDE 3

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

S1: She traveled to France on Friday. S4: On Friday, She took a trip to France.

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

S1: She traveled to France on Friday. S4: On Friday, She took a trip to France.

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

S1: She traveled to France on Friday. S4: On Friday, She took a trip to France. S8: She plans to stay for two weeks.

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

S1: She traveled to France on Friday. S4: On Friday, She took a trip to France. S8: She plans to stay for two weeks.

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

SLIDE 4

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

[Carbonell and Goldstein, 1998]

argmax

i∈D\S

 λ · sim(Si, D) − (1 − λ) · max

j

(sim(Si, Sj))

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

[Carbonell and Goldstein, 1998]

argmax

i∈D\S

 λ · sim(Si, D) − (1 − λ) · max

j

(sim(Si, Sj))

Centrality

Maximum Marginal Relevance

S1 S2 S3 S4 S5 S6 S7 S8 S9

[Carbonell and Goldstein, 1998]

argmax

i∈D\S

 λ · sim(Si, D) − (1 − λ) · max

j

(sim(Si, Sj))

Centrality

Redundancy

Max Coverage

She stopped in France. In France she remained.

SLIDE 5

Max Coverage

She stopped in France. In France she remained. She stopped in France. In France she remained.

Max Coverage

She stopped in France. In France she remained.

Max Coverage

She stopped in France. In France she remained.

Max Coverage

A summary is a set of cuts: s

Max Coverage

She stopped in France. In France she remained.

Max Coverage

She stopped in France. In France she remained.

(she, stopped) (stopped, in) (in, france) (france, she) (she, remained)

SLIDE 6

She stopped in France. In France she remained.

Max Coverage

(she, stopped) (stopped, in) (in, france) (france, she) (she, remained)

She stopped in France. In France she remained.

Max Coverage

(she, stopped) (stopped, in) (in, france) (france, she) (she, remained)

She stopped in France. In France she remained.

Max Coverage

(she, stopped) (stopped, in) (in, france) (france, she) (she, remained)

Set of bigrams covered by summary: B(s)

coverage(s) max

s

Max Coverage

coverage(s) max

s

Max Coverage

s.t. length(s) ≤ Lmax

X

b∈B(s)

value(b)

Max Coverage

s.t. length(s) ≤ Lmax

max

s

SLIDE 7

X

b∈B(s)

value(b)

Max Coverage value(b) = freq(b)

s.t. length(s) ≤ Lmax

max

s

X

b∈B(s)

value(b)

Max Coverage value(b) = freq(b)

s.t. length(s) ≤ Lmax

max

s

[Gillick and Favre 2008]

X

b∈B(s)

value(b)

s.t. length(s) ≤ Lmax

max

s

[Gillick and Favre 2008]

ILP for Decoding

s.t. length(s) ≤ Lmax

max

s

X

b∈B(s)

value(b)

max

s,z

X

b

zb · value(b)

ILP for Decoding

s.t. length(s) ≤ Lmax

max

s,z

X

b

zb · value(b)

bigrams in are covered

s

ILP for Decoding

s.t. length(s) ≤ Lmax

SLIDE 8

max

s,z

X

b

zb · value(b)

bigrams in are covered

s

ILP for Decoding

s.t.

nly bigrams in are covered

s

length(s) ≤ Lmax

max

s,z

X

b

zb · value(b)

ILP for Decoding

X

n

lnsn ≤ Lmax

∀n,b

snQnb ≤ zb s.t.

∀b

X

n

snQnb ≥ zb

Linear Model for Extraction

X

b∈B(s)

value(b)

value(b) = freq(b)

s.t. length(s) ≤ Lmax

max

s

Linear Model for Extraction

X

b∈B(s)

value(b)

Parameterize using features:

value(b) = w>f(b)

s.t. length(s) ≤ Lmax

max

s

Sentence Extraction

Lindsay Lohan pleaded not guilty Wednesday to felony grand theft

f a 2,500 necklace, a case that

could return the troubled starlet to jail rather than the big screen. Saying it appeared that Lohan had violated her probation in a 2007 drunken driving case the judge set bail at $40,000 and warned that if Lohan was accused of breaking the law while free he would have her held without bail. The Mean Girls star is due back in court on Feb. 23 an important hearing in which Lohan could opt to end the case early.

Extraction and Compression

Lindsay Lohan pleaded not guilty Wednesday to felony grand theft

f a 2,500 necklace, a case that

could return the troubled starlet to jail rather than the big screen. Saying it appeared that Lohan had violated her probation in a 2007 drunken driving case the judge set bail at $40,000 and warned that if Lohan was accused of breaking the law while free he would have her held without bail. The Mean Girls star is due back in court on Feb. 23 an important hearing in which Lohan could opt to end the case early.

SLIDE 9

Extraction and Compression

Lindsay Lohan pleaded not guilty Wednesday to felony grand theft

f a 2,500 necklace, a case that

could return the troubled starlet to jail rather than the big screen. Saying it appeared that Lohan had violated her probation in a 2007 drunken driving case the judge set bail at $40,000 and warned that if Lohan was accused of breaking the law while free he would have her held without bail. The Mean Girls star is due back in court on Feb. 23 an important hearing in which Lohan could opt to end the case early. [Martins and Smith 2009] [Woodsend and Lapata 2010]

She stopped in France. In France she remained.

Extraction and Compression Joint Extractive / Compressive Model

She stopped in France. In France she remained.

Joint Extractive / Compressive Model

She stopped in France. In France she remained. She stopped in France. In France she remained.

Joint Extractive / Compressive Model

She stopped in France. In France she remained.

Joint Extractive / Compressive Model

s

SLIDE 10

She stopped in France. In France she remained.

Joint Extractive / Compressive Model

(she, stopped) (stopped, in) (in, france) (france, she) (she, remained)

s

She stopped in France. In France she remained.

Joint Extractive / Compressive Model

(she, stopped) (stopped, in) (in, france) (france, she) (she, remained)

s

B(s)

She stopped in France. In France she remained.

Joint Extractive / Compressive Model

(she, stopped) (stopped, in) (in, france) (france, she) (she, remained)

s

B(s)

X

b∈B(s)

value(b) Joint Extractive / Compressive Model

h

max

s

X

b∈B(s)

value(b) Joint Extractive / Compressive Model X

c∈s

value(c)

i

+

h

max

s

X

b∈B(s)

value(b) Joint Extractive / Compressive Model value(b) = w>f(b)

Parameterize using features:

value(c) = w>f(c) X

c∈s

value(c)

i

+

h

max

s

SLIDE 11

Learning score(s) = w>f(s)

Linear prediction:

Learning score(s) = w>f(s)

Linear prediction:

X

c∈s

f(c) X

b∈B(s)

f(b)

f(s) =

+

Feature function factors:

Features Features

f(b)

Bigram Features

Features

f(b)

Bigram Features Cut Features f(c)

Features

COUNT: Bucketed document counts STOP: Stop word indicators POSITION: First document position indicators CONJ: All two- and three-way conjunctions of above BIAS: Always one

f(b)

Bigram Features Cut Features f(c)

SLIDE 12

Features

COUNT: Bucketed document counts STOP: Stop word indicators POSITION: First document position indicators CONJ: All two- and three-way conjunctions of above BIAS: Always one

f(b)

Bigram Features Cut Features f(c)

COORD: Coordinated phrase, four versions: NP , VP , S, SBAR S-ADJUNCT: Adjunct to matrix verb, four versions: CC, PP , ADVP , SBAR REL-C: Relative clause indicator ATTR-C: Attribution clause indicator ATTR-PP: PP attribution indicator TEMP-PP: Temporal PP indicator TEMP-NP Temporal NP indicator BIAS: Always one

Obtaining Training Data

Lindsay Lohan pleaded not guilty Wednesday to felony grand theft

f a 2,500 necklace, a case that

could return the troubled starlet to jail rather than the big screen. Saying it appeared that Lohan had violated her probation in a 2007 drunken driving case the judge set bail at $40,000 and warned that if Lohan was accused of breaking the law while free he would have her held without bail. The Mean Girls star is due back in court on Feb. 23 an important hearing in which Lohan could opt to end the case early.

s s s

Obtaining Training Data

s s s s s s

Intermediate Extraction TAC 2009: 44 Document Collections

s s s

Obtaining Training Data

s s s s s s

Maximize Gold Recall Intermediate Extraction TAC 2009: 44 Document Collections

s s s

Obtaining Training Data

s s s s s s

Maximize Gold Recall Intermediate Extraction Mechanical Turk TAC 2009: 44 Document Collections

Learning

SLIDE 13

Learning

Structured SVM Training:

Learning

Structured SVM Training:

min

w

kwk2

2 score of gold exceeds score of guess by loss

Learning

for all possible guess summaries:

s.t.

Structured SVM Training:

min

w

kwk2

2

Learning

for all possible guess summaries:

s.t.

Structured SVM Training:

w>f(S⇤) − w>f(S) ≥ loss(S⇤, S)

min

w

kwk2

2

Learning

for all possible guess summaries:

s.t.

Structured SVM Training:

w>f(S⇤) − w>f(S) ≥ loss(S⇤, S) − ξ

min

w

kwk2

2

Learning

for all possible guess summaries:

s.t.

Structured SVM Training:

w>f(S⇤) − w>f(S) ≥ loss(S⇤, S) − ξ

h i

ξ

+

min

w

kwk2

2

SLIDE 14

Learning

for all possible guess summaries:

s.t.

Structured SVM Training:

w>f(S⇤) − w>f(S) ≥ loss(S⇤, S) − ξ End-to-end summarization quality

h i

ξ

+

min

w

kwk2

2

Learning

for all possible guess summaries:

s.t.

Structured SVM Training:

w>f(S⇤) − w>f(S) ≥ loss(S⇤, S) − ξ Exponentially many constraints!

h i

ξ

+

min

w

kwk2

2

Learning

[Tsochantaridis et al. 2004]

Learning

[Tsochantaridis et al. 2004]

Learning

[Tsochantaridis et al. 2004]

Learning

[Tsochantaridis et al. 2004]

[Tsochantaridis et al. 2004]

SLIDE 16

Results

5 12

Rouge-2

Results

5 12

5.9

Rouge-2 Last Doc Baseline

11.1 10.1 5.9

20 42

38.4 35 23.5

6 7.5

6.6 6.2 7.2

Rouge-2 Pyramid Linguistic Quality Last Doc Baseline Extractive Baseline Learned Extractive

SLIDE 17

Results

5 12

11.7 11.1 10.1 5.9

20 42

41.3 38.4 35 23.5

6 7.5

6.5 6.6 6.2 7.2

Rouge-2 Pyramid Linguistic Quality Last Doc Baseline Extractive Baseline Learned Extractive Learned E + C

Coarse-to-fine Decoding

s s s s s s

Coarse-to-fine Decoding

s s s s s s

Coarse-to-fine Decoding

s s s s s s s s s Intermediate Extraction s s s

Coarse-to-fine Decoding

s s s s s s Intermediate Extraction s s s

Coarse-to-fine Decoding

s s s s s s Intermediate Extraction

SLIDE 18

s s s

Coarse-to-fine Decoding

s s s s s s Intermediate Extraction s s s

Coarse-to-fine Decoding

s s s s s s Intermediate Extraction Extraction s s s

Coarse-to-fine Decoding

s s s s s s Intermediate Extraction Exact E + C Extraction s s s

Coarse-to-fine Decoding

s s s s s s Intermediate Extraction Exact E + C Extraction

Approximate E + C

Coarse-to-fine Decoding

240 190 7.6 5.8 7

Avg Objective Avg Recall Seconds per instance Intermediate extraction size

100 2000

Full Extractive

Coarse-to-fine Decoding

240 190 7.6 5.8 7

Avg Objective Avg Recall Seconds per instance Intermediate extraction size

100 2000

Full Exact E+C Full Extractive

SLIDE 19

Coarse-to-fine Decoding

240 190 7.6 5.8 7

Avg Objective Avg Recall Seconds per instance Intermediate extraction size

100 2000

Full Exact E+C Full Extractive Approx E+C

Coarse-to-fine Decoding

240 190 7.6 5.8 7

Avg Objective Avg Recall Seconds per instance Intermediate extraction size

100 2000

LP Relaxation Full Exact E+C Full Extractive Approx E+C

Piano Music Transcription Piano Music Transcription Piano Music Transcription Piano Music Transcription

time note

time freq

SLIDE 25

Polyphony Polyphony Polyphony Polyphony Polyphony Polyphony

. . .

SLIDE 26

Polyphony

. . .

Generative Model

time note

Generative Model

time note

Generative Model

time note Note events

time velocity

Generative Model

time duration velocity

REST PLAY

Mn

µn

Generative Model

Note events

time

Activation

time duration velocity

REST PLAY

Mn

µn

Generative Model

Note events

time

Activation

time duration velocity

REST PLAY

An Mn

µn

Generative Model

Note events

time

Activation

time time duration velocity

REST PLAY

An Mn

µn αn

Generative Model

Note events

time

Activation

time

Component spectrogram

time freq time duration velocity

REST PLAY

An Mn

µn αn

SLIDE 28

Generative Model

Note events

time

Activation

time

Component spectrogram

time freq time duration velocity

REST PLAY

Sn An Mn

µn αn

Generative Model

Note events

time

Activation

time

Component spectrogram

time freq freq time duration velocity

REST PLAY

Sn An Mn

µn αn σn

Generative Model

Note events

time

Activation

time

Component spectrogram

time freq freq time duration velocity

REST PLAY

Sn An Mn

µn αn σn

Generative Model

Note events

time

Activation

time

Spectrogram

time freq

Component spectrogram

time freq freq time duration velocity

REST PLAY

Sn An Mn

µn αn σn

Generative Model

Note events

time

Activation

time

Spectrogram

time freq

Component spectrogram

time freq freq time duration velocity

REST PLAY

X Sn An Mn µn αn σn

Generative Model

Note events

time

Activation

time

Spectrogram

time freq

Component spectrogram

time freq freq time duration velocity

REST PLAY

X

Latent variables

Sn An Mn

µn αn σn

SLIDE 29

Generative Model

Note events

time

Activation

time

Spectrogram

time freq

Component spectrogram

time freq freq time duration velocity

REST PLAY

X

Parameters Latent variables

Sn An Mn

µn αn σn

Note Event Model Note Event Model

Mn

Note Event Model

Mn

Event type

Note Event Model

Mn

E1

Event type

PLAY

Note Event Model

Mn

Note Event Model

REST PLAY

Note Event Model

D1

Duration

D2 D3

Mn

E1 E2 E3

Event type

PLAY REST PLAY duration

REST PLAY

D1

Duration

D2 D3

Mn

SLIDE 32

E1 E2 E3

V2 V3

Event type

V1

Velocity

PLAY REST PLAY duration velocity

REST PLAY

Note Event Model

D1

Duration

D2 D3

Activation Model

V2 V3 V1 D1 D2 D3

An

Activation

Activation Model

V2 V3 V1 D1 D2 D3

V2 V3 V1 D1 D2 D3

αn add Gaussian noise Temporal shape

Component Spectrogram Model

An

Activation

SLIDE 35

Component Spectrogram Model

Note events

time

Activation

time

Spectrogram

time freq

Component spectrogram

time freq freq time duration velocity

REST PLAY

X

Parameters Latent variables

Sn An Mn

µn αn σn

Learning and Inference

Note events

time

Activation

time

Spectrogram

time freq

Component spectrogram

time freq freq time duration velocity

REST PLAY

X

Parameters Latent variables

Sn An Mn

µn αn σn

Learning and Inference

Note events

time

Activation

time

Spectrogram

time freq

Component spectrogram

time freq freq time duration velocity

REST PLAY

X

Parameters Latent variables

Sn An Mn

µn αn σn

Learning and Inference

Note events

time

Activation

time

Spectrogram

time freq

Component spectrogram

time freq freq time duration velocity

REST PLAY

X

Parameters Latent variables

Sn An Mn

µn αn σn

Learning and Inference Learning and Inference

A|M, X, α, σ

Activations update: Note events update:

M|A, α, µ

Learning and Inference

Semi-Markov dynamic program Closed form update Exponentiated gradient ascent

α|A, M

Temporal shapes update:

A|M, X, α, σ

Activations update: Note events update:

M|A, α, µ

Learning and Inference

Semi-Markov dynamic program Closed form update Exponentiated gradient ascent

α|A, M

Temporal shapes update:

A|M, X, α, σ

Activations update:

σ|A, X

Spectral shapes update: Note events update:

M|A, α, µ

SLIDE 38

Learning and Inference

Semi-Markov dynamic program Closed form update Exponentiated gradient ascent Exponentiated gradient ascent

α|A, M

Temporal shapes update:

A|M, X, α, σ

Activations update:

σ|A, X

Spectral shapes update: Note events update:

M|A, α, µ

Evaluation

SLIDE 39

Results

50 60 70 80 69.0 68.6 58.3

Onset F1

MAPS Corpus O’Hanlon 2014 Benetos 2014 Vincent 2013

[Valentin et al. 2010]

Results

50 60 70 80 82.1 69.0 68.6 58.3

Onset F1

MAPS Corpus O’Hanlon 2014 Benetos 2014 Vincent 2013 Our System*

[Berg-Kirkpatrick et al. 2014] [Valentin et al. 2010]

Transcription Transcription

Reference

Transcription

Predicted Reference

Resynthesized Examples