Prosody-Based Unsupervised Speech Summarization with Two-Layer - - PowerPoint PPT Presentation
Prosody-Based Unsupervised Speech Summarization with Two-Layer Mutually Reinforced Random Walk Sujay Kumar Jauhar {sjauhar, yvchen, fmetze}@cs.cmu.edu Yun-Nung (Vivian) Chen The 6th International Joint Conference on Natural Florian Metze
Sujay Kumar Jauhar Yun-Nung (Vivian) Chen Florian Metze
The 6th International Joint Conference on Natural Language Processing – Oct. 14-18, 2013
{sjauhar, yvchen, fmetze}@cs.cmu.edu
Language Technologies Institute School of Computer Science Carnegie Mellon University
2
Introduction Approach Experiments Conclusion
Introduction Approach Experiments Conclusion
O Motivation O Extractive Summarization
3
Introduction Approach Experiments Conclusion
O Motivation O Extractive Summarization
4
O Speech Summarization
O Spoken documents are more difficult to browse than texts
easy to browse, save time, easily get the key points O Prosodic Features
O Speakers may use prosody to implicitly convey the
importance of the speech
5
Introduction Approach Experiments Conclusion
O Motivation O Extractive Summarization
6
O Extractive Speech Summarization
O Select the indicative utterances in a spoken document O Cascade the utterances to form a summary
1st utterance 2nd utterance 3rd utterance 4th utterance : : n-th utterance : :
Extractive Summary 7
O Selection of Indicative Utterances
O Each utterance U in a spoken document d is given an
importance score I(U, d)
O Select the indicative utterances based on I(U,d) O The number of utterances selected as summary is decided
by a predefined ratio
n i
2 1
n i i d
1
utterance term term statistical measure (ex. TF-IDF) Importance score
8
Introduction Approach Experiments Conclusion
O Prosodic Feature Extraction O Graph Construction O Two-Layer Mutually Reinforced Random Walk
9
Introduction Approach Experiments Conclusion
O Prosodic Feature Extraction O Graph Construction O Two-Layer Mutually Reinforced Random Walk
10
O For each pre-segmented audio file, we extract
O number of syllables O number of pauses O duration time: speaking time including pauses O phonation time: speaking time excluding pauses O speaking rate: #syllable / duration time O articulation rate: #syllable / phonation time O fundamental frequency measured in Hz: avg, max, min O energy measured in Pa2/sec O intensity measured in dB
11
Introduction Approach Experiments Conclusion
O Prosodic Feature Extraction O Graph Construction O Two-Layer Mutually Reinforced Random Walk
12
O Utterance-Layer
O Each node is the
utterance in the meeting document
13
U1 U2 U3 U4 U5 U6 U7
Utterance-Layer
O Utterance-Layer
O Each node is the
utterance in the meeting document O Prosody-Layer
O Each node is a
prosodic feature
14
U1 U2 U3 U4 U5 U6 U7
Utterance-Layer
P1 P2 P3 P4 P5 P6
Prosody-Layer
O Utterance-Layer
O Each node is the
utterance in the meeting document O Prosody-Layer
O Each node is a
prosodic feature O Between-Layer
15
U1 U2 U3 U4 U5 U6 U7
Utterance-Layer
P1 P2 P3 P4 P5 P6
Prosody-Layer
O The weight of the edge is the normalized value of the
prosodic feature extracted from the utterance
Introduction Approach Experiments Conclusion
O Prosodic Feature Extraction O Graph Construction O Two-Layer Mutually Reinforced Random Walk
16
O Mathematical Formulation
utterance scores at (t+1)-th iteration 17
U1 U2 U3 U4 U5 U6 U7 Utterance-Layer P1 P2 P3 P4 P5 P6 Prosody-Layer
O Mathematical Formulation
18
O Original importance
O Utterance: equal weight
U1 U2 U3 U4 U5 U6 U7 Utterance-Layer P1 P2 P3 P4 P5 P6 Prosody-Layer
O Mathematical Formulation
scores propagated from prosody nodes weighted by prosodic values 19
O Original importance
O Utterance: equal weight
U1 U2 U3 U4 U5 U6 U7 Utterance-Layer P1 P2 P3 P4 P5 P6 Prosody-Layer
O Mathematical Formulation
prosody scores at (t+1)-th iteration 20
O Original importance
O Utterance: equal weight
U1 U2 U3 U4 U5 U6 U7 Utterance-Layer P1 P2 P3 P4 P5 P6 Prosody-Layer
O Mathematical Formulation
21
O Original importance
O Utterance: equal weight O Prosody: equal weight
U1 U2 U3 U4 U5 U6 U7 Utterance-Layer P1 P2 P3 P4 P5 P6 Prosody-Layer
O Mathematical Formulation
22
O Original importance
O Utterance: equal weight O Prosody: equal weight
U1 U2 U3 U4 U5 U6 U7 Utterance-Layer P1 P2 P3 P4 P5 P6 Prosody-Layer
scores propagated from utterances weighted by prosodic values
23
O Mathematical Formulation Utterance node U can get higher score when
corresponding to utterance U
24
O Mathematical Formulation Utterance node U can get higher score when
corresponding to utterance U Prosody node P can get higher score when
the prosodic feature P Unsupervised learn important utterances/prosodic features
Introduction Approach Experiments Conclusion
O Experimental Setup O Evaluation Metrics O Results O Analysis
25
Introduction Approach Experiments Conclusion
O Experimental Setup O Evaluation Metrics O Results O Analysis
26
O CMU Speech Meeting Corpus
O 10 meetings from 2006/04 – 2006/06 O #Speaker: 6 (total), 2-4 (each meeting) O WER = 44%
O Reference Summaries
O Manually labeled by two annotators as three
“noteworthiness” level (1-3)
O Extract utterances with level 3 as reference summaries
O Parameter Setting
O α = 0.9 O Extractive summary ratio = 10%, 20%, 30%
27
Introduction Approach Experiments Conclusion
O Experimental Setup O Evaluation Metrics O Results O Analysis
28
O ROUGE
O ROUGE-1 O F-measure of matched unigram between extracted
summary and reference summary
O ROUGE-L (Longest Common Subsequence) O F-measure of matched LCS between extracted summary
and reference summary O Average Relevance Score
O Average noteworthiness scores for the extracted
utterances
29
Introduction Approach Experiments Conclusion
O Experimental Setup O Evaluation Metrics O Results O Analysis
30
O Longest
O the longest utterances based on #tokens
O Begin
O the utterances that appear in the beginning
O Latent Topic Entropy (LTE)
O Estimate the “focus” of an utterance O Lower topic entropy represents more topically informative
O TFIDF
O Average TFIDF scores of all words in the utterances
31
2.20 2.25 2.30 2.35 2.40 2.45 2.50 Longest Begin LTE TFIDF Proposed
30.00 35.00 40.00 45.00 50.00 55.00 60.00 65.00 70.00 Longest Begin LTE TFIDF Proposed
ROUGE-1
30.00 35.00 40.00 45.00 50.00 55.00 60.00 65.00 70.00 Longest Begin LTE TFIDF Proposed
ROUGE-L
32
10% For 10% summaries, Begin performs best and proposed performs comparable results
2.20 2.25 2.30 2.35 2.40 2.45 2.50 Longest Begin LTE TFIDF Proposed
30.00 35.00 40.00 45.00 50.00 55.00 60.00 65.00 70.00 Longest Begin LTE TFIDF Proposed
ROUGE-1
30.00 35.00 40.00 45.00 50.00 55.00 60.00 65.00 70.00 Longest Begin LTE TFIDF Proposed
ROUGE-L
33
10% & 20% For 20% summaries, proposed approach outperforms all of the baselines
2.20 2.25 2.30 2.35 2.40 2.45 2.50 Longest Begin LTE TFIDF Proposed
30.00 35.00 40.00 45.00 50.00 55.00 60.00 65.00 70.00 Longest Begin LTE TFIDF Proposed
ROUGE-1
30.00 35.00 40.00 45.00 50.00 55.00 60.00 65.00 70.00 Longest Begin LTE TFIDF Proposed
ROUGE-L
34
10% & 20% & 30% For 30% summaries, proposed approach outperforms all of the baselines
Introduction Approach Experiments Conclusion
O Experimental Setup O Evaluation Metrics O Results O Analysis
35
O Based on converged scores for prosodic features
O Predictive features O number of pauses O min pitch O avg pitch O intensity O Least predictive features O the duration time O the number of syllables O the energy
36
Introduction Approach Experiments Conclusion
O Two-layer mutually reinforced random walk integrates
prosodic knowledge into an unsupervised model for speech summarization
O We show the first attempt at performing unsupervised
speech summarization without using lexical information
O Compared to some lexically derived baselines, the proposed
approach outperforms all of them but one scenario
37
38