Automatic Disfluency Automatic Disfluency Detection in Multi-party - - PowerPoint PPT Presentation

▶
automatic disfluency automatic disfluency detection in
SMART_READER_LITE
LIVE PREVIEW

Automatic Disfluency Automatic Disfluency Detection in Multi-party - - PowerPoint PPT Presentation

Mar 30, 2024 41 likes •324 views

German Research Center for Artificial Intelligence GmbH Automatic Disfluency Automatic Disfluency Detection in Multi-party Detection in Multi-party www.amiproject.org Conversations Conversations Feast, 30th September 2009 , 30th September

slide-1
SLIDE 1

www.amiproject.org

German Research Center for Artificial Intelligence GmbH

Feast Feast, 30th September 2009 , 30th September 2009 Sebastian Sebastian Germesin Germesin

Automatic Disfluency Automatic Disfluency Detection in Multi-party Detection in Multi-party Conversations Conversations

slide-2
SLIDE 2

www.amiproject.org

Sebastian Germesin September 09 2

German Research Center for Artificial Intelligence GmbH

Outline Outline

  • Motivation
  • Theoretical Background
  • Data (AMI Corpus)
  • Disfluency Detection System
  • Hybrid Classification Approach
  • Self-arranging Modules
  • Experimental Results
  • Conclusions & Outlook
slide-3
SLIDE 3

www.amiproject.org

Sebastian Germesin September 09 3

German Research Center for Artificial Intelligence GmbH

Motivation Motivation

Example Example

slide-4
SLIDE 4

www.amiproject.org

Sebastian Germesin September 09 4

German Research Center for Artificial Intelligence GmbH

Motivation Motivation

  • Have to detect (and clean) disfluencies

in the transcribed speech

  • Readability
  • Transcription
  • Extractive Summarization
  • Post-Processing
  • NLP-systems’ performance drop when faced with

disfluent speech

  • Human detector?
  • Too expensive!
  • Too slow!

⇒Automatic Detection System!

slide-5
SLIDE 5

www.amiproject.org

Sebastian Germesin September 09 5

German Research Center for Artificial Intelligence GmbH

Theoretical Background Theoretical Background

“Disfluencies are syntactical and grammatical [speech] errors that occur in spoken but not in written language.” [Besser, 2006]

Definition Definition

slide-6
SLIDE 6

www.amiproject.org

Sebastian Germesin September 09 6

German Research Center for Artificial Intelligence GmbH

Theoretical Background Theoretical Background

“The cat uh the dog sneaks around the corner.”

Terminology Terminology

slide-7
SLIDE 7

www.amiproject.org

Sebastian Germesin September 09 7

German Research Center for Artificial Intelligence GmbH

Theoretical Background Theoretical Background

“The cat uh the dog sneaks around the corner.”

Terminology Terminology

Reparandum

slide-8
SLIDE 8

www.amiproject.org

Sebastian Germesin September 09 8

German Research Center for Artificial Intelligence GmbH

Theoretical Background Theoretical Background

“The cat uh the dog sneaks around the corner.”

Terminology Terminology

Reparandum Interregnum

slide-9
SLIDE 9

www.amiproject.org

Sebastian Germesin September 09 9

German Research Center for Artificial Intelligence GmbH

Theoretical Background Theoretical Background

“The cat uh the dog sneaks around the corner.”

Terminology Terminology

Reparandum Reparans Interregnum

complex

slide-10
SLIDE 10

www.amiproject.org

Sebastian Germesin September 09 10

German Research Center for Artificial Intelligence GmbH

Theoretical Background Theoretical Background

“The d dog sneaks around the corner.”

Terminology Terminology

Reparandum

simple

slide-11
SLIDE 11

www.amiproject.org

Sebastian Germesin September 09 11

German Research Center for Artificial Intelligence GmbH

Theoretical Background Theoretical Background

All Types All Types

Simple disfluencies

slide-12
SLIDE 12

www.amiproject.org

Sebastian Germesin September 09 12

German Research Center for Artificial Intelligence GmbH

Data Data

  • AMI meeting corpus
  • 135 meetings (~ 100 hours speech)
  • 4 participants
  • task: design a remote control
  • freely interaction
  • Many annotations, e.g.:
  • Transcribed speech
  • Dialogue acts
  • Gestures
  • ...

quantitative quantitative

slide-13
SLIDE 13

www.amiproject.org

Sebastian Germesin September 09 13

German Research Center for Artificial Intelligence GmbH

Data Data

  • 45 meeting enriched with disfluency

annotation

  • 31,000 Disfluencies
  • 15.8% erroneous words
  • 41.5% disfluent Dialogue Acts
  • 80% (33) for training
  • 20% (12) for evaluation

quantitative quantitative

slide-14
SLIDE 14

www.amiproject.org

Sebastian Germesin September 09 14

German Research Center for Artificial Intelligence GmbH

Data Data

  • Discovered a heterogeneity towards the

strictness of different disfluency types

  • 1. Some disfluencies have strict structure
  • ex.: Repetition

: “The cat the cat plays “

  • 2. Some other disfluencies have also strict structure but

this structure is very common in natural language

  • ex.: Replacement : “The dog the cat plays“
  • ex.: Fluent

: “The dog the cat and the bird play”

  • 3. Some other disfluencies have no obvious structure
  • ex.: Disruptions

: “The dog the cat and“

  • ex.: Order

: “The plays cat”

qualitative qualitative

slide-15
SLIDE 15

www.amiproject.org

Sebastian Germesin September 09 15

German Research Center for Artificial Intelligence GmbH

Automatic System Automatic System

Design Question Design Question

  • Can we leverage the heterogeneity
  • f disfluencies for their detection?

→Yes!

→ Use modules for subsets of disfluencies → Use different feature-sets for each module (depending on the disfluency types) → Find “optimal” classifier for each module

slide-16
SLIDE 16

www.amiproject.org

Sebastian Germesin September 09 16

German Research Center for Artificial Intelligence GmbH

Automatic System Automatic System

Hybrid Modules Hybrid Modules

  • SHS:
  • Stuttering, Hesitation, Slip-of-the-Tongue
  • REP:
  • Repetition
  • DNE:
  • Discourse Marker, Explicit Editing Term
  • DEL:
  • Deletion
  • REV:
  • Insertion, Replacement, Restart, Other
slide-17
SLIDE 17

www.amiproject.org

Sebastian Germesin September 09 17

German Research Center for Artificial Intelligence GmbH

How to How to combine the modules? combine the modules?

slide-18
SLIDE 18

www.amiproject.org

Sebastian Germesin September 09 18

German Research Center for Artificial Intelligence GmbH

Training Process Training Process

Self-arranging Modules Self-arranging Modules

  • Immense search space
  • #(modules) * #(classifier) * placeInSystem
  • Solution(s):
  • Old system:
  • Choosen manually
  • Current system:
  • Automatically trained

1.Use greedy hill-climbing

– Use weight for errors to improve Precision!

2.Reduce classifier library

– Take 10% results in maximal performance loss

  • f 2.3% (depending on the module)
slide-19
SLIDE 19

www.amiproject.org

Sebastian Germesin September 09 19

German Research Center for Artificial Intelligence GmbH

GroDi GroDi

Greedy Hill-Climbing Greedy Hill-Climbing

slide-20
SLIDE 20

www.amiproject.org

Sebastian Germesin September 09 20

German Research Center for Artificial Intelligence GmbH

Training Process Training Process

Self-arranging Modules Self-arranging Modules

  • Immense search space
  • #(modules) * #(classifier) * placeInSystem
  • Solution(s):
  • Old system:
  • Choosen manually
  • Current system:
  • Automatically trained

1.Use greedy hill-climbing

– Use weight for errors to improve Precision!

2.Reduce classifier library

– Take 10% results in maximal performance loss

  • f 2.3% (depending on the module)
slide-21
SLIDE 21

www.amiproject.org

Sebastian Germesin September 09 21

German Research Center for Artificial Intelligence GmbH

GroDi GroDi

Performance-Curve of J48 Performance-Curve of J48

Best: J48 "-L -U -M 2 -A"

slide-22
SLIDE 22

www.amiproject.org

Sebastian Germesin September 09 22

German Research Center for Artificial Intelligence GmbH

Experimental Results Experimental Results

93.5 % 94.5 % 12 m. 94.7 % 95.1 % 6 m. 33 m. new 0.11 94.8 % 95.3 % 6 m. 22 m. 0.42 90.5 % 92.9 % 6 m. 22 m.

  • ld

83.3 % 88.6 % 12 m. 0.00 85.7 % 90.3 % 6 m.

  • baseline

RT-factor

  • avg. F1

Accuracy Eval. data Train. data System

slide-23
SLIDE 23

www.amiproject.org

Sebastian Germesin September 09 23

German Research Center for Artificial Intelligence GmbH

Conclusions Conclusions

  • Aims:
  • Development of a system that automatically

detects a broad set of disfluencies

  • Fully automatic learning process
  • Robust and Fast
  • Achievements:
  • Stand-alone tool for detection of disfluencies:

GroDi - Get rid of Disfluencies

  • Self-arranging modules
  • Detection rate: 95% Accuracy
  • Real-time factor of 0.11
slide-24
SLIDE 24

www.amiproject.org

Sebastian Germesin September 09 24

German Research Center for Artificial Intelligence GmbH

Outlook Outlook

  • Develop module(s) for the detection of

Mistake, Order, Omission

  • Embed other learning approaches, e.g.:
  • Conditional Random Fields
  • HMMs
  • Use other corpus like, e.g., Switchboard
slide-25
SLIDE 25

www.amiproject.org

Sebastian Germesin September 09 25

German Research Center for Artificial Intelligence GmbH

Thank you! Thank you!

slide-26
SLIDE 26

www.amiproject.org

Sebastian Germesin September 09 26

German Research Center for Artificial Intelligence GmbH

Demo? Demo?

slide-27
SLIDE 27

www.amiproject.org

Sebastian Germesin September 09 27

German Research Center for Artificial Intelligence GmbH

GroDi GroDi

  • Diff. Module Arrangements
  • Diff. Module Arrangements
slide-28
SLIDE 28

www.amiproject.org

Sebastian Germesin September 09 28

German Research Center for Artificial Intelligence GmbH

GroDi GroDi

 Used technologies  WEKA toolkit for machine learning  Maximum Entropy classifier from Stanford NLP group  CRF Tagger from http://crftagger.sourceforge.net/  Features for machine learning:

 Lexical:

words, lexical parallelism, (POS-Tags)

 Prosodic: duration, pauses, pitch, energy  Dynamic:

disfluency types of surrounding words

 Speaker:

age, role in meeting, native language