Agreement Detection in Agreement Detection in Multiparty - - PowerPoint PPT Presentation

▶
agreement detection in agreement detection in multiparty
SMART_READER_LITE
LIVE PREVIEW

Agreement Detection in Agreement Detection in Multiparty - - PowerPoint PPT Presentation

Jun 07, 2023 205 likes •472 views

German Research Center for Artificial Intelligence GmbH www.amiproject.org Agreement Detection in Agreement Detection in Multiparty Conversations Multiparty Conversations FEAST, FEAST, 21st October October 2009 2009 21st Sebastian

slide-1
SLIDE 1

www.amiproject.org

German Research Center for Artificial Intelligence GmbH

FEAST, FEAST, 21st 21st October October 2009 2009 Sebastian Sebastian Germesin Germesin Theresa Wilson Theresa Wilson

Agreement Detection in Agreement Detection in Multiparty Conversations Multiparty Conversations

slide-2
SLIDE 2

www.amiproject.org

Sebastian Germesin November 09 2

German Research Center for Artificial Intelligence GmbH

Motivation Motivation

  • Growing interest in extracting and

summarising information from meetings

  • One important type of information are

agreements / disagreements

⇒ Development of an automatic

detection system!

slide-3
SLIDE 3

www.amiproject.org

Sebastian Germesin November 09 3

German Research Center for Artificial Intelligence GmbH

Agreements in this work Agreements in this work

  • What do we mean by “(dis)agreements”?
  • Utterances where a speaker agrees/disagrees

with an idea/opinion/sentiment of another speaker [Wilson, 2008]

  • Agreements in the context of multi-party

conversations

slide-4
SLIDE 4

www.amiproject.org

Sebastian Germesin November 09 4

German Research Center for Artificial Intelligence GmbH

Example Example

Example:

... Industrial Designer: Finding them is really a pain. Marketing Expert: Hm. Industrial Designer: I mean, when you want it, it’s kicked under the table or so. Project Manager: Yeah, that’s right. ...

agreement target addressee

slide-5
SLIDE 5

www.amiproject.org

Sebastian Germesin November 09 5

German Research Center for Artificial Intelligence GmbH

Data (AMI Corpus) Data (AMI Corpus)

AMI meeting corpus

  • 100 hours of audio

and video recorded meetings

  • 4 participants
  • (guided) task:

“Design a remote control!”

  • Variety of annotations, e.g.:
  • Transcribed speech, ASR output, ...
  • Dialogue Acts, Disfluencies, ...
  • Head- & Hand-Gestures, VFOA, ...
  • and...
slide-6
SLIDE 6

www.amiproject.org

Sebastian Germesin November 09 6

German Research Center for Artificial Intelligence GmbH

Data (AMI Corpus) Data (AMI Corpus)

(Dis)Agreement Annotations

  • 20 AMI meetings have been annotated
  • Word-based annotation scheme
  • 16 for training, 4 for evaluation
  • 636 agreements / 70 disagreements
  • Aligning to DA segments to preserve

comparability with ICSI research ([Hillard 03], [Galley 04], [Hahn 06]):

  • 19,043 segments
  • 876 segments contain agreements
  • 118 segments contain disagreements

4.6% : 0.6% : 94.8% (agree : disagree : other)

slide-7
SLIDE 7

www.amiproject.org

Sebastian Germesin November 09 7

German Research Center for Artificial Intelligence GmbH

Automatic Detection Automatic Detection System System

“Finding them is really a pain!” “Yeah” A B

“agreement”

Target Speaker Detection

B agrees with A

Agree Detection

slide-8
SLIDE 8

www.amiproject.org

Sebastian Germesin November 09 8

German Research Center for Artificial Intelligence GmbH

Automatic Detection Automatic Detection System System

“agreement”, “disagreement”, “other”

Decision Tree

  • Cond. Random

Field Agreement Detection

Cope with skewed classes Use HPR-output as feature

High-Precision Rules

slide-9
SLIDE 9

www.amiproject.org

Sebastian Germesin November 09 9

German Research Center for Artificial Intelligence GmbH

Automatic Detection Automatic Detection System System

HPRs - High Precision Rules

  • Implement prior knowledge in simple

set of rules before actual classification

  • Reduce data skewness
  • Rule-Types:
  • Target-Content
  • Dialogue Act (DA) Labels
  • Subjective Content
  • N-grams
  • ther
  • ther

agree unclass

slide-10
SLIDE 10

www.amiproject.org

Sebastian Germesin November 09 10

German Research Center for Artificial Intelligence GmbH

Automatic Detection Automatic Detection System System

Decision Tree Conditional Random Fields

  • C4.5 implementation from

WEKA Toolkit

  • Lexical features
  • Prosodic features
  • Structural features
  • HPR-output
  • Contextual features
  • CRF implementation from

Stanford NER

  • Lexical features
  • Prosodic features
  • Structural features
  • HPR-output
slide-11
SLIDE 11

www.amiproject.org

Sebastian Germesin November 09 11

German Research Center for Artificial Intelligence GmbH

4 Agreement 1890 Pre-Class. 5 141 Length 1 Silence 2 274 DA-Label (tar) 2 295 DA-Label (src) 12 740 No-Target wrong correct name 4 agree 3362

  • ther

554 “unclass” 3920 segments

High-Precision-Rules

Experimental Results Experimental Results

slide-12
SLIDE 12

www.amiproject.org

Sebastian Germesin November 09 12

German Research Center for Artificial Intelligence GmbH

Experimental Results Experimental Results

Agreement Detection

0.03 0.01 0.005 0.005 0.0 RT Factor 0.40 0.36 0.39 0.40 0.0 Kappa 45.2 36.8 43.5 44.5 0.0 F1 [%] 42.4 31.1 34.6 36.3 0.0 Rec [%] 48.5 45.0 58.8 57.6 0.0 Prec [%] 97.8 97.8 98.1 98.0 97.8 Acc [%] w/o HPRs w/ HPRs w/o HPRs w/ HPRs DTs CRFs Baseline

slide-13
SLIDE 13

www.amiproject.org

Sebastian Germesin November 09 13

German Research Center for Artificial Intelligence GmbH

Automatic Detection Automatic Detection System System

“Finding them is really a pain!” “Yeah” A B

“agreement” B agrees with A

Target Speaker Detection Agree Detection

slide-14
SLIDE 14

www.amiproject.org

Sebastian Germesin November 09 14

German Research Center for Artificial Intelligence GmbH

Automatic Detection Automatic Detection System System

“Finding them is really a pain!” “Yeah” A B

“agreement” B agrees with A

Agree Detection Target Speaker Detection

slide-15
SLIDE 15

www.amiproject.org

Sebastian Germesin November 09 15

German Research Center for Artificial Intelligence GmbH

Automatic Detection Automatic Detection System System

Target Speaker Detection:

  • Novelty in agreement detection
  • Preliminary experiments using

Adjacency Pair-Annotation

slide-16
SLIDE 16

www.amiproject.org

Sebastian Germesin November 09 16

German Research Center for Artificial Intelligence GmbH

Automatic Detection Automatic Detection System System

Internal Representation:

  • Use speaker-dependent (relative) labels
  • 0 for current speaker
  • 1 for previous speaker
  • ...
  • Let’s see this in the example:
slide-17
SLIDE 17

www.amiproject.org

Sebastian Germesin November 09 17

German Research Center for Artificial Intelligence GmbH

Example Example

Example:

... Index ‘1’: Finding them is really a pain. Index ‘2’: Hm. Index ‘1’: I mean, when you want it, it’s kicked under the table or so. Index ‘0’: Yeah, that’s right. ...

agreement target addressee

slide-18
SLIDE 18

www.amiproject.org

Sebastian Germesin November 09 18

German Research Center for Artificial Intelligence GmbH

Automatic Detection Automatic Detection System System

slide-19
SLIDE 19

www.amiproject.org

Sebastian Germesin November 09 19

German Research Center for Artificial Intelligence GmbH

Automatic Detection Automatic Detection System System

Target Detection:

  • Use structural information from Adjacency Pairs

to improve target speaker detection! (backward-window of 10 segments)

  • Fall back to speaker ‘1’ if no AP is available
slide-20
SLIDE 20

www.amiproject.org

Sebastian Germesin November 09 20

German Research Center for Artificial Intelligence GmbH

Experimental Results Experimental Results

Baseline

00.0 12 3 00.0 78 2 0.00 78.0 64.5 164 1 real k F1 Acc 3 2 1 classified as

slide-21
SLIDE 21

www.amiproject.org

Sebastian Germesin November 09 21

German Research Center for Artificial Intelligence GmbH

Experimental Results Experimental Results

Using AP-Information

14.2 1 1 10 3 67.2 40 38 2 0.52 86.9 80.3 1 163 1 real k F1 Acc 3 2 1 classified as

56% improvement

slide-22
SLIDE 22

www.amiproject.org

Sebastian Germesin November 09 22

German Research Center for Artificial Intelligence GmbH

Conclusion Conclusion

Developed a system for agreement detection:

  • Utilized a variety multi-modal, heterogeneous

features (e.g., lexical, prosodic, structural)

  • Investigated the use of High-Precision Rules

to deal with imbalanced class distribution

  • Evaluated two different types of machine learning

techniques

  • Conditional Random Fields
  • Decision Trees
  • Accuracy: 98.1%
  • Kappa

: 0.40

  • CRF: higher Precision
  • DT: higher Recall
slide-23
SLIDE 23

www.amiproject.org

Sebastian Germesin November 09 23

German Research Center for Artificial Intelligence GmbH

Conclusion (cont.) Conclusion (cont.)

Novelty: Target Speaker detection!

  • Introduced preliminary approach, using structural

information from the adjacency pairs

  • 56% relative improvement over the baseline
  • Kappa value of 0.52
slide-24
SLIDE 24

www.amiproject.org

Sebastian Germesin November 09 24

German Research Center for Artificial Intelligence GmbH

Outlook Outlook

  • Separate detection of agreements and

disagreements

  • Separate detection of one-word and

multi-word agreements

  • Use machine learning for addressee

detection

  • Use automatic annotations
  • Use other features (e.g., visual cues)
  • Care about data skewness
slide-25
SLIDE 25

www.amiproject.org

Sebastian Germesin November 09 25

German Research Center for Artificial Intelligence GmbH

Thank you! Thank you!