IMTKU Emotional Dialogue System for Short Text Conversation at - - PowerPoint PPT Presentation

▶
imtku emotional dialogue system for short text
SMART_READER_LITE
LIVE PREVIEW

IMTKU Emotional Dialogue System for Short Text Conversation at - - PowerPoint PPT Presentation

Dec 31, 2023 375 likes •669 views

Tamkang University IMTKU Emotional Dialogue System for Short Text Conversation at NTCIR-14 STC-3 (CECG) Task Department of Information Management Tamkang University, Taiwan Min-Yuh Day Chi-Sheng Hung Yi-Jun Xie Jhih-Yi Chen Yu-Ling Kuo

slide-1
SLIDE 1

IMTKU Emotional Dialogue System for Short Text Conversation at NTCIR-14 STC-3 (CECG) Task

Department of Information Management Tamkang University, Taiwan

Tamkang University

Chi-Sheng Hung Min-Yuh Day Yi-Jun Xie Jhih-Yi Chen Yu-Ling Kuo Jian-Ting Lin

myday@mail.tku.edu.tw NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

slide-2
SLIDE 2

2

IMTKU Emotional Dialogue System for Short Text Conversation at NTCIR-14 STC-3 (CECG) Task

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

Tamkang University

slide-3
SLIDE 3

, -69261011 ,66

myday@mail.tku.edu.tw

Department of Information Management Tamkang University, Taiwan

Chun Tu

Min-Yuh Day

IMTKU Textual Entailment System for Recognizing Inference in Text at NTCIR-9 RITE

Tamkang University Tamkang University

2011

slide-4
SLIDE 4

IMTKU Textual Entailment System for Recognizing Inference in Text at NTCIR-10 RITE-2

Tamkang University

myday@mail.tku.edu.tw NTCIR-10 Conference, June 18-21, 2013, Tokyo, Japan

Department of Information Management Tamkang University, Taiwan

Chun Tu Hou-Cheng Vong Shih-Wei Wu Shih-Jhen Huang

Min-Yuh Day

Tamkang University

2013

slide-5
SLIDE 5

Ya-Jung Wang Min-Yuh Day

Che-Wei Hsu

Huai-Wen Hsu

En-Chun Tu

IMTKU Textual Entailment System for Recognizing Inference in Text at NTCIR-11 RITE-VAL

2014

Yu-Hsuan Tai

Shang-Yu Wu

Cheng-Chia Tsai

NTCIR-11 Conference, December 8-12, 2014, Tokyo, Japan

Tamkang University

Yu-An Lin

slide-6
SLIDE 6

IMTKU Question Answering System for World History Exams at NTCIR-12 QA Lab2

myday@mail.tku.edu.tw NTCIR-12 Conference, June 7-10, 2016, Tokyo, Japan

Min-Yuh Day Cheng-Chia Tsai Wei-Chun Chung Hsiu-Yuan Chang

Yuan-Jie Tsai Jin-Kun Lin Yue-Da Lin Wei-Ming Chen Yun-Da Tsai Cheng-Jhih Han Yi-Jing Lin Yu-Ming Guo Tzu-Jui Sun Yi-Heng Chiang Ching-Yuan Chien

Department of Information Management Tamkang University, Taiwan

Cheng-Hung Lee

Tamkang University

2016

Sagacity Technology

slide-7
SLIDE 7

IMTKU Question Answering System for World History Exams at NTCIR-13 QALab-3

myday@mail.tku.edu.tw NTCIR-13 Conference, December 5-8, 2017, Tokyo, Japan

Department of Information Management Tamkang University, Taiwan

Tamkang University

Yue-Da Lin I-Hsuan Huang Wanchu Huang Shi-Ya Zheng Tz-Rung Chen Min-Chun Kuo

Yi-Jing Lin

2017

Min-Yuh Day

Chao-Yu Chen

slide-8
SLIDE 8

IMTKU Emotional Dialogue System for Short Text Conversation at NTCIR-14 STC-3 (CECG) Task

myday@mail.tku.edu.tw

Department of Information Management Tamkang University, Taiwan

Tamkang University

2019

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

Chi-Sheng Hung Min-Yuh Day Yi-Jun Xie Jhih-Yi Chen Yu-Ling Kuo Jian-Ting Lin

slide-9
SLIDE 9

Outline

  • IMTKU Emotional Dialogue System Architecture
  • IMTKU System Description
  • Performance
  • Discussions and Conclusions

9

Tamkang University

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

slide-10
SLIDE 10

Highlights

  • IMTKU

(Information Management at TamKang University)

Emotional Dialogue System for Short Text Conversation at NTCIR-14 STC-3 (CECG) Task

  • IMTKU Submitted two runs for NTCIR-14 STC-3

(CECG) –IMTKU1: Retrieval-based model –IMTKU2: Generation-based model

10

Tamkang University

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

slide-11
SLIDE 11

IMTKU Emotional Dialogue System Architecture

11

Retrieval-Based Model Generation- Based Model Emotion

Classification

Model Response Ranking

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

4 3 1 2

slide-12
SLIDE 12

The system architecture of IMTKU retrieval-based model for NTCIR-14 STC-3

12

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

Post Retrieval- Based Response Corpus Keyword Boolean Query Solr Matching Distinct Result Data Building Index Word Segmentation Emotion Classification Word2Vec Similarity Ranking Emotion Matching

Retrieval Model

Retrieval-Based Model

1

slide-13
SLIDE 13

The system architecture of IMTKU generation-based model for NTCIR-14 STC-3

13

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

Post Generation-Based Response Training Data Seq2seq model Word Embedding Trained Model Building Word Index Word Segmentation Short Text Emotion Classifier Word2Vec Similarity Ranking Emotion Matching Training Data

Generation-Based Model

Generative Model

2

slide-14
SLIDE 14

The system architecture of IMTKU emotion classification model for NTCIR-14 STC-3

14

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

Corpus Emotion Prediction Emotion Classification Model Testing Dataset Training Dataset MLP LSTM BiLSTM Emotion Classification

Emotion Classification Model

3

slide-15
SLIDE 15

15

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

STC3 Corpus Chinese Segmentation using Jieba Stop Words Removal Word2Vec 1.2 million data (300 dimensions) Vector of Corpus

Response Ranking

The system architecture of IMTKU Response Ranking for NTCIR-14 STC-3

4

slide-16
SLIDE 16

16

Word2Vec Similarity Ranking

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

x = (Sp*α+Sr*β)/2

Sp : The similarity score of the new post and the corpus post Sr : The similarity score of the new post and the corpus response α : Weight = 0.2 β : Weight = 0.8

slide-17
SLIDE 17

STC-3 CECG Evaluation: Total Score and Average Score

!"#$% &'"() = +

,-. /

, ∗ 123, 4 : score corresponding to Label 5678 : total number of questions marked with Label i 9:;<=>; ?@A<; = ∑,-.

/

, ∗ 123, CD i : score corresponding to Label, 5678 : total number of questions marked with Label i Nt: total number of all questions

17

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

slide-18
SLIDE 18

Affective Conversational Robot Index (ACR Index) for Evaluating Emotional Dialogue System

!"# $%&'( = ∑+,-

.

+ ∗ 012+ 34 ∗ 56((+)

i : score corresponding to Label 9:;< : total number of questions marked with Label i Nt : total number of all questions ;=>(?) : maximum value of ?.

18

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

slide-19
SLIDE 19

Emotion Classification Model

19

Emotion Classification Model Accuracy

Multi-Layer Perceptron (MLP)

73.9%

Long Short Term Memory (LSTM)

86.4%

Bi-directional Long Short Term Memory (Bi-LSTM)

87.6%

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

* Five Emotion Classes 1: Like, 2: Sadness, 3: Disgust, 4: Anger, 5: Happiness

slide-20
SLIDE 20

20

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

IMTKU at NTCIR-14 STC-3 Performance (Formal Run)

Run

The Result Like Sad

Overall Score Average Score Overall Score Average Score Overall Score Average Score

IMTKU-1 592 0.592 127 0.635 120 0.6 IMTKU-2 60 0.06 8 0.04 17 0.085

Run

Disgust Anger Happy

Overall Score Average Score Overall Score Average Score Overall Score Average Score

IMTKU-1 97 0.485 88 0.44 160 0.8 IMTKU-2 7 0.035 11 0.055 17 0.085

slide-21
SLIDE 21

21

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

IMTKU at NTCIR-14 STC-3 Evaluators Consistency (Self-Evaluation)

Evaluators

All agree Two agree All disagree

# Pairs

1366 603 31

Percentage (%)

68.3% 30.2% 1.5% 98.5%

slide-22
SLIDE 22

22

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

IMTKU at NTCIR-14 STC-3 Performance (Self-Evaluation)

Run

Label0 Label1 Label2

Total Score Average Score ACR Index

Retrieval-based Model

304 209 487 1183 1.183 0.591

Generation-based Model

875 90 35 160 0.16 0.08

slide-23
SLIDE 23
  • IMTKU Submitted two runs for STC-3 (CECG)

– IMTKU-1: Retrieval-based model – IMTKU-2: Generation-based model

  • The performance of retrieval-based model is

superior to the generation-based model.

23

Conclusions

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

slide-24
SLIDE 24
  • We have developed an emotional dialogue

system that integrates retrieval-based model, generation-based model, and emotion classification model with deep learning approach for short text conversation.

  • We have proposed an Affective Conversational

Robot Index (ACR Index) for evaluating emotional dialogue system.

24

Contributions

NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

slide-25
SLIDE 25

IEEE EM-RITE 2019

The 8th IEEE International Workshop on Empirical Methods for Recognizing Inference in TExt (IEEE EM-RITE 2019)

In conjunction with IEEE IRI 2019 Los Angeles, California, USA July 30 - August 1, 2019 https://sites.google.com/view/emrite2019

25

slide-26
SLIDE 26

MSNDS 2019

The 10th International Workshop on Mining and Analyzing Social Networks for Decision Support (MSNDS 2019)

In conjunction with IEEE/ACM ASONAM 2019 Vancouver, Canada 27-30 August, 2019

https://sites.google.com/view/msnds2019/

26

slide-27
SLIDE 27

The 31st Annual Conference on Computational Linguistics and Speech Processing (ROCLING 2019)

New Taipei City, Taiwan, October 3-5, 2019

27

ROCLING 2019

https://sites.google.com/view/rocling2019/

slide-28
SLIDE 28

IMTKU Emotional Dialogue System for Short Text Conversation at NTCIR-14 STC-3 (CECG) Task

Department of Information Management Tamkang University, Taiwan

Tamkang University

2019

Q&A

myday@mail.tku.edu.tw NTCIR-14 Conference, June 10-13, 2019, Tokyo, Japan

Chi-Sheng Hung Min-Yuh Day Yi-Jun Xie Jhih-Yi Chen Yu-Ling Kuo Jian-Ting Lin