IIT-UHH at SemEval-2017 Task 3: Exploring Multiple Features for - - PowerPoint PPT Presentation

▶
iit uhh at semeval 2017 task 3 exploring multiple
SMART_READER_LITE
LIVE PREVIEW

IIT-UHH at SemEval-2017 Task 3: Exploring Multiple Features for - - PowerPoint PPT Presentation

Feb 23, 2024 339 likes •691 views

IIT-UHH at SemEval-2017 Task 3: Exploring Multiple Features for Community Question Answering and Implicit Dialogue Identification Titas Nandi 1 , Chris Biemann 2 , Seid Muhie Yimam 2 , Deepak Gupta 1 , Sarah Kohail 2 , Asif Ekbal 1 and Pushpak

slide-1
SLIDE 1

1/34

IIT-UHH at SemEval-2017 Task 3: Exploring Multiple Features for Community Question Answering and Implicit Dialogue Identification

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1

1Indian Institute of Technology Patna, India 2Universit¨

at Hamburg, Germany {titas.ee13,deepak.pcs16,asif,pb}@iitp.ac.in {biemann,yimam,kohail}@informatik.uni-hamburg.de

Presented by Alexander Panchenko2 August 3, 2017

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-2
SLIDE 2

2/34

Outline

1

Task Description Structure of the Task Related Work

2

System Description Basic Features Implicit Dialogue Identification Statistical Model

3

Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017

4

Conclusions

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-3
SLIDE 3

3/34

Outline

1

Task Description Structure of the Task Related Work

2

System Description Basic Features Implicit Dialogue Identification Statistical Model

3

Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017

4

Conclusions

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-4
SLIDE 4

4/34

SemEval 2017 Task 3: the Three Sub-Tasks

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-5
SLIDE 5

5/34

Outline

1

Task Description Structure of the Task Related Work

2

System Description Basic Features Implicit Dialogue Identification Statistical Model

3

Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017

4

Conclusions

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-6
SLIDE 6

6/34

Related Work

Useful ideas from the best systems of 2015 and 2016 tasks:

Belinkov (2015): word vectors and meta-data features Nicosia (2015): derived features from a comment in the context of the entire thread Filice (2016): stacking classifiers across subtasks

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-7
SLIDE 7

7/34

Outline of the Method

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-8
SLIDE 8

8/34

Outline

1

Task Description Structure of the Task Related Work

2

System Description Basic Features Implicit Dialogue Identification Statistical Model

3

Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017

4

Conclusions

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-9
SLIDE 9

9/34

String Similarity Features

String similarity between a question-comment/question pair:

Jaro-Winkler Levenshtein Jaccard Sorensen-Dice n-gram LCS

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-10
SLIDE 10

10/34

Domain (Task) Specific Features

If a comment by asker of the question is an acknowledgement Position of comment in the thread Coverage (the ratio of the number of tokens) of question by the comment and comment by the question Presence of URLs, emails or HTML tags

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-11
SLIDE 11

11/34

Word Embedding Features

Trained word embedding model using Word2Vec on unannotated data Sentence vectors

averaging word vectors wscore = wquestion − wcomment

Distance scores

Based on the computed sentence vectors

Cosine Distance (1 − cos) Manhattan Distance Euclidean Distance

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-12
SLIDE 12

12/34

Topic Modeling Features

Trained LDA Topic model using Mallet tool on training data Extracted the 20 most relevant topics for the data Topic Vector of a Question/Comment

wscore = wquestion − wcomment

Topic Vocabulary of a Question/Comment

Vocabulary(T) =

10

  • i=1

topic words(ti) where ti is one of the top topics for comment/question T.

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-13
SLIDE 13

13/34

Keyword and Named Entity Features

Extracted keywords or focus words from question and comment using the RAKE algorithm (Rose et al., 2010)

Keyword match between question and comment

Extracted Named Entities from question and comment Entity tags consisted of LOCATION, PERSON, ORGANIZATION, DATE, MONEY, PERCENT and TIME

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-14
SLIDE 14

14/34

Outline

1

Task Description Structure of the Task Related Work

2

System Description Basic Features Implicit Dialogue Identification Statistical Model

3

Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017

4

Conclusions

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-15
SLIDE 15

15/34

Implicit Dialogue Identification

Identified implicit dialogues among users

User Interaction Graph

Each user is in dialogue with some other user who came before him/her

Asker - desirable Other users - not desirable

Vertices - Users in a comment thread Edges - Directed edges showing interaction Edge weight - the level of interaction

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-16
SLIDE 16

16/34

Implicit Dialogue Identification: an Example

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-17
SLIDE 17

17/34

Implicit Dialogue Identification: an Example

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-18
SLIDE 18

18/34

Implicit Dialogue Identification: an Example

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-19
SLIDE 19

19/34

Implicit Dialogue Identification: an Example

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-20
SLIDE 20

20/34

Implicit Dialogue Identification: an Example

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-21
SLIDE 21

21/34

Implicit Dialogue Identification: an Example

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-22
SLIDE 22

22/34

Implicit Dialogue Identification: an Example

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-23
SLIDE 23

23/34

Computing Edge Weights

The edge weight is computed (or revised) on the basis of:

Explicit dialogue score. If one user refers the other explicitly, then add 1.0 to the edge score. Embedding score. For each word in a comment, find the word in the other comment that has maximum cosine similarity with it. Then finally average all those max cosine scores to get a value. Topic score. The cosine of topic vectors of the two comments.

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-24
SLIDE 24

24/34 Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-25
SLIDE 25

25/34

Outline

1

Task Description Structure of the Task Related Work

2

System Description Basic Features Implicit Dialogue Identification Statistical Model

3

Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017

4

Conclusions

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-26
SLIDE 26

26/34

Classification Model

Normalized all feature values with Z-scores Feature Selection using wrapper methods to maximize accuracy on the development set Used SVM confidence probabilities for ranking (RBF kernel)

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-27
SLIDE 27

27/34

Subtask C: Similarity of Questions and External Comments

Oversample the data using the SMOTE (Chawla, 2002) technique and run classifier on original question - external comment pair Stacking across tasks: the SVM scores of all three subtasks are combined: Score C = log(SVM Score) + log(Score A) + log(Score B)

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-28
SLIDE 28

28/34

Outline

1

Task Description Structure of the Task Related Work

2

System Description Basic Features Implicit Dialogue Identification Statistical Model

3

Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017

4

Conclusions

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-29
SLIDE 29

29/34

Feature Ablation Results: Impact of Different Feature Sets

Features Development Set 2017 Subtask A MAP P R F1 Acc All Features 65.50 58.43 62.71 60.50 72.54 All — string 65.53 57.84 62.71 60.18 72.17 All — embedding 62.11 53.03 53.42 53.23 68.52 All — domain 61.85 54.46 54.52 54.49 69.47 All — topic 65.15 59.02 61.98 60.47 72.83 All — keyword 65.73 57.98 62.59 60.20 72.25 IR Baseline 53.84

  • Runs

Test Set 2017 Subtask A MAP P R F1 Acc Primary 86.88 73.37 74.52 73.94 72.70 Contrastive 1 86.35 79.42 51.94 62.80 68.02 Contrastive 2 85.24 81.22 57.65 67.43 71.06

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-30
SLIDE 30

30/34

Outline

1

Task Description Structure of the Task Related Work

2

System Description Basic Features Implicit Dialogue Identification Statistical Model

3

Results Results on Different Feature Sets Comparison with Other Teams at SemEval 2017

4

Conclusions

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-31
SLIDE 31

31/34

Comparison of Results on Subtask A at SemEval 2017

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-32
SLIDE 32

32/34

Comparison of Results on Subtask C at SemEval 2017

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-33
SLIDE 33

33/34

Observations and Conclusions

Small in-domain texts are better for training, compared to large out-of-domain pre-trained GoogleNews embeddings Most instrumental are features based on:

User dialogues Word embeddings

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34

slide-34
SLIDE 34

34/34

Thank you! Any questions from the community?

Titas Nandi1, Chris Biemann2, Seid Muhie Yimam2, Deepak Gupta1, Sarah Kohail2, Asif Ekbal1 and Pushpak Bhattacharyya1 (IIT Patna) Answer Selection and Ranking in CQA sites Presented by Alexander Panchenko2 August / 34