Social Media Argumentation Mining: The Quest for Deliberateness in - - PowerPoint PPT Presentation

Social Media Argumentation Mining: The Quest for Deliberateness in Raucousnes

Jan Šnajder Joint work with Filip Boltuži´ c

Text Analysis and Knowledge Engineering Lab FER, University of Zagreb

Dagstuhl 19 April 2016

1 / 36

Social media argumentation mining

Much work on AM focuses on well-structured, edited text: e.g. legal documents [Walton, 2005] and scientific documents [Jiménez-Aleixandre and Erduran, 2007] Recent interest in AM from social media texts: online debates [Cabrio and Villata, 2012, Habernal et al., , Boltuži´ c and Šnajder, 2014], discussions on regulations [Park and Cardie, 2014], and product reviews [Ghosh et al., 2014]

Online debates – a somewhat controlled setting Comment boards, product reviews, microblogs – less controlled

2 / 36

Outline

1

Argumentation in social media

2

Argument recognition

3

Argument clustering

4

Argument similarity

3 / 36

Outline

1

Argumentation in social media

2

Argument recognition

3

Argument clustering

4

Argument similarity

4 / 36

User comments

Our interest: AM from user comments (not necessarily debates)

Yahoo News: User comment on Trump rally event

The President we have now divided our country and put his ego first instead of the people. Trump hasn’t divided the country that’s why he has so many people behind him. We want someone who is not afraid

• f the politics in Washington and change our policies with dealing with
• ther countries.

No predefined topic (topics emerge ad hoc) Mostly monological

5 / 36

Why analyze this?

To the extent in which we are interested in analyzing opinion of

• ther people, we should also be interested in analyzing the

underlying reasons to fully apprehend their opinions

Our (long term) goal

Analyze, on a large scale, what arguments people use to express their stance, including the faulty arguments, as well as all the (often implicit) premises (reflecting beliefs, policies, value systems, . . . ) these arguments build on.

6 / 36

Challenges of social media AM

Noisy text Vague claims (unclear, ambiguous, poorly worded) Vague/incomplete argument structure (esp. true of short texts)

7 / 36

Main tasks?

Component identification – the task of detecting the premises and conclusion of an argument, as found in a text of discourse Relation prediction – identifying the relations between components [Habernal and Gurevych, 2016]: a (slightly modified) Toulmin model may be suitable for short documents, such as article comments or forum posts Relevant tasks, but it’s not obvious how they help in analyzing user arguments on a large scale, where we need to be able to determine the identity of arguments (expressed in text)

8 / 36

Main tasks for social media AM?

Identify the main arguments – identify the main (central, most prominent, most often used) arguments that the users use when discussing a certain topic Classify opinionated posts – given an opinionated user comment, identify the main arguments used in it

9 / 36

Example

Yahoo News: User comment on Trump rally event

The President we have now divided our country and put his ego first instead of the people. Trump hasn’t divided the country that’s why he has so many people behind him. We want someone who is not afraid

• f the politics in Washington and change our policies with dealing with
• ther countries.

Main argument: “Donald Trump would make a good president” Main claim: “Donald Trump will change the foreign policy for the better” Premises: “Existing foreign policy is bad”, “Trump is not afraid to take on the Establishment”, etc.

10 / 36

Machine learning perspective

Argument clustering – grouping of similar arguments, so that the main arguments/claims can be identified

[Boltuži´ c and Šnajder, 2015]

Argument classification – given an opinionated comment, classify it into one or many classes, each corresponding to one main argument (obtained either manually or using argument clustering)

[Hasan and Ng, 2014]: “reason classification”, [Boltuži´ c and Šnajder, 2014]: “argument recognition”

11 / 36

Outline

1

Argumentation in social media

2

Argument recognition

3

Argument clustering

4

Argument similarity

12 / 36

Task Description

Argument Recognition

Identifying what arguments, from a set of predefined arguments, are used in a comment, and how. Input:

1 Prominent arguments from past two-sided debates 2 Users’ comments from on-line discussion boards

Output (for each comment):

1 Is an argument used in a comment? 2 Does the comment support or attack the argument?

13 / 36

Should gay marriage be legal?

Comment

Gay marriages must be legal in all 50 states. 2 people regardless of their genders. Discrimination against gay marriage is unconstitutional and biased. Tolerance, education and social justice make our world a better place.

Supported argument

It is discriminatory to refuse gay couples the right to marry

Attacked argument

Marriage should be between a man and a woman.

14 / 36

COMARG Corpus

COMARG: Corpus of comments, arguments, and manually annotated comment–argument pairs Source: procon.org and idebate.org Under God in Pledge (UGIP) Gay Marriages (GM) # Arguments 6 7 # Comments 175 198 # Pairs 1,050 1,386 Five-point scale:

A – comment explicitly attacks the argument a – comment vaguely/implicitly attacks the argument N – comment makes no use of the argument s – comment vaguely/implicitly supports the argument S – comment explicitly supports the argument

15 / 36

Annotation Example

Comment

I believe that the statement about God in the pledge should be eliminated. In order to create unity in our nation we shouldn’t be forcing someone else’s God onto people. Also, adding the phrase Under God" was a decision made to widen the gap between us and the Soviet Union. It wasn’t put there to "honor god" or make us any better. Furthermore, we should seperate church from state. Its the law.

S (explicitly supported)

Separation of state and religion.

a (vaguely/implicitly attacked)

Under God is part of American tradition and history.

N (not used)

Likely to be seen as a state sanctioned condemnation of religion.

16 / 36

Argument Recognition Model

Argument Recognition framed as multiclass classification Features:

1 Textual Entailment (TE)

Excitement Open Platform: 7 pre-trained decision algorithms (14 features)

2 Semantic Text Similarity (STS)

TakeLab STS sentence-level, comment-level (32 features)

3 Stance Alignment (SA)

Binary feature: 1 if argument and comment have same stance

No lexical features ⇒ topic independence

17 / 36

Results

Micro-averaged F1-score

A-a-N-s-S Aa-N-sS A-N-S Model UGIP GM UGIP GM UGIP GM MCC baseline 68.2 69.4 68.2 69.4 79.5 76.6 BoWO baseline 68.2 69.4 67.8 69.5 79.6 76.9 TE 69.1 81.1 69.6 72.3 80.1 73.4 STS 67.8 68.7 67.3 69.9 79.2 75.8 SA 68.2 69.4 68.2 69.4 79.5 76.6 STS+SA 68.2 69.5 67.5 68.7 79.6 76.1 TE+SA 68.9 72.4 71.0 73.7 81.8 80.3 TE+STS+SA 70.5 72.5 68.9 73.4 81.4 79.7

STS or STS+SA not good TE outperforms baseline from 0.6% to 11.7% F1 TE+SA overall best SA helps distinguish entailment/contradiction

18 / 36

Outline

1

Argumentation in social media

2

Argument recognition

3

Argument clustering

4

Argument similarity

19 / 36

Should marijuana be legalized?

User comment 1

No, because marijuana lessen the brain’s ability for cognitive thinking.

User comment 2

There have been plenty of highway deaths associated with marajuanna use.

User comment 3

The Legalization of marijuana would lower are crime rates in the United States of America by at least 15 to 20

User comment 4

Marijuana is proven to cause depression and change brain patterns in

• dd ways among other things

20 / 36

Should marijuana be legalized?

“Damages health" (CON) User comment 1

No, because marijuana lessen the brain’s ability for cognitive thinking.

User comment 4

Marijuana is proven to cause depression and change brain patterns in odd ways among other things

21 / 36

Task Description

Identifying Prominent Arguments

Identifying reasonings and opinions to cluster into arguments. Input:

1 Users’ comments from on-line discussions

Output:

1 Set of argument clusters 2 Representative argument of each cluster

22 / 36

Corpus

Threaded debated annotated with arguments at sentence level [Hasan and Ng, 2014] Four topics

Should gay marriage be legal? Should marijuana be legalized? Is Obama a good president? Should abortion be legalized?

GM MAR OBA ABO Pro Con Pro Con Pro Con Pro Con #Arguments 5 4 5 5 8 8 7 5 #Comments 639 197 585 239 358 272 446 368

23 / 36

Argument similarity

1 Vector-space similarity

Bag-of-words (BoW)

Inverse sentence frequency weight

Neural network skip-gram [Mikolov et al., 2013]

Word-vector sum for sentences

Cosine distance

2 Semantic textual similarity (STS) [Šari´

c et al., 2012]

Text comparison features Output real valued similarity score

Hierarhical agglomerative clustering (HAC) [Xu et al., 2005]

Input: Distance matrix Output: Hierarhical structures

Linkage criterion: Complete linkage, Ward’s method

24 / 36

Clustering evaluation

Comparison against gold labels

OBA MAR GM ABO Model (linkage) V ARI V ARI V ARI V ARI STS (Complete) .11 .02 .05 .03 .05 .01 .06 .02 BoW (Complete) .15 .03 .04 .00 .04 .01 .04 .01 BoW (Ward’s) .27 .04 .17 .02 .15 .04 .24 .07 Skip-gram (Complete) .21 .04 .13 .02 .10 .04 .20 .03 Skip-gram (Ward’s) .30 .10 .25 .19 .15 .07 .23 .08 Skip-gram (Ward’s) pro/con .24 .08 .25 .20 .16 .07 .20 .07

Ward’s linkage best performance Word embeddings best performance Clustering separately based on stance improves performance on two topics

25 / 36

Error analysis: Background knowledge

Comment

Pot is also one of the most high priced exports of Central American Countries and the Carribean Predicted: “Not addictive (PRO)” True: “Legalized marijuana can be controlled and regulated by the government (PRO)”

26 / 36

Error analysis: Argument granularity

Gold labels are too specific?

Damages our bodies Responsible for brain damage Damaging our bodies

Gold labels are too general?

the economy would get billions of dollars (...) no longer would this revenue go di- rectly into the black market. If the tax

• n

cigarettes can be $5.00/pack imagine what we could tax pot for! Economy profits Tax benefits Legalized marijuana can be controlled and regulated by the government

27 / 36

Outline

1

Argumentation in social media

2

Argument recognition

3

Argument clustering

4

Argument similarity

28 / 36

Argument similarity

Common to both task: requirement to compute similarity of arguments Also considered by [Swanson et al., 2015, Misra et al., 2015] under the name argument facet similarity Existing approaches to RTE/STS provide only limited means to measure argument similarity

29 / 36

Argument similarity

Comment

Now it is not taxed, and those who sell it are usually criminals of some sort (though many are harmless). If a thing is not taxed, criminals can sell it Criminals should be stopped from selling things Things that are taxed are controlled and regulated by the government

Main claim

Legalized marijuana can be controlled and regulated by the government.

30 / 36

Argument similarity

Current approaches to argument similarity are not generative in nature and cannot generate a chain of implicit premises Yet this seems to be a key ingredient of an argumentation mining system capable of large-scale analysis of social media arguments If micro-level argumentation focuses on the components of a single argument, then nano-level argumentation is what we need

31 / 36

References I

Boltuži´ c, F . and Šnajder, J. (2014). Back up your stance: Recognizing arguments in online discussions. In Proceedings of the First Workshop on Argumentation Mining, pages 49–58. Boltuži´ c, F . and Šnajder, J. (2015). Identifying prominent arguments in online debates using semantic textual similarity. In Proceedings of the 2nd Workshop on Argumentation Mining, pages 110–115.

32 / 36

References II

Cabrio, E. and Villata, S. (2012). Combining textual entailment and argumentation theory for supporting online debates interactions. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Short Papers-Volume 2, pages 208–212. Ghosh, D., Muresan, S., Wacholder, N., Aakhus, M., and Mitsui, M. (2014). Analyzing argumentative discourse units in online interactions. In Proceedings of the First Workshop on Argumentation Mining, pages 39–48. Habernal, I., Eckle-Kohler, J., and Gurevych, I. Argumentation mining on the web from information seeking perspective.

33 / 36

References III

Habernal, I. and Gurevych, I. (2016). Argumentation mining in user-generated web discourse. arXiv preprint arXiv:1601.02403. Hasan, K. S. and Ng, V. (2014). Why are you taking this stance? Identifying and classifying reasons in ideological debates. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 751–762. Jiménez-Aleixandre, M. P . and Erduran, S. (2007). Argumentation in science education: An overview. In Argumentation in Science Education, pages 3–27. Springer. Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013). Efficient estimation of word representations in vector space. In Proceedings of ICLR, Scottsdale, AZ, USA.

34 / 36

References IV

Misra, A., Anand, P ., Tree, J., and Walker, M. (2015). Using summarization to discover argument facets in online idealogical dialog. In NAACL HLT, pages 430–440. Park, J. and Cardie, C. (2014). Identifying appropriate support for propositions in online user comments. ACL 2014, pages 29–38. Šari´ c, F ., Glavaš, G., Karan, M., Šnajder, J., and Dalbelo Baši´ c, B. (2012). Takelab: Systems for measuring semantic text similarity. In Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 441–448, Montréal, Canada.

35 / 36

References V

Swanson, R., Ecker, B., and Walker, M. (2015). Argument mining: Extracting arguments from online dialogue. In 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, page 217. Walton, D. (2005). Argumentation methods for artificial intelligence in law. Springer. Xu, R., Wunsch, D., et al. (2005). Survey of clustering algorithms. Neural Networks, IEEE Transactions on, 16(3):645–678.

36 / 36