Argumentative Writing Support: Structure Identification and Quality - - PowerPoint PPT Presentation

▶
argumentative writing support
SMART_READER_LITE
LIVE PREVIEW

Argumentative Writing Support: Structure Identification and Quality - - PowerPoint PPT Presentation

Jan 24, 2024 641 likes •918 views

Argumentative Writing Support: Structure Identification and Quality Assessment of Arguments Dagstuhl Seminar: Natural Language Argumentation: Mining, Processing, and Reasoning over Textual Arguments Christian Stab Iryna Gurevych Ubiquitous

slide-1
SLIDE 1

1 Dagstuhl Seminar: “Natural Language Argumentation: Mining, Processing, and Reasoning over Textual Arguments”

Argumentative Writing Support:

Structure Identification and Quality Assessment of Arguments

2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Christian Stab Iryna Gurevych Ubiquitous Knowledge Processing Lab Technische Universität Darmstadt

slide-2
SLIDE 2

2 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Argumentation Mining Research @ Darmstadt

  • Analyzing argumentation structures in the discourse context, e.g. user-

generated Web content (Habernal & Gurevych, 2015), scientific articles, (Kirschner et al., 2015), student essays (Stab & Gurevych, 2014)

  • Projects:
  • Argumentative Writing Support (AWS): a writing assistance system to

support authors in writing persuasive arguments and to improve writing skills

  • Large-scale argumentation mining on the Web: e.g. comments to articles,

discussion forums, or blogs (joint project with Ivan, Benno, Henning)

  • Information Consolidation on the Web: correspondence between proposition-

level semantic relations and argumentative structures (joint project with Ido)

  • Free resources, e.g. argument-annotated corpora, guidelines,

software: https://www.ukp.tu-darmstadt.de/research/research- areas/argumentation-mining/ This talk: looking at persuasive essays (PhD thesis by Christian Stab)

slide-3
SLIDE 3

5 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Outline

Parsing Argumentation Structures Myside Bias Recognition Detecting Insufficiently Supported Arguments

slide-4
SLIDE 4

6 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Parsing Argumentation Structures

Example

Museums and art galleries will disappear soon? It is quite common that more and more people can watch exhibitions through television or internet at home due to modern technology; therefore, some people think museums and art galleries will disappear soon. However, I still believe [some museums and art galleries will not disappear]1. [Technology indeed simplifies people's life all the time]2. Obviously, [people who watch exhibitions on TV or internet at home, save the time and money on the road, which is increasingly significant particularly to people in modern society]3. However, [in accordance with recent research, experts suggest the lifestyle of individuals in modern society is unhealthy]4 because [they lack of physical exercise and face- to-face communication]5. [The importance of museums and art galleries is plain in terms

  • f education and culture]6. First of all, [authentic exhibits cannot

be completely displayed only by images and videos]7. [Travelling to a place is much better than viewing the landscape

  • f that place on TV or photos]8, so [the best method to learn
  • ne thing is to experience it]9. Furthermore, [museums and art

galleries preserve some culture heritages]10; therefore, [these buildings will not disappear unless people abandon their culture]11. In conclusion, I admit that [modern technology has provided a more convenient and comfortable manner for people to watch exhibitions]12 but [museums and art galleries are necessary to be preserved for its importance of education and culture]13.

Paragraph 4 Paragraph 2 Paragraph 3

1 &13 3 2 4 5 6 7 8 9 11 10 12

Major Claim(s) Claims Premises

Example Essay: Argumentation Structure:

slide-5
SLIDE 5

7 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Parsing Argumentation Structures

Data

Persuasive essays

  • Written for e.g. IELTS, TOEFL, etc.
  • Collected from www.essayforum.com
  • 402 essays; 7,116 sentences; 147,271 tokens

Annotation scheme

  • Argumentative structure as tree structure
  • Argument components: Major Claim (751), Claim (1,506) and Premise (3,832)
  • Argumentative relations: Support (3,613) and Attack (219)

Inter-Annotator Agreement*

  • Argument components:

αU =.767

  • Argumentative relations: α = .723 (avg. of support & attack)

*determined among three annotators on a subset of 80 essays

slide-6
SLIDE 6

8 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Parsing Argumentation Structures

Pipeline

Argument Component Classification

  • Classify each argument component as major claim, claim or premise

Argumentative Relation Identification

  • Classify argument component pairs as argumentatively related or not

Problem: Result is an arbitrary graph NOT a tree Solution: Joint Modeling (Tree generation)

slide-7
SLIDE 7

9 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Parsing Argumentation Structures

Joint Modeling

Component types and argumentative relations share mutual information

Idea:

Jointly model argument component types and argumentative relations to find an optimal tree

ILP-based joint model

  • Finds the tree structure which optimizes previous analysis results
  • Allows to find several trees (arguments) in a paragraph

Component Type Argumentative Relation

Claim Premise Claim Premise No outgoing relations (root node) Exhibits outgoing relations More incoming relations Fewer incoming relations

slide-8
SLIDE 8

10 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych |

Parsing Argumentation Structures

Joint Modeling

component classification relation identification statistics F1 MC Cl Pr F1 NoLi Link Cl → Pr Pr → Cl Trees Baseline heuristics .724 .740 .560 .870 .660 .885 .436

  • 100%

Base classifier † .773 .865 .592 .861

†

.736 .917 .547

  • 20.9%

IncBaseline

†

.776 .865 .601 .861

†

.739 .917 .555 206 1,144 24.2% ILP Joint Model

†‡

.823 .865 .701 .904

†

.752 .913 .591 297 283 100%

Baseline Heuristic

  • Last component in introduction and first component in conclusion as major claim
  • First component in paragraph as claim, remaining as premise
  • Link all premises to first component in paragraph

Base classifier

  • Argument Component Classification (major claim, claim, premise)
  • Classification of argument component pairs (linked, unlinked)

IncBaseline

  • Incorporates baseline in base classifiers if both base classifiers fail to predict claims or relations in a

paragraph

† significant improvement over baseline heuristic; ‡ significant improvement over base classifier

slide-9
SLIDE 9

11 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Outline

Parsing Argumentation Structures Myside Bias Recognition Detecting Insufficiently Supported Arguments

slide-10
SLIDE 10

12 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Myside Bias Recognition

Introduction & Motivation

Myside Bias

  • Tendency to ignore evidence against one’s own position
  • Argumentation is biased towards own prior beliefs

 Weak arguments

Considering opposing viewpoints is crucial

  • Improves precision of claims
  • Better elaboration of reasons
  • Significantly improves the argumentation quality (Wolfe and Britt, 2009)

Myside Bias Recognition

  • Detecting text/arguments that anticipate opposing viewpoints
  • Applications: filtering bad arguments, writing support, etc.
slide-11
SLIDE 11

13 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Myside Bias Recognition

Data & Approach Data

  • 402 persuasive essays

Inter-Annotator Agreement

  • κ = .786
  • α = .787

Class Distribution

  • 37.6% unbiased
  • 62.4% biased

 myside bias is a frequent flaw

Task

  • Binary Document Classification

Learner

  • SVM with polykernel

Features

  • Unigrams
  • Dependencies
  • Production Rules
  • Adversative transitions
  • Sentiment features
  • Discourse features
slide-12
SLIDE 12

14 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Myside Bias Recognition

Results – Feature Analysis on Development Set

0,709 0,657 0,569 0,75 0,385 0,52 0,2 0,4 0,6 0,8 Unigram Depencies Producation Rules Adverstative Transitions Sentiment Features Discourse Features Macro F1

  • Adversative Transitions and unigrams perform best
  • Sentiment features are not informative; same results as majority baseline

 Myside bias is indicated by lexical features

Dependencies

slide-13
SLIDE 13

15 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Myside Bias Recognition

Results – Model Assessment

0,894 0,384 0,679 0,734 0,2 0,4 0,6 0,8 1 Human Upper Bound Baseline Majority Baseline Heuristics SVM uni+pr+adv Macro F1

  • Baseline Majority:

Classifies each text as biased

  • Baseline Heuristics: All texts with ‘Admittedly’ or ‘argue that’ are unbiased
  • SVM uni+pr+adv:

SVM with unigrams, production rules and adversative transitions

  • Yields best performance (75.6% accuracy; 0.734 macro F1)
  • Significantly outperforms heuristic baseline (Wilcoxon sign ranked test; sign level = 0.005)
  • Achieves 84% of human performance (accuracy)
slide-14
SLIDE 14

16 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Outline

Parsing Argumentation Structures Myside Bias Recognition Detecting Insufficiently Supported Arguments

slide-15
SLIDE 15

17 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Detecting Insufficiently Supported Arguments

Introduction A good argument exhibits sufficient support to accept the claim

slide-16
SLIDE 16

18 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Detecting Insufficiently Supported Arguments

Introduction A good argument exhibits sufficient support to accept the claim Which argument has stronger support?

“It is an undeniable fact that tourism harms the natural habitats of the destination countries. As Australia’s Great Barrier Reef has shown, the visitors cause immense destruction by breaking corals as souvenirs, throwing boat anchors or droping fuel and other sorts of pollution.” “Cloning will be beneficial for people who are in need of organ transplants. Cloned

  • rgans will match perfectly to the blood

group and tissue of patients since they can be raised from cloned stem cells of the

  • patient. In addition, it shortens the healing

process.”

A B

slide-17
SLIDE 17

19 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Detecting Insufficiently Supported Arguments

Introduction A good argument exhibits sufficient support to accept the claim Which argument has stronger support?

“It is an undeniable fact that tourism harms the natural habitats of the destination countries. As Australia’s Great Barrier Reef has shown, the visitors cause immense destruction by breaking corals as souvenirs, throwing boat anchors or droping fuel and other sorts of pollution.” “Cloning will be beneficial for people who are in need of organ transplants. Cloned

  • rgans will match perfectly to the blood

group and tissue of patients since they can be raised from cloned stem cells of the

  • patient. In addition, it shortens the healing

process.”

A B

Bad argument!

  • A very general claim is inferred from only
  • ne instance (Hasty generalization)

 Reasons are insufficient

Good Argument

  • It is reasonable to accept the claim by

means of the provided reasons  Reasons are sufficient

slide-18
SLIDE 18

20 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Detecting Insufficiently Supported Arguments

Data & Task Data

  • 1,029 arguments from persuasive

essays

Inter-Annotator Agreement

  • multi-π = .7672
  • α = .7673

Class Distribution

  • 66.2% sufficient
  • 38.8% insufficient

 sufficiency occurs relatively frequently

Task

  • Binary classification
  • (sufficient / insufficient)

Models

  • SVM with manual features
  • MLP: Multilayer Perceptron
  • CNN: Convolutional Neural

Network with word embeddings

slide-19
SLIDE 19

21 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Detecting Insufficiently Supported Arguments

Results

  • Baseline Majority:

Classifies each text as sufficient

  • Baseline Heuristics: All texts including ‘exampl’ are insufficient
  • Baseline SVM-bow: SVM with unigrams
  • CNN performs best and significantly outperforms all other approaches
  • It yields 84.8% accuracy and 0.836 macro F1 score
  • Achieves 93.1% human performance

0,887 0,398 0,733 0,775 0,79 0,778 0,836 0,2 0,4 0,6 0,8 1 Human Upper Bound Baseline Majority Baseline Heuristics Baseline SVM-bow SVM-mf MLP CNN Macro F1

slide-20
SLIDE 20

22 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Summary

Parsing Argumentation Structures

  • Joint-Modeling simultaneously improves classification and relation identification
  • Results are promising for persuasive essays

Argumentation Quality

  • Lexical and adversative transitions perform best for finding myside biases
  • Convolutional Neural Networks achieve 93.1% of human performance for

recognizing insufficiently supported arguments.

Future Work

  • Scale argumentation structure parser to heterogeneous texts
  • Integrate models in writing environments
  • Evaluation of effectiveness for supporting authors
slide-21
SLIDE 21

23 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Thank you for your attention! Questions?

Contact

Iryna Gurevych

Technische Universität Darmstadt Ubiquitous Knowledge Processing Lab  Hochschulstr. 10, 64289 Darmstadt, Germany +49 (0)6151 16 – 25290  +49 (0)6151 16 - 5455  gurevych (at) ukp.informatik.tu-darmstadt.de

Christian Stab

Technische Universität Darmstadt Ubiquitous Knowledge Processing Lab  Hochschulstr. 10, 64289 Darmstadt, Germany  +49 (0)6151 16 - 4197  +49 (0)6151 16 - 5455  stab (at) ukp.informatik.tu-darmstadt.de

slide-22
SLIDE 22

24 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Annotation Tools

WebAnno: Flexible and Web-based Annotation Tool (Yimam et al. 2013)

  • General purpose annotation tool
  • Enables to annotate argumentation structures
  • https://webanno.github.io/webanno/

DiGAT: Discourse Graph Annotation Tool (Kirschner et al. 2015)

  • Web-based annotation tool
  • Tailored to argument structure annotation
  • https://github.com/judithek/DiGAT

DKPro Statistics: Agreement measures (Meyer et al. 2014)

  • Java Implementation
  • Freely available https://github.com/dkpro/dkpro-statistics
slide-23
SLIDE 23

25 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

DKPro-argumentation

Unified type system for modeling argumentation

  • Based on UIMA
  • Easy to expand

Enables cross-resource-experiments

  • Seamless integration in DKPro and DKPro-TC
  • Several data sets using it

Supports all subtasks of AM

  • Component identification on diff. granularities
  • Component classification
  • Relation identification

Available here: https://github.com/dkpro/dkpro-argumentation

slide-24
SLIDE 24

26 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

References

Christian Stab and Iryna Gurevych. 2014. Annotating argument components and relations in persuasive essays. In Proceedings of the 25th International Conference on Computational Linguistics, COLING 2014, pages 1501–1510, Dublin, Ireland, August Christian Stab and Iryna Gurevych. 2014b. Identifying argumentative discourse structures in persuasive essays. In Conference on Empirical Methods in Natural Language Processing, EMNLP ‘14, pages 46–56, Doha, Qatar Christian Stab and Iryna Gurevych. 2016. Parsing Argumentation Structures in Persuasive Essays. Under review in Computational Linguistics

slide-25
SLIDE 25

27 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Features for Myside Bias Classification

  • Production rules
  • Binary features extracted from constituent parse trees of sentences
  • The feature set includes e.g. rules like “VP -> VBG,NP” or “PP -> IN,NP”
  • Adversative transitions
  • Lexicon including five categories from: www.msu.edu/~jdowell/135/transw.html
  • Concession, conflict, dismissal, emphasis and replacement
  • Considered each category as a binary feature
  • Discourse relations
  • Extracted using a Discourse-Parser from Lin et al. (2014)
  • Binary features encoding the type of discourse relations and if the relation is

implicit or explicit, e.g. Concession_imp or Contrast_exp

slide-26
SLIDE 26

28 2016 | Computer Science Department | UKP Lab - Prof. Dr. Iryna Gurevych | Christian Stab |

Comparison Argumentation Structure Parsers

  • Peldszus
  • Based on Minimal Spanning Trees (MST)
  • Presupposes that text is already segmented
  • Recognizes exactly one tree per paragraph even if several arguments are

present

  • Stab
  • Based on Integer Linear Programming (ILP)
  • Requires only 2 base classifiers
  • Includes a module for segmenting the text
  • Capable of separating argumentative from non-argumentative text units
  • Can recognize several arguments in a paragraph -> it is capable of separating

several arguments