Slides available at http://bit.ly/cl-scisumm16-slides and will be - PowerPoint PPT Presentation

Slides available at http://bit.ly/cl-scisumm16-slides and will be filed in GitHub.

Slides available at http://bit.ly/cl-scisumm16-slides and will be filed in GitHub. 2 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

Slides available at http://bit.ly/cl-scisumm16-slides and will be filed in GitHub. 3 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

4 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

5 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

6 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

0.2 0.5 0.4 0.3 F1 Score F1 Score 0.1 0.2 0.1 0 0 7 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016 CEUR version (all system runs averaged)

Task 1a Best performing System ID Avg StDev Systems performance 16 0.114941 0.038295 8 0.102306 0.056893 6 0.100184 0.056926 13 0.063622 0.050519 9 0.056172 0.053044 5 0.054283 0.028954 12 0.034219 0.020178 15 0.034122 0.014837 10 0.03073 0.023688 8 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

Task 1b System ID Avg StDev Best performing performance System 16 0.1696516 0.0860830 8 0.264754 01473109 13 0.10294 0.0236852 5 0.088737 0.0617396 12 0.052747 0.0341898 Best performing 15 0.152984 0.0870947 System 10 0.168061 0.122391 9 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

Best performing System ID Approaches Comments Systems 3 • NNMF for BioMedSumm The best for human summaries 8 • hLDA topic modeling The best for abstract • Sentence length/position and community • Cited text spans summaries • RST 15 • Tkern1-1 Kernel-based • Tkern1-1ce approaches are worthy • Tkern1-4 of exploration • Tkern1-4ce • Tkern1-8 • Tkern1-8ce 16 • Manifold Ranking System Ranking approaches do not seem to work 10 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

0.3 0.25 0.2 F1 Score 0.15 0.1 0.05 0 -0.05 11 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

System ID Approach Task 1a Comments 5 • Discourse profiling, similarity • 0.03 Some assumptions function might be misplaced Best performing Systems 6 • Tfidf + neural network, dissimilarity • 0.10 Tfidf approach score performed among the best, like last year Best performing Systems 8 • Sentence fusion • 0.12 Second best • Jaccard Cascade • 0.09 performance, second • Jaccard Focused • 0.12 highest deviation • SVM method • 0.04 • Voting Method 1 • 0.11 • Voting Method 2 • 0.10 9 • Sect-class TSR • 0.00 Ranking methods have • Modified TSR • 0.05 not worked well • TSR-sent-class • 0.00 12 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

ID Approach Task 1A Comments 10 WEKA + SUMMA • 0.02 • Regression did not perform • Method 1 • 0.01 well • Method 2 12 • Ranking problem, Text classification • 0.02 • Suggests that Task 1a is not problem IR 13 • Unsupervised bigram overlap method • 0.04 • Middle order performance Best performing in Task 1a Systems 15 • Tfidf+st+sl • 0.13 • Best performance, most • Tkern1-1 • 0.01 deviation • Tkern1-1ce • 0.01 • Tkern1-4 • 0.01 • Tkern1-4ce • 0.01 • Tkern1-8 • 0.01 • Tkern1-8ce • 0.01 Best performing Systems 16 • SVMRank, Manifold Ranking System • 0.10 • Most consistent out of top performing systems 13 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

0.5 0.45 0.4 0.35 F1 Score 0.3 0.25 0.2 0.15 0.1 0.05 0 14 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

ID Approach Task 1B Comments Best 5 • Transdisciplinary Scientific 0.06 Dependency on Task 1A hurts performing Lexicon performance Systems 8 • Sentence fusion • 0.29 Combinations of Voting • Jaccard Cascade • 0.25 methods with Task 1A • Jaccard Focused • 0.31 approaches worked well • SVM method • 0.17 • Voting Method 1 • 0.28 • Voting Method 2 • 0.26 10 WEKA + SUMMA • 0.13 Domain knowledge improves • Text classification 1 • 0.06 classification • Text classification 2 12 • Text classification • 0.01 Citation context is not enough; More features need to be explored 13 • Rule-based approach • 0.05 Dependency on Task 1A and paper structure 16 • Manifold Ranking System • 0.15 Ranking did not perform well 15 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

Abstract summaries Human summaries Community summaries F1 Score F1 Score F1 Score 0.05 0.15 0.25 0.35 0.05 0.15 0.25 0.35 0.45 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.1 0.2 0.3 0.1 0.2 0.3 0.4 0 0 0 8$JACCARD … 8$JACCARD … 15$TFIDF+… 8$JACCARD … 3$LMKL2 10$RUN1 3$LMEQUAL 3$LMEQUAL 8$VOTING … 8$SVM … 3$LMKL1 10$RUN2 3$LMKL1 8$SVM … 8$SVM … 8$VOTING … 8$JACCARD … 8$VOTING … 3$LMKL2 3$TF 8$JACCARD … 8$VOTING … 8$VOTING … 15$TKERN1… 3$TF 15$TFIDF+S… 15$TKERN1… 10$RUN1 15$TKERN1-8 15$TKERN1… 10$RUN2 10$RUN2 8$JACCARD … 15$TKERN1- … 8$VOTING … 15$TKERN1… 15$TKERN1-1 10$RUN1 15$TKERN1… 15$TKERN1- … 15$TKERN1-… 16$DEFAULT 15$TKERN1-8 15$TKERN1-… 15$TKERN1… 15$TKERN1-4 15$TKERN1-1 3$LMEQUAL 15$TKERN1-4 3$LMKL2 15$TKERN1- … 15$TKERN1-… 3$LMKL1 15$TFIDF+S … 16$DEFAULT 3$TF 16$DEFAULT F1 Score F1 Score F1 Score -0.05 -0.05 -0.1 0.05 0.15 0.25 0.05 0.15 0.25 0.35 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.1 0.2 0.3 0.1 0.2 0.3 0.4 0 0 0 16 16

17 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

18 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

Other shared tasks have a notebook version of the proceedings. Authors wishing to revise should submit a revised version of their paper to the ACL Anthology. We also encourage extended versions (e.g., with more detailed analyses) to the IJDL special issue: http://bit.ly/birndl-ijdl First submission deadline: 30 September Notification: 15 November 19 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

This task was possible through the generous support of 20 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

Slides available at http://bit.ly/cl-scisumm16-slides and will be filed in GitHub. 21 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

22 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

23 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

24 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

25 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

OCR & Section Parse CLAIR -Umich’s Python module Annotation! Post Processing with U-Colorado’s python scripts 26 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016

………….. ……….. 27 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016