Slides available at http://bit.ly/cl-scisumm16-slides and will be - - PowerPoint PPT Presentation

▶
slides available at http bit ly cl scisumm16 slides and
SMART_READER_LITE
LIVE PREVIEW

Slides available at http://bit.ly/cl-scisumm16-slides and will be - - PowerPoint PPT Presentation

Oct 29, 2022 132 likes •418 views

Slides available at http://bit.ly/cl-scisumm16-slides and will be filed in GitHub. Slides available at http://bit.ly/cl-scisumm16-slides and will be filed in GitHub. 2 BIRNDL 2016: CL-SciSumm 16 Overview 23 June 2016 Slides available at

slide-1
SLIDE 1

Slides available at http://bit.ly/cl-scisumm16-slides and will be filed in GitHub.

slide-2
SLIDE 2

2

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

Slides available at http://bit.ly/cl-scisumm16-slides and will be filed in GitHub.

slide-3
SLIDE 3

3

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

Slides available at http://bit.ly/cl-scisumm16-slides and will be filed in GitHub.

slide-4
SLIDE 4

4

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

slide-5
SLIDE 5

5

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

slide-6
SLIDE 6

6

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

slide-7
SLIDE 7

7

CEUR version (all system runs averaged)

0.1 0.2

F1 Score 0.1 0.2 0.3 0.4 0.5 F1 Score

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

slide-8
SLIDE 8

System ID Task 1a Avg performance StDev

16 0.114941 0.038295 8 0.102306 0.056893 6 0.100184 0.056926 13 0.063622 0.050519 9 0.056172 0.053044 5 0.054283 0.028954 12 0.034219 0.020178 15 0.034122 0.014837 10 0.03073 0.023688

Best performing Systems

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

8

slide-9
SLIDE 9

System ID Task 1b Avg performance StDev

16 0.1696516 0.0860830 8 0.264754 01473109 13 0.10294 0.0236852 5 0.088737 0.0617396 12 0.052747 0.0341898 15 0.152984 0.0870947 10 0.168061 0.122391

Best performing System Best performing System

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

9

slide-10
SLIDE 10

10

System ID Approaches Comments 3

  • NNMF for BioMedSumm

The best for human summaries 8

  • hLDA topic modeling
  • Sentence length/position
  • Cited text spans
  • RST

The best for abstract and community summaries 15

  • Tkern1-1
  • Tkern1-1ce
  • Tkern1-4
  • Tkern1-4ce
  • Tkern1-8
  • Tkern1-8ce

Kernel-based approaches are worthy

  • f exploration

16

  • Manifold Ranking System

Ranking approaches do not seem to work

Best performing Systems

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

slide-11
SLIDE 11

11

  • 0.05

0.05 0.1 0.15 0.2 0.25 0.3

F1 Score 23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

slide-12
SLIDE 12

System ID Approach Task 1a Comments

5

  • Discourse profiling, similarity

function

  • 0.03

Some assumptions might be misplaced 6

  • Tfidf + neural network, dissimilarity

score

  • 0.10

Tfidf approach performed among the best, like last year 8

  • Sentence fusion
  • Jaccard Cascade
  • Jaccard Focused
  • SVM method
  • Voting Method 1
  • Voting Method 2
  • 0.12
  • 0.09
  • 0.12
  • 0.04
  • 0.11
  • 0.10

Second best performance, second highest deviation 9

  • Sect-class TSR
  • Modified TSR
  • TSR-sent-class
  • 0.00
  • 0.05
  • 0.00

Ranking methods have not worked well

Best performing Systems Best performing Systems

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

12

slide-13
SLIDE 13

ID Approach Task 1A Comments

10 WEKA + SUMMA

  • Method 1
  • Method 2
  • 0.02
  • 0.01
  • Regression did not perform

well 12

  • Ranking problem, Text classification

problem

  • 0.02
  • Suggests that Task 1a is not

IR 13

  • Unsupervised bigram overlap method
  • 0.04
  • Middle order performance

in Task 1a 15

  • Tfidf+st+sl
  • Tkern1-1
  • Tkern1-1ce
  • Tkern1-4
  • Tkern1-4ce
  • Tkern1-8
  • Tkern1-8ce
  • 0.13
  • 0.01
  • 0.01
  • 0.01
  • 0.01
  • 0.01
  • 0.01
  • Best performance, most

deviation 16

  • SVMRank, Manifold Ranking System
  • 0.10
  • Most consistent out of top

performing systems

Best performing Systems Best performing Systems

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

13

slide-14
SLIDE 14

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

F1 Score 23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

14

slide-15
SLIDE 15

15

ID Approach Task 1B Comments 5

  • Transdisciplinary Scientific

Lexicon 0.06 Dependency on Task 1A hurts performance 8

  • Sentence fusion
  • Jaccard Cascade
  • Jaccard Focused
  • SVM method
  • Voting Method 1
  • Voting Method 2
  • 0.29
  • 0.25
  • 0.31
  • 0.17
  • 0.28
  • 0.26

Combinations of Voting methods with Task 1A approaches worked well 10 WEKA + SUMMA

  • Text classification 1
  • Text classification 2
  • 0.13
  • 0.06

Domain knowledge improves classification 12

  • Text classification
  • 0.01

Citation context is not enough; More features need to be explored 13

  • Rule-based approach
  • 0.05

Dependency on Task 1A and paper structure 16

  • Manifold Ranking System
  • 0.15

Ranking did not perform well

Best performing Systems

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

slide-16
SLIDE 16

16

0.05 0.1 0.15 0.2 0.25 0.3 0.35 8$JACCARD … 3$LMKL2 3$LMEQUAL 3$LMKL1 8$SVM … 8$JACCARD … 3$TF 8$VOTING … 15$TFIDF+S… 15$TKERN1-8 10$RUN2 8$VOTING … 10$RUN1 15$TKERN1-… 15$TKERN1-… 15$TKERN1-1 15$TKERN1-4 15$TKERN1-… 16$DEFAULT

F1 Score

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 15$TFIDF+… 10$RUN1 8$VOTING … 10$RUN2 8$SVM … 8$VOTING … 8$JACCARD … 15$TKERN1… 15$TKERN1… 15$TKERN1… 8$JACCARD … 15$TKERN1… 15$TKERN1… 16$DEFAULT 15$TKERN1… 3$LMEQUAL 3$LMKL2 3$LMKL1 3$TF

F1 Score

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 8$JACCARD … 8$JACCARD … 3$LMEQUAL 8$SVM … 3$LMKL1 8$VOTING … 3$LMKL2 8$VOTING … 3$TF 10$RUN1 10$RUN2 15$TKERN1- … 15$TKERN1-1 15$TKERN1- … 15$TKERN1-8 15$TKERN1-4 15$TKERN1- … 15$TFIDF+S … 16$DEFAULT

F1 Score

  • 0.1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8

F1 Score

  • 0.05

0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

F1 Score

  • 0.05

0.05 0.1 0.15 0.2 0.25 0.3

F1 Score

Abstract summaries Community summaries Human summaries

16

slide-17
SLIDE 17

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

17

slide-18
SLIDE 18

18

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

slide-19
SLIDE 19

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

19

Other shared tasks have a notebook version of the proceedings. Authors wishing to revise should submit a revised version of their paper to the ACL Anthology. We also encourage extended versions (e.g., with more detailed analyses) to the IJDL special issue: http://bit.ly/birndl-ijdl First submission deadline: 30 September Notification: 15 November

slide-20
SLIDE 20

20

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

This task was possible through the generous support of

slide-21
SLIDE 21

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

21

Slides available at http://bit.ly/cl-scisumm16-slides and will be filed in GitHub.

slide-22
SLIDE 22

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

22

slide-23
SLIDE 23

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

23

slide-24
SLIDE 24

24

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

slide-25
SLIDE 25

25

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

slide-26
SLIDE 26

26

Annotation! Post Processing with U-Colorado’s python scripts OCR & Section Parse

CLAIR -Umich’s Python module

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview

slide-27
SLIDE 27

27

………….. ………..

23 June 2016 BIRNDL 2016: CL-SciSumm 16 Overview