Vandalism Detection in Wikidata Stefan Heindorf 1 , Martin Potthast 2 - - PowerPoint PPT Presentation

▶
vandalism detection in wikidata
SMART_READER_LITE
LIVE PREVIEW

Vandalism Detection in Wikidata Stefan Heindorf 1 , Martin Potthast 2 - - PowerPoint PPT Presentation

Oct 21, 2023 257 likes •951 views

Vandalism Detection in Wikidata Stefan Heindorf 1 , Martin Potthast 2 , Benno Stein 2 , Gregor Engels 1 CIKM 2016 October 25, 2016 1 2 Motivation Vandalism Detection in Wikidata Stefan Heindorf 2 Motivation Vandalism Detection in Wikidata

slide-1
SLIDE 1

Vandalism Detection in Wikidata

Stefan Heindorf1, Martin Potthast2, Benno Stein2, Gregor Engels1 CIKM 2016 October 25, 2016

1 2

slide-2
SLIDE 2

Motivation

Vandalism Detection in Wikidata Stefan Heindorf 2

slide-3
SLIDE 3

Motivation

Vandalism Detection in Wikidata Stefan Heindorf 2

slide-4
SLIDE 4

Motivation

Vandalism Detection in Wikidata Stefan Heindorf 2

slide-5
SLIDE 5

Motivation

Vandalism Detection in Wikidata Stefan Heindorf 2

slide-6
SLIDE 6

Motivation

Vandalism Detection in Wikidata Stefan Heindorf 2

slide-7
SLIDE 7

Motivation

Vandalism Detection in Wikidata Stefan Heindorf 2

slide-8
SLIDE 8

Motivation

Vandalism Detection in Wikidata Stefan Heindorf 2

slide-9
SLIDE 9

Motivation

Vandalism Detection in Wikidata Stefan Heindorf 2

slide-10
SLIDE 10

3 Stefan Heindorf Vandalism Detection in Wikidata

slide-11
SLIDE 11

3 Stefan Heindorf Vandalism Detection in Wikidata

Item head

slide-12
SLIDE 12

3 Stefan Heindorf Vandalism Detection in Wikidata

Item head Item body

slide-13
SLIDE 13

3 Stefan Heindorf Vandalism Detection in Wikidata

Item head Item body Revisions

slide-14
SLIDE 14

3 Stefan Heindorf Vandalism Detection in Wikidata

(Feb 22, 2013) (May 13, 2013) (May 30, 2013) Item head Item body Revisions

slide-15
SLIDE 15

3 Stefan Heindorf Vandalism Detection in Wikidata

(Feb 22, 2013) (May 13, 2013) (May 30, 2013) Item head Item body Revisions

slide-16
SLIDE 16

3 Stefan Heindorf Vandalism Detection in Wikidata

(Feb 22, 2013) (May 13, 2013) (May 30, 2013) Item head Item body Revisions

slide-17
SLIDE 17

3 Stefan Heindorf Vandalism Detection in Wikidata

Item head Item body Revisions

slide-18
SLIDE 18

Why is it a problem?

4

Patrolling Reverting Warning Blocking Protecting

  • Over 2 Mio manual edits per month
  • A lot of tedious work
  • Vandalism is not detected in time

Stefan Heindorf Vandalism Detection in Wikidata

slide-19
SLIDE 19

Research Question

How to detect damaging changes to crowdsourced knowledge bases?

5 Stefan Heindorf Vandalism Detection in Wikidata

slide-20
SLIDE 20

Our Approach

Vandalism Detection in Wikidata Stefan Heindorf 6

slide-21
SLIDE 21

Our Approach

  • 1. Label Dataset

 Vandalism Corpus [SIGIR’15]

Vandalism Detection in Wikidata Stefan Heindorf 6

slide-22
SLIDE 22

Our Approach

  • 1. Label Dataset

 Vandalism Corpus [SIGIR’15]

  • 2. Study Vandalism Characteristics  47 Features

Vandalism Detection in Wikidata Stefan Heindorf 6

slide-23
SLIDE 23

Our Approach

  • 1. Label Dataset

 Vandalism Corpus [SIGIR’15]

  • 2. Study Vandalism Characteristics  47 Features
  • 3. Experiment with ML

 Multiple-Instance Learning

Vandalism Detection in Wikidata Stefan Heindorf 6

slide-24
SLIDE 24

Our Approach

  • 1. Label Dataset

 Vandalism Corpus [SIGIR’15]

  • 2. Study Vandalism Characteristics  47 Features
  • 3. Experiment with ML

 Multiple-Instance Learning

  • 4. Compare with state of the art

 2 Baselines

Vandalism Detection in Wikidata Stefan Heindorf 6

slide-25
SLIDE 25

Corpus [SIGIR ’15]

Revisions over time

7

slide-26
SLIDE 26

Corpus [SIGIR ’15]

Revisions over time

7

Month

slide-27
SLIDE 27

Corpus [SIGIR ’15]

Revisions over time

7

Month

slide-28
SLIDE 28

Corpus [SIGIR ’15]

Revisions over time

7

Month

103,000 vandalism revisions

slide-29
SLIDE 29

Corpus [SIGIR ’15]

Revisions over time

7

Month

103,000 vandalism revisions 24 million manual revisions

slide-30
SLIDE 30

Corpus [SIGIR ’15]

Revisions over time

7

Month

103,000 vandalism revisions 24 million manual revisions  0.4% vandalism

slide-31
SLIDE 31

Corpus [SIGIR ’15]

Revisions over time

7

Item head (1.3% vandalism) Month

103,000 vandalism revisions 24 million manual revisions  0.4% vandalism

slide-32
SLIDE 32

Corpus [SIGIR ’15]

Revisions over time

7

Item head (1.3% vandalism) Item body (0.2% vandalism) Month

103,000 vandalism revisions 24 million manual revisions  0.4% vandalism

slide-33
SLIDE 33

Corpus [SIGIR ’15]

Revisions over time

7

Item head (1.3% vandalism) Item body (0.2% vandalism) Training Month

103,000 vandalism revisions 24 million manual revisions  0.4% vandalism

slide-34
SLIDE 34

Corpus [SIGIR ’15]

Revisions over time

7

Item head (1.3% vandalism) Item body (0.2% vandalism) Training Validation Month

103,000 vandalism revisions 24 million manual revisions  0.4% vandalism

slide-35
SLIDE 35

Corpus [SIGIR ’15]

Revisions over time

7

Item head (1.3% vandalism) Item body (0.2% vandalism) Training Test Validation Month

103,000 vandalism revisions 24 million manual revisions  0.4% vandalism

slide-36
SLIDE 36

Content Features

11 Character features (e.g., lowerCaseRatio, digitRatio) 9 Word features (e.g., badWordRatio) 4 Sentence features (e.g., commentSitelinkSimilarity) 3 Statement features (e.g., propertyFrequency)

Context Features

10 User features (e.g., userCountry) 2 Item features (e.g., logItemFrequency) 8 Revision features (e.g., revisionTag, revisionLanguage)

Features (47 in total)

Stefan Heindorf 8 Vandalism Detection in Wikidata

slide-37
SLIDE 37

Features (47 in total)

Stefan Heindorf 8 Vandalism Detection in Wikidata

revisionTag

slide-38
SLIDE 38

Features (47 in total)

Stefan Heindorf 8 Vandalism Detection in Wikidata

revisionTag Vand. Total Prob.

  • Rev. with tags

52 T 8,619 T 0.60% By abuse filter 49 T 122 T 39.90% By editing tools 3 T 8,496 T 0.03%

  • Rev. w/o tags

52 T 15,386 T 0.34% revisionTag

slide-39
SLIDE 39

Features (47 in total)

Stefan Heindorf 8 Vandalism Detection in Wikidata

revisionTag Vand. Total Prob.

  • Rev. with tags

52 T 8,619 T 0.60% By abuse filter 49 T 122 T 39.90% By editing tools 3 T 8,496 T 0.03%

  • Rev. w/o tags

52 T 15,386 T 0.34% revisionTag

slide-40
SLIDE 40

Features (47 in total)

Stefan Heindorf 8 Vandalism Detection in Wikidata

revisionTag Vand. Total Prob.

  • Rev. with tags

52 T 8,619 T 0.60% By abuse filter 49 T 122 T 39.90% By editing tools 3 T 8,496 T 0.03%

  • Rev. w/o tags

52 T 15,386 T 0.34% revisionTag

slide-41
SLIDE 41

Features (47 in total)

Stefan Heindorf 8 Vandalism Detection in Wikidata

revisionTag Vand. Total Prob.

  • Rev. with tags

52 T 8,619 T 0.60% By abuse filter 49 T 122 T 39.90% By editing tools 3 T 8,496 T 0.03%

  • Rev. w/o tags

52 T 15,386 T 0.34% revisionTag

slide-42
SLIDE 42

Multiple-Instance Learning

Vandalism Detection in Wikidata Stefan Heindorf 9

slide-43
SLIDE 43

Multiple-Instance Learning

  • Observation: Vandalism seldom occurs in isolation

Vandalism Detection in Wikidata Stefan Heindorf 9

slide-44
SLIDE 44

Multiple-Instance Learning

  • Observation: Vandalism seldom occurs in isolation

Vandalism Detection in Wikidata Stefan Heindorf 9

22:35, 11 September 2013 184.19.64.111 (talk) . . (Changed English label: Barack Obama Aloha) 22:35, 11 September 2013 184.19.64.111 (talk) . . (Added English alias: Lulu:):):):):):):)) 12:05, 11 September 2013 MatmaBot (talk | contribs) . . (Changed Polish description: imported

slide-45
SLIDE 45

Multiple-Instance Learning

  • Observation: Vandalism seldom occurs in isolation

Vandalism Detection in Wikidata Stefan Heindorf 9

22:35, 11 September 2013 184.19.64.111 (talk) . . (Changed English label: Barack Obama Aloha) 22:35, 11 September 2013 184.19.64.111 (talk) . . (Added English alias: Lulu:):):):):):):)) 12:05, 11 September 2013 MatmaBot (talk | contribs) . . (Changed Polish description: imported

slide-46
SLIDE 46

Multiple-Instance Learning

  • Observation: Vandalism seldom occurs in isolation

Vandalism Detection in Wikidata Stefan Heindorf 9

22:35, 11 September 2013 184.19.64.111 (talk) . . (Changed English label: Barack Obama Aloha) 22:35, 11 September 2013 184.19.64.111 (talk) . . (Added English alias: Lulu:):):):):):):)) 12:05, 11 September 2013 MatmaBot (talk | contribs) . . (Changed Polish description: imported

Session 1 Session 2

slide-47
SLIDE 47

Multiple-Instance Learning

  • Observation: Vandalism seldom occurs in isolation
  • Idea: Apply Multiple-Instance Learning

Vandalism Detection in Wikidata Stefan Heindorf 9

22:35, 11 September 2013 184.19.64.111 (talk) . . (Changed English label: Barack Obama Aloha) 22:35, 11 September 2013 184.19.64.111 (talk) . . (Added English alias: Lulu:):):):):):):)) 12:05, 11 September 2013 MatmaBot (talk | contribs) . . (Changed Polish description: imported

Session 1 Session 2

slide-48
SLIDE 48

Multiple-Instance Learning

  • Observation: Vandalism seldom occurs in isolation
  • Idea: Apply Multiple-Instance Learning

Vandalism Detection in Wikidata Stefan Heindorf 9

22:35, 11 September 2013 184.19.64.111 (talk) . . (Changed English label: Barack Obama Aloha) 22:35, 11 September 2013 184.19.64.111 (talk) . . (Added English alias: Lulu:):):):):):):)) 12:05, 11 September 2013 MatmaBot (talk | contribs) . . (Changed Polish description: imported

Session 1 Session 2

slide-49
SLIDE 49

Multiple-Instance Learning

  • Observation: Vandalism seldom occurs in isolation
  • Idea: Apply Multiple-Instance Learning

Vandalism Detection in Wikidata Stefan Heindorf 9

22:35, 11 September 2013 184.19.64.111 (talk) . . (Changed English label: Barack Obama Aloha) 22:35, 11 September 2013 184.19.64.111 (talk) . . (Added English alias: Lulu:):):):):):):)) 12:05, 11 September 2013 MatmaBot (talk | contribs) . . (Changed Polish description: imported

Session 1 Session 2

slide-50
SLIDE 50

WDVD vs. Baselines

  • WDVD (our approach)

Wikidata Vandalism Detector

10 Vandalism Detection in Wikidata Stefan Heindorf

slide-51
SLIDE 51

WDVD vs. Baselines

  • WDVD (our approach)

Wikidata Vandalism Detector

  • FILTER (baseline)

Wikidata Abuse Filter

10 Vandalism Detection in Wikidata Stefan Heindorf

slide-52
SLIDE 52

WDVD vs. Baselines

  • WDVD (our approach)

Wikidata Vandalism Detector

  • FILTER (baseline)

Wikidata Abuse Filter

  • ORES (baseline)

Objective Revision Evaluation Service

10 Vandalism Detection in Wikidata Stefan Heindorf

slide-53
SLIDE 53

WDVD vs. Baselines

  • WDVD (our approach)

Wikidata Vandalism Detector

  • FILTER (baseline)

Wikidata Abuse Filter

  • ORES (baseline)

Objective Revision Evaluation Service

10 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

Precision Recall

Vandalism Detection in Wikidata Stefan Heindorf

Test Dataset (0.2% vandalism)

slide-54
SLIDE 54

WDVD vs. Baselines

  • WDVD (our approach)

Wikidata Vandalism Detector

  • FILTER (baseline)

Wikidata Abuse Filter

  • ORES (baseline)

Objective Revision Evaluation Service

10 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

Precision Recall

Vandalism Detection in Wikidata Stefan Heindorf

FILTER Test Dataset (0.2% vandalism)

slide-55
SLIDE 55

WDVD vs. Baselines

  • WDVD (our approach)

Wikidata Vandalism Detector

  • FILTER (baseline)

Wikidata Abuse Filter

  • ORES (baseline)

Objective Revision Evaluation Service

10 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

Precision Recall

Vandalism Detection in Wikidata Stefan Heindorf

ORES FILTER Test Dataset (0.2% vandalism)

slide-56
SLIDE 56

WDVD vs. Baselines

  • WDVD (our approach)

Wikidata Vandalism Detector

  • FILTER (baseline)

Wikidata Abuse Filter

  • ORES (baseline)

Objective Revision Evaluation Service

10 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

Precision Recall

Vandalism Detection in Wikidata Stefan Heindorf

ORES FILTER Test Dataset (0.2% vandalism)

slide-57
SLIDE 57

WDVD vs. Baselines

  • WDVD (our approach)

Wikidata Vandalism Detector

  • FILTER (baseline)

Wikidata Abuse Filter

  • ORES (baseline)

Objective Revision Evaluation Service

10 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

Precision Recall

Vandalism Detection in Wikidata Stefan Heindorf

ORES FILTER Test Dataset (0.2% vandalism)

slide-58
SLIDE 58

WDVD vs. Baselines

  • WDVD (our approach)

Wikidata Vandalism Detector

  • FILTER (baseline)

Wikidata Abuse Filter

  • ORES (baseline)

Objective Revision Evaluation Service

10 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

Precision Recall

Vandalism Detection in Wikidata Stefan Heindorf

ORES FILTER PR-AUC: 0.491 ROC-AUC: 0.991 Test Dataset (0.2% vandalism)

slide-59
SLIDE 59

WDVD vs. Baselines

  • WDVD (our approach)

Wikidata Vandalism Detector

  • FILTER (baseline)

Wikidata Abuse Filter

  • ORES (baseline)

Objective Revision Evaluation Service

10 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

Precision Recall

Vandalism Detection in Wikidata Stefan Heindorf

 Detect and revert 30% vandalism

fully automatically

ORES FILTER Test Dataset (0.2% vandalism)

slide-60
SLIDE 60

WDVD vs. Baselines

  • WDVD (our approach)

Wikidata Vandalism Detector

  • FILTER (baseline)

Wikidata Abuse Filter

  • ORES (baseline)

Objective Revision Evaluation Service

10 0.2 0.4 0.6 0.8 1 0.2 0.4 0.6 0.8 1

Precision Recall

Vandalism Detection in Wikidata Stefan Heindorf

 Detect and revert 30% vandalism

fully automatically

ORES FILTER

  • Reduce workload by factor 10

(precision 2% instead of 0.2%)  Still find 98.8% of all vandalism

Test Dataset (0.2% vandalism)

slide-61
SLIDE 61

Conclusion and Outlook

Stefan Heindorf 11 Vandalism Detection in Wikidata

slide-62
SLIDE 62

Conclusion and Outlook

Conclusion

  • Vandalism: Concentration on item heads (currently)
  • Features:

Content & Context

  • Model:

Multiple-Instance

  • PR-AUC:

0.491

  • ROC-AUC: 0.991

Stefan Heindorf 11 Vandalism Detection in Wikidata

slide-63
SLIDE 63

Conclusion and Outlook

Conclusion

  • Vandalism: Concentration on item heads (currently)
  • Features:

Content & Context

  • Model:

Multiple-Instance

  • PR-AUC:

0.491

  • ROC-AUC: 0.991

Stefan Heindorf 11 Vandalism Detection in Wikidata

Code + Data: http://www.heindorf.me/ wdvd.html

slide-64
SLIDE 64

Conclusion and Outlook

Conclusion

  • Vandalism: Concentration on item heads (currently)
  • Features:

Content & Context

  • Model:

Multiple-Instance

  • PR-AUC:

0.491

  • ROC-AUC: 0.991

Outlook

  • Goal:

Better detection (on item bodies)

  • Idea:

Double-check with other sources

Stefan Heindorf 11 Vandalism Detection in Wikidata

Code + Data: http://www.heindorf.me/ wdvd.html

slide-65
SLIDE 65

Conclusion and Outlook

Conclusion

  • Vandalism: Concentration on item heads (currently)
  • Features:

Content & Context

  • Model:

Multiple-Instance

  • PR-AUC:

0.491

  • ROC-AUC: 0.991

Outlook

  • Goal:

Better detection (on item bodies)

  • Idea:

Double-check with other sources

Stefan Heindorf 11 Vandalism Detection in Wikidata

Code + Data: http://www.heindorf.me/ wdvd.html Join the competition: Vandalism Detection @WSDM Cup 2017 http://www.wsdm-cup-2017.org/

slide-66
SLIDE 66

Conclusion and Outlook

Conclusion

  • Vandalism: Concentration on item heads (currently)
  • Features:

Content & Context

  • Model:

Multiple-Instance

  • PR-AUC:

0.491

  • ROC-AUC: 0.991

Outlook

  • Goal:

Better detection (on item bodies)

  • Idea:

Double-check with other sources

Acknowledgement

  • German Research Foundation (DFG)
  • SIGIR Student Travel Grant

Stefan Heindorf 11 Vandalism Detection in Wikidata

Code + Data: http://www.heindorf.me/ wdvd.html Join the competition: Vandalism Detection @WSDM Cup 2017 http://www.wsdm-cup-2017.org/

slide-67
SLIDE 67

Conclusion and Outlook

Conclusion

  • Vandalism: Concentration on item heads (currently)
  • Features:

Content & Context

  • Model:

Multiple-Instance

  • PR-AUC:

0.491

  • ROC-AUC: 0.991

Outlook

  • Goal:

Better detection (on item bodies)

  • Idea:

Double-check with other sources

Acknowledgement

  • German Research Foundation (DFG)
  • SIGIR Student Travel Grant

Stefan Heindorf 11 Vandalism Detection in Wikidata

Code + Data: http://www.heindorf.me/ wdvd.html Join the competition: Vandalism Detection @WSDM Cup 2017 http://www.wsdm-cup-2017.org/

Thank you!