Don't Bite the Newbies: How Reverts affect the Quantity and Quality - - PowerPoint PPT Presentation

▶
don t bite the newbies how reverts affect the quantity
SMART_READER_LITE
LIVE PREVIEW

Don't Bite the Newbies: How Reverts affect the Quantity and Quality - - PowerPoint PPT Presentation

Mar 07, 2023 145 likes •563 views

Don't Bite the Newbies: How Reverts affect the Quantity and Quality of Wikipedia Work Aaron Halfaker, Niki Kittur and John Riedl Wikipedia:Volunteer participation Decreasing barriers Increases the number of participants Allows

slide-1
SLIDE 1

Don't Bite the Newbies: How Reverts affect the Quantity and Quality of Wikipedia Work

Aaron Halfaker, Niki Kittur and John Riedl

slide-2
SLIDE 2

Wikipedia:Volunteer participation

  • Decreasing barriers
  • Increases the number of participants
  • Allows malicious changes to be made
slide-3
SLIDE 3

Wikipedia:Volunteer participation

  • Decreasing barriers
  • Increases the number of participants
  • Allows malicious changes to be made
slide-4
SLIDE 4

Wiki solution: The revert

  • The revert
slide-5
SLIDE 5

What does a revert look like?

time

slide-6
SLIDE 6

Revert – Not just for vandalism

  • Reverts can represent something more

substantial

  • ~40% of reverts in English Wikipedia

– Not related to vandalism – Not self-reverts (mistakes)

  • Prior work
  • Editors are demotivated when their work is edited[1].

– Newbies especially

  • Newbies are getting reverted more & this has been

implicated in their recent decrease[2].

1.X. Zhang and F. Zhu. Intrinsic motivation of open content contributors: The case of Wikipedia. WISE, 2006. 2.B. Suh et al. The singularity is not near: Slowing growth of Wikipedia. WikiSym’09, pages 1–10, NY, NY, USA, 2009.

slide-7
SLIDE 7

Research Question:

What effect does being reverted have on editor activity?

slide-8
SLIDE 8

The Message

(thanks John!)

  • Editors appear to be demotivated by being

reverted

  • This is especially true for newcomers
  • and especially true for those reverted by old-timers
  • But, reverts appear to be an effective training

mechanism

  • Reverts appear to have a net positive effect on

editor productivity.

slide-9
SLIDE 9

Outline

  • 1. Research Questions & Metrics
  • 2. Measuring changes in editor activity
  • 3. Results & Implications
slide-10
SLIDE 10

Outline

  • 1. Research Questions & Metrics
  • 2. Measuring changes in editor activity
  • 3. Results & Implications
slide-11
SLIDE 11

Outline

  • 1. Research Questions & Metrics
  • 2. Measuring changes in editor activity
  • 3. Results & Implications
slide-12
SLIDE 12

Research Questions & Metrics

slide-13
SLIDE 13

Research Questions & Metrics

RQ1: How does being reverted affect the quantity of editor work?

slide-14
SLIDE 14

RQ1: How does being reverted affect the quantity of editor work?

Research Questions & Metrics

RQ2: How does being reverted affect the quality of editor work?

slide-15
SLIDE 15

RQ1: How does being reverted affect the quantity of editor work? RQ2: How does being reverted affect the quality of editor work?

Research Questions & Metrics

Persistent Word Revisions

  • Quality of an edit be measured by observing how other editors react.
  • The longer and editor's contributions last, the higher the quality of work.

Problem: Safer edits appear to be higher quality.

slide-16
SLIDE 16

Research Questions & Metrics

RQ1: How does being reverted affect the quantity of editor work? RQ2: How does being reverted affect the quality of editor work?

slide-17
SLIDE 17

Research Questions & Metrics

RQ1: How does being reverted affect the quantity of editor work? RQ2: How does being reverted affect the quality of editor work?

An edit is bold when it is more likely to be noticed by other editors.

  • Bold edits change more article content.
  • Bold edits change more established content.

Problem: Does the change in boldness account for the change in quality?

slide-18
SLIDE 18

Metrics

RQ1: How does being reverted affect the quantity of editor work? RQ2: How does being reverted affect the quality of editor work?

slide-19
SLIDE 19

Research Questions & Metrics

RQ1: How does being reverted affect the quantity of editor work? RQ2: How does being reverted affect the quality of editor work?

Productivity: The product of the quantity and quality of an editor's work

  • Lacks the problem of boldness.
slide-20
SLIDE 20

Research Questions & Metrics

RQ1: How does being reverted affect the quantity of editor work? RQ2: How does being reverted affect the quality of editor work? RQ3: How does being reverted affect editor communication?

slide-21
SLIDE 21

Research Questions & Metrics

RQ1: How does being reverted affect the quantity of editor work? RQ2: How does being reverted affect the quality of editor work? RQ3: How does being reverted affect editor communication? RQ4: How does editor experience moderate these effects?

Tenure: Time since editor's first edit in Wikipedia.

slide-22
SLIDE 22

Research Questions & Metrics

RQ1: How does being reverted affect the quantity of editor work? RQ2: How does being reverted affect the quality of editor work? RQ3: How does being reverted affect editor communication?

slide-23
SLIDE 23

Measuring changes in editor activity: The “activity delta”

slide-24
SLIDE 24

Measuring Changes: Activity Deltas

slide-25
SLIDE 25

Measuring Changes: Activity Deltas

slide-26
SLIDE 26

Measuring Changes: Activity Deltas

slide-27
SLIDE 27

Measuring Changes: Activity Deltas

Sample: 400,000 revisions:

  • 200,000 reverted (not vandal or self-reverts)
  • 200,000 not-reverted

Controlled measures of activity by sd(week -1)

  • Δ/σ AKA “effect size”
  • Allows direct comparison of all editors, regardless of

edit rate

slide-28
SLIDE 28

Discovering likely effects

  • Linear and Logistic regressions identify effect
  • Determines direction and strength of correlation
  • Allows us to control for confounding factor by

including then in them as explanatory variables to control for their effects.

– Reverted editor experience – Reverting editor experience – Characteristics of the reverted edit

  • Words added/removed, Establishment of words removed, etc.
  • Plots help us reason about the shape of effect
slide-29
SLIDE 29

Results & Implications

slide-30
SLIDE 30

RQ1: Effect on Quantity of work

Newbie: Less than one month of tenure Old-timer: More than one year of tenure

slide-31
SLIDE 31

RQ1: Effect on Quantity of work

Newbie: Less than one month of tenure Old-timer: More than one year of tenure Reverted editor Reverting editor

slide-32
SLIDE 32

RQ1: Effect on Quantity of work

  • Lower probability of survival (survival = edit for 8 more

weeks)

  • Reduced quantity of work even for those who survive
  • Reverted newbies decrease their quantity of work more
  • Editors reverted by old-timers decrease their quantity of

work more

slide-33
SLIDE 33

RQ2: Effect on Quality of work

(quality: reverts/revision)

slide-34
SLIDE 34

RQ2: Effect on Quality of work

(boldness)

slide-35
SLIDE 35

RQ2: Effect on Quality of work

(productivity: PWR/day)

slide-36
SLIDE 36

RQ2: Effect on Quality of work

  • Quality (acceptability to others) increased
  • Boldness of work decreased
  • Productivity is higher – including editors who

leave or decrease their work

  • Especially for newbies and less productive editors
slide-37
SLIDE 37

RQ3: Effect on Communication

(Controlled for editor tenure)

slide-38
SLIDE 38

RQ3: Effect on Communication

  • Reduced personal communications to other

editors

  • Could be a bad sign. Commiseration with other

editors could reinforce motivation.

  • Maintain communication over article content
  • Wikipedia's Bold, Revert, Discuss cycle encourages

this behavior.

– Could support learning – Essential for finding consensus after conflict

slide-39
SLIDE 39

Summary

  • Being reverted predicts:
  • Decreases in quantity of work
  • Increase in quality of work
  • Both predictions are more extreme for reverted

newcomers.

➢ Special concern should be granted for reverted

newbies

  • Being reverted by an old-timer intensifies the

decrease in quantity, no change to quality

➢ Highly experienced editors should be kept away

from newbies.

slide-40
SLIDE 40

Acknowledgments

  • My co-authors
  • Niki Kittur - nkittur@cs.cmu.edu
  • John Riedl - riedl@cs.umn.edu
  • The Wikipedia group in the GroupLens lab
  • This work has been financially supported by the

National Science Foundation: IIS 08-08692 and 09-68483.

Aaron Halfaker halfaker@cs.umn.edu http://halfaker.info