Validating every change Vandalism As online communities grow, - - PowerPoint PPT Presentation

▶
validating every change vandalism
SMART_READER_LITE
LIVE PREVIEW

Validating every change Vandalism As online communities grow, - - PowerPoint PPT Presentation

Aug 21, 2023 444 likes •764 views

Validating every change Vandalism As online communities grow, destructive actors increase OSM is vulnerable Mapbox protects the users from harmful data Incorrect/poor quality Harmful data Graffiti Showdown Creative labels

slide-1
SLIDE 1
slide-2
SLIDE 2

Validating every change

slide-3
SLIDE 3

Vandalism

  • As online communities

grow, destructive actors increase

  • OSM is vulnerable
  • Mapbox protects the

users from harmful data

slide-4
SLIDE 4

Incorrect/poor quality

slide-5
SLIDE 5

Harmful data

slide-6
SLIDE 6

Graffiti Showdown

slide-7
SLIDE 7

Creative labels

slide-8
SLIDE 8

Creative labels

slide-9
SLIDE 9

Creative labels

slide-10
SLIDE 10

Creative labels

slide-11
SLIDE 11

Creative labels

slide-12
SLIDE 12

Creative labels

slide-13
SLIDE 13

Creative labels

slide-14
SLIDE 14

Creative labels

slide-15
SLIDE 15

Creative labels

slide-16
SLIDE 16

Statistics (per million changes)

  • 570 incorrect labels
  • 300 editing failures

(dragged nodes)

  • 160 spam incidents
  • 100 harmful deletions
  • 50 obscene labels
  • 20 graffiti
  • ...
slide-17
SLIDE 17

Daily change statistics

  • 2 million features get touched
  • 10k label edits
  • 30k changesets

○ 0.2% is vandalism ○ 2% are low quality

  • 20k new contributors join

monthly

○ 30% of new users make a mistake in their first 10 edits

slide-18
SLIDE 18

Daily touched features by data layer

slide-19
SLIDE 19

Sharp angles

Past approaches at Mapbox

  • Validating changesets
  • Relying only on

algorithms

  • Monitor new users
  • Building blacklists

One approach does not address all cases of vandalism.

Potential vandalism Profanity check Vandalism Human review

slide-20
SLIDE 20

Approach

Step 1

Split the OSM mono layer into data layers

Step 2 Step 3

Cluster daily changes into deltas. Diff the changes per day.

slide-21
SLIDE 21

A new unit of change

slide-22
SLIDE 22

Approach

Step 4

Review the daily changes

Step 5 Step 6

Share harmful changes and fix them Apply the updates to the map and protect from harmful changes

slide-23
SLIDE 23

Machine review

  • Profanity checking in 100

languages for labels

  • Use NLP to determine how

likely a label is a place name

  • Shape classifiers for likelihood
  • f a shape being a building
  • Drastic changes to stable

features

  • ...
slide-24
SLIDE 24

Human review

  • Review changes in

○ geometry ○ labels ○ hierarchy ○ primary tags

  • Classify harmful

changes

slide-25
SLIDE 25

Isolate changes

slide-26
SLIDE 26

QA of reviews

  • Review team regularly

gets sampled

  • >99% accuracy for

selected cases

  • Expert mappers double

check each review and single out the problematic features

slide-27
SLIDE 27

Review statistics from Mapbox

  • Our review team reviews all 80’000

changes on a daily basis

  • We flag around 1000-2000 changes

a day

  • We fix >200 defects on a daily basis
  • 50% of issues are fixed by OSM
slide-28
SLIDE 28

Daily catch

slide-29
SLIDE 29

Sharing vandalism detections

  • smcha.mapbox.com

is the one stop shop for OSM validation

  • All our harmful

detections are made public

  • Mapbox regularly fixes

harmful data

slide-30
SLIDE 30

Sharing harmful edits

slide-31
SLIDE 31

Takeaways

  • Only 0.2% of edits are vandalism
  • OSM is eventually consistent
  • Mapbox provides you a validated view of

OSM

  • Let’s protect the future of OSM before

vandalism becomes a bigger problem

  • We need better shared monitoring

efforts