How Good are Humans at Solving d r CAPTCHAs? A Large Scale - - PowerPoint PPT Presentation

▶
how good are humans at solving
SMART_READER_LITE
LIVE PREVIEW

How Good are Humans at Solving d r CAPTCHAs? A Large Scale - - PowerPoint PPT Presentation

Feb 23, 2024 141 likes •1.2k views

b a L y t i r u c e S r e t u p m o C How Good are Humans at Solving d r CAPTCHAs? A Large Scale Evaluation o f n a Elie Bursztein, Steven Bethard, Celine Fabry, John t S Mitchell, Dan Jurafsky, http://ly.tl/p11 E.

slide-1
SLIDE 1

S t a n f

  • r

d C

  • m

p u t e r S e c u r i t y L a b

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Elie Bursztein, Steven Bethard, Celine Fabry, John Mitchell, Dan Jurafsky,

slide-2
SLIDE 2
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-3
SLIDE 3
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

?

slide-4
SLIDE 4
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

?

users

slide-5
SLIDE 5
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

?

users bots

slide-6
SLIDE 6
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

?

users bots CAPTCHA Completely Automated Public Turing test to tell Computers and Humans Apart

slide-7
SLIDE 7
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-8
SLIDE 8
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-9
SLIDE 9
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-10
SLIDE 10
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-11
SLIDE 11
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-12
SLIDE 12
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-13
SLIDE 13
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-14
SLIDE 14
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-15
SLIDE 15
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-16
SLIDE 16
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-17
SLIDE 17
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-18
SLIDE 18
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-19
SLIDE 19
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-20
SLIDE 20
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-21
SLIDE 21
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-22
SLIDE 22
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-23
SLIDE 23
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-24
SLIDE 24
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-25
SLIDE 25
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-26
SLIDE 26
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-27
SLIDE 27
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-28
SLIDE 28
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-29
SLIDE 29
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

93%

slide-30
SLIDE 30
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

86% 93%

slide-31
SLIDE 31
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

70% 86% 93%

slide-32
SLIDE 32
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-33
SLIDE 33
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-34
SLIDE 34
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-35
SLIDE 35
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-36
SLIDE 36
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-37
SLIDE 37
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

7.3 sec

slide-38
SLIDE 38
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

7.3 sec 8.2 sec

slide-39
SLIDE 39
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

9.3 sec 7.3 sec 8.2 sec

slide-40
SLIDE 40
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

10.6 sec 9.3 sec 7.3 sec 8.2 sec

slide-41
SLIDE 41
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Outline

  • Study methodology
  • Population demography
  • Captcha measures
slide-42
SLIDE 42
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Accuracy

slide-43
SLIDE 43
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Solving time Accuracy

slide-44
SLIDE 44
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

The supply chain

Websites Paper

slide-45
SLIDE 45
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

The supply chain

Scraping

Websites Paper

slide-46
SLIDE 46
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

The supply chain

Scraping

Websites

Solving

Paper

slide-47
SLIDE 47
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

The supply chain

Scraping

Websites

Solving Data Mining

Paper

slide-48
SLIDE 48
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Scraping

  • Alexa top 50
  • 23 scheme
  • 10 000 captcha samples
  • Custom scraper
  • Cookies
  • Javascript events
  • Ip rate limiting
  • User agent
slide-49
SLIDE 49
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Scraping

  • Alexa top 50
  • 23 scheme
  • 10 000 captcha samples
  • Custom scraper
  • Cookies
  • Javascript events
  • Ip rate limiting
  • User agent
slide-50
SLIDE 50
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-51
SLIDE 51
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize

slide-52
SLIDE 52
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu

slide-53
SLIDE 53
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu captcha.net

slide-54
SLIDE 54
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu captcha.net eBay

slide-55
SLIDE 55
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu captcha.net eBay Digg

slide-56
SLIDE 56
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu captcha.net eBay Digg Google

slide-57
SLIDE 57
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu captcha.net eBay Digg Google Blizzard

slide-58
SLIDE 58
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu captcha.net eBay Digg Google Blizzard Yahoo

slide-59
SLIDE 59
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu captcha.net eBay Digg Google Blizzard Yahoo Microsoft

slide-60
SLIDE 60
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu captcha.net eBay Digg Google Blizzard Yahoo Microsoft recaptcha

slide-61
SLIDE 61
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu captcha.net eBay Digg Google Blizzard Yahoo Microsoft recaptcha Skyrock

slide-62
SLIDE 62
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu captcha.net eBay Digg Google Blizzard Yahoo Microsoft recaptcha Skyrock Slashdot

slide-63
SLIDE 63
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu captcha.net eBay Digg Google Blizzard Yahoo Microsoft recaptcha Skyrock Slashdot mail.ru

slide-64
SLIDE 64
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation
slide-65
SLIDE 65
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize

slide-66
SLIDE 66
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Digg

slide-67
SLIDE 67
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Digg eBay

slide-68
SLIDE 68
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Digg eBay Google

slide-69
SLIDE 69
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Digg eBay Google Microsoft

slide-70
SLIDE 70
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Digg eBay Google Microsoft recaptcha

slide-71
SLIDE 71
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Digg eBay Google Microsoft recaptcha Slashdot

slide-72
SLIDE 72
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Digg eBay Google Microsoft recaptcha Slashdot Yahoo

slide-73
SLIDE 73
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Precision is costly

slide-74
SLIDE 74
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Precision is costly

1000

0.1% Precision accuracy

slide-75
SLIDE 75
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Precision is costly

1000

0.1% Precision accuracy Knowing the probable answer

3 x

slide-76
SLIDE 76
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Precision is costly

1000

0.1% Precision accuracy Knowing the probable answer

3 x 3000

by scheme

slide-77
SLIDE 77
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Precision is costly

slide-78
SLIDE 78
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Precision is costly

63000 captcha

slide-79
SLIDE 79
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Underground API

slide-80
SLIDE 80
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

MTurk by Amazon

Worker(s) Requester (us)

slide-81
SLIDE 81
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Requester interface

slide-82
SLIDE 82
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Worker interface

slide-83
SLIDE 83
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever

slide-84
SLIDE 84
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever

  • 8 audio schemes
slide-85
SLIDE 85
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever

  • 8 audio schemes
  • 13 images schemes
slide-86
SLIDE 86
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever

  • 8 audio schemes
  • 13 images schemes
  • 1000 x 3 image captchas / scheme (bypass-captcha)
slide-87
SLIDE 87
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever

  • 8 audio schemes
  • 13 images schemes
  • 1000 x 3 image captchas / scheme (bypass-captcha)
  • 3500 x 3 audio captchas / scheme (MTurk)
slide-88
SLIDE 88
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever

  • 8 audio schemes
  • 13 images schemes
  • 1000 x 3 image captchas / scheme (bypass-captcha)
  • 3500 x 3 audio captchas / scheme (MTurk)
  • 5000 x 3 image captchas / scheme (MTurk)
slide-89
SLIDE 89
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever

  • 8 audio schemes
  • 13 images schemes
  • 1000 x 3 image captchas / scheme (bypass-captcha)
  • 3500 x 3 audio captchas / scheme (MTurk)
  • 5000 x 3 image captchas / scheme (MTurk)
  • 318 000 captchas annotated overall
slide-90
SLIDE 90
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

3502 2791 625 578 331 95 71 64 52 51 39 33 30 28 23 21 21 19 17 16 15 15 15 15 13 12 149

Tamil English Malayalam Hindi/Urdu Telugu Romanian Spanish Kannada Bengali Mandarin Marathi Slovene Gujarati German French Dutch Macedonian Arabic Cebuano Bikol Vietnamese Punjabi Hebrew Portuguese Balochi Russian Other 500 1000 1500 2000 2500 3000 3500 4000

Language repartition

slide-91
SLIDE 91
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

!!"# $%"# &'"# &"#&"# ()*+,-./0# 123+#4*+..-# 5)06,/# 7.#8./9)-#,:;*)<.=# >+?@#

Repartition by education

slide-92
SLIDE 92
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

205 239 429 686 727 658 647 708 318 361 388 364 351 268 182 177 133 230 106 113 137 91 132 76 104 38 37 103 63 70 46 26 56 36 42 26 58 35 26 18 11 10 21 2 6 1 2 16 15 2 2 1 Number of users

100 200 300 400 500 600 700 800 18192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646667687172

Age repartition

slide-93
SLIDE 93
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Nb of user distinct answer for image scheme (bp)

85.484 73.793 43.852 64.253 80.672 70.667 20.048 68.49 37.694 79.557 46.855 71.635 70.492 13.825 21.839 42.008 30.543 16.597 24 42.512 25.821 40.576 18.966 33.839 24.519 25.615 0.69124 4.3678 14.139 5.2036 2.7311 5.3333 37.44 5.6893 21.729 1.4778 19.306 3.8462 3.8934

Authorize Baidu Captchas.net Digg eBay Google Mail.ru Microsoft Recaptcha Skyrock Slashdot Blizzard Yahoo 0% 20% 40% 60% 80%

1 answer 2 answer 3 answer

slide-94
SLIDE 94
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

93.251 80.978 59.949 78.495 82.271 66.718 34.608 53.248 42.747 87.11 68.117 85.494 68.705 6.5178 16.319 32.113 19.031 15.827 25.385 41.004 33.633 38.565 11.995 26.132 13.632 26.032 0.23095 2.7027 7.9385 2.4745 1.9013 7.8974 24.388 13.119 18.689 0.89514 5.7506 0.87449 5.2632

Authorize Baidu captchas.net Digg eBay Google Mail.ru Microsoft Recaptcha Skyrock Slashdot Blizzard Yahoo 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

1 answer 2 answer 3 answer

Nb of user distinct answer for image scheme (mk)

slide-95
SLIDE 95
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation 19.113 1.0129 24.787 0.40965 1.5264 7.8678 32.486 34.363 39.232 12.615 38.797 4.1875 10.823 25.512 39.474 36.439 41.655 86.372 36.417 95.403 87.65 66.62 28.04 29.197

Authorize Digg eBay Google Microsoft Recaptcha Slashdot Yahoo 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

1 answer 2 answer 3 answer

Nb of user distinct answer for audio scheme

slide-96
SLIDE 96
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Image solving time

100 100 200 100 100 100 50 100 200 200 100 200 200 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

authorize baidu captchas.net digg ebay google mailru mslive recaptcha skyrock slashdot blizzard yahoo

slide-97
SLIDE 97
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Audio solving time

50 50 50 50 50 50 50 50 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031323334353637383940

Authorize Digg eBay Google Microsoft Recaptcha Slashdot Yahoo

slide-98
SLIDE 98
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Accuracy by education

0.87 0.88 0.88 0.87 0.85 0.51 0.54 0.52 0.51 0.54 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Not formal High School Bachelor Master Phd

Image captcha Audio captcha

slide-99
SLIDE 99
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Solving time by education

9.6 8.49 9.36 9.16 7.64 19.75 19.44 23.67 23.25 21.33 seconds 10 20 30 40 50 60 Not formal High School Bachelor Master Phd Solving time for image Solving time for audio

slide-100
SLIDE 100
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Image captcha accuracy time by language

0.979 0.928 0.848 0.925 0.935 0.873 0.704 0.794 0.772 0.956 0.890 0.955 0.874 0.976 0.927 0.837 0.919 0.935 0.861 0.700 0.804 0.738 0.954 0.867 0.946 0.881

Authorize Baidu captchas.net Digg eBay Google Mail.ru Microsoft Recaptcha Skyrock Slashdot Blizzard Yahoo 0.70 0.75 0.80 0.85 0.90 0.95 1.00

Native speakers Non-Native speakers

slide-101
SLIDE 101
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Image captcha solving time by language

5.9 6.3 7.0 7.2 6.7 8.0 11.1 11.5 9.4 7.0 6.2 8.3 9.4 7.1 7.4 8.7 8.6 7.5 10.4 13.5 13.7 12.8 8.3 8.3 9.7 11.1

Authorize Baidu captchas.net Digg eBay Google Mail.ru Microsoft Recaptcha Skyrock Slashdot Blizzard Yahoo 0 s 2 s 4 s 6 s 8 s 10 s 12 s 14 s

Native speakers Non-native speakers

slide-102
SLIDE 102
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

0.63 0.39 0.65 0.35 0.39 0.50 0.73 0.71 0.57 0.37 0.62 0.35 0.37 0.45 0.65 0.67

Authorize Digg eBay Google Microsoft Recaptcha Slashdot Yahoo 0.2 0.3 0.4 0.5 0.6 0.7

Native speakers Non-native speakers

Audio captcha accuracy time by language

slide-103
SLIDE 103
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Audio captcha solving time by language

11.94 14.83 11.84 35.20 16.63 30.12 11.70 25.00 15.41 18.89 14.49 33.21 19.48 30.63 18.39 27.27

Authorize Digg eBay Google Microsoft Recaptcha Slashdot Yahoo 0 s 5 s 10 s 15 s 20 s 25 s 30 s 35 s

Native speakers Non-native speakers
slide-104
SLIDE 104
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Conclusion / ongoing work

  • Captcha security rely on many different techniques
  • Not clear how to design them well
  • Ongoing work
  • What makes a captcha hard/slow for human ?
  • What makes a captcha secure ?
slide-105
SLIDE 105
  • E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky

http://ly.tl/p11

How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Features interactions