The End is Nigh Generic Solving of Text-based Elie Bursztein, - - PowerPoint PPT Presentation

▶
the end is nigh generic solving of text based
SMART_READER_LITE
LIVE PREVIEW

The End is Nigh Generic Solving of Text-based Elie Bursztein, - - PowerPoint PPT Presentation

Jan 19, 2023 99 likes •1.09k views

The End is Nigh Generic Solving of Text-based Elie Bursztein, Jonathan Aigrain, Angelika Mosciki, John Mitchell \ https://www.elie.net/tag/captcha Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs

slide-1
SLIDE 1

\

The End is Nigh Generic Solving of Text-based

Elie Bursztein, Jonathan Aigrain, Angelika Mosciki, John Mitchell

slide-2
SLIDE 2

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

slide-3
SLIDE 3

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

slide-4
SLIDE 4

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

slide-5
SLIDE 5

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

slide-6
SLIDE 6

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

slide-7
SLIDE 7

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

slide-8
SLIDE 8

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

slide-9
SLIDE 9

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

slide-10
SLIDE 10

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

slide-11
SLIDE 11

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Real world Captchas

Baidu (2011) Baidu (2013) eBay ReCaptcha (2011) ReCaptcha (2013) Wikipedia Yahoo CNN

slide-12
SLIDE 12

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Outline

slide-13
SLIDE 13

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Outline

Old approach: segment then recognize

slide-14
SLIDE 14

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Outline

Old approach: segment then recognize Our new approach: single machine learning step

slide-15
SLIDE 15

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Outline

Old approach: segment then recognize Our new approach: single machine learning step Evaluation: efficiency on real world captchas

slide-16
SLIDE 16

\

Segment then recognize

How to break captchas the old way

slide-17
SLIDE 17

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Think lego

slide-18
SLIDE 18

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Slashdot captcha

The segment then recognize approach

slide-19
SLIDE 19

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Preprocessing

The segment then recognize approach

slide-20
SLIDE 20

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Preprocessing

The segment then recognize approach

slide-21
SLIDE 21

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Preprocessing Segmentation

The segment then recognize approach

slide-22
SLIDE 22

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Preprocessing Segmentation

The segment then recognize approach

slide-23
SLIDE 23

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Preprocessing Segmentation Post- segmentation

The segment then recognize approach

slide-24
SLIDE 24

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Preprocessing Segmentation Post- segmentation

The segment then recognize approach

slide-25
SLIDE 25

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Preprocessing Segmentation Post- segmentation Recognition

The segment then recognize approach

slide-26
SLIDE 26

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Preprocessing Segmentation Post- segmentation Recognition

f a e t e s t

The segment then recognize approach

slide-27
SLIDE 27

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Preprocessing Segmentation Post- segmentation Recognition

f a e t e s t

Post-recognition

The segment then recognize approach

slide-28
SLIDE 28

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Preprocessing Segmentation Post- segmentation Recognition

f a e t e s t f a s t e s t

Post-recognition

The segment then recognize approach

slide-29
SLIDE 29

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

How about collapsing?

Baidu (2011) Baidu (2013) eBay ReCaptcha (2011) ReCaptcha (2013) Wikipedia Yahoo CNN

slide-30
SLIDE 30

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

ad hoc/complicated heuristics

Figure 7. Detecting characters with a cross shape (the Figure 8. S Vertical Histogram. Identification o

” shape. Figure 13(b) shows a segmentation result. (a) (b)

The Robustness of Google CAPTCHAs

Ahmad S El Ahmad, Jeff Yan, Mohamad Tayara

slide-31
SLIDE 31

Elie Bursztein Captcha (in)security Confidential

I don’t know how to do it (in the general case)

slide-32
SLIDE 32

\

Captcha breaking reloaded

When AI over-come human limitations

slide-33
SLIDE 33

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

What is wrong with the previous approach?

slide-34
SLIDE 34

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

What is wrong with the previous approach?

Segmentation requires to find an invariant to exploit color, cluster size, number of characters, shape ...

slide-35
SLIDE 35

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

What is wrong with the previous approach?

Segmentation requires to find an invariant to exploit color, cluster size, number of characters, shape ...

slide-36
SLIDE 36

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

What is wrong with the previous approach?

Segmentation requires to find an invariant to exploit color, cluster size, number of characters, shape ... Captcha breaking is more exploit writing than machine learning

slide-37
SLIDE 37

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Classifiers are extremely accuracy (e.g KNN/SVM)

% success

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Trainning set size

10 20 50 100 200 500

09 AZ09 azAZ09 Distortion 3 fonts 5 fonts Angles

slide-38
SLIDE 38

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

If classifiers are that good, why are we doing pre-processing ?

slide-39
SLIDE 39

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

There is no segmentation

slide-40
SLIDE 40

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Approach overview

Cut-points detection

slide-41
SLIDE 41

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Approach overview

Cut-points detection Slicer

slide-42
SLIDE 42

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Approach overview

Cut-points detection Slicer Scorer

slide-43
SLIDE 43

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Approach overview

Cut-points detection Slicer Scorer Arbiter

slide-44
SLIDE 44

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

New approach

Captcha

slide-45
SLIDE 45

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

New approach

Captcha Slicer

slide-46
SLIDE 46

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

New approach

Captcha Slicer

slide-47
SLIDE 47

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Scorer

slide-48
SLIDE 48

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Scorer

slide-49
SLIDE 49

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Scorer

slide-50
SLIDE 50

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Scorer

slide-51
SLIDE 51

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Scorer

slide-52
SLIDE 52

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Scorer

slide-53
SLIDE 53

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Arbiter

slide-54
SLIDE 54

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Arbiter

ilebe

slide-55
SLIDE 55

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Arbiter

ilebe kinedi

slide-56
SLIDE 56

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Arbiter

ilebe kinedi ilebe

slide-57
SLIDE 57

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Arbiter

ilebe kinedi ilebe ilebe

slide-58
SLIDE 58

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Improving accuracy using reinforcement learning

ilebe ikebe ilabe

slide-59
SLIDE 59

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Improving accuracy using reinforcement learning

ilebe ikebe ilabe The classifier guide the learning process and tell humans when it need help

slide-60
SLIDE 60

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Reinforcement illustrated

Bad segmentation Bad recognition

4 1 t 4

  • 7

: : : : : :

A K E

slide-61
SLIDE 61

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Reinforcement illustrated

Bad segmentation Bad recognition

4 1 t 4

  • 7

: : : : : :

A K E

slide-62
SLIDE 62

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Reinforcement illustrated

Bad segmentation Bad recognition

4 1 t 4

  • 7

: : : : : :

A K E

slide-63
SLIDE 63

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Reducing complexity

Algorithm complexity is extremely high up to 9h on a single captcha

slide-64
SLIDE 64

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Reducing complexity

Algorithm complexity is extremely high up to 9h on a single captcha

Solutions

slide-65
SLIDE 65

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Reducing complexity

Algorithm complexity is extremely high up to 9h on a single captcha Reduce the number of possible cuts inflection points, smart elimination

Solutions

slide-66
SLIDE 66

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Reducing complexity

Algorithm complexity is extremely high up to 9h on a single captcha Reduce the number of possible cuts inflection points, smart elimination

Solutions

Faster and less accurate recognition sequential recognition, left to right

slide-67
SLIDE 67

\

Evaluation on real world captchas

slide-68
SLIDE 68

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Yahoo

slide-69
SLIDE 69

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Yahoo

Inflection points

slide-70
SLIDE 70

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Yahoo

Potential cuts Inflection points

slide-71
SLIDE 71

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Yahoo

Removing bad cuts Potential cuts Inflection points

slide-72
SLIDE 72

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Yahoo

Removing bad cuts Potential cuts Compatible cuts with start Inflection points

slide-73
SLIDE 73

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Yahoo

Removing bad cuts Potential cuts Compatible cuts with start Iterative 4cz8jyaz Inflection points

slide-74
SLIDE 74

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Recaptcha example

slide-75
SLIDE 75

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Recaptcha example

Inflection points

slide-76
SLIDE 76

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Recaptcha example

Inflection points Potential cuts

slide-77
SLIDE 77

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Recaptcha example

Inflection points Potential cuts

slide-78
SLIDE 78

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Recaptcha example

Inflection points Potential cuts

slide-79
SLIDE 79

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Recaptcha example

Inflection points Potential cuts Removing bad cuts

slide-80
SLIDE 80

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Recaptcha example

Inflection points Potential cuts Removing bad cuts Compatible cuts with start

slide-81
SLIDE 81

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Recaptcha example

Inflection points Potential cuts Removing bad cuts Compatible cuts with start Iterative adaclv

slide-82
SLIDE 82

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Recaptcha example

Inflection points Potential cuts Removing bad cuts Compatible cuts with start full graph adachi Iterative adaclv

slide-83
SLIDE 83

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

How about lines ?

Realize that lines are known shape Train classifier to recognize them as empty character

slide-84
SLIDE 84

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Baidu example

slide-85
SLIDE 85

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Baidu example

Inflection points

slide-86
SLIDE 86

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Baidu example

Inflection points Potential cuts

slide-87
SLIDE 87

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Baidu example

Inflection points Removing bad cuts Potential cuts

slide-88
SLIDE 88

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Baidu example

Inflection points Removing bad cuts Potential cuts

slide-89
SLIDE 89

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Baidu example

Inflection points Removing bad cuts Potential cuts

slide-90
SLIDE 90

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Baidu example

Inflection points Removing bad cuts Compatible cuts with start Potential cuts

slide-91
SLIDE 91

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Baidu example

Inflection points Removing bad cuts Compatible cuts with start Potential cuts Iterative a6yk--

slide-92
SLIDE 92

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Baidu example

Inflection points Removing bad cuts Compatible cuts with start Potential cuts Iterative a6yk-- full graph a6yk--

slide-93
SLIDE 93

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Overall results

Reinforcement learning Simple learning Previous Work Full L-R L-R Time Seq. Full L-R Seq Accuracy Delta Ref. Baidu (2011) 38.68% 33.42% 3.94 s 36.58% 17.27% 16.55% 16.69% 5% +33.6% [10] Baidu (2013) 55.22% 54.38% 1.9 s 54.38%

  • 51%

+4.22% [21] CNN

  • 51.09%

4.9 s 48.54%

  • 46.40%

45.96% 16% +35.09% [10] eBay 51.39% 47.92% 2.31 s 48.61% 39.43% 40.14% 36.29% 43% +11.4% [10] ReCaptcha (2011) 22.67% 21.74% 7.16 s 19.25% 19.86% 18.25% 17.10% 40.4%

  • 17.73%

[15] ReCaptcha (2013) 22.34% 19.22% 4.59 s 19.74% 20% 14.61% 12.77% Wikipedia

  • 28.29%
  • 26.36%
  • 27.02%

26.24% 25% +3.3% [10] Yahoo

  • 3.67%

7.95 s 5.33%

  • 2.72%

2.29%

Generic approach: Succeed where the old approach fail and most often beat it where it worked

slide-94
SLIDE 94

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Human vs computers

Human accuracy Breaker accuracy (left-right) Accuracy 0% 20% 40% 60% 80% 100% Spacing between letters 0px

  • 1px
  • 2px
  • 3px
  • 4px
  • 5px
  • 6px
  • 7px

The gap between Human and computer recognition is now too narrow to be considered secure

slide-95
SLIDE 95

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

The end of an era, the dawn of a new one

slide-96
SLIDE 96

Elie Bursztein The End is Nigh: Generic Solving of Text-based CAPTCHAs https://www.elie.net/tag/captcha

Summary New generation

Deprecated Segment and Recognize attacks Risk analysis defense Purely based AI Attack Text distortions defense

slide-97
SLIDE 97

Elie Bursztein, Matthieu Martin, John C. Mitchell Text-based CAPTCHAS strenghts and weaknesses http://ly.tl/p22

Captcha research https://www.elie.net/tag/captcha Follow-me on Twitter @elie

Questions ?