Toward a Field Study on the Impact of Hacking Competitions on Secure - - PowerPoint PPT Presentation

▶
toward a field study on the impact of hacking
SMART_READER_LITE
LIVE PREVIEW

Toward a Field Study on the Impact of Hacking Competitions on Secure - - PowerPoint PPT Presentation

Apr 08, 2023 161 likes •574 views

Toward a Field Study on the Impact of Hacking Competitions on Secure Development Daniel Votipka , Hongyi Hu, Bryan Eastes, and Michelle L. Mazurek 12 Aug 2018 SECURE DEVELOPMENT 2 SECURE DEVELOPMENT 2 SECURE DEVELOPMENT 2 SECURE

slide-1
SLIDE 1

Toward a Field Study on the Impact of Hacking Competitions on Secure Development

Daniel Votipka, Hongyi Hu, Bryan Eastes, and Michelle L. Mazurek

12 Aug 2018

slide-2
SLIDE 2

SECURE DEVELOPMENT

2

slide-3
SLIDE 3

SECURE DEVELOPMENT

2

slide-4
SLIDE 4

SECURE DEVELOPMENT

2

slide-5
SLIDE 5

SECURE DEVELOPMENT

2

slide-6
SLIDE 6

SECURE DEVELOPMENT

2

  • Non-experts lack experience
  • Experts learn through CTFs

[Votipka et al., 2018]

slide-7
SLIDE 7

3

  • Attack-oriented competitions
  • Goal: find and exploit

vulnerabilities

  • Simple, vulnerable programs
  • Expose competitors to several

classes of vulnerabilities

CAPTURE THE FLAG (CTF)

slide-8
SLIDE 8

3

  • Attack-oriented competitions
  • Goal: find and exploit

vulnerabilities

  • Simple, vulnerable programs
  • Expose competitors to several

classes of vulnerabilities

CAPTURE THE FLAG (CTF)

Do these help in practice?

slide-9
SLIDE 9
  • 1. Do CTFs improve prevention of

security issues?

  • 2. Do CTFs improve recognition of

security issues?

4

RESEARCH QUESTIONS

slide-10
SLIDE 10

5

Time

Dropbox Capture- the-Flag

(1 week)

PILOT STUDY OVERVIEW

slide-11
SLIDE 11

5

Time

Dropbox Capture- the-Flag

(1 week)

Diary Study

Diary Surveys

(10 mins: 6 weeks, 2x/week, 1x/day)

PILOT STUDY OVERVIEW

slide-12
SLIDE 12

5

Time

Dropbox Capture- the-Flag

(1 week)

Diary Study

Diary Surveys

(10 mins: 6 weeks, 2x/week, 1x/day)

Knowledge Assessment

Pre-CTF Assessment

(60 mins)

Post-CTF Assessment

(60 mins)

PILOT STUDY OVERVIEW

slide-13
SLIDE 13

6

Time

Pre-CTF Assessment

(60 mins)

Dropbox Capture- the-Flag

(1 week)

Diary Surveys

(10 mins: 6 weeks, 2x/week, 1x/day)

Post-CTF Assessment

(60 mins)

  • Survey regarding recent commit
  • Issues considered
  • Reasons for considering each issue
  • Actions taken to resolve
  • Not security specific
  • Open to all Dropbox developers

DIARY SURVEYS

slide-14
SLIDE 14

6

Time

Pre-CTF Assessment

(60 mins)

Dropbox Capture- the-Flag

(1 week)

Diary Surveys

(10 mins: 6 weeks, 2x/week, 1x/day)

Post-CTF Assessment

(60 mins)

  • Survey regarding recent commit
  • Issues considered
  • Reasons for considering each issue
  • Actions taken to resolve
  • Not security specific
  • Open to all Dropbox developers

DIARY SURVEYS

Measure impact of CTF

  • n day-to-day decisions
slide-15
SLIDE 15

7

Time

Pre-CTF Assessment

(60 mins)

Dropbox Capture- the-Flag

(1 week)

Diary Surveys

(10 mins: 6 weeks, 2x/week, 1x/day)

Post-CTF Assessment

(60 mins)

  • Part 1: Find vulnerabilities in insecure code
  • Copy of the Dropbox codebase
  • 4 known vulnerabilities
  • Part 2: Write a secure program
  • Only CTF participants

KNOWLEDGE ASSESSMENT

slide-16
SLIDE 16

7

Time

Pre-CTF Assessment

(60 mins)

Dropbox Capture- the-Flag

(1 week)

Diary Surveys

(10 mins: 6 weeks, 2x/week, 1x/day)

Post-CTF Assessment

(60 mins)

  • Part 1: Find vulnerabilities in insecure code
  • Copy of the Dropbox codebase
  • 4 known vulnerabilities
  • Part 2: Write a secure program
  • Only CTF participants

KNOWLEDGE ASSESSMENT

Measure improvements to secure development in a controlled setting

slide-17
SLIDE 17

8

  • Number of flagged commits
  • Communication with the Dropbox

security team

ADDITIONAL METRICS

slide-18
SLIDE 18

9

  • Diary Surveys
  • 28 participants (12 CTF)
  • 169 surveys
  • Knowledge Assessment
  • 7 participants

PILOT PARTICIPATION

slide-19
SLIDE 19

9

  • Diary Surveys
  • 28 participants (12 CTF)
  • 169 surveys
  • Knowledge Assessment
  • 7 participants

PILOT PARTICIPATION

  • Small sample
  • Methodological

issues addressed in future iterations

slide-20
SLIDE 20

10

  • Security considered in 17/124

functionality changes

  • 19% CTF, 13% non-CTF

DIARY SURVEYS

slide-21
SLIDE 21

10

  • Security considered in 17/124

functionality changes

  • 19% CTF, 13% non-CTF

DIARY SURVEYS

CTF participants considered security more often

slide-22
SLIDE 22

11

Auth Bug Local File Disclosure Logic Privacy SQLi SSRF XSS

20 40 60 Percentage of functionality changes

CTF Non-CTF CSRF XSS SQLi Privacy Logic Local File Disclosure Auth Bug

VULNERABILITIES CONSIDERED

slide-23
SLIDE 23

11

Auth Bug Local File Disclosure Logic Privacy SQLi SSRF XSS

20 40 60 Percentage of functionality changes

CTF Non-CTF CSRF XSS SQLi Privacy Logic Local File Disclosure Auth Bug

VULNERABILITIES CONSIDERED

Everyone considered logic- based vulnerabilities

slide-24
SLIDE 24

11

Auth Bug Local File Disclosure Logic Privacy SQLi SSRF XSS

20 40 60 Percentage of functionality changes

CTF Non-CTF CSRF XSS SQLi Privacy Logic Local File Disclosure Auth Bug

VULNERABILITIES CONSIDERED

CTF participants considered non- functionality vulnerabilities from the CTF Everyone considered logic- based vulnerabilities

slide-25
SLIDE 25

Hacker Sensitive Data Similar Standard Practice eammate

  • ol

12

Tool Teammate Standard Practice Similar Exp. Sensitive Data Hacker

20 40 60 80 Percentage of functionality changes

CTF Non-CTF

REASONS FOR CONSIDERING ISSUES

slide-26
SLIDE 26

Hacker Sensitive Data Similar Standard Practice eammate

  • ol

12

Tool Teammate Standard Practice Similar Exp. Sensitive Data Hacker

20 40 60 80 Percentage of functionality changes

CTF Non-CTF

REASONS FOR CONSIDERING ISSUES

CTF participants adopted an adversarial mindset

slide-27
SLIDE 27

Expert External Doc Later Review Previous Experience System Doc eammate 10 20 30 40 50

13

10 20 30 40 50 Percentage of functionality changes

CTF Non-CTF System Doc Teammate Previous Exp. Later Review External Doc Expert

ACTIONS TAKEN

slide-28
SLIDE 28

Expert External Doc Later Review Previous Experience System Doc eammate 10 20 30 40 50

13

10 20 30 40 50 Percentage of functionality changes

CTF Non-CTF System Doc Teammate Previous Exp. Later Review External Doc Expert

ACTIONS TAKEN

CTF participants sought help outside of their team

slide-29
SLIDE 29

14

CTF Scores

2.5 5

0.0 2.5 5.0

2000 4000 6000

KNOWLEDGE ASSESSMENT

Change in Assessment Score

slide-30
SLIDE 30

15

CTF Scores

2.5 5

0.0 2.5 5.0

2000 4000 6000

KNOWLEDGE ASSESSMENT

Average CTF Score 1306

Change in Assessment Score

slide-31
SLIDE 31

16

CTF Scores

2.5 5

0.0 2.5 5.0

2000 4000 6000

KNOWLEDGE ASSESSMENT

Average Change in Assessment Score 1.36

Change in Assessment Score

slide-32
SLIDE 32

17

CTF Scores

2.5 5

Change in Assessment Score

0.0 2.5 5.0

2000 4000 6000

KNOWLEDGE ASSESSMENT

slide-33
SLIDE 33

17

CTF Scores

2.5 5

Change in Assessment Score

0.0 2.5 5.0

2000 4000 6000

KNOWLEDGE ASSESSMENT

Participants with higher than average CTF scores also had higher than average changes in assessment scores

slide-34
SLIDE 34

17

CTF Scores

2.5 5

Change in Assessment Score

0.0 2.5 5.0

2000 4000 6000

KNOWLEDGE ASSESSMENT

Participants with higher than average CTF scores also had higher than average changes in assessment scores Perfect score on both assessments

slide-35
SLIDE 35

18

  • Non-CTF participants’ commits were

flagged slightly more often

  • 2/17 Non-CTF participants flagged
  • 1/18 CTF participants flagged
  • 4 CTF participants alerted security

team to potential vulnerability

ADDITIONAL METRICS

slide-36
SLIDE 36
  • 2. Do CTFs improve recognition of

security issues?

  • 1. Do CTFs improve prevention of

security issues?

SUMMARY

19

slide-37
SLIDE 37
  • Participants who solved more challenges

improved in the knowledge assessment

  • Exposure to non-functionality vulnerabilities
  • 2. Do CTFs improve recognition of

security issues?

  • 1. Do CTFs improve prevention of

security issues?

SUMMARY

19

slide-38
SLIDE 38
  • Participants who solved more challenges

improved in the knowledge assessment

  • Exposure to non-functionality vulnerabilities
  • 2. Do CTFs improve recognition of

security issues?

  • 1. Do CTFs improve prevention of

security issues?

SUMMARY

19

  • Increased consideration of security
  • Improved security team engagement
slide-39
SLIDE 39
  • Participants who solved more challenges

improved in the knowledge assessment

  • Exposure to non-functionality vulnerabilities
  • 2. Do CTFs improve recognition of

security issues?

  • 1. Do CTFs improve prevention of

security issues?

SUMMARY

19 dvotipka@cs.umd.edu vulnstudy.cs.umd.edu

Questions:

  • Increased consideration of security
  • Improved security team engagement