Hackers vs Testers: A Comparison of Software Vulnerability - - PowerPoint PPT Presentation

Hackers vs Testers: A Comparison of Software Vulnerability Discovery Processes

Daniel Votipka, Rock A. Stevens, Elissa M. Redmiles, Jeremy Hu, and Michelle L. Mazurek

22 May 2018

VULNERABILITY DISCOVERY

2

3

Testers:

• Functionality
• Performance
• Security

VULNERABILITY DISCOVERY

Generalists

4

Hackers:

• Security Team
• Contracted Review
• Bug Bounty

VULNERABILITY DISCOVERY

Experts

CHALLENGES

5

• Timeliness
• Cognitive Diversity
• Communication

RESEARCH QUESTIONS

• 1. How do testers and hackers

search for vulnerabilities?

• 2. What are the differences

between testers and hackers?

6

Interview study:

• Task Analysis
• Tools, Skills, and Communities

RECRUITMENT

Hacker Groups:

• Bug Bounty Programs
• Top Hacking Teams

Tester Groups:

• Meetup and LinkedIn
• IEEE and AST
• Ministry of Testing

7

106 total groups

PARTICIPANTS

8 10 15 0-3 26-50 Participants Vulnerabilities Found Vulnerability Finding Time 5-10 hrs/w 10-20 hrs/w

RESEARCH QUESTIONS

• 1. How do testers and hackers

search for vulnerabilities?

• 2. What are the differences

between testers and hackers?

9

RESEARCH QUESTIONS

• 1. How do testers and hackers

search for vulnerabilities?

• 2. What are the differences

between testers and hackers?

10

11

Info Gathering Program Understanding Attack Surface Exploration Vulnerability Recognition Reporting

HACKER AND TESTER PROCESS

12

Info Gathering Program Understanding Attack Surface Exploration Vulnerability Recognition Reporting

Information Gathering

• Build context prior to reading or

executing code

• Example actions:
• Identifying libraries
• Update history
• Previous bug reports

“There were…other functional issues, so I figured that was probably where there was most likely to be security issues as well. Bugs tend to cluster.”

13

Info Gathering Program Understanding Attack Surface Exploration Vulnerability Recognition Reporting

Program Understanding

• Determine how the program operates
• Interaction between components
• Interaction with the environment

“You’re touching a little bit everything, and then you are organizing that into a structure in your head.”

14

Info Gathering Program Understanding Attack Surface Exploration Vulnerability Recognition Reporting

Attack Surface

• Identify how user interacts with program
• Direct and indirect inputs

15

Info Gathering Program Understanding Attack Surface Exploration Vulnerability Recognition Reporting

Exploration

• Possible inputs to the attack surface
• Example actions:
• Fuzzing
• Reading code

16

Info Gathering Program Understanding Attack Surface Exploration Vulnerability Recognition Reporting

Vulnerability Recognition

• Notice a problem when exploring
• Typically described as intuition-based

17

Info Gathering Program Understanding Attack Surface Exploration Vulnerability Recognition Reporting

Reporting

• Tell developers about the problem
• Advocate for remediation
• Critical aspects:
• Make report understandable
• Importance of fixing

“You do have to [convince] someone that there’s a risk. …It’s quite timely [time consuming], running a ticket.”

RESEARCH QUESTIONS

• 1. How do testers and hackers

search for vulnerabilities?

• 2. What are the differences

between testers and hackers?

18

19

Info Gathering Program Understanding Attack Surface Exploration Vulnerability Recognition Reporting Vulnerability Discovery Experience Access to Development Process Underlying System Knowledge Motivation

20

Info Gathering Program Understanding Attack Surface Exploration Vulnerability Recognition Reporting Vulnerability Discovery Experience Access to Development Process Underlying System Knowledge Motivation

Attack Surface

21

Exploration Vulnerability Recognition Vulnerability Discovery Experience

Exploration

• Informs test case selection

Attack Surface

• More likely to consider

indirect input paths Vulnerability Recognition

• Know vulnerability patterns
• List of common vulnerabilities

“As soon as I found the LinkedIn problem, I made sure to test [FB and Twitter input] to make sure [they were processed correctly]. And if we did allow login with another 3rd party in the future, I would check that too.”

22

Employment Hacking Exercises Community Bug Reports Attack Surface Exploration Vulnerability Recognition Vulnerability Discovery Experience

23

Employment Hacking Exercises Community Bug Reports Vulnerability Discovery Experience

AMOUNT OF EXPERIENCE

24

Info Gathering Program Understanding Reporting Access to Development Process

Internal

• Communicate with

developers

• Documentation

External

• Reverse engineering
• Develop exploits

“You can give feedback to your

• developers. . . .You’re coming back with

information, and then they react on it.” “It’s hard to ignore certain details once you know about certain areas already.”

RECOMMENDATIONS

25

• Provide training in known contexts
• Hire hackers into the testing team
• Bug report-based exercises
• Improve hacker communication
• Single point of contact

SUMMARY

26

• Similar processes
• Impacted by:
• Vulnerability Discovery Experience
• Underlying System Knowledge,
• Access to the Development Process
• Motivation
• Biggest difference in amount of experience and

relationship with the developers Recommendations:

• Training in a known context
• Hacker/company communication

dvotipka@cs.umd.edu vulnstudy.cs.umd.edu

Questions: