Security Testing fuzzing protocol fuzzing m odel-based testing - PowerPoint PPT Presentation

Security Testing fuzzing protocol fuzzing m odel-based testing autom ated reverse engineering autom ated reverse engineering Erik Poll Erik Poll Radboud University Nijmegen

Testing Ingredients T ti I di t Two things are needed to test a SUT ( System Under Test) Two things are needed to test a SUT ( System Under Test) 1 . test suite , ie collection of input data 2 . a test oracle that decides if a test was ok or reveals an error, , i.e. some way to decide if the SUT behaves as we want A nice & simple test oracle: just seeing if the SUT crashes A nice & simple test oracle: just seeing if the SUT crashes Both defining test suite and test oracles can be a lot of work: for each individual test case the test oracle may need to be y tweaked by specifying exactly what should happen 2

C Coverage Criteria C it i Measures of how good a test suite is Measures of how good a test suite is statement coverage • branch coverage • Statement coverage does not imply branch coverage; eg for g p y g ; g void f (int x, y) { if (x>0) {y++}; y--; } statement coverage needs 1 test case, branch coverage needs 2 More complex coverage criteria exists, eg MCDC (Modified • condition/ decision coverage) which is used in avionics g ) <#>

P Possible Perverse Effect of Coverage Criteria ibl P Eff t f C C it i High coverage criteria may discourage defensive programming High coverage criteria may discourage defensive programming void m(File f){ if <security_check_fails> {throw (SecurityException)} try { <the main part of the method> } catch (SomeException) { <take some measures>; throw (SecurityException) } ( y p ) } } If th If the green defensive code is hard to trigger in tests, programmers d f i d i h d t t i i t t may be tempted (or forced) to remove it to improve coverage in testing... testing... 4

S Security testing is HARD, in general it t ti i HARD i l Normal testing will look at right wanted behaviour for sensible Normal testing will look at right, wanted behaviour for sensible • • inputs, and some inputs on borderline conditions Security testing also involves looking for the wrong unwanted Security testing also involves looking for the wrong, unwanted • behaviour for really silly inputs Similarly, normal use of a system is more likely to reveal Si il l l f t i lik l t l • functional problems (users will complain) than than security problems (hackers won’t complain) 5

S Security testing is HARD, in general it t ti i HARD i l all possible inputs normal . . . . input that triggers security bug . . . . inputs inputs 6

JML JML annotations as test oracle t ti t t l Tools for runtime assertion checking of JML annotations can be used Tools for runtime assertion checking of JML annotations can be used when testing code instrumented with check to test annotations, which throw code instrumented with check to test annotations which throw • • special exceptions for violations effectively the annotations serve as test oracle effectively, the annotations serve as test oracle • • Benefits: Test oracle for free: you can test by sending random data T l f f b di d d • More precise and detailed feedback: adding • //@ invariant contents != null; //@ invariant contents != null; an application may crash with an Invariant Violation in line 18000 after 1 minute with runtime assertion checking, whereas otherwise it would crash NullpointerException in line 12000 after 5 minutes - pointing to the real origin of the problem, not the eventual effect 7

S Symbolic Execution for test suites b li E ti f t t it Symbolic execution can be used to generate test suites with good Symbolic execution can be used to generate test suites with good • • coverage Basic idea symbolic execution: Basic idea symbolic execution: • instead of giving variables a concrete value (say 42), variables are given a symbolic value (say N), and the program is executed with i b li l ( N) d th i t d ith these symbolic values to see when certain program points are reached reached 8

S Symbolic Execution b li E ti m(int x y){ m(int x,y){ x = x + y; y = y – x; if (2*y > 8) { .... ( y ) { } else if (3*x < 10){ ... } } 9

S Symbolic Execution b li E ti m(int x y){ m(int x,y){ // let x == N and y == M // let x == N and y == M x = x + y; // x becomes N+M y = y – x; // y becomes M-(N+M) == -N if (2*y > 8) { .... ( y ) { // taken if 2*-N > 8, ie N < -4 // , } else if (3*x < 10){ ... // taken if N>=-4 and 3(M+N)<10 } } There are tools that given such sets of constraints try to produce test There are tools that, given such sets of constraints, try to produce test data that meets these constraints 10

S Symbolic Execution b li E ti Symbolic execution can also be used for program verification: Symbolic execution can also be used for program verification: 1. symbolically execute a method (or piece of code) 2. assuming precondition (and invariant) on initial values, prove postcondition (and invariant) for final values p p ( ) 11

12 Fuzzing

F Fuzzing i Fuzzing Fuzzing try really long inputs for string arguments to trigger segmentation faults and hence find buffer overflows faults and hence find buffer overflows Benefit: can be automated, because test suite of long inputs can be automatically generated, and test oracle is trivial: looking if the b t ti ll t d d t t l i t i i l l ki if th program crashes This original idea has been generalised to other settings: The general idea of fuzzing: using semi-random, automatically generated test data that is likely to trigger security problems g y gg y p 13

Fuzzing in memory safe languages F i i f l For memory safe languages such as Java or C(+ + ) fuzzing can still For memory safe languages such as Java or C(+ + ), fuzzing can still reveal bugs in a VM, bytecode verifier, or libraries with native code Eg, fast graphics libraries often rely on native code Eg fast graphics libraries often rely on native code CVE reference: CVE-2007-0243 Release Date: 2007 01 17 Release Date: 2007-01-17 Sun Java JRE GIF Image Processing Buffer Overflow Vulnerability Critical: Highly critical Impact: System access Where: From remote g y p y Description: A vulnerability has been reported in Sun Java Runtime Environment (JRE), which can be exploited by malicious people to compromise a vulnerable system. The vulnerability is caused due to an i l bl t Th l bilit i d d t error when processing GIF images and can be exploited to cause a heap- based buffer overflow via a specially crafted GIF image with an image width of 0 width of 0. Successful exploitation allows execution of arbitrary code. 14

Fil f File format fuzzing t f i Incorrectly formatted files or corner cases in file formats can cause Incorrectly formatted files, or corner cases in file formats can cause trouble Eg Eg GIF image with width 0 on previous slide • Microsoft Security Bulletin MS04-028 • Buffer Overrun in JPEG Processing (GDI+ ) Could Allow Code Execution Impact of Vulnerability : Remote Code Execution Maximum Severity Rating: Critical Recommendation: Customers should apply the update immediately Recommendation: Customers should apply the update immediately Root cause: a zero sized comment field, without content. 15

F Fuzzing web-applications? i b li ti ? Could we fuzz a web application in the hope to find security flaws? Could we fuzz a web application in the hope to find security flaws? • • • SQL injection • XSS • ... What would be needed? • • test inputs that trigger these security flaws • test inputs that trigger these security flaws • some way of detecting if a security flaw occurred • looking at website response, or log files 16

Fuzzing web-applications F i b li ti There are many tools to fuzz web-applications There are many tools to fuzz web-applications • • • Spike proxy, HP Webinspect, AppScan, WebScarab, Wapiti, w3af RFuzz WSFuzzer SPI Fuzzer Burp Mutilidae w3af, RFuzz, WSFuzzer, SPI Fuzzer Burp, Mutilidae, ... Some fuzzers crawl a website, generating traffic themselves, • other fuzzers modify traffic generated by some other means other fuzzers modify traffic generated by some other means. As usual, there will be false positives & negatives, eg • • false negative for SQL injection due to not recognizing some SQL database errors • false positives for XSS due to signalling a correctly quoted echoed response as XSS [ Frank van der Loo, Comparison of penentration testing tools for web applications, MSc thesis] 17

P Protocol Fuzzing t l F i Protocol fuzzing based on known protocol format Protocol fuzzing based on known protocol format ie format of packets or messages Typical things to try in protocol fuzzing: trying out many/ all possible value for specific fields t i t / ll ibl l f ifi fi ld • esp undefined values, or values Reserved for Future Use (RFU) giving incorrect lengths, length that are zero, or payloads that are • too short/ long Tools for protocol fuzzing exist, eg SNOOZE Tools for protocol fuzzing exist, eg SNOOZE 18

E Example : GSM protocol fuzzing l GSM t l f i GSM is a extremely rich & complicated protocol GSM is a extremely rich & complicated protocol • • 19