on static malware detection
play

On Static Malware Detection Tayssir Touili LIPN, CNRS & Univ. - PowerPoint PPT Presentation

On Static Malware Detection Tayssir Touili LIPN, CNRS & Univ. Paris 13 Motivation: Malware Detection The number of new malware exceeds 75 million by the end of 2011, and is still increasing. The number of malware that produced


  1. On Static Malware Detection Tayssir Touili LIPN, CNRS & Univ. Paris 13

  2. Motivation: Malware Detection • The number of new malware exceeds 75 million by the end of 2011, and is still increasing. • The number of malware that produced incidents in 2010 is more than 1.5 billion. • The worm MyDoom slowed down global internet access by 10% in 2004. • Authorities investigating the 2008 crash of Spanair flight 5022 have discovered a central computer system used to monitor technical problems in the aircraft was infected with malware

  3. Motivation: Malware Detection • The number of new malware exceeds 75 million by the end of 2011, and is still increasing. • The number of malware that produced incidents in 2010 is more than 1.5 billion. • The worm MyDoom slowed down global internet access by 10% in 2004. • Authorities investigating the 2008 crash of Spanair flight 5022 have discovered a Malware detection is central computer system used to monitor technical problems in the aircraft was infected with malware important!!

  4. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature

  5. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature o Easy to get around o New variants of viruses with the same behavior cannot be detected by these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable

  6. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature o Easy to get around o New variants of viruses with the same behavior cannot be detected by these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable • Code emulation: Executes binary code in a virtual environment

  7. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature o Easy to get around o New variants of viruses with the same behavior cannot be detected by these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable • Code emulation: Executes binary code in a virtual environment o Checks program’s behavior only in a limited time interval

  8. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature Solution: o Easy to get around Check the behavior (not the syntax) of o New variants of viruses with the same behavior cannot be detected by these techniques the program without executing it o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable • Code emulation: Executes binary code in a virtual environment o Checks program’s behavior only in a limited time interval Static Analysis and Model Checking are good candidates

  9. Goal: Static Analysis and Model- checking for malware detection Binary code ╞ Malicious behavior ? Model? Specification formalism? Existing works: use finite automata to model the programs Stack?

  10. Stack: important for malware detection • To achieve their goal, malware have to call functions of the operating system • Antiviruses determine malware by checking the calls to the operating systems. • Virus writers try to hide these calls. L0 : push L1 L0 : call f L’0: jmp f L1: … L1: … … … … … f : function f f : function f

  11. Stack: important for malware detection • To achieve their goal, malware have to call functions of the operating system Important to analyse the program’s • Antiviruses determine malware by checking the calls stack to the operating systems. • Virus writers try to hide these calls. L0 : push L1 Solution: L0 : call f L’0: jmp f Use pushdown systems to model L1: … L1: … programs … … … … f : function f f : function f

  12. Pushdown Systems PDS = finite automaton + Stack P =(P, Г , Δ), • P is a finite set of control states • Г is the stack alphabet • Δ ⊆ (P× Г) × (P×Г*) is a finite set of transitions • A configuration is a pair <p,ω> ∈ × Г * P • If <p, α> → <p’,ω> ∈ Δ, then, for every u ∈ Г*, <p, αu> => <p’,ωu>

  13. From Binary Codes to PDSs

  14. Difficulty: mov eax, 1 0 is pushed dec eax It’s non-trival to get onto the stack push eax registers’ values call GetModuleHandleA

  15. Computing Registers’ Values We need an oracle that computes the values of the registers mov eax, 1 eax’s value dec eax is 0 push eax call GetModuleHandleA We use Jakstab [Kinder-Veith 2008] to implement the oracle Jakstab (Java Toolkit for static analysis of binaries) does a kind of constant propagation to determine registers’ values

  16. From Binary Codes to PDSs l 1 : mov eax, 1 l 2 : dec eax g 0 = entry point of l 3 : push eax GetModuleHandeA l 4 : call GetModuleHandleA l 5 : ... Control states of PDS = control points of program Stack alphabet = return addresses+ registers’ values l 1 l 2 Push 0 Push l 5 l 3

  17. Malicious behaviors? Binary code ╞ Malicious behavior ? Specification PDS formalism?

  18. Specification of malicious behaviors? Example: fragment of email worm Avron Call the API GetModuleHandleA mov eax, 0 with 0 as parameter. push eax This returns the entry address of its call GetModuleHandleA own executable. Copy itself to other locations.

  19. Specification of malicious behaviors? Example: fragment of email worm Avron Call the API GetModuleHandleA mov eax, 0 with 0 as parameter. push eax This returns the entry address of its call GetModuleHandleA own executable. Copy itself to other locations. How to describe this specification?

  20. Specification of malicious behaviors? Example: fragment of email worm Avron EX p mov eax, 0 p push eax call GetModuleHandleA In CTL (Branching-time temporal logic) : mov(eax,0) ˄ EX ( push(eax) ˄ EX call GetModuleHandleA ) EX p : there is a path where p holds at the next state

  21. Specification of malicious behaviors? Example: fragment of email worm Avron EX p mov eax, 0 p push eax call GetModuleHandleA In CTL (Branching-time temporal logic) : mov(eax,0) ˄ EX ( push(eax) ˄ EX call GetModuleHandleA ) ˅ mov(ebx,0) ˄ EX ( push(ebx) ˄ EX call GetModuleHandleA ) ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX call GetModuleHandleA ) ˅ ….. all the other registers EX p : there is a path where p holds at the next state

  22. Specification of malicious behaviors? Example: fragment of email worm Avron EX p mov eax, 0 p push eax call GetModuleHandleA Huge! In CTL (Branching-time temporal logic) : mov(eax,0) ˄ EX ( push(eax) ˄ EX call GetModuleHandleA ) ˅ mov(ebx,0) ˄ EX ( push(ebx) ˄ EX call GetModuleHandleA ) ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX call GetModuleHandleA ) ˅ ….. all the other registers EX p : there is a path where p holds at the next state

  23. Specification of malicious behaviors? Example: fragment of email worm Avron mov eax, 0 CTPL = CTL + ∃ , ∀ push eax variables + call GetModuleHandleA In CTL: mov(eax,0) ˄ EX ( push(eax) ˄ EX callGetModuleHandleA ) ˅ mov(ebx,0) ˄ EX ( push(ebx) ˄ EX callGetModuleHandleA ) ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX callGetModuleHandleA ) ˅ ….. all the other registers In CTPL: ᴲ ( mov(r,0) ˄ EX ( push(r) ˄ EX call GetModuleHandleA ) ) r

  24. Specification of malicious behaviors? Example: fragment of email worm Avron mov eax, 0 CTPL = CTL + ∃ , ∀ push eax variables + call GetModuleHandleA In CTL: CTPL cannot describe the stack: mov(eax,0) ˄ EX ( push(eax) ˄ EX callGetModuleHandleA ) ˅ needed for malicious behaviors mov(ebx,0) ˄ EX ( push(ebx) ˄ EX callGetModuleHandleA ) description ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX callGetModuleHandleA ) ˅ ….. all the other registers In CTPL: ᴲ ( mov(r,0) ˄ EX ( push(r) ˄ EX call GetModuleHandleA ) ) r

  25. Specification of malicious behaviors? Example: fragment of email worm Avron Call the API GetModuleHandleA mov eax, 0 with 0 as parameter. push eax This returns the entry address of its call GetModuleHandleA own executable. Copy itself to other locations. In CTPL: ᴲ ( mov(r,0) ˄ EX ( push(r) ˄ EX call GetModuleHandleA ) ) r

  26. Specification of malicious behaviors? Example: fragment of email worm Avron Call the API GetModuleHandleA mov eax, 0 with 0 as parameter. push ebx This returns the entry address of its pop ebx own executable. push eax Copy itself to other locations. call GetModuleHandleA In CTPL: ᴲ ( mov(r,0) ˄ EX ( push(r) ˄ EX call GetModuleHandleA ) ) the head of r stack is 0 Our solution: Consider predicates over the stack In SCTPL: EF ( call GetModuleHandleA ˄ 0 Г* ) EF p : there is a path where p holds in the future

  27. SCTPL Logic  ::= b |¬  |  ∧  | EX  | E [  U  ] | EG 

  28. SCTPL Logic  ::= b(y 1 ,…,y n ) |¬  |  ∧  | EX  | E [  U  ] | EG  • y ∈ Y , a set of variables over a finite domain D

  29. SCTPL Logic  ::= b(y 1 ,…,y n ) |¬  |  ∧  | EX  | E [  U  ] | EG  |  y  • y ∈ Y , a set of variables over a finite domain D

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend


More recommend