efficient malware detection using model checking
play

Efficient Malware Detection using Model-Checking Tayssir Touili - PowerPoint PPT Presentation

Efficient Malware Detection using Model-Checking Tayssir Touili LIPN, CNRS & Univ. Paris 13 Motivation: Malware Detection The number of new malware exceeds 75 million by the end of 2011, and is still increasing. The number of


  1. Efficient Malware Detection using Model-Checking Tayssir Touili LIPN, CNRS & Univ. Paris 13

  2. Motivation: Malware Detection • The number of new malware exceeds 75 million by the end of 2011, and is still increasing. • The number of malware that produced incidents in 2010 is more than 1.5 billion. • The worm MyDoom slowed down global internet access by 10% in 2004. • Authorities investigating the 2008 crash of Spanair flight 5022 have discovered a central computer system used to monitor technical problems in the aircraft was infected with malware

  3. Motivation: Malware Detection • The number of new malware exceeds 75 million by the end of 2011, and is still increasing. • The number of malware that produced incidents in 2010 is more than 1.5 billion. • The worm MyDoom slowed down global internet access by 10% in 2004. • Authorities investigating the 2008 crash of Spanair flight 5022 have discovered a Malware detection is central computer system used to monitor technical problems in the aircraft was infected with malware important!!

  4. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature

  5. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature o Easy to get around o New variants of viruses with the same behavior cannot be detected by these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable

  6. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature o Easy to get around o New variants of viruses with the same behavior cannot be detected by these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable • Code emulation: Executes binary code in a virtual environment

  7. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature o Easy to get around o New variants of viruses with the same behavior cannot be detected by these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable • Code emulation: Executes binary code in a virtual environment o Checks program’s behavior only in a limited time interval

  8. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature Solution: o Easy to get around Check the behavior (not the syntax) of o New variants of viruses with the same behavior cannot be detected by the program without executing it these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable • Code emulation: Executes binary code in a virtual environment o Checks program’s behavior only in a limited time interval

  9. Limitations of classic anti-virus techniques • Signature (pattern) matching: Every known malware has one signature Solution: o Easy to get around Check the behavior (not the syntax) of o New variants of viruses with the same behavior cannot be detected by the program without executing it these techniques o Nop insertion, code reordering, variable renaming, etc o Virus writers frequently update there viruses to make them undetectable • Code emulation: Executes binary code in a virtual environment o Checks program’s behavior only in a limited time interval Model Checking is a good candidate

  10. Goal: Model-checking for malware detection

  11. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ?

  12. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ? Model?

  13. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ? Model? Specification formalism?

  14. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ? Model? Specification formalism? Existing works: use finite automata to model the programs

  15. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ? Model? Specification formalism? Existing works: use finite automata to model the programs Stack?

  16. Stack: important for malware detection • To achieve their goal, malware have to call functions of the operating system • Antiviruses determine malware by checking the calls to the operating systems. • Virus writers try to hide these calls.

  17. Stack: important for malware detection • To achieve their goal, malware have to call functions of the operating system • Antiviruses determine malware by checking the calls to the operating systems. • Virus writers try to hide these calls. L0 : push L1 L0 : call f L’0: jmp f L1: … L1: … … … … … f : function f f : function f

  18. Stack: important for malware detection • To achieve their goal, malware have to call functions of the operating system Important to analyse the program’s • Antiviruses determine malware by checking the calls stack to the operating systems. • Virus writers try to hide these calls. L0 : push L1 L0 : call f L’0: jmp f L1: … L1: … … … … … f : function f f : function f

  19. Stack: important for malware detection • To achieve their goal, malware have to call functions of the operating system Important to analyse the program’s • Antiviruses determine malware by checking the calls stack to the operating systems. • Virus writers try to hide these calls. L0 : push L1 Solution: L0 : call f L’0: jmp f Use pushdown systems to model L1: … L1: … programs … … … … f : function f f : function f

  20. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ? Pushdown Specification Systems formalism?

  21. Goal: Model-checking for malware detection Binary code ╞ Malicious behavior ? Specification Pushdown Systems formalism?

  22. Specification of malicious behaviors? Example: fragment of email worm Avron Call the API GetModuleHandleA mov eax, 0 with 0 as parameter. push eax This returns the entry address of its call GetModuleHandleA own executable. Copy itself to other locations.

  23. Specification of malicious behaviors? Example: fragment of email worm Avron Call the API GetModuleHandleA mov eax, 0 with 0 as parameter. push eax This returns the entry address of its call GetModuleHandleA own executable. Copy itself to other locations. How to describe this specification?

  24. Specification of malicious behaviors? Example: fragment of email worm Avron mov eax, 0 push eax call GetModuleHandleA In CTL (Branching-time temporal logic) : mov(eax,0) ˄ EX ( push(eax) ˄ EX call GetModuleHandleA )

  25. Specification of malicious behaviors? Example: fragment of email worm Avron EX p mov eax, 0 p push eax call GetModuleHandleA In CTL (Branching-time temporal logic) : mov(eax,0) ˄ EX ( push(eax) ˄ EX call GetModuleHandleA ) EX p : there is a path where p holds at the next state

  26. Specification of malicious behaviors? Example: fragment of email worm Avron EX p mov eax, 0 p push eax call GetModuleHandleA In CTL (Branching-time temporal logic) : mov(eax,0) ˄ EX ( push(eax) ˄ EX call GetModuleHandleA ) ˅ mov(ebx,0) ˄ EX ( push(ebx) ˄ EX call GetModuleHandleA ) ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX call GetModuleHandleA ) ˅ … .. all the other registers EX p : there is a path where p holds at the next state

  27. Specification of malicious behaviors? Example: fragment of email worm Avron EX p mov eax, 0 p push eax call GetModuleHandleA Huge! In CTL (Branching-time temporal logic) : mov(eax,0) ˄ EX ( push(eax) ˄ EX call GetModuleHandleA ) ˅ mov(ebx,0) ˄ EX ( push(ebx) ˄ EX call GetModuleHandleA ) ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX call GetModuleHandleA ) ˅ … .. all the other registers EX p : there is a path where p holds at the next state

  28. Specification of malicious behaviors? Example: fragment of email worm Avron mov eax, 0 CTPL = CTL + push eax ∃ , variables + ∀ call GetModuleHandleA In CTL: mov(eax,0) ˄ EX ( push(eax) ˄ EX callGetModuleHandleA ) ˅ mov(ebx,0) ˄ EX ( push(ebx) ˄ EX callGetModuleHandleA ) ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX callGetModuleHandleA ) ˅ … .. all the other registers

  29. Specification of malicious behaviors? Example: fragment of email worm Avron mov eax, 0 CTPL = CTL + push eax ∃ , variables + ∀ call GetModuleHandleA In CTL: mov(eax,0) ˄ EX ( push(eax) ˄ EX callGetModuleHandleA ) ˅ mov(ebx,0) ˄ EX ( push(ebx) ˄ EX callGetModuleHandleA ) ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX callGetModuleHandleA ) ˅ … .. all the other registers In CTPL: ᴲ r ( mov(r,0) ˄ EX ( push(r) ˄ EX call GetModuleHandleA ) )

  30. Specification of malicious behaviors? Example: fragment of email worm Avron mov eax, 0 CTPL = CTL + push eax ∃ , variables + ∀ call GetModuleHandleA In CTL: CTPL cannot describe the stack: mov(eax,0) ˄ EX ( push(eax) ˄ EX callGetModuleHandleA ) ˅ needed for malicious behaviors mov(ebx,0) ˄ EX ( push(ebx) ˄ EX callGetModuleHandleA ) description ˅ mov(ecx,0) ˄ EX ( push(ecx) ˄ EX callGetModuleHandleA ) ˅ … .. all the other registers In CTPL: ᴲ r ( mov(r,0) ˄ EX ( push(r) ˄ EX call GetModuleHandleA ) )

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend


More recommend