Efficient Malware Detection using Model-Checking Tayssir Touili - - PowerPoint PPT Presentation

▶
efficient malware detection using model checking
SMART_READER_LITE
LIVE PREVIEW

Efficient Malware Detection using Model-Checking Tayssir Touili - - PowerPoint PPT Presentation

Sep 25, 2022 508 likes •1.04k views

Efficient Malware Detection using Model-Checking Tayssir Touili LIPN, CNRS & Univ. Paris 13 Motivation: Malware Detection The number of new malware exceeds 75 million by the end of 2011, and is still increasing. The number of

slide-1
SLIDE 1

Efficient Malware Detection using Model-Checking

Tayssir Touili LIPN, CNRS & Univ. Paris 13

slide-2
SLIDE 2

Motivation: Malware Detection

  • The number of new malware exceeds 75 million by the end of 2011, and is still

increasing.

  • The number of malware that produced incidents in 2010 is more than 1.5 billion.
  • The worm MyDoom slowed down global internet access by 10% in 2004.
  • Authorities investigating the 2008 crash of Spanair flight 5022 have discovered a

central computer system used to monitor technical problems in the aircraft was infected with malware

slide-3
SLIDE 3

Motivation: Malware Detection

  • The number of new malware exceeds 75 million by the end of 2011, and is still

increasing.

  • The number of malware that produced incidents in 2010 is more than 1.5 billion.
  • The worm MyDoom slowed down global internet access by 10% in 2004.
  • Authorities investigating the 2008 crash of Spanair flight 5022 have discovered a

central computer system used to monitor technical problems in the aircraft was infected with malware

Malware detection is

important!!

slide-4
SLIDE 4

Limitations of classic anti-virus techniques

  • Signature (pattern) matching: Every known malware

has one signature

slide-5
SLIDE 5

Limitations of classic anti-virus techniques

  • Signature (pattern) matching: Every known malware

has one signature

  • Easy to get around
  • New variants of viruses with the same behavior cannot

be detected by these techniques

  • Nop insertion, code reordering, variable renaming, etc
  • Virus writers frequently update there viruses to make

them undetectable

slide-6
SLIDE 6

Limitations of classic anti-virus techniques

  • Signature (pattern) matching: Every known malware

has one signature

  • Easy to get around
  • New variants of viruses with the same behavior cannot

be detected by these techniques

  • Nop insertion, code reordering, variable renaming, etc
  • Virus writers frequently update there viruses to make

them undetectable

  • Code emulation: Executes binary code in a virtual

environment

slide-7
SLIDE 7

Limitations of classic anti-virus techniques

  • Signature (pattern) matching: Every known malware

has one signature

  • Easy to get around
  • New variants of viruses with the same behavior cannot

be detected by these techniques

  • Nop insertion, code reordering, variable renaming, etc
  • Virus writers frequently update there viruses to make

them undetectable

  • Code emulation: Executes binary code in a virtual

environment

  • Checks program’s behavior only in a limited time interval
slide-8
SLIDE 8

Limitations of classic anti-virus techniques

  • Signature (pattern) matching: Every known malware has one

signature

  • Easy to get around
  • New variants of viruses with the same behavior cannot be detected by

these techniques

  • Nop insertion, code reordering, variable renaming, etc
  • Virus writers frequently update there viruses to make them undetectable
  • Code emulation: Executes binary code in a virtual environment
  • Checks program’s behavior only in a limited time interval

Solution:

Check the behavior (not the syntax) of the program without executing it

slide-9
SLIDE 9

Limitations of classic anti-virus techniques

  • Signature (pattern) matching: Every known malware has one

signature

  • Easy to get around
  • New variants of viruses with the same behavior cannot be detected by

these techniques

  • Nop insertion, code reordering, variable renaming, etc
  • Virus writers frequently update there viruses to make them undetectable
  • Code emulation: Executes binary code in a virtual environment
  • Checks program’s behavior only in a limited time interval

Solution:

Check the behavior (not the syntax) of the program without executing it

Model Checking is a good candidate

slide-10
SLIDE 10

Goal: Model-checking for malware detection

slide-11
SLIDE 11

Goal: Model-checking for malware detection

Binary code ╞ Malicious behavior ?

slide-12
SLIDE 12

Goal: Model-checking for malware detection

Binary code ╞ Malicious behavior ? Model?

slide-13
SLIDE 13

Goal: Model-checking for malware detection

Binary code ╞ Malicious behavior ? Model? Specification

formalism?

slide-14
SLIDE 14

Goal: Model-checking for malware detection

Existing works: use finite automata to model the programs Binary code ╞ Malicious behavior ? Model? Specification

formalism?

slide-15
SLIDE 15

Goal: Model-checking for malware detection

Existing works: use finite automata to model the programs

Stack?

Binary code ╞ Malicious behavior ? Model? Specification

formalism?

slide-16
SLIDE 16

Stack: important for malware detection

  • To achieve their goal, malware have to call functions of

the operating system

  • Antiviruses determine malware by checking the calls

to the operating systems.

  • Virus writers try to hide these calls.
slide-17
SLIDE 17

Stack: important for malware detection

  • To achieve their goal, malware have to call functions of

the operating system

  • Antiviruses determine malware by checking the calls

to the operating systems.

  • Virus writers try to hide these calls.

L0 : call f L1: … … … f : function f L0 : push L1 L’0: jmp f L1: … … … f : function f

slide-18
SLIDE 18

Stack: important for malware detection

  • To achieve their goal, malware have to call functions of

the operating system

  • Antiviruses determine malware by checking the calls

to the operating systems.

  • Virus writers try to hide these calls.

L0 : call f L1: … … … f : function f L0 : push L1 L’0: jmp f L1: … … … f : function f

Important to analyse the program’s

stack

slide-19
SLIDE 19

Stack: important for malware detection

  • To achieve their goal, malware have to call functions of

the operating system

  • Antiviruses determine malware by checking the calls

to the operating systems.

  • Virus writers try to hide these calls.

L0 : call f L1: … … … f : function f L0 : push L1 L’0: jmp f L1: … … … f : function f

Important to analyse the program’s

stack

Solution:

Use pushdown systems to model programs

slide-20
SLIDE 20

Goal: Model-checking for malware detection

Binary code ╞ Malicious behavior ? Pushdown Systems Specification formalism?

slide-21
SLIDE 21

Goal: Model-checking for malware detection

Binary code ╞ Malicious behavior ? Pushdown Systems

Specification formalism?

slide-22
SLIDE 22

Specification of malicious behaviors? Example: fragment of email worm Avron

Call the API GetModuleHandleA with 0 as parameter. This returns the entry address of its

  • wn executable.

Copy itself to other locations. mov eax, 0 push eax call GetModuleHandleA

slide-23
SLIDE 23

Specification of malicious behaviors? Example: fragment of email worm Avron

Call the API GetModuleHandleA with 0 as parameter. This returns the entry address of its

  • wn executable.

Copy itself to other locations. mov eax, 0 push eax call GetModuleHandleA

How to describe this specification?

slide-24
SLIDE 24

Specification of malicious behaviors? Example: fragment of email worm Avron

mov eax, 0 push eax call GetModuleHandleA

In CTL (Branching-time temporal logic) : mov(eax,0)˄EX (push(eax)˄EX call GetModuleHandleA)

slide-25
SLIDE 25

Specification of malicious behaviors? Example: fragment of email worm Avron

mov eax, 0 push eax call GetModuleHandleA

In CTL (Branching-time temporal logic) : mov(eax,0)˄EX (push(eax)˄EX call GetModuleHandleA) EX p: there is a path where p holds at the next state p EX p

slide-26
SLIDE 26

Specification of malicious behaviors? Example: fragment of email worm Avron

mov eax, 0 push eax call GetModuleHandleA

In CTL (Branching-time temporal logic) : mov(eax,0)˄EX (push(eax)˄EX call GetModuleHandleA) ˅ mov(ebx,0)˄EX (push(ebx)˄EX call GetModuleHandleA) ˅ mov(ecx,0)˄EX (push(ecx)˄EX call GetModuleHandleA) ˅ ….. all the other registers EX p: there is a path where p holds at the next state p EX p

slide-27
SLIDE 27

Specification of malicious behaviors? Example: fragment of email worm Avron

mov eax, 0 push eax call GetModuleHandleA

In CTL (Branching-time temporal logic) : mov(eax,0)˄EX (push(eax)˄EX call GetModuleHandleA) ˅ mov(ebx,0)˄EX (push(ebx)˄EX call GetModuleHandleA) ˅ mov(ecx,0)˄EX (push(ecx)˄EX call GetModuleHandleA) ˅ ….. all the other registers EX p: there is a path where p holds at the next state p EX p

Huge!

slide-28
SLIDE 28

Specification of malicious behaviors? Example: fragment of email worm Avron

mov eax, 0 push eax call GetModuleHandleA

In CTL: mov(eax,0)˄EX (push(eax)˄EX callGetModuleHandleA) ˅ mov(ebx,0)˄EX (push(ebx)˄EX callGetModuleHandleA) ˅ mov(ecx,0)˄EX (push(ecx)˄EX callGetModuleHandleA) ˅ ….. all the other registers

∀ ∃,

CTPL = CTL + variables +

slide-29
SLIDE 29

Specification of malicious behaviors? Example: fragment of email worm Avron

mov eax, 0 push eax call GetModuleHandleA

In CTL: mov(eax,0)˄EX (push(eax)˄EX callGetModuleHandleA) ˅ mov(ebx,0)˄EX (push(ebx)˄EX callGetModuleHandleA) ˅ mov(ecx,0)˄EX (push(ecx)˄EX callGetModuleHandleA) ˅ ….. all the other registers

∀ ∃,

CTPL = CTL + variables + In CTPL: ᴲ r (mov(r,0)˄EX (push(r)˄ EX call GetModuleHandleA))

slide-30
SLIDE 30

Specification of malicious behaviors? Example: fragment of email worm Avron

mov eax, 0 push eax call GetModuleHandleA

In CTL: mov(eax,0)˄EX (push(eax)˄EX callGetModuleHandleA) ˅ mov(ebx,0)˄EX (push(ebx)˄EX callGetModuleHandleA) ˅ mov(ecx,0)˄EX (push(ecx)˄EX callGetModuleHandleA) ˅ ….. all the other registers

∀ ∃,

CTPL = CTL + variables + In CTPL: ᴲ r (mov(r,0)˄EX (push(r)˄ EX call GetModuleHandleA))

CTPL cannot describe the stack:

needed for malicious behaviors description

slide-31
SLIDE 31

Specification of malicious behaviors? Example: fragment of email worm Avron

In CTPL: ᴲ r (mov(r,0)˄EX (push(r)˄ EX call GetModuleHandleA))

mov eax, 0 push eax call GetModuleHandleA Call the API GetModuleHandleA with 0 as parameter. This returns the entry address of its

  • wn executable.

Copy itself to other locations.

slide-32
SLIDE 32

Specification of malicious behaviors? Example: fragment of email worm Avron

In CTPL: ᴲ r (mov(r,0)˄EX (push(r)˄ EX call GetModuleHandleA))

mov eax, 0 push ebx pop ebx push eax call GetModuleHandleA Call the API GetModuleHandleA with 0 as parameter. This returns the entry address of its

  • wn executable.

Copy itself to other locations.

slide-33
SLIDE 33

Specification of malicious behaviors? Example: fragment of email worm Avron

In CTPL: ᴲ r (mov(r,0)˄EX (push(r)˄ EX call GetModuleHandleA))

mov eax, 0 push ebx pop ebx push eax call GetModuleHandleA Call the API GetModuleHandleA with 0 as parameter. This returns the entry address of its

  • wn executable.

Copy itself to other locations.

Our solution: Consider predicates over the stack

slide-34
SLIDE 34

Specification of malicious behaviors? Example: fragment of email worm Avron

In CTPL: ᴲ r (mov(r,0)˄EX (push(r)˄ EX call GetModuleHandleA))

mov eax, 0 push ebx pop ebx push eax call GetModuleHandleA Call the API GetModuleHandleA with 0 as parameter. This returns the entry address of its

  • wn executable.

Copy itself to other locations.

Our solution: Consider predicates over the stack In SCTPL: EF ( call GetModuleHandleA ˄ (head_stack = 0) ) EF p: there is a path where p holds in the future

slide-35
SLIDE 35

Expressing Obfuscated Calls in SCTPL

L0 : call f L: … … … f : function f L0 : push L L’0: jmp f L: … … … f : function f

slide-36
SLIDE 36

Expressing Obfuscated Calls in SCTPL

L0 : call f L: … … … f : function f L0 : push L L’0: jmp f L: … … … f : function f ᴲL ( E !( ᴲ f call(f) ˄ AX (head_stack=L)) U (ret ˄ (head_stack= L)))

slide-37
SLIDE 37

Expressing Obfuscated Calls in SCTPL

L0 : call f L: … … … f : function f L0 : push L1 L’0: jmp f L: … … … f : function f ᴲL ( E !( ᴲ f call(f) ˄ AX (head_stack=L)) U (ret ˄ (head_stack= L))) L is not a return address of a function call

slide-38
SLIDE 38

Expressing Obfuscated returns in SCTPL

L0 : call f a : … … f: … … pop eax jmp eax

h_s= a h_s= a

h_s : head-stack

!

slide-39
SLIDE 39

Expressing Appending Viruses in SCTPL

L0 : call f a : … f: pop eax

An appending virus append itself at the end of the host file The virus has to compute its address in memory

h_s = a h_s = a

h_s : head-stack

slide-40
SLIDE 40

Goal: Model-checking for malware detection

Binary code ╞ Malicious behavior ?

slide-41
SLIDE 41

Goal: Model-checking for malware detection

Binary code ╞ Malicious behavior ? Pushdown Systems SCTPL

slide-42
SLIDE 42

Goal: Model-checking for malware detection

Binary code ╞ Malicious behavior ? Pushdown Systems SCTPL Pushdown System╞ SCTPL ?

slide-43
SLIDE 43

SCTPL model-checking for Pushdown Systems Non trivial: stack can be unbounded

slide-44
SLIDE 44

SCTPL model-checking for Pushdown Systems Non trivial: stack can be unbounded Theorem: Given a Pushdown System P and a SCTPL formula ᵠ, whether P satisfies ᵠ can be effectively decided.

slide-45
SLIDE 45

Implementation

We implemented our techniques in a tool for virus detection IdaPro + Jakstab CFG Binary Code Translator PDS Model-checking engine Malware behaviors as SCTPL formulas YES/NO

slide-46
SLIDE 46

Experiments of POMMADE

  • 1. Our tool was able to detect more than 800 malwares

2.We checked 400 real benign programs from Windows XP system. Benign programs are proved benign with only three false positives. 3.Our tool was able to detect all the 200 new malwares generated by two malware creators 4.Analyze the Flame malware that was not detected for more than 5 years by any anti-virus

slide-47
SLIDE 47

Our tool vs. known anti-viruses

NGVCK and VCL32 malware generators 1.generate 200 new malwares 2.the best malware generators 3.generate complex malwares

Gener ator No. Of Vari ant s PO MM

ADE

Avi ra Kasp ersk y Ava st Qiho

  • 360

McA fee AVG BitDef ender Eset Nod3 2 F- Sec ure Nort

  • n

Pan da Tre nd Micr

  • NGVC

K

100 100 % 0% 23% 18 % 68% 100 % 11% 97% 81% 0% 46% 0% 0%

VCL32 100 100

% 0% 2% 100 % 99% 0% 100 % 100% 76% 0% 30% 0% 0%

slide-48
SLIDE 48

Analyze The Flame Malware

Flame is being used for targeted cyber espionage in Middle Eastern countries. It can 1.sniff the network traffic 2.take screenshots 3.record audio conversations 4.intercept the keyboard 5.and so on

It was not detected by any anti-virus for 5 years

slide-49
SLIDE 49

Analyze The Flame Malware

Flame is being used for targeted cyber espionage in Middle Eastern countries. It can 1.sniff the network traffic 2.take screenshots 3.record audio conversations 4.intercept the keyboard 5.and so on

It was not detected by any anti-virus for 5 years

Our tool can detect this malware Flame

slide-50
SLIDE 50

Conclusion

  • We introduced a new logic SCTPL to precisely specify malicious

behaviors

  • We proposed efficient SCTPL model-checking algorithms for

pushdown systems.

  • We implemented our techniques in a tool for malware detection:

POMMADE

  • POMMADE was able to detect more than 800 malwares, several
  • f them cannot be detected by well-known anti-viruses, such as,

Avast, Kaspersky, McAfee, Norton, Avira, etc

slide-51
SLIDE 51
slide-52
SLIDE 52

Questions?