Effective and Efficient Malware Detection at the End Host Clemens - - PowerPoint PPT Presentation

▶
effective and efficient malware detection at the end host
SMART_READER_LITE
LIVE PREVIEW

Effective and Efficient Malware Detection at the End Host Clemens - - PowerPoint PPT Presentation

Aug 29, 2022 16 likes •446 views

Secure Systems Lab Technical University Vienna Effective and Efficient Malware Detection at the End Host Clemens KOLBITSCH, Paolo MILANI COMPARETTI, Engin KIRDA, Christopher KRUEGEL, Xiaoyong ZHOU, XiaoFeng WANG ck@iseclab.org Secure Systems

slide-1
SLIDE 1

Secure Systems Lab Technical University Vienna 1

Effective and Efficient Malware Detection at the End Host

Clemens KOLBITSCH, Paolo MILANI COMPARETTI, Engin KIRDA, Christopher KRUEGEL, Xiaoyong ZHOU, XiaoFeng WANG

ck@iseclab.org Secure Systems Lab [TU Vienna, Institute Eurecom Sophia Antipolis, UC Santa Barbara] Indiana University at Bloomington

slide-2
SLIDE 2

Secure Systems Lab Technical University Vienna 2

Motivation

  • Binary signature based detection inherently

ineffective

– We all know the problems... – Arms-race, pretty much a lost battle

  • Network based approaches evadable

– Systems scan for communication artifacts – Encryption / blending thwart detection

Why do we propose yet another malware detection scheme (yamds)?

slide-3
SLIDE 3

Secure Systems Lab Technical University Vienna 3

Motivation

  • Don't rely on artifacts of malware instances

– Instead focus on generic patterns

  • Proposed solution:

– Detection based on malware's behavior – Behavior is hard to obfuscate – Behavior is hard to randomize – Behavior is often stable across various malware version

Why do we propose yet another malware detection scheme (yamds)?

slide-4
SLIDE 4

Secure Systems Lab Technical University Vienna 4

Motivation

  • Behavior-based detection received some attention
  • ver last couple of years
  • Despite promising detection results, binary signatures

remain the method of choice

slide-5
SLIDE 5

Secure Systems Lab Technical University Vienna 5

Motivation

  • Behavior-based detection received some attention
  • ver last couple of years
  • Despite promising detection results, binary signatures

remain the method of choice binary signatures + efficiency

  • evasion
  • emulation

behavior + effectiveness

slide-6
SLIDE 6

Secure Systems Lab Technical University Vienna 6

Motivation

  • Behavior-based detection received some attention
  • ver last couple of years
  • Despite promising detection results, binary signatures

remain the method of choice binary signatures

  • evasion
  • emulation

+ efficiency behavior + effectiveness

slide-7
SLIDE 7

Secure Systems Lab Technical University Vienna 7

Outline

  • Motivation
  • Detecting Behavior

– Motivating example (Agent)

  • Matching Behavior Graphs
  • Extracting Behavior Graphs
  • Evaluation
slide-8
SLIDE 8

Secure Systems Lab Technical University Vienna 8

Detecting Behavior

slide-9
SLIDE 9

Secure Systems Lab Technical University Vienna 9

Detecting Behavior

  • Characteristic malware behavior

– Manifest on system (i.e., survive reboot)

  • (Over-) write system executables, dlls, files
  • Create registry entries (autorun)
  • Register as Windows (startup) service

– Conceal from being detected

  • Restart under some stealthy name (e.g., svchost.exe)
  • Inject into legitimate processes

– Replicate

  • Send eMails ('check out this picture I found: pic.jpg.exe')
  • Copy to Samba shares, USB drives, etc.
  • Scan and exploit services on LAN or WAN
slide-10
SLIDE 10

Secure Systems Lab Technical University Vienna 10

Detecting Behavior

System Overview

  • Detection based on execution characteristics

– Execute malware in full system emulator (Anubis) – Monitor interaction with the operating system – Perform detailed (taint-) analysis – Generate detection graphs

  • Describe sequence of required system calls leading to security

relevant system activity

  • Include dependencies to related, previous calls (using taint

dependencies)

  • Detect described behavior on end host

– Log system call activity of unknown executable – Match against behavior graph

slide-11
SLIDE 11

Secure Systems Lab Technical University Vienna 11

Detecting Behavior

Developer Perspective

  • Example: Agent (trojan horse)
  • As part of its system manifestation, it

– Reads content from binary image – Decrypts binary content

  • Proprietary decryption routine
  • Simple, XOR based algorithm

– Stores binary in system file (C:\Wind...\drivers\ip6fw.sys) – Later, restarts IPv6 firewall

  • Turns itself into a system service
slide-12
SLIDE 12

Secure Systems Lab Technical University Vienna 12

Detecting Behavior

Taint-Trace Perspective

GetModuleFileNameA NtCreateFile NtCreateSection NtMapViewOfSection NtWriteFile NtCreateFile

Name FileHandle SectionHandle FileHandle (read & decrypt buffer)

Mode: Open Mode: Create C:\Win... \ip6fw.sys

slide-13
SLIDE 13

Secure Systems Lab Technical University Vienna 13

Detecting Behavior

System Perspective

GetModuleFileNameA

Name (read & decrypt buffer)

NtCreateFile NtCreateSection NtMapViewOfSection

FileHandle SectionHandle

Mode: Open NtWriteFile NtCreateFile

FileHandle

Mode: Create C:\Win... \ip6fw.sys

slide-14
SLIDE 14

Secure Systems Lab Technical University Vienna 14

Detecting Behavior

System Perspective

GetModuleFileNameA

Name (read & decrypt buffer)

NtWriteFile NtCreateFile

FileHandle

Mode: Create C:\Win... \ip6fw.sys NtCreateFile NtCreateSection NtMapViewOfSection

FileHandle SectionHandle

Mode: Open

slide-15
SLIDE 15

Secure Systems Lab Technical University Vienna 15

Detecting Behavior

System Perspective

GetModuleFileNameA

Name (read & decrypt buffer)

NtCreateFile NtCreateSection NtMapViewOfSection

FileHandle SectionHandle

Mode: Open NtWriteFile NtCreateFile

FileHandle

Mode: Create C:\Win... \ip6fw.sys

slide-16
SLIDE 16

Secure Systems Lab Technical University Vienna 16

Detecting Behavior

System Perspective

NtCreateFile NtCreateSection NtMapViewOfSection

SectionHandle

Mode: Open

FileHandle

NtWriteFile NtCreateFile Mode: Create C:\Win... \ip6fw.sys

FileHandle

slide-17
SLIDE 17

Secure Systems Lab Technical University Vienna 17

Detecting Behavior

System Perspective

NtCreateFile NtCreateSection NtMapViewOfSection

SectionHandle

Mode: Open

FileHandle

NtWriteFile NtCreateFile Mode: Create C:\Win... \ip6fw.sys

FileHandle

slide-18
SLIDE 18

Secure Systems Lab Technical University Vienna 18

Detecting Behavior

  • Detection based on execution characteristics

– Works well as long as we can see all types of dependencies between system calls – Handle dependencies

  • Insufficient for detection
  • Behavior graphs break into trivial subgraphs

– Data dependencies

  • Convenient for behavior graph generation
  • Necessary for behavior detection
slide-19
SLIDE 19

Secure Systems Lab Technical University Vienna 19

Matching Behavior Graphs

slide-20
SLIDE 20

Secure Systems Lab Technical University Vienna 20

Matching Behavior Graphs

  • Maintaining dependencies using taint propagation

– Performance overhead: Extended emulation engine – Memory overhead: Shadow memory – Not applicable to production systems / end hosts

  • Maintaining dependencies without taint propagation

– Handle dependencies

  • Direct value propagation
  • System provided identifiers

– File, section, process, thread handles – Registry keys – Socket identifiers

– Must be constant between call invocations

slide-21
SLIDE 21

Secure Systems Lab Technical University Vienna 21

Matching Behavior Graphs

  • Maintaining dependencies without taint propagation

– Data dependencies

  • Arbitrary data (& control) dependency between system calls
  • Might modify values between system calls

– Our proposal: Anticipate precise call arguments

  • Use recorded execution semantics
  • Extract data propagation/manipulation formulas
  • Emulate taint dependency between system call A and B

– Log outgoing parameters of call A – Use as input to propagation formula – Predicted incoming parameters for system call B – Compare predicted and monitored input parameters – Assume dependency between A and B if prediction holds

slide-22
SLIDE 22

Secure Systems Lab Technical University Vienna 22

Matching Behavior Graphs

System Perspective

GetModuleFileNameA

Name (read & decrypt buffer)

NtCreateFile NtCreateSection NtMapViewOfSection

FileHandle SectionHandle

Mode: Open NtWriteFile NtCreateFile

FileHandle

Mode: Create C:\Win... \ip6fw.sys

slide-23
SLIDE 23

Secure Systems Lab Technical University Vienna 23

Matching Behavior Graphs

System Perspective

GetModuleFileNameA

Name f1, data (read & decrypt buffer) f4, data

NtCreateFile NtCreateSection NtMapViewOfSection

FileHandle f2, handle SectionHandle f3, handle

Mode: Open NtWriteFile NtCreateFile

FileHandle f5, handle

Mode: Create C:\Win... \ip6fw.sys

slide-24
SLIDE 24

Secure Systems Lab Technical University Vienna 24

Matching Behavior Graphs

System Perspective

GetModuleFileNameA

Name f1, data (read & decrypt buffer) f4, data

NtCreateFile NtCreateSection NtMapViewOfSection

FileHandle f2, handle SectionHandle f3, handle

Mode: Open NtWriteFile NtCreateFile

FileHandle f5, handle

Mode: Create C:\Win... \ip6fw.sys

(read & decrypt buffer) f4, data

NtMapViewOfSection NtWriteFile

slide-25
SLIDE 25

Secure Systems Lab Technical University Vienna 25

Matching Behavior Graphs

System Perspective

NtMapViewOfSection NtWriteFile

f5, handle

NtMapViewOfSection(out m_buffer[0...size],

  • ut m_size)

NtCreateFile(out c_handle)

slide-26
SLIDE 26

Secure Systems Lab Technical University Vienna 26

Matching Behavior Graphs

System Perspective

NtMapViewOfSection NtWriteFile

f5, handle

NtMapViewOfSection(out m_buffer[0...size],

  • ut m_size)

NtCreateFile(out c_handle) NtWriteFile(in w_handle, in w_buffer[0...size], in w_size) (p_buffer, p_size) = f4(m_buffer, m_size)

slide-27
SLIDE 27

Secure Systems Lab Technical University Vienna 27

Matching Behavior Graphs

System Perspective

NtMapViewOfSection NtWriteFile

f5, handle

NtMapViewOfSection(out m_buffer[0...size],

  • ut m_size)

NtCreateFile(out c_handle) NtWriteFile(in w_handle, in w_buffer[0...size], in w_size) (p_buffer, p_size) = f4(m_buffer, m_size)

slide-28
SLIDE 28

Secure Systems Lab Technical University Vienna 28

Extracting Behavior Graphs

slide-29
SLIDE 29

Secure Systems Lab Technical University Vienna 29

Extracting Behavior Graphs

  • Analyze executable in Anubis sandbox

– Obtain instruction level log

  • Defeats packers

– Obtain program flow log – Obtain memory access log – Generate precise taint propagation trees

  • Data/control dependencies
  • Instructions that access/generate tainted data
  • Link system calls consuming data (sinks) with all taint

generating calls (sources)

Anubis Scanner Slicer

slide-30
SLIDE 30

Secure Systems Lab Technical University Vienna 30

Extracting Behavior Graphs

  • Scan logs for security relevant behavior

– Provided with a list of interesting system calls

  • Extract graphs matching behavior

– Include triggering system call X – Link in system calls providing tainted data to X – Analyze dependencies:

  • Label edges with handle dependencies
  • Call slicer for all data dependencies

Anubis Scanner Slicer

slide-31
SLIDE 31

Secure Systems Lab Technical University Vienna 31

Extracting Behavior Graphs

  • Find encoding formula for each data dependency
  • Binary program slicing

– Resolve def-use chains

  • Starting at selected call invocation
  • Iterate backwards (using program flow logs)
  • Aided by taint information and memory access logs

– Optional:

  • Symbolic execution to simplify encoding function

– Embed into dynamically loadable library (dll) – Label graph edges with appropriate function (dll)

Anubis Scanner Slicer

slide-32
SLIDE 32

Secure Systems Lab Technical University Vienna 32

Extracting Behavior Graphs

  • Resolving def-use chains

– Three possible sources 1) Previous system call invocation

  • Replaced with stub
  • Provides input values to slice (i.e., recorded, outgoing system

call parameters)

2) Immediate values

  • Implicitly encoded in binary slice (e.g., push $0x3)

Anubis Scanner Slicer

slide-33
SLIDE 33

Secure Systems Lab Technical University Vienna 33

Extracting Behavior Graphs

  • Resolving def-use chains

– Three possible sources 3) Preinitialized data segments

  • BSS section

– Constants – Static strings

  • Two-sided approach:

– Use static values from Anubis analysis – Dynamically inspect running process

Anubis Scanner Slicer

slide-34
SLIDE 34

Secure Systems Lab Technical University Vienna 34

Extracting Behavior Graphs

  • Fully automated process

– Analyze binary – Generate behavior graph(s) – Extract propagation formulas – Verify graph on binary

  • Run binary & scanner on real host
  • Verify behavior graph matches (only) on intended

executable

Anubis Scanner Slicer

slide-35
SLIDE 35

Secure Systems Lab Technical University Vienna 35

Evaluation

slide-36
SLIDE 36

Secure Systems Lab Technical University Vienna 36

Evaluation

  • Effectiveness of behavior graphs

– Applicable to polymorphic variants of a malware sample? – General enough for whole malware families?

  • Efficiency of behavior graph matching

– Overhead through system call logging – Additional system load through dependency verification

slide-37
SLIDE 37

Secure Systems Lab Technical University Vienna 37

Effectiveness

  • Six current threats / threat families
  • Identified using AV (binary) signature
  • Encountered 0 false positives

Name Type Samples Variants Samples AV Our detected Exploit-based worm 50 2 1 50 1.00 Beagle Mass-mailing worm 50 20 14 46 0.92 Mass-mailing worm 50 32 12 47 0.94 Mass-mailing worm 50 20 2 41 0.82 Mass-mailing worm 50 22 12 46 0.92 Agent Trojan horse 50 6 3 49 0.98 Total 300 102 44 279 0.93 Eff. Allaple Mydoom Mytob Netsky

slide-38
SLIDE 38

Secure Systems Lab Technical University Vienna 38

Effectiveness

  • Experiment:

Can the system detect malware instances never seen by the graph generator?

Name Samples AV variants Samples New Known detected 50 50 45 0.90 Beagle 50 24 26 30 0.60 50 24 26 36 0.72 50 46 4 5 0.10 13 8 5 7 0.54 Agent 50 6 44 45 0.90 Total 263 108 155 168 0.63 Eff. Allaple Mydoom Mytob Netsky

slide-39
SLIDE 39

Secure Systems Lab Technical University Vienna 39

Effectiveness

  • Experiment:

Name Samples AV variants Samples New Known detected New Known 50 50 45 0.90 Beagle 50 24 26 30 0.60 50 24 26 36 0.72 50 46 4 5 0.10 13 8 5 7 0.54 Agent 50 6 44 45 0.90 Total 263 108 155 168 0.23 0.92 Eff. Allaple Mydoom Mytob Netsky

Can the system detect malware instances never seen by the graph generator?

slide-40
SLIDE 40

Secure Systems Lab Technical University Vienna 40

Efficiency

  • I-O bound activity

– Compressing, archiving

  • CPU bound computation

– Compilation, rendering

Test Baseline Log Full scanner 7-zip (benchmark) 114 sec 117 sec 2.3 % 118 sec 2.4 % 7-zip (compress) 318 sec 328 sec 3.1 % 333 sec 4.7 % 7-zip (archive) 213 sec 225 sec 6.2 % 231 sec 8.4 % IE (rendering) 0.41 pages/s 0.39 pages/s 4.4 % 0.39 pages/s 4.4 % VC++ (compile) 104 sec 117 sec 12.2 % 146 sec 39.8 %

slide-41
SLIDE 41

Secure Systems Lab Technical University Vienna 41

Summary

  • Behavior can be detected

– Monitor from system perspective – Match against behavior graphs – Link graph nodes through argument dependencies

  • Handle dependencies

– Vital for checking – BUT not specific enough for doing detection

  • Data dependencies

– Anticipate future call arguments – Efficient replacement for taint dependencies – Provided through slicing malware semantics

slide-42
SLIDE 42

Secure Systems Lab Technical University Vienna 42

Summary

  • Evaluation

– Behavior detection is fast enough for end hosts – Approach intrinsically robust against polymorphism and metamorphism – To some extent, behavior graphs are usable across malware variants

slide-43
SLIDE 43

Secure Systems Lab Technical University Vienna 43

Thanks for your attention!