B l a c k B o x Lightweight Security Monitoring for COTS - - PowerPoint PPT Presentation

B l a c k B

• x

Lightweight Security Monitoring for COTS Binaries

Byron Hawkins and Brian Demsky

University of California, Irvine, USA

Michael B. Taylor

University of California, San Diego, USA

Why Security Monitoring?

Motjvatjon #1: Exploits occur frequently.

• Thousands of vulnerabilitjes reported in 2015.
• Millions of new malware released every year.
• Over 90% of atuacks target 10+ year old bugs.
• Windows XP stjll claims over 10% market share!

Why Security Monitoring?

Motjvatjon #2: Exploit recovery requires informatjon.

• Identjfy which machines were afgected:
• Repair corrupted fjles.
• Restore failed or crippled services.
• Remove persistent malware.
• Learn how the atuacker gained control.
• Prevent recurrence of the same exploit.

Why Security Monitoring?

Motjvatjon #3: Automated security may never be feasible.

Why Security Monitoring?

Motjvatjon #3: Automated security may never be feasible.

• COTS rewritjng approaches have been defeated.

Why Security Monitoring?

Motjvatjon #3: Automated security may never be feasible.

• COTS rewritjng approaches have been defeated.
• Recompilatjon required, but opportunity is limited:
• Slow adoptjon of security tools
• Legacy platgorms maintain signifjcant market share

Why Security Monitoring?

Motjvatjon #3: Automated security may never be feasible.

• COTS rewritjng approaches have been defeated.
• Recompilatjon required, but opportunity is limited:
• Slow adoptjon of security tools
• Legacy platgorms maintain signifjcant market share
• Is it possible to build security into the compiler?

Exploit Example: Bufger Overfmow

Parse the argument by copying up to the space into the bufger.

Exploit Example: Bufger Overfmow

User input can change the return address of this functjon!

Automated Security

Defense concept:

• Identjfy vulnerable program instructjons

x 7 f f f f f f f d 4 1 8 a r g # 2 : * *

• p

t i

• n

_

• u

t x 7 f f f f f f f d a e 7 a r g # 1 : * i n p u t x 1 l

• c

a l # 2 : i x 7 7 2 d 6 e 6 f 6 9 7 4 7 6 f l

• c

a l # 1 : b u f f e r x 6 1 7 6 6 e 6 9 2 d 6 8 7 4 6 9 < p r e v i

• u

s s t a c k f r a m e b a s e > x 7 5 6 c 6 1 7 6 2 d 6 4 6 9 6 c < r e t u r n a d d r e s s > A d v e r s a r y c

• n

t r

• l

s t h e p r

• g

r a m !

Automated Security

Defense concept:

• Identjfy vulnerable program instructjons
• Detect adversarial manipulatjon of instructjon operands

x 7 f f f f f f f d 4 1 8 a r g # 2 : * *

• p

t i

• n

_

• u

t x 7 f f f f f f f d a e 7 a r g # 1 : * i n p u t x 1 l

• c

a l # 2 : i x 7 7 2 d 6 e 6 f 6 9 7 4 7 6 f l

• c

a l # 1 : b u f f e r x 6 1 7 6 6 e 6 9 2 d 6 8 7 4 6 9 < p r e v i

• u

s s t a c k f r a m e b a s e > x 7 5 6 c 6 1 7 6 2 d 6 4 6 9 6 c < r e t u r n a d d r e s s > A d v e r s a r y c

• n

t r

• l

s t h e p r

• g

r a m !

Automated Security

Defense problem:

• How to distjnguish adversarial infmuence?

x 7 f f f f f f f d 4 1 8 a r g # 2 : * *

• p

t i

• n

_

• u

t x 7 f f f f f f f d a e 7 a r g # 1 : * i n p u t x 1 l

• c

a l # 2 : i x 7 7 2 d 6 e 6 f 6 9 7 4 7 6 f l

• c

a l # 1 : b u f f e r x 6 1 7 6 6 e 6 9 2 d 6 8 7 4 6 9 < p r e v i

• u

s s t a c k f r a m e b a s e > x 7 5 6 c 6 1 7 6 2 d 6 4 6 9 6 c < r e t u r n a d d r e s s > A d v e r s a r y c

• n

t r

• l

s t h e p r

• g

r a m !

Automated Security

Defense proposal:

• Detect unintended data fmows...

x 7 f f f f f f f d 4 1 8 a r g # 2 : * *

• p

t i

• n

_

• u

t x 7 f f f f f f f d a e 7 a r g # 1 : * i n p u t x 1 l

• c

a l # 2 : i x 7 7 2 d 6 e 6 f 6 9 7 4 7 6 f l

• c

a l # 1 : b u f f e r x 6 1 7 6 6 e 6 9 2 d 6 8 7 4 6 9 < p r e v i

• u

s s t a c k f r a m e b a s e > x 7 5 6 c 6 1 7 6 2 d 6 4 6 9 6 c < r e t u r n a d d r e s s > A d v e r s a r y c

• n

t r

• l

s t h e p r

• g

r a m !

Automated Security

Defense proposal:

• Detect unintended data fmows…
• Intended data fmows can be exploited!

x 7 f f f f f f f d 4 1 8 a r g # 2 : * *

• p

t i

• n

_

• u

t x 7 f f f f f f f d a e 7 a r g # 1 : * i n p u t x 1 l

• c

a l # 2 : i x 7 7 2 d 6 e 6 f 6 9 7 4 7 6 f l

• c

a l # 1 : b u f f e r x 6 1 7 6 6 e 6 9 2 d 6 8 7 4 6 9 < p r e v i

• u

s s t a c k f r a m e b a s e > x 7 5 6 c 6 1 7 6 2 d 6 4 6 9 6 c < r e t u r n a d d r e s s > A d v e r s a r y c

• n

t r

• l

s t h e p r

• g

r a m !

Automated Security

Defense proposal:

• Detect unintended data fmows…
• Intended data fmows can be exploited!
• Calculate intended operand values
• e.g. legitjmate return addresses

Automated Security

Defense proposal:

• Detect unintended data fmows…
• Intended data fmows can be exploited!
• Calculate intended operand values
• e.g. legitjmate return addresses
• Counter example: Microsofu media licensing

Automated Security Limitatjons

At startup of Microsofu Word, a media license module dynamically generates a small routjne on the heap:

ipcsecproc.dll

DGC

56 basic blocks

code generator

Automated Security Limitatjons

It pushes a placeholder on the call stack...

ipcsecproc.dll

DGC

56 basic blocks create fake stack frame

Call Stack

Automated Security Limitatjons

…followed by a non-conventjonal “return” to enter the DGC.

ipcsecproc.dll

DGC

56 basic blocks

Call Stack

"incorrect" return

Automated Security Limitatjons

Calculatjng this return address requires calculatjng heap states.

ipcsecproc.dll

DGC

56 basic blocks

Call Stack

"incorrect" return

Automated Security Limitatjons

The DGC routjne calls several security-sensitjve functjons.

ipcsecproc.dll

DGC

56 basic blocks

"incorrect" return

VirtualProtect() DeviceIoControl() CreateFile() Kernel32.dll CryptUnprotectData() Crypt32.dll

Automated Security Limitatjons

Defense proposal:

• Detect unintended data fmows…
• Intended data fmows can be exploited!
• Calculate intended operand values
• e.g. legitjmate return addresses
• Counter example: Microsofu media licensing
• Counter example: Windows thread injectjon

Automated Security Limitatjons

D W O R D y

• u

r A p p P r

• c

e s s I d = G e t P r

• c

e s s B y N a m e ( " Y

• u

r A p p . e x e " ) ;

Automated Security Limitatjons

D W O R D y

• u

r A p p P r

• c

e s s I d = G e t P r

• c

e s s B y N a m e ( " Y

• u

r A p p . e x e " ) ; H A N D L E y

• u

r A p p P r

• c

e s s = O p e n P r

• c

e s s ( P R O C E S S _ A L L _ A C C E S S , y

• u

r A p p P r

• c

e s s I d ) ;

Automated Security Limitatjons

D W O R D y

• u

r A p p P r

• c

e s s I d = G e t P r

• c

e s s B y N a m e ( " Y

• u

r A p p . e x e " ) ; H A N D L E y

• u

r A p p P r

• c

e s s = O p e n P r

• c

e s s ( P R O C E S S _ A L L _ A C C E S S , y

• u

r A p p P r

• c

e s s I d ) ; H M O D U L E k e r n e l 3 2 = G e t M

• d

u l e H a n d l e ( " k e r n e l 3 2 . d l l " ) ;

Automated Security Limitatjons

D W O R D y

• u

r A p p P r

• c

e s s I d = G e t P r

• c

e s s B y N a m e ( " Y

• u

r A p p . e x e " ) ; H A N D L E y

• u

r A p p P r

• c

e s s = O p e n P r

• c

e s s ( P R O C E S S _ A L L _ A C C E S S , y

• u

r A p p P r

• c

e s s I d ) ; H M O D U L E k e r n e l 3 2 = G e t M

• d

u l e H a n d l e ( " k e r n e l 3 2 . d l l " ) ; L P T H R E A D _ S T A R T _ R O U T I N E l

• a

d L i b r a r y = G e t P r

• c

A d d r e s s ( k e r n e l 3 2 , " L

• a

d L i b r a r y A " ) ;

Automated Security Limitatjons

D W O R D y

• u

r A p p P r

• c

e s s I d = G e t P r

• c

e s s B y N a m e ( " Y

• u

r A p p . e x e " ) ; H A N D L E y

• u

r A p p P r

• c

e s s = O p e n P r

• c

e s s ( P R O C E S S _ A L L _ A C C E S S , y

• u

r A p p P r

• c

e s s I d ) ; H M O D U L E k e r n e l 3 2 = G e t M

• d

u l e H a n d l e ( " k e r n e l 3 2 . d l l " ) ; L P T H R E A D _ S T A R T _ R O U T I N E l

• a

d L i b r a r y = G e t P r

• c

A d d r e s s ( k e r n e l 3 2 , " L

• a

d L i b r a r y A " ) ; H A N D L E h T h r e a d = C r e a t e R e m

• t

e T h r e a d ( y

• u

r A p p P r

• c

e s s , l

• a

d L i b r a r y , m y M a l w a r e D l l P a t h ) ;

Why Security Monitoring?

Motjvatjon #3: Automated security may never be feasible.

• COTS rewritjng approaches have been defeated.
• Recompilatjon required, but opportunity is limited:
• Slow adoptjon of security tools
• Legacy platgorms maintain signifjcant market share
• Indirect branch targets may be diffjcult to calculate:
• Microsofu media licensing
• Windows thread injectjon

Goals of BlackBox

• Log malicious and abnormal program actjvity
• Minimize logging of ordinary control fmow
• Block known exploits and protect known vulnerabilitjes

Naïve Applicatjon Monitoring

Log all branches executed by the monitored program.

Naïve Applicatjon Monitoring

Problem: Adversary can hide evidence afuer a successful exploit.

Component: Remote Logging

Write logs over a TCP socket to a remote server.

Component: Remote Logging

Problem: Billions of branch history events per hour.

Component: Binary Translatj tjon

Execute code from a cached copy, logging each CFG edge once.

Next call to foo() will not log A → C → D → E again.

Component: Binary Translatjon

Problem: Stjll up to millions of CFG edges per hour.

Component: Trusted Profj fjle

Learn normal program behavior during offmine profjling.

Offmine Profjling

Component: Trusted Profjle

Filter trusted CFG edges before logging.

Component: Trusted Profjle

Shadow Stack: Always trust conventjonal return edges.

Trustjng Dynamic Code

• 1. Large desktop applicatjons typically use JIT engines:

Microsofu Offjce JScript9 Adobe PDF Reader JScript9, ShockwaveFlash Chrome V8, PepperFlash

Trustjng Dynamic Code

• 1. Large desktop applicatjons typically use JIT engines:
• 2. Observatjonally equivalent executjons frequently

exhibit low-level difgerences in generated code.

Microsofu Offjce JScript9 Adobe PDF Reader JScript9, ShockwaveFlash Chrome V8, PepperFlash

Component: Dynamic Code Abstractjon

JIT engines may generate millions of basic blocks.

Component: Dynamic Code Abstractjon

Random factors afgect low-level code, e.g.:

• OS thread scheduling
• Webserver response tjme
• Internal tjmers

Component: Dynamic Code Abstractjon

JIT Profj fjle: Learn trusted entry and exit points (API).

Component: Dynamic Code Abstractjon

Encapsulate JIT code in a singleton CFG node.

Component: Dynamic Code Abstractjon

Learn the trusted API for the JIT code.

Improve System Call Visibility

• Exploits rely on system calls to carry out malicious

actjons such as creatjng or deletjng fjles.

Improve System Call Visibility

• Exploits rely on system calls to carry out malicious

actjons such as creatjng or deletjng fjles.

• System calls are usually invoked from trampolines

in core system libraries like ntdll:

Improve System Call Visibility

• Exploits rely on system calls to carry out malicious

actjons such as creatjng or deletjng fjles.

• System calls are usually invoked from trampolines

in core system libraries like ntdll: Executjon of a trusted trampoline does not always imply executjon of a trusted system call.

Improve System Call Visibility

The ZwCreateFile trampoline is trusted—what can go wrong? A malicious font could cause a malware library to be loaded!

Improve System Call Visibility

The ZwCreateFile trampoline is trusted—what can go wrong?

Component: Stack Spy

Suspicious syscall: infmuenced by an untrusted branch.

Component: Stack Spy

Suspicious syscall: infmuenced by an untrusted branch.

Component: Stack Spy

Suspicious syscall: infmuenced by an untrusted branch. Log this syscall 0x52 as suspicious, along with the untrusted branch.

BlackBox Log

What’s in the log fjle?

BlackBox Log

Suspicious syscalls occur where the player handles a media format not covered in the Trusted Profjle. Log sample of Adrenalin Media Player:

R a n k E v e n t

BlackBox Log

Log sample of an ROP exploit on Adrenalin Player:

R a n k E v e n t

BlackBox Log

How can this exploit be prevented?

R a n k E v e n t

Blacklist

Log entries can be converted into blacklist entries: This blacklist prohibits the exploited basic block from ever making an entry into dynamic code:

R a n k E v e n t

Blacklist

Prohibit the exploited basic block from entering DGC:

Blacklist

Prohibit abnormal return sites in an entjre module:

Blacklist

Prohibit abnormal return sites in the whole program:

Blacklist

Prohibit abnormal return sites in the whole program: Not viable for all programs!

• Adobe PDF Reader plugins contain abnormal returns

Benchmark: SPEC CPU 2006

Geometric Mean: 14.5%

Problem: Return address points to F on the left.

Correlate indirect branch targets via hashtable.

Hot paths are compiled into traces (10% speedup).

Benchmark: SPEC CPU 2006

Overhead corresponds roughly to cumulatjve indirect branch degree.

Log Reductjon

Program All Branches Binary Translatjon Trusted Profjle Chrome 485,251,278,660 6,137,106 7 Adobe PDF 34,075,711,128 2,292,342 4 Word 603,491,452,236 580,655 24 PowerPoint 251,845,377,624 1,335,817 50 Excel 198,427,776,372 561,401 28 Outlook 547,678,615,056 615,708 4 SciTE 61,325,719,872 124,013 33 pldfmatex 23,504,352,560 64,290 43 Notepad++ 129,695,545,404 589,155 24 Adrenalin 48,881,533,212 791,847 603 mp3info 2,080,031,200 4,339,200 3

Total log entries in one hour of normal program use.

Related Work: Other Defenses

• Code Pointer Integrity, Kuznetsov et al, USENIX '14
• SafeDispatch, Jang et al, NDSS '14
• Inference of Peak Density of Indirect Branches,

Tymburiba et al, CGO '16

• Automated Sofuware Diversity, Larsen et al, S&P '14
• Librando, Homescu et al, CCS '13
• Mining Sandboxes, Jamrozik et al, ICSE '16

Related Work: CFI

• Control-Flow Integrity, Abadi et al, CCS '05
• Control Data Isolatjon, Arthur et al, CGO '15
• Context-Sensitjve CFI, van der Veen et al, CCS '15
• Cryptographic CFI, Mashtjzadeh et al, CCS '15
• Opaque CFI, Mohan et al, NDSS '15
• RockJIT, Niu et al, CCS '14

Related Work: Other Defenses

• Code Pointer Integrity, Kuznetsov et al, USENIX '14
• SafeDispatch, Jang et al, NDSS '14
• Inference of Peak Density of Indirect Branches,

Tymburiba et al, CGO '16

• Automated Sofuware Diversity, Larsen et al, S&P '14
• Librando, Homescu et al, CCS '13
• Mining Sandboxes, Jamrozik et al, ICSE '16

Related Work: Atuacks

• Control Flow Bending, Carlini et al, USENIX '15
• Control Flow Jujutsu, Evans et al, CCS '15
• Counterfeit Object Oriented Programming,

Schuster et al, S&P '15

• Losing Control, Contj et al, CCS '15

Conclusion

• Sofuware exploits are an ongoing problem.
• Automated security contjnues to face challenges.
• BlackBox provides an efgectjve alternatjve to both

automated security and antj-virus:

• Isolates the pivotal actjons of exploits.
• Prevents repeated and foreseen exploits.