Dynamic VM Monitoring using Hypervisor Probes Z. J. Estrada , C. - - PowerPoint PPT Presentation

▶
dynamic vm monitoring using hypervisor probes
SMART_READER_LITE
LIVE PREVIEW

Dynamic VM Monitoring using Hypervisor Probes Z. J. Estrada , C. - - PowerPoint PPT Presentation

May 12, 2023 303 likes •1.08k views

Dynamic VM Monitoring using Hypervisor Probes Z. J. Estrada , C. Pham, F. Deng, L. Yan, Z. Kalbarczyk, R. K. Iyer European Dependable Computing Conference 2015-09-09 ECE ILLINOIS 1 Department of Electrical and Computer Engineering Dynamic VM

slide-1
SLIDE 1

Dynamic VM Monitoring using Hypervisor Probes

  • Z. J. Estrada, C. Pham, F. Deng, L. Yan,
  • Z. Kalbarczyk, R. K. Iyer

European Dependable Computing Conference 2015-09-09

ECE ILLINOIS

Department of Electrical and Computer Engineering

1

slide-2
SLIDE 2

Dynamic VM Monitoring

Goal

On-demand VM Monitoring to reduce the effort required to harden computing systems against failures and attacks. Uptime requirements Effort required QA concerns Lack of knowledge

ECE ILLINOIS

Department of Electrical and Computer Engineering

2

slide-3
SLIDE 3

VM Monitoring

Reliability & Security Monitoring

Recording and analyzing a computer system to detect failures and attacks.

◮ Passive - polling based ◮ Active - event based

ECE ILLINOIS

Department of Electrical and Computer Engineering

3

slide-4
SLIDE 4

VM Monitoring

Hypervisor OS Applications VM KVM ECE ILLINOIS

Department of Electrical and Computer Engineering

4

slide-5
SLIDE 5

VM Monitoring

Hypervisor OS Applications VM KVM

Monitor

ECE ILLINOIS

Department of Electrical and Computer Engineering

4

slide-6
SLIDE 6

VM Monitoring

Hypervisor OS Applications VM KVM ECE ILLINOIS

Department of Electrical and Computer Engineering

4

slide-7
SLIDE 7

VM Monitor Monitor is running inside the hypervisor

ECE ILLINOIS

Department of Electrical and Computer Engineering

5

slide-8
SLIDE 8

VM Monitor VM execution reaches a hook

ECE ILLINOIS

Department of Electrical and Computer Engineering

5

slide-9
SLIDE 9

VM Monitor Control is transferred to the monitor

ECE ILLINOIS

Department of Electrical and Computer Engineering

5

slide-10
SLIDE 10

VM Monitor The monitor performs its monitoring function

ECE ILLINOIS

Department of Electrical and Computer Engineering

5

slide-11
SLIDE 11

VM Monitor Control is transferred back to the VM

ECE ILLINOIS

Department of Electrical and Computer Engineering

5

slide-12
SLIDE 12

VM Monitor The VM resumes normal execution

ECE ILLINOIS

Department of Electrical and Computer Engineering

5

slide-13
SLIDE 13

Hook-Based VM Monitoring

Previous techniques: + Active monitoring + Protected hooks − Guest OS only - no userspace − Not dynamic - boot time config − Require guest OS modifications

ECE ILLINOIS

Department of Electrical and Computer Engineering

6

slide-14
SLIDE 14

Goals

Hook-based monitoring should: + be protected from attacks in the VM + be simple to use + not require guest OS modification + be runtime adaptable + allow for arbitrary hook placement

ECE ILLINOIS

Department of Electrical and Computer Engineering

7

slide-15
SLIDE 15

Hypervisor Probes

ECE ILLINOIS

Department of Electrical and Computer Engineering

8

slide-16
SLIDE 16

Hardware Assisted Virt.

VMEntry VMExit Guest Mode (non-root) Host Mode (root) User Kernel User Kernel

ECE ILLINOIS

Department of Electrical and Computer Engineering

9

slide-17
SLIDE 17

Hypervisor Probes

◮ Event on guest execution

◮ Event transfers control to hypervisor (VM Exit) ◮ Perform monitoring after that event

◮ Hooks added/removed at runtime ◮ Monitors applications and the guest OS

ECE ILLINOIS

Department of Electrical and Computer Engineering

10

slide-18
SLIDE 18

Hprobe Architecture

KVM Hypervisor

VM

Event Forwarder

Host Linux kernel

Hprobe Kernel agent Hprobe user agent

Host System

Helper APIs Set/Remove probes Insert/Remove probes Set single step

Probe Probe Probe ioctl(…)

Detector 1 Detector 2 Detector n Status Checker

ECE ILLINOIS

Department of Electrical and Computer Engineering

11

slide-19
SLIDE 19

Hprobes API

int HPROBE_add_probe( ); int HPROBE_remove_probe( );

◮ addr info: gva+cr3 ◮ vmid: unique id for VM ◮ vcpu type: vcpu state

ECE ILLINOIS

Department of Electrical and Computer Engineering

12

slide-20
SLIDE 20

Probe ⇒ Event Forwarder

VM Hypervisor ... pushl %eax incl %eax decl %ebx ...

ECE ILLINOIS

Department of Electrical and Computer Engineering

13

slide-21
SLIDE 21

Probe ⇒ Event Forwarder

VM Hypervisor ... pushl %eax int3 decl %ebx ...

ECE ILLINOIS

Department of Electrical and Computer Engineering

13

slide-22
SLIDE 22

Probe ⇒ Event Forwarder

VM Hypervisor ... pushl %eax int3 decl %ebx ...

probe hit (int3) handler()

Detector

ECE ILLINOIS

Department of Electrical and Computer Engineering

13

slide-23
SLIDE 23

Probe ⇒ Event Forwarder

VM Hypervisor ... pushl %eax incl %eax decl %ebx ...

probe hit (int3) handler() Reset inst.

ECE ILLINOIS

Department of Electrical and Computer Engineering

13

slide-24
SLIDE 24

Probe ⇒ Event Forwarder

VM Hypervisor ... pushl %eax incl %eax decl %ebx ...

probe hit (int3) handler() Reset inst. single step execute inst.

ECE ILLINOIS

Department of Electrical and Computer Engineering

13

slide-25
SLIDE 25

Probe ⇒ Event Forwarder

VM Hypervisor ... pushl %eax int3 decl %ebx ...

probe hit (int3) handler() Reset inst. single step execute inst. trap rewrite int3

ECE ILLINOIS

Department of Electrical and Computer Engineering

13

slide-26
SLIDE 26

Probe ⇒ Event Forwarder

VM Hypervisor ... pushl %eax int3 decl %ebx ...

probe hit (int3) handler() Reset inst. single step execute inst. trap rewrite int3 resume ...

ECE ILLINOIS

Department of Electrical and Computer Engineering

13

slide-27
SLIDE 27

Userspace Probe Challenge

Guest Page Tables

ECE ILLINOIS

Department of Electrical and Computer Engineering

14

slide-28
SLIDE 28

Userspace Probe Challenge

Guest Page Tables

ECE ILLINOIS

Department of Electrical and Computer Engineering

14

slide-29
SLIDE 29

Userspace Probe Challenge

Guest Page Tables

ECE ILLINOIS

Department of Electrical and Computer Engineering

14

slide-30
SLIDE 30

Extended Page Tables (EPT)

[1]

◮ Guest OS has full control over PTs ◮ 2nd set of HW PTs for GPA→HPA ◮ Use EPT to write-protect Guest Page Table

[1] http://www-archive.xenproject.org/files/xensummit 4/VT roadmap d Nakajima.pdf

ECE ILLINOIS

Department of Electrical and Computer Engineering

15

slide-31
SLIDE 31

Goals

Hook-based monitoring should: + be protected from attacks in the VM + be simple to use + not require guest OS modification + be runtime adaptable + allow for arbitrary hook placement

ECE ILLINOIS

Department of Electrical and Computer Engineering

16

slide-32
SLIDE 32

Goals

Hook-based monitoring should: be protected from attacks in the VM + be simple to use + not require guest OS modification + be runtime adaptable + allow for arbitrary hook placement

ECE ILLINOIS

Department of Electrical and Computer Engineering

16

slide-33
SLIDE 33

Goals

Hook-based monitoring should: be protected from attacks in the VM be simple to use + not require guest OS modification + be runtime adaptable + allow for arbitrary hook placement

ECE ILLINOIS

Department of Electrical and Computer Engineering

16

slide-34
SLIDE 34

Goals

Hook-based monitoring should: be protected from attacks in the VM be simple to use not require guest OS modification + be runtime adaptable + allow for arbitrary hook placement

ECE ILLINOIS

Department of Electrical and Computer Engineering

16

slide-35
SLIDE 35

Goals

Hook-based monitoring should: be protected from attacks in the VM be simple to use not require guest OS modification be runtime adaptable + allow for arbitrary hook placement

ECE ILLINOIS

Department of Electrical and Computer Engineering

16

slide-36
SLIDE 36

Goals

Hook-based monitoring should: be protected from attacks in the VM be simple to use not require guest OS modification be runtime adaptable allow for arbitrary hook placement

ECE ILLINOIS

Department of Electrical and Computer Engineering

16

slide-37
SLIDE 37

Hprobe Microbenchmarks

◮ probe @ noop kernel function ◮ execute 1M times

kernel user kernel user

Hypervisor VM

insert probe start/stop hypercall record time

[2] [2] Adapted from an image by Fei Deng

ECE ILLINOIS

Department of Electrical and Computer Engineering

17

slide-38
SLIDE 38

2.6GHz E5430 Harpertown (2007) 2.2-3.0GHz E5-2660 Sandy Bridge (2012) 2.0 2.5 3.0 3.5 4.0 4.5 Time (µs)

Hprobe Single Probe Latency

ECE ILLINOIS

Department of Electrical and Computer Engineering

18

slide-39
SLIDE 39

Hook-based VM Monitoring

Name Latency User Dynamic Modifications Lares 28µs No No Hypervisor/Guest SIM 0.40µs No No Hypervisor/Guest hprobes 2.6µs Yes Yes Hypervisor

ECE ILLINOIS

Department of Electrical and Computer Engineering

19

slide-40
SLIDE 40

Hook-based VM Monitoring

Name Latency User Dynamic Modifications Lares 28µs No No Hypervisor/Guest SIM 0.40µs No No Hypervisor/Guest hprobes 2.6µs Yes Yes Hypervisor

◮ as-a-Service is worth slight performance cost

ECE ILLINOIS

Department of Electrical and Computer Engineering

19

slide-41
SLIDE 41

Detectors

What detectors can we build with hprobes?

ECE ILLINOIS

Department of Electrical and Computer Engineering

20

slide-42
SLIDE 42

Detectors

What detectors can we build with hprobes?

◮ Arbitrarily chose events ◮ On-demand ◮ Access to VM memory & CPU state

ECE ILLINOIS

Department of Electrical and Computer Engineering

20

slide-43
SLIDE 43

Heartbeat/watchdog

App Detector

ECE ILLINOIS

Department of Electrical and Computer Engineering

21

slide-44
SLIDE 44

Heartbeat/watchdog

App Detector I n s e r t P r

  • b

e

ECE ILLINOIS

Department of Electrical and Computer Engineering

21

slide-45
SLIDE 45

Heartbeat/watchdog

App Detector I n s e r t P r

  • b

e P r

  • b

e H i t

ECE ILLINOIS

Department of Electrical and Computer Engineering

21

slide-46
SLIDE 46

Heartbeat/watchdog

App Detector I n s e r t P r

  • b

e P r

  • b

e H i t reset timer

ECE ILLINOIS

Department of Electrical and Computer Engineering

21

slide-47
SLIDE 47

Heartbeat/watchdog

App Detector I n s e r t P r

  • b

e P r

  • b

e H i t P r

  • b

e H i t reset timer

ECE ILLINOIS

Department of Electrical and Computer Engineering

21

slide-48
SLIDE 48

Heartbeat/watchdog

App Detector I n s e r t P r

  • b

e P r

  • b

e H i t P r

  • b

e H i t reset timer reset timer

ECE ILLINOIS

Department of Electrical and Computer Engineering

21

slide-49
SLIDE 49

Heartbeat/watchdog

  • App

Detector I n s e r t P r

  • b

e P r

  • b

e H i t P r

  • b

e H i t reset timer reset timer

ECE ILLINOIS

Department of Electrical and Computer Engineering

21

slide-50
SLIDE 50

Heartbeat/watchdog

  • App

Detector I n s e r t P r

  • b

e P r

  • b

e H i t P r

  • b

e H i t reset timer reset timer timer expires declare failure

ECE ILLINOIS

Department of Electrical and Computer Engineering

21

slide-51
SLIDE 51

Watchdog - Performance

1x 2x 3x 4x 5x 6x 7x 8x 9x

Internal Sample Loop Size

24000 26000 28000 30000 32000 34000

Time (msec)

PI-QMC Main Loop Runtime

No Detector With Detector

ECE ILLINOIS

Department of Electrical and Computer Engineering

22

slide-52
SLIDE 52

Detectors

Infinite Loop Detector

ECE ILLINOIS

Department of Electrical and Computer Engineering

23

slide-53
SLIDE 53

Detectors

Infinite Loop Detector

◮ Kernel or App-level ◮ Previously determined

threshold

◮ Or register

ECE ILLINOIS

Department of Electrical and Computer Engineering

23

slide-54
SLIDE 54

Infinite Loop Detector

for(i=0; i<10; i++) { ... } //after loop

ECE ILLINOIS

Department of Electrical and Computer Engineering

24

slide-55
SLIDE 55

Infinite Loop Detector

for(i=0; i<10; i++) { ... } //after loop

1st Probe

(counter)

ECE ILLINOIS

Department of Electrical and Computer Engineering

24

slide-56
SLIDE 56

Infinite Loop Detector

for(i=0; i<10; i++) { ... } //after loop

1st Probe

(counter)

2nd Probe

(reset)

ECE ILLINOIS

Department of Electrical and Computer Engineering

24

slide-57
SLIDE 57

Without Infinite Loop

Application Time (s) 95% CI (s) % overhead Normal 1.13 0.0325 N/A Na¨ ıve ILD - Page 1.26 0.0229 11.5 Na¨ ıve ILD - No Page 1.26 0.0265 11.8 Smart ILD - Page 1.14 0.0267 1.15 Smart ILD - No Page 1.15 0.0215 1.9

ECE ILLINOIS

Department of Electrical and Computer Engineering

25

slide-58
SLIDE 58

Consider this situation

ECE ILLINOIS

Department of Electrical and Computer Engineering

26

slide-59
SLIDE 59

A vulnerability is announced

ECE ILLINOIS

Department of Electrical and Computer Engineering

26

slide-60
SLIDE 60

At a later time, a patch is released

ECE ILLINOIS

Department of Electrical and Computer Engineering

26

slide-61
SLIDE 61

?

What can we do?

ECE ILLINOIS

Department of Electrical and Computer Engineering

26

slide-62
SLIDE 62

?

We may have to follow a maintenance window

ECE ILLINOIS

Department of Electrical and Computer Engineering

26

slide-63
SLIDE 63

?

Even when the bug and patch are coreleased

ECE ILLINOIS

Department of Electrical and Computer Engineering

26

slide-64
SLIDE 64

STOPGAP

To mitigate risk, we would like a stopgap

ECE ILLINOIS

Department of Electrical and Computer Engineering

26

slide-65
SLIDE 65

STOPGAP

Solution

Use an Hprobe-based Detector

ECE ILLINOIS

Department of Electrical and Computer Engineering

26

slide-66
SLIDE 66

Emergency Detector

Should be...

◮ easier than a patch ◮ simpler than a patch ◮ less disruptive than a patch ◮ less risky than a patch

ECE ILLINOIS

Department of Electrical and Computer Engineering

27

slide-67
SLIDE 67

Emergency Detector

◮ CVE-2008-0600 - Privilege Escalation in

vmsplice() [3]

◮ Integer overflow in a struct iovec

argument

◮ Corrupts OS (kernel) stack ◮ Execute attack payload

struct iovec { void *iov_base; size_t iov_len; };

[3] http://www.win.tue.nl/∼aeb/linux/hh/hh-12.html#ss12.4

ECE ILLINOIS

Department of Electrical and Computer Engineering

28

slide-68
SLIDE 68

Emergency Detector

◮ Added to running guest OS ◮ Detects malicious value that causes

  • verflow

◮ Two modes of operation

◮ Read-only mode: does not change anything ◮ Fix mode: malicious value ⇒ benign value

ECE ILLINOIS

Department of Electrical and Computer Engineering

29

slide-69
SLIDE 69

Emergency Detector

◮ Probe at vmsplice() syscall ◮ Get value of iov len off of the stack

ECE ILLINOIS

Department of Electrical and Computer Engineering

30

slide-70
SLIDE 70

Emergency Detector

procedure VMSPLICE HANDLER(vcpu) iov pointer ← read guest(esp+arg offset) iov len ← read guest virt(iov pointer) if iov len ≥ BAD VALUE then HANDLE EXPLOIT ATTEMPT(vcpu) end if end procedure

ECE ILLINOIS

Department of Electrical and Computer Engineering

31

slide-71
SLIDE 71

Detector Performance

◮ Checkpoint/Restart In Userspace ◮ Two scientific computing applications

◮ Folding @ Home ◮ Path-integral Quantum Monte Carlo

◮ Three cases:

◮ Normal: base case without monitoring ◮ hprobe: only monitor sys vmsplice ◮ Na¨

ıve: monitor all system calls

ECE ILLINOIS

Department of Electrical and Computer Engineering

32

slide-72
SLIDE 72

Detector Performance

Application Runtime ± 95% CI (s)

  • verhead (%)

F@H Normal 0.221 ± 0.0092 F@H w/hprobe 0.228 ± 0.012 3.30 F@H w/Na¨ ıve 0.253 ± 0.0085 14.4 pi-qmc Normal 0.137 ± 0.0063 pi-qmc w/hprobe 0.140 ± 0.0073 1.73 pi-qmc w/Na¨ ıve 0.152 ± 0.0051 11.1

ECE ILLINOIS

Department of Electrical and Computer Engineering

33

slide-73
SLIDE 73

Thoughts

◮ Zero overhead without vmsplice() ◮ Cloud provider doesn’t need tenant to

update

◮ Can be used while official fix is in QA ◮ Don’t need full understanding of bug

ECE ILLINOIS

Department of Electrical and Computer Engineering

34

slide-74
SLIDE 74

VM Monitoring Techniques

H p r

  • b

e s

( E D C C ' 1 5 )

H y p e r T a p

( D S N ' 1 4 )

L i v e W i r e

( N D S S ’ 3 )

L i b V M I

( A C S A C ’ 7 )

L a r e s

( S P ’ 8 )

S I M

( C C S ’ 9 )

A n t f a r m

( A T C ’ 6 )

L y c

  • s

i d

( V E E ’ 8 )

O s c k

( A S P L O S ’ 1 1 )

V i r t u

  • s
  • (

S P ’ 1 1 )

V M S T

( S P ’ 1 2 )

On-demand Add/Remove ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ Changes to VM ✗ ✗ ✗ ✗ ✓ ✗ ✗ ✗ ✗ Userspace Monitoring ✓ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ Root-of-trust (invariant) OS HW OS OS OS HW OS OS OS Active/Passive Mon. A (Hook) A P P A (Hook) A P P P Auto-generate Monitoring ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✓ ✓ This Presentation Desirable Feature ✓ = Supported Feature ✗ = Unsupported Feature

ECE ILLINOIS

Department of Electrical and Computer Engineering

35

slide-75
SLIDE 75

Acknowledgements

◮ Collaborators:

Cuong Pham, Fei Deng, Dr. Lok Yan,

  • Prof. Zbigniew Kalbarczyk, Prof. Ravi Iyer

ECE ILLINOIS

Department of Electrical and Computer Engineering

36

slide-76
SLIDE 76

Summary

◮ VM Monitoring ◮ How hprobes work ◮ Microbenchmarks ◮ Emergency Detector

ECE ILLINOIS

Department of Electrical and Computer Engineering

37