Eleos: Exit-Less OS Services for SGX Enclaves
Meni Orenbach Marina Minkin Pavel Lifshits Mark Silberstein Accelerated Computing Systems Lab
Haifa, Israel
Eleos: Exit-Less OS Services for SGX Enclaves Meni Orenbach Marina - - PowerPoint PPT Presentation
Eleos: Exit-Less OS Services for SGX Enclaves Meni Orenbach Marina Minkin Pavel Lifshits Mark Silberstein Accelerated Computing Systems Lab Haifa, Israel What do we do? Improve performance: I/O intensive & memory demanding SGX enclaves
Haifa, Israel
22 May@Systor' 2017 Meni Orenbach, Technion 2
Improve performance: I/O intensive & memory demanding SGX enclaves
Cost of SGX execution for these applications is high
In-enclave System Calls & User Managed Virtual Memory
Eleos vs vanilla SGX
2x Throughput: memcached & face verification servers Even for 5x available enclave memory
Available for Linux, Windows* (*) Without Eleos, these applications crash in Windows enclaves
22 May@Systor' 2017 Meni Orenbach, Technion 3
22 May@Systor' 2017 Meni Orenbach, Technion 4
– Confidentiality – Integrity – Freshness
Operating system Application Enclave Enclave
22 May@Systor' 2017 Meni Orenbach, Technion 5
– Confidentiality – Integrity – Freshness
Operating system Application Enclave Enclave
22 May@Systor' 2017 Meni Orenbach, Technion 6
– Confidentiality – Integrity – Freshness
Operating system Application Enclave Enclave
22 May@Systor' 2017 Meni Orenbach, Technion 7
– Confidentiality – Integrity – Freshness
Operating system Application Enclave Enclave
22 May@Systor' 2017 Meni Orenbach, Technion 8
– Confidentiality – Integrity – Freshness
Operating system Application Enclave Enclave
22 May@Systor' 2017 Meni Orenbach, Technion 9
– Confidentiality – Integrity – Freshness
Operating system Application Enclave Enclave
Lets look at How to secure server applications with enclaves
22 May@Systor' 2017 Meni Orenbach, Technion 10
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 11
Untrusted memory Unsecured access
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 12
Untrusted memory Unsecured access
Untrusted (Host & OS) Trusted (Enclave)
Dedicated SGX mem Limited to: 128 MB Secured access
22 May@Systor' 2017 Meni Orenbach, Technion 13
Host app
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 14
Host app
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 15
Enter enclave Host app
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 16
Enter enclave
Host app
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 17
Enter enclave
Host app
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 18
Enter enclave
Exit enclave Host app
Untrusted (Host & OS) Trusted (Enclave)
22 May@Systor' 2017 Meni Orenbach, Technion 19
22 May@Systor' 2017 Meni Orenbach, Technion 20
22 May@Systor' 2017 Meni Orenbach, Technion 21
22 May@Systor' 2017 Meni Orenbach, Technion 22
64 MB 512 MB 5 10 15 20 25 30 35 40 Memory footprint
Throughput: Slowdown factor
11X 34X
22 May@Systor' 2017 Meni Orenbach, Technion 23
64 MB 512 MB 5 10 15 20 25 30 35 40 Memory footprint
Throughput: Slowdown factor
11X 34X
Crashes in Windows
22 May@Systor' 2017 Meni Orenbach, Technion 24
22 May@Systor' 2017 Meni Orenbach, Technion 25
Send responses Wait for network requests
Enter enclave Exit enclave Host app
Untrusted (Host & OS) Trusted (Enclave)
Decrypt requests 150 cycles/32B Process requests
Encrypt responses
22 May@Systor' 2017 Meni Orenbach, Technion 26
Enter enclave Exit enclave Host app
Untrusted (Host & OS) Trusted (Enclave)
Send responses Wait for network requests
Enter enclave Exit enclave Host app
Decrypt requests 150 cycles/32B Process requests
Encrypt responses
~3,300 cycles
22 May@Systor' 2017 Meni Orenbach, Technion 27
Enter enclave Exit enclave Host app
Untrusted (Host & OS) Trusted (Enclave)
Send responses Wait for network requests
Enter enclave Exit enclave Host app
Decrypt requests
Process requests
Encrypt responses
~3,300 cycles ~3,800 cycles
22 May@Systor' 2017 Meni Orenbach, Technion 28
Enter enclave Exit enclave Host app
Untrusted (Host & OS) Trusted (Enclave)
Send responses Wait for network requests
Enter enclave Exit enclave Host app
Decrypt requests 150 cycles/32B Process requests
Encrypt responses
~3,300 cycles ~3,800 cycles Exits causes indirect costs: 1.5X – 5X slower execution FlexSC [OSDI'10] syscall analysis
22 May@Systor' 2017 Meni Orenbach, Technion 29
Enter enclave Exit enclave Host app
Untrusted (Host & OS) Trusted (Enclave)
Send responses Wait for network requests
Enter enclave Exit enclave Host app
Decrypt requests 150 cycles/32B Process requests
Encrypt responses
~3,300 cycles ~3,800 cycles Exits causes indirect costs: 1.5X – 5X slower execution FlexSC [OSDI'10] syscall analysis
22 May@Systor' 2017 Meni Orenbach, Technion 30
64 MB 512 MB 5 10 15 20 25 30 35 40 SGX Eleos Memory footprint
3.5x 5x
Throughput: Slowdown factor
22 May@Systor' 2017 Meni Orenbach, Technion 31
64 MB 512 MB 5 10 15 20 25 30 35 40 SGX Eleos Memory footprint
3.5x 5x
Throughput: Slowdown factor
22 May@Systor' 2017 Meni Orenbach, Technion 32
22 May@Systor' 2017 Meni Orenbach, Technion 33
22 May@Systor' 2017 Meni Orenbach, Technion 34
System mem SGX mem Dedicated memory Enclave code & data Limited to 128 MB
22 May@Systor' 2017 Meni Orenbach, Technion 35
System mem
secret_foo(): ... *p = 1;
SGX mem Enclave Trusted Untrusted
22 May@Systor' 2017 Meni Orenbach, Technion 36
System mem
secret_foo(): ... *p = 1;
SGX mem Hardware Address translation Enclave Trusted Untrusted
22 May@Systor' 2017 Meni Orenbach, Technion 37
System mem
secret_foo(): ... *p = 1;
Encrypted SGX mem Page table Hardware Address translation Enclave Trusted Untrusted
22 May@Systor' 2017 Meni Orenbach, Technion 38
System mem
secret_foo(): ... *p = 1;
Encrypted SGX mem Page table Hardware Address translation Swapped-out Enclave Trusted Untrusted
22 May@Systor' 2017 Meni Orenbach, Technion 39
System mem Fault handler
secret_foo(): ... *p = 1;
Encrypted SGX mem Page table Hardware Address translation Swapped-out Enclave Trusted Untrusted SGX-driver
22 May@Systor' 2017 Meni Orenbach, Technion 40
System mem Fault handler
secret_foo(): ... *p = 1;
Encrypted
Integrity validation Decrypted
SGX mem Page table Hardware Address translation Swapped-out Enclave Trusted Untrusted SGX-driver
22 May@Systor' 2017 Meni Orenbach, Technion 41
System mem Fault handler
secret_foo(): ... *p = 1; *(++p) = 2;
Encrypted
Decrypted
SGX mem Page table Hardware Address translation Enclave Trusted SGX driver Untrusted
22 May@Systor' 2017 Meni Orenbach, Technion 42
System mem Fault handler
secret_foo(): ... *p = 1; *(++p) = 2;
Encrypted
Decrypted
SGX mem Page table Hardware Address translation Enclave Trusted SGX driver Untrusted Fast path
22 May@Systor' 2017 Meni Orenbach, Technion 43
System mem Fault handler
secret_foo(): ... *p = 1; *(++p) = 2;
Encrypted
Decrypted
SGX mem Page table Hardware Address translation Enclave Trusted SGX driver Untrusted
Fast path
22 May@Systor' 2017 Meni Orenbach, Technion 44
System mem Fault handler
secret_foo(): ... *p = 1; *(++p) = 2;
Encrypted
Decrypted
SGX mem Page table Hardware Address translation Enclave Trusted SGX driver Untrusted
22 May@Systor' 2017 Meni Orenbach, Technion 45
System mem Fault handler
secret_foo(): ... *p = 1; *(++p) = 2;
Encrypted
Decrypted
SGX mem Page table Hardware Address translation Enclave Trusted SGX driver Untrusted Enclave resume Enclave exit
22 May@Systor' 2017 Meni Orenbach, Technion 46
System mem Fault handler
secret_foo(): ... *p = 1; *(++p) = 2;
Encrypted
Decrypted
SGX mem Page table Hardware Address translation Enclave Trusted SGX driver Untrusted Enclave resume Enclave exit Indirect costs
22 May@Systor' 2017 Meni Orenbach, Technion 47
System mem Fault handler
secret_foo(): ... *p = 1; *(++p) = 2;
Encrypted
Decrypted
SGX mem Page table Hardware Address translation Enclave Trusted SGX driver Untrusted Enclave resume Enclave exit Overaheads: Untrusted software manages enclave memory Indirect costs
22 May@Systor' 2017 Meni Orenbach, Technion 48
System mem Fault handler
secret_foo(): ... *p = 1; *(++p) = 2;
Encrypted
Decrypted
SGX mem Page table Hardware Address translation Enclave Trusted SGX driver Untrusted Enclave resume Enclave exit Overaheads: Untrusted software manages enclave memory Indirect costs
22 May@Systor' 2017 Meni Orenbach, Technion 49
22 May@Systor' 2017 Meni Orenbach, Technion 50
System mem Fault handler
secret_foo(): ... *p = 1; *(++p) = 2;
SGX mem Page table Hardware Address translation Enclave Trusted SGX driver Untrusted Hardware Address translation
22 May@Systor' 2017 Meni Orenbach, Technion 51
System mem Fault handler
secret_foo(): ... *p = 1; *(++p) = 2;
SGX mem Page table Hardware Address translation Enclave Trusted Hardware Address translation
22 May@Systor' 2017 Meni Orenbach, Technion 52
System mem Fault handler
secret_foo(): ... *p = 1; *(++p) = 2;
SGX mem Page table Hardware Address translation Enclave Trusted Hardware Address translation
No available hardware
22 May@Systor' 2017 Meni Orenbach, Technion 53
System mem Fault handler
secret_foo(): ... *p = 1; *(++p) = 2;
SGX mem Page table Hardware Address translation Enclave Trusted Software Address translation
22 May@Systor' 2017 Meni Orenbach, Technion 54
System mem Fault handler
secret_foo(): s_ptr<int> p = suvm_malloc(1024); ... *p = 1;
SGX mem Page table Enclave Trusted Software Address translation
22 May@Systor' 2017 Meni Orenbach, Technion 55
System mem Fault handler
secret_foo(): s_ptr<int> p = suvm_malloc(1024); ... *p = 1;
SGX mem Page table Enclave Trusted Software Address translation
Template class: SecuredPointer.
22 May@Systor' 2017 Meni Orenbach, Technion 56
System mem Fault handler
secret_foo(): s_ptr<int> p = suvm_malloc(1024); ... *p = 1;
Encrypted SGX mem Page table Enclave Trusted Software Address translation
Template class: SecuredPointer.
22 May@Systor' 2017 Meni Orenbach, Technion 57
System mem Fault handler
secret_foo(): s_ptr<int> p = suvm_malloc(1024); ... *p = 1;
Encrypted SGX mem Page table Enclave Trusted Software Address translation
Template class: SecuredPointer.
Swapped-out
22 May@Systor' 2017 Meni Orenbach, Technion 58
System mem Fault handler
secret_foo(): s_ptr<int> p = suvm_malloc(1024); ... *p = 1;
Encrypted SGX mem Page table Enclave Trusted Software Address translation
Template class: SecuredPointer.
Swapped-out
22 May@Systor' 2017 Meni Orenbach, Technion 59
System mem Fault handler
secret_foo(): s_ptr<int> p = suvm_malloc(1024); ... *p = 1;
Encrypted
Decrypted
SGX mem Page table Enclave Trusted Software Address translation
Template class: SecuredPointer.
Swapped-out
Integrity validation
22 May@Systor' 2017 Meni Orenbach, Technion 60
System mem Fault handler
secret_foo(): s_ptr<int> p = suvm_malloc(1024); ... *p = 1;
Encrypted
Decrypted
SGX mem Page table Enclave Trusted Software Address translation
Template class: SecuredPointer.
Swapped-out
Integrity validation
Control path in-enclave
22 May@Systor' 2017 Meni Orenbach, Technion 61
System mem Fault handler
secret_foo(): s_ptr<int> p = suvm_malloc(1024); ... *p = 1; *(++p) = 2;
Encrypted
Decrypted
SGX mem Page table Enclave Trusted Software Address translation
22 May@Systor' 2017 Meni Orenbach, Technion 62
System mem Fault handler
secret_foo(): s_ptr<int> p = suvm_malloc(1024); ... *p = 1; *(++p) = 2;
Encrypted
Decrypted
SGX mem Page table Enclave Trusted Software Address translation
22 May@Systor' 2017 Meni Orenbach, Technion 63
System mem Fault handler
secret_foo(): s_ptr<int> p = suvm_malloc(1024); ... *p = 1; *(++p) = 2;
Encrypted
Decrypted
SGX mem Page table Enclave Trusted Software Address translation
Fast path No page table Lookup!
22 May@Systor' 2017 Meni Orenbach, Technion 64
22 May@Systor' 2017 Meni Orenbach, Technion 65
Throughput speedup
22 May@Systor' 2017 Meni Orenbach, Technion 66
22 May@Systor' 2017 Meni Orenbach, Technion 67
Virtual Machine ballooning
22 May@Systor' 2017 Meni Orenbach, Technion 68
Virtual Machine ballooning
22 May@Systor' 2017 Meni Orenbach, Technion 69
22 May@Systor' 2017 Meni Orenbach, Technion 70
Face verification server Workload generator
22 May@Systor' 2017 Meni Orenbach, Technion 71
1 2 4 0.5 1 1.5 2 2.5 3 3.5 Eleos Native Server threads
22 May@Systor' 2017 Meni Orenbach, Technion 72
1 2 4 0.5 1 1.5 2 2.5 3 3.5 Eleos Native Server threads
22 May@Systor' 2017 Meni Orenbach, Technion 73
1 2 4 0.5 1 1.5 2 2.5 3 3.5 Eleos Native Server threads
Eleos scales better than vanilla-SGX: Saves inter-processor-interrupts
22 May@Systor' 2017 Meni Orenbach, Technion 74
1 2 4 0.5 1 1.5 2 2.5 3 3.5 Eleos Native Server threads
Eleos scales better than vanilla-SGX: Saves inter-processor-interrupts Saturate 10Gb network
22 May@Systor' 2017 Meni Orenbach, Technion 75
Workload Generator (memaslap)
~75 LOC modification for SUVM
Memcached Graphene LibOS [Eurosys'2014]
22 May@Systor' 2017 Meni Orenbach, Technion 76
1 Thread 4 Threads 0.5 1 1.5 2 2.5 3 E l e
( 5 MB D B ) v a n i l l a S G X ( 2 MB D B ) Server threads
No SGX Faults No SGX Faults
22 May@Systor' 2017 Meni Orenbach, Technion 77
1 Thread 4 Threads 0.5 1 1.5 2 2.5 3 E l e
( 5 MB D B ) v a n i l l a S G X ( 2 MB D B ) Server threads
Disclaimer: Eleos+Graphene is 3x slower than native No SGX Faults No SGX Faults
22 May@Systor' 2017 Meni Orenbach, Technion 78
– Makes memory demanding applications available
– Memory demanding app? Link to SUVM – I/O intensive app? Link to RPC – Maintaining small TCB
22 May@Systor' 2017 Meni Orenbach, Technion 79
22 May@Systor' 2017 Meni Orenbach, Technion 80
Get data
22 May@Systor' 2017 Meni Orenbach, Technion 81
Get data Data Unavailable
22 May@Systor' 2017 Meni Orenbach, Technion 82
Fetch data Get data Data Unavailable
22 May@Systor' 2017 Meni Orenbach, Technion 83
Fetch data Get data Data Unavailable
22 May@Systor' 2017 Meni Orenbach, Technion 84
Get data Fetch data
22 May@Systor' 2017 Meni Orenbach, Technion 85
– System calls: GPUfs [ASPLOS'13], GPUnet [OSDI'14] – Virtual memory: ActivePointers [ISCA'16]
– Asynchronous DMA host copies – Non-blocking enclave launches
More information at: “SGX Enclaves as Accelerators" [Systex'16]
22 May@Systor' 2017 Meni Orenbach, Technion 86
22 May@Systor' 2017 Meni Orenbach, Technion 87