Shoot4U: Using VMM Assists to Optimize TLB Operations on Preempted - - PowerPoint PPT Presentation
Shoot4U: Using VMM Assists to Optimize TLB Operations on Preempted vCPUs Jiannan Ouyang , John Lange Haoqiang Zheng University of Pittsburgh VMware Inc. VEE16 04/02/2016 CPU Consolidation in the Cloud CPU Consolidation: multiple virtual
VEE’16 04/02/2016
2
Figure 1. Average activity distribution of a typical shared Google clusters including Online Services, each containing over 20,000 servers, over a period of 3 months [Barroso 13].
3 P P P P A B A B B A A B A
vCPU of VM-A
B
vCPU of VM-B Preempted Running pCPU
P
Lock Holder Preemption problem Lock Waiter Preemption problem TLB Shootdown Preemption problem
4
A preempted vCPU is holding a spinlock Causes dramatically longer lock waiting time
context switch latency + CPU shares allocated to other vCPUs
co-scheduling, relaxed co-scheduling [VMware 10] Adaptive co-scheduling [Weng HPDC11] Balanced scheduling [Sukwong EuroSys11] Demand-based coordinated scheduling [Kim ASPLOS13]
Intel Pause-Loop Exiting (PLE) [Riel 11]
5
i i+1 i+2 i+3
A lock waiter is preempted, and blocks the queue P(waiter preemption) > P(holder preemption)
Key idea: proportional timeout
6
VMM maintains vCPU preemption states and shares with the
Use conventional approach if the remote vCPU is running. Defer TLB flush if the remote vCPU is preempted. Cons: preemption state may change after checking.
Goal: eliminate the problem Key idea: invalidate guest TLB entries from the VMM
7
An analysis of the impact that various low level
Shoot4U: A novel virtualized TLB architecture that ensures
An evaluation of the performance benefits achieved by
8
9
2 4 6 8 10 12 14 16 18 20 blackscholes bodytrack canneal dedup ferret freqmine raytrace streamcluster swaptions vips x264 Slowdown 70.6 max ideal slowdown
(12-core VMs, measured on Linux/KVM, with PLE disabled)
10
10 20 30 40 50 60 70 80 90 100 blackscholes bodytrack canneal dedup ferret freqmine raytrace streamcluster swaptions vips x264 blackscholes bodytrack canneal dedup ferret freqmine raytrace streamcluster swaptions vips x264 Percentage (%) 1VM 2VM
k:lock k:tlb k:other u:*
11
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 100 101 102 103 104 105 106 Cumulative Percent Latency (us) 1VM 2VM
12
TLB (Translation Lookaside Buffer)
a per-core hardware cache for page table translation results
TLB coherence is managed by the OS
TLB shootdown operations: IPI + invlpg
Linux TLB shootdown is busy-waiting based
VMM
P P P P A A B A
(TLB Shootdown Preemption)
B B A B
13
14
Tell the VMM what TLB entries and vCPUs to invalidate (hypervall) The VMM invalidates and returns, no interrupt injection and waiting
<vcpu set, addr range>
VMM
P P P P A A B A B B A B
15
https://github.com/ouyangjn/shoot4u
use hypercall for TLB shootdowns
hypercall handler: vCPU set => pCPU set, and send IPIs IPI handler: invalidate guest TLB entries with invpid
kvm_hypercall3(unsigned long KVM_HC_SHOOT4U, unsigned long vcpu_bitmap, unsigned long start_addr, unsigned long end_addr);
Shoot4U API
16
6-core Intel “Ivy-Bridge” Xeon processors with hyperthreading 24 GB RAM split across two NUMA domains. CentOS 7 (Linux 3.16)
12 vCPUs, 4G RAM on the same socket Fedora 19 (Linux 4.0) VM1: PARSEC Benchmark Suite, VM2 sysbench CPU test
baseline: unmodified Linux kernel kvmtlb [kvmtlb 12] Shoot4U Pause-Loop Exiting (PLE) [Riel 11] Preemptable Ticket Spinlock (PMT) [Ouyang VEE ’13]
17
18
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 100 101 102 103 104 105 106 Cumulative Percent Latency (us) shoot4u-1VM shoot4u-2VM kvmtlb-1VM kvmtlb-2VM baseline-1VM baseline-2VM
19
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 1.1 b l a c k s c h
e s b
y t r a c k c a n n e a l d e d u p f e r r e t f r e q m i n e r a y t r a c e s t r e a m c l u s t e r s w a p t i
s v i p s x 2 6 4 Normalized Execution Time
baseline ple pmt pmt+kvmtlb pmt+shoot4u ple+pmt+shoot4u
20
2 4 6 8 10 12 14 16 18 20 blackscholes bodytrack canneal dedup ferret freqmine raytrace streamcluster swaptions vips x264 Slowdown 70.6
baseline ple+pmt+shoot4u
21
10 20 30 40 50 60 70 80 90 100 blackscholes bodytrack canneal dedup ferret freqmine raytrace streamcluster swaptions vips x264 blackscholes bodytrack canneal dedup ferret freqmine raytrace streamcluster swaptions vips x264 Percentage (%) baseline 2VM ple+pmt+shoot4u 2VM
k:lock k:tlb k:other u:*
22
We conducted a set of experiments in order to provide a
We Shoot4U, an optimization for TLB shootdown operations that
Our evaluation demonstrates the effectiveness of our approach,
23
24
http://www.cs.pitt.edu/~ouyang/
http://www.prognosticlab.org Pisces Co-Kernel Kitten Lightweight Kernel Palacios VMM
25
[Ouyang 13] Jiannan Ouyang and John R. Lange. Preemptable Ticket
Spinlocks: Improving Consolidated Performance in the Cloud. In Proc. 9th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE), 2013.
[Uhlig 04] Volkmar Uhlig, Joshua LeVasseur, Espen Skoglund, and
[Friebel 08] Thomas Friebel. How to deal with lock-holder
[Kim ASPLOS’13] H. Kim, S. Kim, J. Jeong, J. Lee, and S. Maeng.
26
[VMware 10] VMware(r) vSphere(tm): The cpu scheduler in
[Barroso 13] L. A. Barroso, J. Clidaras, and U. Holzle. The
[Weng HPDC’11] C. Weng, Q. Liu, L.
[Sukwong EuroSys’11] O. Sukwong and H. S. Kim. Is Co-
27
[Riel 11] R. v. Riel. Directed yield for pause loop exiting, 2011.
[kvmtlb 12] KVM Paravirt Remote Flush TLB. https://lwn.net/