Nested Virtualization on ARM NEVE: Nested Virtualization Extensions - PowerPoint PPT Presentation

Nested Virtualization on ARM NEVE: Nested Virtualization Extensions Jin Tack Lim Christoffer Dall Shih-Wei Li 
 Jason Nieh Marc Zyngier LEADING 
 jitack@cs.columbia.edu christo ff er.dall@linaro.org shih-wei@cs.columbia.edu, COLLABORATION 
 nieh@cs.columbia.edu marc.zyngier@arm.com IN THE ARM 
 ECOSYSTEM connect.linaro.org

Nested Virtualization VM VM VM App App App App VM Kernel Kernel App App Hypervisor Kernel Hypervisor Hardware

Terminology VM Nested VM Nested VM App App App App VM Kernel Kernel App App Guest Hypervisor Kernel Host Hypervisor Hardware

Use Cases 1. Run guest operating systems with built-in virtualization. 2. IaaS hosting private clouds 3. Test your hypervisor in a VM 4. Debug your hypervisor in a VM 5. Develop hypervisors using a cloud

ARM Virtualization Extensions VM VM User Space User Space EL0 Kernel Kernel EL1 Hypervisor EL2

ARM Nested Virtualization VM VM User Space User Space EL0 Kernel Kernel EL1 Virtual 
 Guest Hypervisor Guest Hypervisor EL2 Host Hypervisor EL2

ARM Nested Virtualization VM VM User Space User Space EL0 Kernel Kernel EL1 Guest Hypervisor EL ?? Guest Hypervisor Host Hypervisor EL2

ARMv8.0 Nested Virtualization VM VM User Space User Space EL0 Kernel Kernel EL1 Guest Hypervisor EL0 Guest Hypervisor Trap-and-emulate Host Hypervisor EL2

ARMv8.0 Nested Virtualization VM VM User Space User Space EL0 Kernel Kernel EL1 Guest Hypervisor EL1 Guest Hypervisor ?? -and-emulate Host Hypervisor EL2

ARMv8.3 Nested Virtualization • Gives you software emulation of VM vEL2 in EL1 User Space EL0 • HCR_EL2.NV: Kernel EL1 • Traps EL2 operations executed in EL1 to EL2 Guest Hypervisor EL1 • Traps eret to EL2 Trap -and-emulate Host Hypervisor EL2 • CurrentEL reports EL2 even in EL1

KVM/ARM Nested Virtualization Implementation • EL2 Emulation • Stage 2 MMU Virtualization • Hyp Timer Virtualization • Nested Virtual Interrupts

Nested CPU Virtualization struct kvm_cpu_context { u64 sys_regs[NR_SYS_REGS]; + u64 el2_regs[NR_EL2_REGS]; } struct kvm_vcpu_arch { … struct kvm_cpu_context ctxt; }

Hypervisor-VM Switch Host VM App App App App EL0 Kernel EL1 Save EL1 sys_regs Linux KVM EL2 Restore EL1 sys_regs

Hypervisor-Hypervisor Switch Host VM App App EL0 Kernel EL1 Guest Hypervisor Save/restore EL1 sys_regs Linux KVM EL2 Save/restore el2_regs

Emulating EL2 in EL1 • Define mapping of EL2 registers to EL1 registers • Example: TTBR0_EL2 to TTBR0_EL1 • Example: SCTLR_EL2 adapted to SCTLR_EL1 • Shadow EL1 registers

Nested CPU Virtualization struct kvm_cpu_context { u64 sys_regs[NR_SYS_REGS]; + u64 el2_regs[NR_EL2_REGS]; + u64 shaow_sys_regs[NR_SYS_REGS]; } struct kvm_vcpu_arch { … struct kvm_cpu_context ctxt; }

Shadow Registers &sys_regs &shadow_sys_regs PSTATE.mode == EL0/1 PSTATE.mode == EL2 u64 *vcpu->ctxtx.hw_regs

Virtual Exceptions VM EL0 User Space • Trap to virtual EL2 EL1 Kernel • “Forward” exceptions vEL2 Guest KVM • Emulate virtual exceptions Host KVM EL2

Virtual Exceptions VM EL0 User Space • Returning from virtual EL2 EL1 Kernel • Trap eret to EL2 (ARMv8.3) vEL2 Guest KVM • Emulate virtual exception return Host KVM EL2

KVM/ARM Nested Virtualization Implementation • EL2 Emulation • Stage 2 MMU Virtualization • Hyp Timer Virtualization • Nested Virtual Interrupts

Memory Virtualization EL0 User Space Stage 1: VA -> IPA Kernel EL1

Memory Virtualization VM EL0 User Space Stage 1: VA -> IPA Kernel EL1 Stage 2: IPA -> PA Host Hypervisor EL2

Memory Virtualization VM Nested VM EL0 User Space Stage 1: VA -> IPA Kernel EL1 ???? Guest Hypervisor Stage 2: IPA -> PA Host Hypervisor EL2

Memory Virtualization VM Nested VM EL0 User Space Stage 1: VA -> IPA Kernel EL1 Virtual stage 2 Guest Hypervisor Shadow Stage 2: 
 IPA -> PA Host Hypervisor EL2

KVM/ARM Nested Virtualization Implementation • EL2 Emulation • Stage 2 MMU Virtualization • Hyp Timer Virtualization • Nested Virtual Interrupts

Nested Timer Virtualization • ARM provides a virtual and physical timer in EL1 • EL2 provides a separate EL2 “hyp” timer • Nested KVM/ARM supports a virtual CPU with EL2 and the hyp timer

KVM/ARM Nested Virtualization Implementation • EL2 Emulation • Stage 2 MMU Virtualization • Hyp Timer Virtualization • Nested Virtual Interrupts

ARM Generic Interrupt Controller (GIC) IRQ CPU Interface Device 
 Interrupt 
 Dist. CPU ACK/EOI Lines CPU Interface GIC

ARM Generic Interrupt Controller (GIC) List Registers (LRs) IRQ CPU Interface VIRQ Virtual CPU Interface Dist. CPU ACK/EOI CPU Interface ACK/EOI Virtual CPU Interface GIC List Registers (LRs)

Nested Interrupt Virtualization VM Nested VM User Space • Deliver virtual interrupts Virtual CPU Kernel from the host to the VM Interface Guest VMM LRs Host VMM

Nested Interrupt Virtualization VM • Deliver virtual interrupts Nested VM from the guest hypervisor User to the nested VM Space Virtual CPU • Shadow list registers Kernel Interface • The nested VM can ACK Guest VMM and EOI virtual interrupts LRs without trapping Host VMM

Performance Evaluation • Problem: No ARMv8.3 hardware available. • Solution: Use ARMv8.0 hardware with the software modification

Emulating v8.3 on v8.0 VM Nested VM App App App EL0 OS Kernel EL1 Guest Hypervisor HVCHVC HVC Paravirtualization HVC HVC Host Hypervisor EL2 ARMv8.0 Hardware

Hypercall MicroBenchmark VM Nested VM VM App App App EL0 App App App EL0 OS Kernel EL1 OS Kernel EL1 Hypercall Return Guest Hypervisor Hypercall Return Hypervisor EL2 Host Hypervisor EL2

Hypercall MicroBenchmark ARMv8.3 VM Nested VM 2,729 422,720 Cycle counts 1 155x Ratio to VM

Application Benchmarks Normalized overhead (lower is better) 50 ARMv8.3 VM 40 ARMv8.3 Nested 30 20 10 0 5 4 3 2 1 0 Kernbench Hackbench SPECjvm2008 TCP RR TCP STREAM TCP MAERTS Apache Nginx Memcached MySQL

Nested VM Exit/Entry on ARM VM Nested VM App App App EL0 OS Kernel EL1 VM Exit VM Entry EL1 Registers Guest Hypervisor … Host Hypervisor EL2 Registers EL2 > 120 traps

NEVE: NEsted Virtualization Extensions for ARM • Supports unmodified guest hypervisors and OSes • Improves performance by providing register redirection

Register Classification • VM registers: EL1 registers only a ff ecting the nested VM’s execution • Hypervisor registers: EL2 registers a ff ecting the hypervisor’s execution

VM Registers VM Nested VM App App App EL0 OS Kernel EL1 VM Exit VM Entry EL1 Registers Guest Hypervisor … Host Hypervisor EL2 This is when VM register states are used

VM Registers: Logging to Memory VM Register Trap! Without NEVE msr x0, TTBR0_EL1 Memory

VM Registers: Logging to Memory VM Register msr x0, TTBR0_EL1 Memory With NEVE TTBR0_EL1

Hypervisor control registers • Can’t apply the technique for VM registers • They have an immediate impact (EL2 system registers) • Traps are handled by redirecting to EL1 registers in software Guest Hypervisor EL1 Registers EL1 Host Hypervisor EL2 EL2 Registers

Hypervisor control registers • Can’t apply the technique for VM registers • They have an immediate impact (EL2 system registers) • Traps are handled by redirecting to EL1 registers in software • Redirect in hardware instead! Guest Hypervisor EL1 Registers EL1 Host Hypervisor EL2 EL2 Registers

Hypercall MicroBenchmark ARMv8.3 NEVE VM Nested VM Nested VM 92,385 2,729 422,720 Cycle counts 34x 155x Ratio to VM 15 1 126 Trap counts

Application Workloads Application Description Application Description Netperf TCP_RR Kernbench Kernel compile Network performance Hackbench Scheduler stress Netperf TCP STREAM Network performance SPECjvm2008 Java Runtime Netperf TCP MAERTS Network performance MySQL Database management Apache Web server stress Memcached Key-Value store Nginx Web server stress

Experimental Setup • Software • ARM Hardware • Native/VM/Nested VM • APM X-Gene (ARMv8.0) • KVM on KVM • 4-way SMP • 8-way SMP • 64 GB RAM • v4.10 • 12 GB RAM • x86 Hardware • Virt I/O 
 • Intel E5-2630 v3 (VM/nested VM) • VMCS Shadowing • 10 Gb Ethernet • 8-way SMP • 128 GB RAM

Application Benchmarks Normalized overhead (lower is better) 50 ARMv8.3 VM 40 ARMv8.3 Nested NEVE Nested 30 20 10 0 5 4 3 2 1 0 Kernbench Hackbench SPECjvm2008 TCP RR TCP STREAM TCP MAERTS Apache Nginx Memcached MySQL

Application Benchmarks Normalized overhead (lower is better) 50 ARMv8.3 VM 40 ARMv8.3 Nested NEVE Nested 30 x86 Nested VM 20 10 0 5 4 3 2 1 0 Kernbench Hackbench SPECjvm2008 TCP RR TCP STREAM TCP MAERTS Apache Nginx Memcached MySQL