NOVA Microhypervisor on ARMv8-A FOSDEM 2020 Udo Steinberg BedRock - - PowerPoint PPT Presentation

NOVA Microhypervisor on ARMv8-A

FOSDEM 2020 Udo Steinberg

BedRock Systems, Inc.

February 2, 2020

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 1 / 17

Outline

1

NOVA Microhypervisor

2

ARMv8-A Virtualization

3

Current Status, Demo, Roadmap

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 2 / 17

NOVA: System Architecture – x86

OS Kernel App App OS Kernel App App

Guest VM Guest VM

VMM VMM

x86 Hardware Guest VM Guest VM Guest VM

VMM VMM VMM OS Kernel OS Kernel OS Kernel App App App App App App

Ring3 (H) Ring3 (G) Ring0 (G) Ring0 (H)

App App

Host NOVA Microhypervisor

Partition Manager

The microhypervisor is the only privileged component 1 Every virtual machine has its own VMM instance

1Ignoring SMM and Firmware, which are beyond our control

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 3 / 17

NOVA: System Architecture – ARMv8-A

EL2 EL3

Monitor

EL0 EL1

App VMM VMM OS Kernel App App OS Kernel App App

secure non-secure

Host Guest VM Guest VM TEE ARM Hardware NOVA Microhypervisor

Partition Manager

The microhypervisor is the only privileged component 1 Every virtual machine has its own VMM instance

1Ignoring TF-A, Monitor and TEE, which are beyond our control

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 4 / 17

NOVA: Capability-Based Access Control

1 2 3 4 5 6 7

PD Capability SC Capability PT Capability

1 2 3 4 5 6

PD Capability

7

PT Capability EC Capability PT Capability PT Capability PDA PT PT EC SC

Object Space Object Space

PDB Protection Domain A Protection Domain B

Selector Selector

2 6

Capability is pointer to KObject or PFrame + permissions Protection Domain has Object Space, Memory Space, ... Hypercall ctrl pd with take/grant semantics replaces MDB

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 5 / 17

NOVA: Basic Abstractions

UTCB UTCB SC UTCB SC UTCB P PDB ECcallee PDB ECcallee ECcaller PDA ECcaller PDA

NOVA Microhypervisor NOVA Microhypervisor ipc reply (MTD); ipc call (P, MTD);

Protection Domains, Execution+Scheduling Contexts, Portals Semaphores for Synchronization and Interrupt Delivery Hypercall interface uses capabilities for all operations Synchronous IPC with timeslice donation ⇒ priority inheritance MTD defines number of words to copy UTCBcaller ⇄ UTCBcallee

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 6 / 17

NOVA: Handling VM Exits / Exceptions

UTCB UTCB SC SC VMCB VMCB P VMM EChandler VMM EChandler VM

NOVA Microhypervisor NOVA Microhypervisor

VM vCPU vCPU

call (P, MTDARCH); VMRESUME/ERET VMEXIT/EXCEPTION ipc reply (MTDARCH);

vCPU state saved to / restored from VMCB Microhypervisor synthesizes IPC call on behalf of vCPU Destination portal selected based on type of event IPC reply from VMM provides updated architectural state MTDARCH defines state to copy VMCB ⇄ UTCBhandler

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 7 / 17

CPU Virtualization: Architectural State

ARMv8-A Message Transfer Descriptor (MTDARCH)

GIC TMR EL2 HCR EL2 HPFAR EL2 ESR FAR EL2 ELR SPSR EL2 IDR EL1 SCTLR EL1 VBAR EL1 MAIR EL1 TCR EL1 TTBR EL1 AFSR EL1 ESR FAR EL1 ELR SPSR EL1 IDR EL1 SP A32 DACR IFSR A32 SPSR EL0 IDR EL0 SP FPR GPR POISON

1 2 3 4 7 8 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 30 31

x86 Message Transfer Descriptor (MTDARCH)

FPU TSC STA INJ CTRL QUAL LBR DR CR IDTR GDTR LDTR TR CS/SS FS/GS DS/ES FLAGS IP GPR8−15 GPR4−7 GPR0−3

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 31

For every bit set to 1, the corresponding architectural state is transmitted from the vCPU to the VMM handler or vice versa.

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 8 / 17

ARM: FPU Virtualization

Hypervisor context-switches FPU state (32x128bit SIMD registers) between ECs lazily FPU Access Disabling/Enabling

Switch away from FPU owner ⇒ disable FPU Switch back to FPU owner ⇒ enable FPU

FPU switch moved out of critical IPC path using hazard tricks CPU Hazard Bit EC Hazard Bit FPU is disabled (0) EC is not FPU owner (0) FPU is enabled (1) EC is FPU owner (1) Slow path taken only if CPU Hazard ⊕ EC Hazard is 1 FPU use must be explicitly declared during EC creation

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 9 / 17

ARM: Interrupt Virtualization

Redistributor Redistributor Redistributor Redistributor

GICD GICR

Hyp IF vCPU IF Processing Element

SPI PPI EOI vINTID

GICH GICV

VMM Guest VM

program inject

Distributor

vIRQ EOI pIRQ

pCPU IF

pINTID vINTID→pINTID deactivate

GICC

Unified interrupt injection interface for GICv2/GICv3 Hypercall assign int for configuring and routing SPIs

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 10 / 17

ARM: Timer Virtualization

System Counter CNTVOFF CNTVCT CNTPCT

System Time Distribution

–

managed by Microhypervisor

Physical Timer Real system counter Can be trapped

⇒ Trap & emulate timer

pTimer interrupt emulated with semaphore timeouts

⇒ Asynchronous delivery

Virtual Timer System counter - offset Cannot be trapped

⇒ Context-switch timer

vTimer interrupt temporarily belongs to current VM

⇒ Synchronous via Portal

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 11 / 17

ARM: System MMU

CTX CTX PTAB PTAB

VMID

SMG SMG

0x860 0x404 0x405

TLB

VMID:VA ⇒ PA:PERM Miss Fill Miss Fill VA PA VMID Stream Mapping Groups Translation Contexts Translation Lookaside Buffer TTBR TTBR SID

SMG Translation Configuration

VMID:VA ⇒ PA:PERM VMID:VA ⇒ PA:PERM DMA Page Tables

System MMU protects against rogue DMA Limited number of stream mapping groups and translation contexts managed by partition manager Hypercall assign dev for configuring SID/SMG/CTX and binding a device to a protection domain

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 12 / 17

Currently Supported ARM Platforms

Avnet Xilinx Ultra 96 4x Cortex-A53 GICv2 SMMUv2 NXP i.MX 8MQuad 4x Cortex-A53 GICv3 Renesas R-Car M3/H3 4x Cortex-A53 4x Cortex-A57 GICv2 Raspberry Pi 4B 4x Cortex-A72 GICv2 QEMU Virt Platform Cortex-A GICv2/GICv3

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 13 / 17

Current Status: Demo

Core Core Display SD USB WiFi

Guest Host

Core Core UART

VM VM VM

vCPU vCPU vCPU vCPU

VMM VMM VMM UART Multiplexer Virtual Ethernet Switch Partition Manager NOVA Microhypervisor UART Driver

vETH vETH vETH vUART vUART vUART vGIC vGIC vGIC

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 14 / 17

Roadmap

Architecture Unification

Merge significant portions of the x86 and ARMv8 source code

{src/x86 64, src/aarch64} ⇒ src {inc/x86 64, inc/aarch64} ⇒ inc Support for newer ARM features (ARMv8.1 – ARMv8.6) Additional NOVA functionality

Relocatable microhypervisor binary VM introspection support Improved kernel resource management Useful external features and bug fixes

Performance Optimizations Formal Verification of the NOVA microhypervisor

... and of components running on top of it

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 15 / 17

Formal Verification

/* * \arg{v1} "x" (Vint v1) * \arg{v2} "y" (Vint v2) * \pre empSP * \post{}[Vint (trim 32 (v1+v2))] empSP */ auto add_func (uint32 x, uint32 y) { return x + y; }

foo cpp.v foo cpp spec.v coqc cpp2v foo.cpp

Source Code clang Plugin Code Specifications AST Coq Representation

foo cpp proof.vo Proof Automation C++ Semantics

Machine-Checked Proof

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 16 / 17

Source code available under GPLv2 license at:

https://github.com/bedrocksystems/NOVA https://github.com/udosteinberg/NOVA

Checkout the ”arm” branch. Further information (papers, links) at:

https://bedrocksystems.com http://hypervisor.org

Udo Steinberg (BedRock Systems, Inc.) NOVA Microhypervisor on ARMv8-A February 2, 2020 17 / 17