SLIDE 1
1
Light Weight Virtualization with QEMU/KVM
Anthony Xu <anthony.xu@intel.com> Chao Peng <chao.p.peng@intel.com> Haozhong Zhang <haozhong.zhang@intel.com>
2
Light Weight Virtualization with QEMU/KVM Anthony Xu - - PowerPoint PPT Presentation
Light Weight Virtualization with QEMU/KVM Anthony Xu - - PowerPoint PPT Presentation
Light Weight Virtualization with QEMU/KVM Anthony Xu <anthony.xu@intel.com> Chao Peng <chao.p.peng@intel.com> Haozhong Zhang <haozhong.zhang@intel.com> 1 Outline Background Optimization for VM Launch Time Memory Footprint
SLIDE 2
SLIDE 3
3
Container vs. Traditional VM
Container Traditional VM
Launch Time
✓
Resource Efficiency
✓
Density
✓
Deployment
✓
Security
✓
SLIDE 4
4
Light Weight VM
Container Traditional VM
Light Weight VM Launch Time
✓ ✓
Resource Efficiency
✓ ✓
Density
✓ ✓
Deployment
✓ ✓
Security
✓ ✓
SLIDE 5
5
QEMU Lite
- Target for container-like usage scenarios
- Optimized for the launch time
- 1276 ms 335 ms
- Smaller memory footprint
- More optimization on the way
- Integrated with Intel Clear Container
- Docker-like engine
SLIDE 6
6
Outline
Background Optimization for VM Launch Time Memory Footprint Integration with Intel Clear Container Upstream Status & TODO
SLIDE 7
7
Baseline
- Host
- Intel Xeon E5-2698 v3 @ 2.30 GHz / [Memory]
- Guest (QEMU configurations)
- Q35 / 6 VCPUs / 1G RAM / virtio-blk / -kernel / no network
283 (22%) 43 (3%) 468 (37%) 211 (17%) 271 (21%)
200 400 600 800 1000 1200 1400
Baseline
QEMU SeaBIOS OptionROM (linuxboot) Linux Kernel Userspace (systemd)
Total: 1276 ms
SLIDE 8
8
Tweak QEMU & Kernel Configurations
- Disable QEMU features at compile time
- https://github.com/chao-p/qemu-lite-tools/blob/master/qemu-config.sh
- Tune guest kernel boot parameters
- rcupdate.rcu_expedited=1
- pci=lastbus=0
- …
SLIDE 9
9
QEMU Optimizations
- Cache KVM_GET_SUPPORTED_CPUID
- 5922 duplicated calls during QEMU initialization
- QEMU commit 494e95e “target-i386: kvm: cache KVM_GET_SUPPORTED_CPUID data”
- QEMU: 272 ms 90 ms
- Parallelize VCPU initialization
- QEMU: 90 ms 54 ms
SLIDE 10
10
New Machine Type pc-lite
- Minimum Devices
- APIC
- PCI Host
- Virtio Console
- NVDIMM
- …
- SeaBIOS: 43 ms 29 ms
- Guest kernel: 151 ms 142 ms
SLIDE 11
11
And more …
- Use vNVDIMM device as guest disk drive
- DAX
- Guest kernel: 142 ms 124 ms
- Guest userspace: 240 ms 195 ms
- Remove guest BIOS completely
- Patch guest ACPI in QEMU
- Load guest ELF kernel in QEMU
- SeaBIOS: 29 ms 0 ms
- OptionROM: 468 ms 0 ms
- Guest kernel: 124 ms 72 ms
SLIDE 12
12
Final Result
283 68 43 468 211 72 271 195
200 400 600 800 1000 1200 1400
Baseline Optimized QEMU SeaBIOS OptionROM (linuxboot) Linux Kernel Userspace (systemd)
Total: reduced by 74% (1276 ms335 ms)
SLIDE 13
13
283 232 232 90 54 54 54 68 43 43 43 43 43 29 29 468 468 468 468 468 468 468 211 211 155 155 155 142 124 72 271 271 240 240 240 240 195 195 200 400 600 800 1000 1200 1400 Baseline QEMU Config Kernel Parameter Cache CPUID Parallelize VCPU Init Q35 Opt vNVDIMM Remove BIOS QEMU SeaBIOS OptionROM (linuxboot) Linux Kernel Userspace (systemd)
SLIDE 14
14
Outline
Background Optimization for VM Launch Time Memory Footprint Integration with Intel Clear Container Upstream Status
SLIDE 15
15
Memory Consumption
- 1 x VM: 6 vCPU, 1G RAM
- Read /dev/$pid/status
117872 134348 (+14%) 164968 (+40%) 20000 40000 60000 80000 100000 120000 140000 160000 180000 kvmtool QEMU Lite QEMU Lite
VmRSS
SLIDE 16
16
Potential Optimizations
- Still ongoing
- Remove unnecessary devices
- Reduce memory consumption by QEMU itself
- COW share ROM
- e.g. ROM for guest ELF kernel
- Reduce duplicated host/guest copies if (almost) no modifications
- Lazy creation of dirty memory bitmap
- Defer to its first usage (migration)
- KSM
- VMFork/Clone
- Tools to fine-grained profile/trace the memory consumption
SLIDE 17
17
Outline
Background Optimization for VM Launch Time Memory Footprint Integration with Intel Clear Container Upstream Status
SLIDE 18
18
Integration with Intel Clear Container
- QEMU Lite in Intel Clear Container v2.0
- Replace kvmtool used in Intel Clear Container v1.0
- Benefit from Docker/rkt compatible interface
- Support docker images
- Better deployment
SLIDE 19
19
Outline
Background Optimization for VM Launch Time Memory Footprint Integration with Intel Clear Container Upstream Status
SLIDE 20
20
PC-lite or Q35
- [RFC 0/9] Introduce light weight PC platform pc-lite
- https://lists.nongnu.org/archive/html/qemu-devel/2016-06/msg04842.html
- All optimizations currently go into a new machine type pc-lite.
- Less pollutions to other machine types
- Easier to optimize
- Extra cost for maintenance
- Optimize Q35?
- Make the work more widely useful
- Add ability to disable (more) features and devices
SLIDE 21
21
No Firmware or Light-weight Firmware
- Completely remove guest firmware currently.
- Significant speedup
- Specific to Linux
- Light-weight firmware
- e.g. qboot
- Support more guest operating systems
- More optimizations maybe needed
- Keep no firmware as an option
- For usage scenarios requiring extreme launch time
SLIDE 22
22
Outline
Background Optimization for VM Launch Time Memory Footprint Integration with Intel Clear Container Upstream Status
SLIDE 23
23
Status
- [RFC 0/9] Introduce light weight PC platform pc-lite
- https://lists.nongnu.org/archive/html/qemu-devel/2016-06/msg04842.html
- Github repo
- Patches: https://github.com/chao-p/qemu/tree/pc-lite-v1
- Tools/Guides: https://github.com/chao-p/qemu-lite-tools
- Integrated in Intel Clear Container v2
- Ongoing
- More optimizations for launch time
- Optimizations for memory footprint
- Upstream
SLIDE 24