vNUMA in Xen Wei Liu & Elena Ufimtseva Chicago August 18, 2014 - PowerPoint PPT Presentation

vNUMA in Xen Wei Liu & Elena Ufimtseva Chicago – August 18, 2014

Agenda Motives History and status Design Problems Benchmark Future Work Agenda ◮ Motives ◮ History and status ◮ Design ◮ Problems ◮ Preliminary benchmark results ◮ Future work Chicago – August 18, 2014 vNUMA in Xen 2 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Motives ◮ Cross NUMA node memory access is expensive ◮ Need to avoid cross node memory access ◮ Xen is NUMA aware ◮ NUMA aware scheduling ◮ NUMA aware guest memory placement ◮ Operating system like Linux is NUMA aware ◮ NUMA aware scheduling ◮ NUMA aware memory allocation / migration ◮ The missing bits ◮ Memory layout information ◮ CPU topology Chicago – August 18, 2014 vNUMA in Xen 3 / 19

Agenda Motives History and status Design Problems Benchmark Future Work History and status ◮ PV vNUMA presented in Xen Summit 2010 by Dulloor Rao http://slidesha.re/1AXsFbu ◮ HVM vNUMA patches posted by Andre Przywara circa 2010 ◮ Elena Ufimtseva has been working on upstreamable PV vNUMA since 2013 Chicago – August 18, 2014 vNUMA in Xen 4 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Design: PV and PVH ◮ Toolstack puts enlightenment information in hypervisor ◮ Guest memory allocation in accordance with enlightenment information ◮ Guest retrieves enlightenment information via hypercall during boot up Chicago – August 18, 2014 vNUMA in Xen 5 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Design: PV and PVH Enlightenment information structure struct vnuma_info { nr_vnodes; vdistance[nr_vnodes * nr_vnodes]; vcpu_to_vnode[nr_vnodes]; vnode_to_pnode[nr_vnodes]; vmemrange[nr_vnodes]; } Chicago – August 18, 2014 vNUMA in Xen 6 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Design: HVM ◮ Toolstack puts enlightenment information in hypervisor ◮ Toolstack arranges ACPI tables ◮ Guest memory allocation in accordance with enlightenment information ◮ Guest retrieves layout information via ACPI tables during boot up Chicago – August 18, 2014 vNUMA in Xen 7 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Problems: vNUMA and other features PV PVH HVM Ballooning Y Y* N PoD N/A ? N Chicago – August 18, 2014 vNUMA in Xen 8 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Problems: CPU topology Chicago – August 18, 2014 vNUMA in Xen 9 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Benchmark ◮ Host ◮ 2 sockets, 12 PCPUs, HT disabled ◮ 36GB RAM, 2 NUMA nodes ◮ NUMA balancing enabled ◮ Guest ◮ 12 VCPUs ◮ 16GB RAM, 2 virtual NUMA nodes ◮ vnodes mapped to different pnodes, vcpu pinned to pnode ◮ NUMA balancing enabled ◮ Benchmarks to run ◮ Autonuma ◮ SPECJBB ◮ STREAM Chicago – August 18, 2014 vNUMA in Xen 10 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Benchmark: Autonuma Chicago – August 18, 2014 vNUMA in Xen 11 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Benchmark: Autonuma Chicago – August 18, 2014 vNUMA in Xen 12 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Benchmark: Autonuma Chicago – August 18, 2014 vNUMA in Xen 13 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Benchmark: SPECJBB Chicago – August 18, 2014 vNUMA in Xen 14 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Benchmark: SPECJBB Chicago – August 18, 2014 vNUMA in Xen 15 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Benchmark: STREAM Chicago – August 18, 2014 vNUMA in Xen 16 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Benchmark: Conclusion ◮ vNUMA improves performance for PV guest ◮ vNUMA has weird result in SPECJBB for PVH guest, but other two benchmarks have good results Chicago – August 18, 2014 vNUMA in Xen 17 / 19

Agenda Motives History and status Design Problems Benchmark Future Work Future Work ◮ basic vNUMA support for all guest types ◮ Dom0 vNUMA ◮ address vNUMA compatibility issues with PoD and ballooning for HVM guest ◮ address performance issue for PVH Chicago – August 18, 2014 vNUMA in Xen 18 / 19

Thank you! Chicago – August 18, 2014 vNUMA in Xen 19 / 19