Scheduling in Xen: Past, Present and Future Dario Faggioli - PowerPoint PPT Presentation

Scheduling in Xen: Past, Present and Future Dario Faggioli dario.faggioli@citrix.com Seattle, WA – 18th of August, 2015

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Introduction Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 2 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Welcome ◮ Hello, my name is Dario ◮ Working on Xen (from within Citrix) since 2011 ◮ Mostly hypervisor stuff, but also toolstack: NUMA, cpupools, scheduling Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 3 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Scheduling & Scheduling in Xen Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 4 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Scheduling in General Is scheduling really important, e.g., in OSes? Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 5 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Scheduling in General Is scheduling really important, e.g., in OSes? Well... Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 5 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Scheduling in General Is scheduling really important, e.g., in OSes? Well... ◮ most of the time, there isn’t even any CPU overbooking ◮ when there’s overbooking, not everything is equally important ◮ I/O is more important! Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 5 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Scheduling in Virtualization Is scheduling really important on a virtualized host? ◮ There is pretty much always CPU overbooking ◮ All activities (i.e., all VMs) are (potentially) equally important Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 6 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Scheduling in Virtualization Is scheduling really important on a virtualized host? ◮ There is pretty much always CPU overbooking ◮ All activities (i.e., all VMs) are (potentially) equally important Scheduler hacker: ≪ Hey, this sounds pretty cool... ≫ Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 6 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Scheduling in Virtualization Is scheduling really important on a virtualized host? ◮ There is pretty much always CPU overbooking ◮ All activities (i.e., all VMs) are (potentially) equally important Scheduler hacker: ≪ Hey, this sounds pretty cool... ≫ Well, it’s not! I/O is still more important! Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 6 / 39

Control Domain Control Domain NetBSD or Linux NetBSD or Linux device model device model (qemu) (qemu) toolstack toolstack Paravirtualized Paravirtualized Driver Domain Fully (PV) (PV) Virtualized Domain: Domain: (HVM) NetBSD or Linux NetBSD or Linux Domain: Hardware netback netfront Hardware netback netfront netback Windows, Drivers blkback blkback blkfront Drivers blkback blkfront FreeBSD... Xen Hypervisor Xen Hypervisor Hardware Hardware I/O Devices CPU Memory I/O Devices CPU Memory Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Wait a Second... And we, on Xen, are even more special : Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 7 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Wait a Second... And we, on Xen, are even more special : ◮ How does Xen hadles I/O: Control Domain Control Domain NetBSD or Linux NetBSD or Linux device model device model (qemu) (qemu) toolstack toolstack Paravirtualized Paravirtualized Driver Domain Fully (PV) (PV) Virtualized Domain: Domain: (HVM) NetBSD or Linux NetBSD or Linux Domain: Hardware netback netfront Hardware netback netfront netback Windows, Drivers blkback blkback blkfront Drivers blkback blkfront FreeBSD... Xen Hypervisor Xen Hypervisor Hardware Hardware I/O Devices CPU Memory I/O Devices CPU Memory Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 7 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Wait a Second... (cont. II) So, I/O, turns out you actually need scheduling, eh? Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 8 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions History of Xen Scheduling (Past) Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 9 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions I Mean... Honestly... Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 10 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions I Mean... Honestly... WHO CARES ?!?! Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 10 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions I Mean... Honestly... WHO CARES ?!?! No, seriously: Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 10 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions I Mean... Honestly... WHO CARES ?!?! No, seriously: ◮ some info at http://wiki.xen.org/wiki/Xen Project Schedulers Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 10 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions I Mean... Honestly... WHO CARES ?!?! No, seriously: ◮ some info at http://wiki.xen.org/wiki/Xen Project Schedulers ◮ we just killed SEDF in 4.6 (yay!!) check RTDS for something similar (but better!) Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 10 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions I Mean... Honestly... WHO CARES ?!?! No, seriously: ◮ some info at http://wiki.xen.org/wiki/Xen Project Schedulers ◮ we just killed SEDF in 4.6 (yay!!) check RTDS for something similar (but better!) ◮ that’s it Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 10 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Xen’s Scheduler Features (Preset) Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 11 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Hard and Soft Affinity ◮ pinning: you can run there and only there! #xl vcpu-pin vm1 0 2 ◮ hard affinity: you can’t run outside of that spot # xl vcpu-pin vm1 all 8-12 ◮ soft affinity: you can’t run outside of that spot and, preferably , you should ru there # xl vcpu-pin vm1 all - 10,11 Same achieved with cpus= and cpus soft= in config file. cpus= or cpus soft= in config file control where memory is allocated Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 12 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions Hard and Soft Affinity (cont.) Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 13 / 39

Introduction Sched. in Virt. History of (Xen) Scheduling Scheduler Features Workin’ On Conclusions What about NUMA Automatic placement policy in libxl (since Xen 4.2) ◮ acts at domain creation time, important for memory allocation! ◮ easy to tweak (at libxl build time, for now) heuristics: ◮ use the smallest possible set of nodes (ideally, just one) ◮ use the (set of) node(s) with fewer vCPUs bound to it ([will] consider both hard and soft affinity) ◮ use the (set of) node(s) with the most free RAM (mimics the “ worst fit ”algorithm) Seattle, WA – 18th of August, 2015 Scheduling in Xen: Past, Present and Future 14 / 39