IO virtualization Michael Kagan Mellanox Technologies
IO Virtualization • Mission – non-stop services to consumers � Flexibility – assign IO resources to consumer as needed � Agility – assignment of IO resources to consumer when needed � Isolation – deliver IO services to consumer without interference � Fault-tolerance – deliver IO services despite HW failures Reliability, flexibility & efficiency • Mean – de-couple service delivery from the Hardware � Overcome Hardware boundary constrains � Resource assignment is a matter of mouse-click � Consumer-transparent hardware replacement De-couple consumers’ execution environment from underlying Hardware 2
Simple (monolithic) IO model cAPI – soc, MPI etc app app mAPI – PCI cfg, pwr etc cAPI user kernel app app cAPI mngr mAPI mng service Driver IO device • Kernel-resident mngr owns HW Initialization, configuration � Management interface (API) � • OS is part of IO subsystem Isolation, protection, QoS � • Constrained performance Kernel call, data copy etc. � • Low QoS granularity Agility? Flexibility? Fault tolerance? � • • Dumb IO device 3
Advanced IO model cAPI – soc, MPI etc app mAPI – PCI cfg, pwr etc app cAPI user app app app app kernel cAPI cAPI user user app kernel kernel app app app app app cAPI cAPI cAPI mngr mAPI mngr mngr mAPI mAPI mng mng service service mng Driver Driver Driver IO device IO device IO device • Kernel-resident mngr owns HW Initialization, configuration � IO delivery policy provisioning (QoS, protection) � • IO delivered to consumers via IO channels – OS is not a part of IO subsystem Abstract interface to post IO operation requests � HW-enforced cross-channel protection and isolation � Unlimited scalability � • High performance Kernel bypass, zero copy etc. � • Fine QoS granularity Agility? Flexibility? Fault tolerance? � • • Smart IO device 4
IO in virtual machines – take 1 VM VM app app app app cAPI cAPI user user app app kernel kernel app app cAPI cAPI app app user user app app kernel kernel cAPI cAPI app app app app cAPI cAPI mngr mngr mngr mngr mAPI mAPI mAPI mAPI mngr mngr mng mng mng mng mAPI mAPI Driver Driver Driver Driver mng mng service service Driver Driver Hypervisor mngr IO device IO device mAPI mng service Driver • Hypervisor-resident mngr owns HW IO delivered to guest via IO device emulation � � “mng” part is faked; “service” path used for IO operation IO device • Improved agility, flexibility and fault tolerance • Additional SW layer on the IO service path Cross-domain isolation and protection in Hypervisor � • Virtualization cost – Guest OS and Hypervisor are part of IO subsystem Lower IO performance � Higher CPU overhead � Limited scalability – architecture tied to HW � Low QoS granularity � • Dumb IO device 5
IO in virtual machines – take 2 cAPI – soc, MPI etc VM VM mAPI – PCI cfg, pwr etc app app app app cAPI cAPI user user app app app app app app app app cAPI cAPI kernel kernel cAPI cAPI user user user user kernel kernel kernel kernel app app app app app app app app app app app app cAPI cAPI cAPI cAPI cAPI cAPI mngr mngr mAPI mAPI mngr mngr mngr mngr mngr mngr mAPI mAPI mng mng mAPI mAPI mAPI mAPI mng mng mng mng Driver Driver mng mng service service Driver Driver Driver Driver Driver Driver Hypervisor IO device IO device IO device IO device mngr mAPI mng • Hypervisor-resident mngr owns HW Driver IO device(s) configuration, IO policy provisioning � • IO delivered to consumers via IO channels IO delivery is an integral part of the platform IO device � • Uncompromised IO services delivery Unlimited scalability (“IO channel for every socket”) � HW-enforced isolation and protection � Direct IO channel access (pass-through) � � Hypervisor offload on data path, zero copy etc. Fine QoS granularity � Allows on-demand resource allocation � • Standard IO interface to enable transparent servers’ management • Smart IO device 6
Open Fabrics Channel IO Interface • Industry-standard channel IO interface � Included in Linux kernel � Endorsed by Microsoft • Supports multiple IO devices � InfiniBand HCAs � Ethernet NICs • Standard IO provisioning interface � Assign QoS for an IO channel or group of channels • Efficient and versatile IO delivery � Enables direct access to IO HW – full Hypervisor offload � Raw datagram IO channels � Transport-offloaded IO channels � RDMA IO channels • Unlimited scalability � Unlimited number of IO channels 7
Channel IO in virtual machines VM VM VM VM VM VM app app app app app app app app app app app app core (framework) core (framework) core (framework) core (framework) core (framework) core (framework) WR formatter WR formatter WR formatter WR formatter WR formatter WR formatter user user user user user user kernel kernel kernel kernel kernel kernel app app app app app app app app app app app app core (framework) core (framework) core (framework) core (framework) core (framework) core (framework) WR formatter WR formatter WR formatter WR formatter WR formatter WR formatter Hypervisor Hypervisor Hypervisor mngr mngr mngr mng mng mng service WR formatter Driver Driver Driver IO device IO device IO device Pass- -through through Hypervisor acceleration Simple IO device Pass Hypervisor acceleration Simple IO device • Hardware-independent consumer-transparent guest migration � Support direct access to HW 8
Summary • Channel IO architecture implements IO virtualization mission � Flexibility, agility, isolation, management at high performance • Channel IO production HW available � Multiple generations, multiple vendors • Channel IO API standard materializing � Native support by standard OSes � Supported on proprietary OSes • Channel IO serves IO consolidation trend � e.g. FC over Ethernet • Channel IO enables on-demand resource allocation � IO channels, IO resources, memory 9
Recommend
More recommend
