Octavian Ciuhandu Peter Pouliot Cloudbase Solutions Microsoft + - - PowerPoint PPT Presentation

Octavian Ciuhandu Peter Pouliot

Cloudbase Solutions Microsoft

+ Built and maintained by a small team of highly skilled engineers + Undercloud of KVM on Centos providing virtualized Ubuntu Devstack Controllers + Two Physical Hyper-V nodes per Controller dynamically assembled + All layers automated through native

• perating system tools, puppet, or shell

scripting. + Executes a Tempest run for every upstream commit + Automation and Scripts: + https://github.com/openstack-hyper-v + https://github.com/cloudbase

2012 + Began recycling hardware originally used for the first Hyper-V/ OpenStack cloud deployment from 2011. + Combination of HP Dell servers, blades and storage. + Acquired 3 Racks located in corporate LAB environment within the Microsoft NERD Center + Acquire dedicated physical network infrastructure including internet connectivity allowing for complete isolation of infrastructure and full control of all network infrastructure dedicated to OpenStack CI efforts. 2013 + OpenStack Hyper-V CI begins operation Mid Year + First official votes “IceHouse” + 6 additional racks located 5 blocks from original site acquired for expansion of additional compute capacity. + Metropolitian Area Network deployed between NERD site and nearby MS colocation facility using previously dark fiber establishing a 1g connection between the sites. 120 nodoes of HP 1u nodes recycled from new facility and added as Hyper-V/KVM compute capacity.

2014 - 2015 + Additional server and networking resources acquired allowing for expansion in the remaining 4 racks as well increase capacity to 17 Racks of server equipment. + Network resources were exhausted preventing use of all equipment. + Microsoft purchases new network infrastructure allowing us to upgrade site to site communication to 10G backplane add per rack physical network segmentation and consumption of compute resources + Some newer hardware acquired through recycling allowing for replacement of some of the older infrastructure. + Mid 2015 begin acquiring 10G HBA interfacing allowing for Tesing

• f OVS on Windows and 10G to the Host

2016 + Acquire quanta high density compute and storage, as well as more 10/40G network infrastructure. + Begin rebuild of CI in parallel to the operation of existing CI. + Network backplane upgraded to 40G, and 10G to the host for CI Operations. + Undercloud KVM compute operating on all SSD, Hyper-V compute upgraded to single disk 4TB high capacity storage.

+ Needed to be able to automate multiple linux distributions, Windows Flavors, Hardware Models, Supporting applications, processes. + Because of lack of Nesting for Windows Physical compute was required. + Due to old hardware needed to be able to rebuild supporting components immediately from scratch if hardware failed. + Workload platforms = Centos (KVM physical)+Ubuntu(Devstack vm)+ Windows 2012 (Two Physical Hyper-V).

+ Began automating deployment and configuration of Hyper-V using puppet 2.7+ on windows server 2012. + Created a basic framework for deploying OpenStack nova using public python binaries, with locally served files on Windows, vswitch and NTP configuration as well as installation of supporting binaries. + After initial configuration moved to building supporting linux deployment automation + Created dynamic PXE tool kit by which any linux distribution can be automated using simple ERB templates across versions

+ Automated IPAM infrastructure to allow for bulk loading MAC address information for hosts via yaml and dynamically create network segments, static leases and names for all host network interfaces, and ip subnet ranges. + Created a Jenkins infrastructure to allow for automation of Operational tasks within the infrastructure, such switch/pdu manipulation, hardware/device firmware upgrades. + Added formatting/sanity/deployment testing of existing automation to help with maintaining puppet automation using containers on Linux/Windows.

OpenStack Community 44% Cloudbase 14% EMC 9% VMware 6% Big Switch 6% Mellanox 5% Nuage Networks 5% Freescale 4% Cisco 4% Dell 3%

+ 11 CI’s: nova, neutron, compute-hyperv, networking-hyperv, cinder (iSCSI, SMB- Windows, SMB Linux), manilla, os-win, ovs + We currently maintain a production OpenStack CI facility with 700+ servers

http://stackalytics.com/?release=all&project_type=all&metric=ci

As of 27 Jan. 2016

Icehouse Juno Kilo Liberty Mitaka Newton Ocata Pike Community: 3381 5446 6049 5701 6296 6988 3200 1892 Cloudbase: 810 1498 1599 2050 3205 3537 1236 1145 Percentage: 23.96% 27.51% 26.43% 35.96% 50.91% 50.62% 38.63% 60.52%

+ Nova + Neutron Networking-HyperV Agent OVS (on Windows) – silent runs to validate the CI + Cinder iSCSI SMB3 Windows + Compute-HyperV + Networking-HyperV + Manila + OS-Win + OS-Brick + Oslo projects – Unit tests – silent runs to validate the CI + Cloudbase-init

+ Upgrade to:

• quanta servers
• 10 Gbps networking
• 40 Gbps backplane
• Dedicated dataplane, completely isolated

+ Redesigned the CI functionality:

• MaaS for bare-metal deployments
• Juju for orchestrating the application

deployments + Separate CI accounts and distributed zuul + Part of CI on upgraded undercloud-

• vercloud model while others are direct on

bare-metal deployments

MaaS + deployed in HA mode using 2 bare-metal servers + HAProxy with floating IP, used for both MaaS services and PosgreSQL backend + 3 KVM nodes to provide VMs for MaaS for services that do not require a lot of resources Juju + deployed in HA mode using 3 VMs + Separate users for different teams managing CIs + Separate models for different infrastructure services and Cis Jumpbox + VM with separate users and each user has access granted only to the Juju model he requires

User Admin + controller + default + infra + undercloud + Zuul User OVS + neutron-ovs + Ovs User ci-deployer + ci-deployer User cloudbaseinit + cloudbaseinit + cloudbaseinit-dev

+ Extended existing zuul charm from charm-store + One zuul instance as gearman server:

• deployed on bare-metal

+ Two zuul instances as zuul mergers as lxd containers + One zuul server instance per CI as lxd containers + https://github.com/cloudbase/zuul-charm

Charms used to deploy + Jenkins + Logs server + Ubuntu repository cache + Pypi cache using devpi in HA mode on 2 nodes + Windows Server 2016 instances + Jenkins slaves for each Windows Server 2016 instance

+ Single openstack controller + Controller services deployed in lxd containers + 25 kvm compute nodes + Updated the existing charms to support linux-bridge with flat networking for the undercloud + MySQL DB for managing the VLAN ranges allocation for all overcloud tests

+ Active for most CIs + Workflow:

• VLAN range is reserved
• VM is launched for devstack controller
• 2 bare-metal Hyper-V nodes are

allocated Cinder also uses a third windows node for cinder volume service

• Parallel build for all 3 nodes

+ Image is periodically updated with latest github repos changes + Apt and Pypi caches are used + Hyper-V nodes are re-used:

• Python installation is fully removed
• All build, logs, VMs and other possible

artifacts are cleaned up

• OpenStack Windows services are set to

manual, created only if not already present

+ Full bare-metal deployments for each run + Active-directory node re-used + No nodes are re-used + CI-Deployer – adds resiliency and node removal features to the charms deployment workflow + Custom charms for maximum flexibility:

• Hyper-V CI Charm
• Devstack CI Charm

+ Nodepool – ensures that a minimum number of nodes have the OS already deployed and are ready to be consumed by the charms

+ Polling the github repo, will implement a github hook for job triggering + Contains 2 steps:

• Unit testing

Builds project and installer Runs unit tests Uploads unit test logs If successful uploads installer and triggers integration tests

• Integration tests

Deploys a controller and 2 Windows Server 2016 compute nodes using the OVS installer Runs full tempest suite tests Uploads integration test logs

• Currently Unit tests report upstream,

integration tests are run silently for CI reliability testing

+ Zabbix:

• OS Level
• OpenStack services
• Hyper-V nodes
• Networking

+ Racktables for CI hardware inventory and status, integrated with Zabbix + In process of deployment using charms in dedicated model

+ Support tools:

• https://github.com/cloudbase/ci-deployer

+ Charms:

• https://github.com/cloudbase/zuul-charm
• https://github.com/cloudbase/active-directory-charm
• https://github.com/cloudbase/devstack-charm
• https://github.com/cloudbase/hyperv-charm

+ CI code:

• https://github.com/cloudbase/?utf8=%E2%9C%93&q=-ci
• https://github.com/cloudbase/common-ci/pull/11/files

www.cloudbase.it

`