Ironic Grenade Blowing up our upgrades Vlad Drok (vdrok) - Mirantis Vasyl Saienko (vsaienko) - Mirantis John Villalovos (jlvillal) - Intel Corporation mirantis.com intel.com
How we made Grenade work in Ironic ● What is Grenade and Ironic ● Grenade as a plug-in ● Grenade phases ● Networking of Ironic in Devstack and Grenade ● Grenade testing difficulties for Ironic ● Future work
What is Grenade? Why do we want to use it? ● Grenade is a test harness to exercise the OpenStack * upgrade process between releases. ● It allows us to test a “Cold Upgrade” and is a step in our end-goal of supporting “Rolling Upgrades.” ● Grenade docs: http://docs.openstack.org/developer/grenade/ ● Ironic is a service for bare-metal provisioning. ● Ironic + Grenade allows us to verify that cold-upgrade works and new patches don’t break cold-upgrades.
Grenade as a plug-in To use Grenade as a plug-in a few things are required: ● Setup the openstack/ironic/devstack/upgrade/settings file ● Create a ‘grenade’ job in openstack-infra/project-config ● Update openstack-infra/project-config ○ The GRENADE_PLUGINRC environment variable must be setup ○ export GRENADE_PLUGINRC="enable_grenade_plugin ironic https://git.openstack.org/openstack/ironic"
Grenade Resource Phases Base devstack setup of current stable branch, start services, and smoke test early_create phase verify_noapi destroy phase phase create phase Upgrade to current master with proposed patch and verify pre-upgrade Final smoke test start services phase verify State is saved post-upgrade and services phase shutdown
Networking of Ironic and Devstack ● L3 (Layer 3) access from the ironic-conductor service to the Ironic Python Agent (IPA) ramdisk during provisioning ● L3 access from the IPA ramdisk to the ironic-api service Ironic operates with bare-metal servers, but in the gate we use VMs to emulate them. Due to this network setup in the gate it looks very complex, but it is only first look :)
Phase: base stack.sh node-0 node-0 node-N node-N Ironic nodes (KVM VMs) 1. Create brbm br-node-0 br-node-0 br-node-N br-node-N 2. Create VMs (ironic nodes) 3. Add brbm <----> br-int Baremetal bridge: brbm 4. Connect nodes to private net ovs-node-0 ovs-node-0 ovs-node-N ovs-node-N brbm-tap brbm-tap (set tag: 10 on ovs-tap) Neutron Integration bridge: br-int Port: tap Port: tap Port: ovs-tap Port: ovs-tap tag: 10 5. Run base smoke test tag: 10 tag: 10 DHCP Router private Namespace Namespace tap tap 10.1.0.0/20 qdhcp qrouter
Phase: resources.sh early_create node-0 node-0 node-0 node-N node-N node-N Ironic nodes (KVM VMs) br-node-0 br-node-0 br-node-0 br-node-N br-node-N br-node-N 1. Create network ironic_grenade Baremetal bridge: brbm 2. Connect nodes to ironic_grenade ovs-node-0 ovs-node-0 ovs-node-0 ovs-node-N ovs-node-N ovs-node-N brbm-tap brbm-tap brbm-tap net (set tag: 20 on ovs-tap) Neutron Integration bridge: br-int Port: tap Port: tap Port: ovs-tap Port: ovs-tap Port: ovs-tap Port: tap Port: tap tag: 10 tag: 10 tag: 20 tag: 10 tag: 10 tag: 20 tag: 20 DHCP Router private Namespace Namespace tap tap 10.1.0.0/20 qdhcp qrouter DHCP Router ironic_grenade Namespace Namespace tap tap 10.2.0.0/20 qdhcp qrouter
Instance Phase: resources.sh create/verify node-0 node-N Ironic nodes (KVM VMs) br-node-0 br-node-N 1. Create resources (boot Instance) 2. Verify (ping instance) Baremetal bridge: brbm ovs-node-0 ovs-node-N brbm-tap Neutron Integration bridge: br-int Port: tap Port: tap Port: ovs-tap Port: tap Port: tap tag: 20 tag: 10 tag: 10 tag: 20 tag: 20 DHCP Router private Namespace Namespace tap tap 10.1.0.0/20 qdhcp qrouter DHCP Router ironic_grenade Namespace Namespace tap tap 10.2.0.0/20 qdhcp qrouter
Instance Phase: resources.sh shutdown node-0 node-N Ironic nodes services/verify (KVM VMs) br-node-0 br-node-N 1. Shutdown all services 2. Verify (ping instance) Baremetal bridge: brbm ovs-node-0 ovs-node-N brbm-tap Neutron Integration bridge: br-int Port: tap Port: tap Port: ovs-tap Port: tap Port: tap tag: 20 tag: 10 tag: 10 tag: 20 tag: 20 DHCP Router private Namespace Namespace tap tap 10.1.0.0/20 qdhcp qrouter DHCP Router ironic_grenade Namespace Namespace tap tap 10.2.0.0/20 qdhcp qrouter
Instance Phase: resources.sh upgrade node-0 node-0 node-N node-N Ironic nodes services/verify (KVM VMs) br-node-0 br-node-0 br-node-N br-node-N 1. Upgrade services/run db migrations Baremetal bridge: brbm 2. Start services ovs-node-0 ovs-node-0 ovs-node-N ovs-node-N brbm-tap brbm-tap Neutron picked new tag for networks tag: 10 → tag: 11 Neutron Integration bridge: br-int Port: tap Port: tap Port: tap Port: tap Port: ovs-tap Port: ovs-tap tag: 20 → tag: 21 Port: tap Port: tap Port: tap Port: tap tag: 21 tag: 20 tag: 11 tag: 10 tag: 11 tag: 10 tag: 20 tag: 21 tag: 20 tag: 21 3. Update tag on ovs-tap to connect nodes back to ironic_grenade DHCP Router private Namespace Namespace tap tap tap tap 10.1.0.0/20 qdhcp qrouter 4. Verify (ping instance) DHCP Router ironic_grenade Namespace Namespace tap tap tap tap 10.2.0.0/20 qdhcp qrouter
Instance Phase: resources.sh destroy node-0 node-0 node-N node-N Ironic nodes (KVM VMs) br-node-0 br-node-0 br-node-N br-node-N 1. Destroy resources that were created during resources.sh create Baremetal bridge: brbm 2. Update tag on ovs-tap to connect ovs-node-0 ovs-node-0 ovs-node-N ovs-node-N brbm-tap brbm-tap nodes back to private (set tag: 11 ) Neutron Integration bridge: br-int Port: tap Port: tap Port: ovs-tap Port: ovs-tap Port: tap Port: tap 3. Run target smoke test tag: 11 tag: 21 tag: 11 tag: 11 tag: 21 tag: 21 DHCP Router private Namespace Namespace tap tap 10.1.0.0/20 qdhcp qrouter DHCP Router ironic_grenade Namespace Namespace tap tap 10.2.0.0/20 qdhcp qrouter
Ironic hypervisor Since we can not create additional bare-metal resources, we have to use what was created during devstack setup. In the case of the grenade job, 7 nodes are created. Ironic nodes node-0 node-N (KVM VMs) devstack VM We have a process called cleaning that happens after the instance deletion request.
Tests Resource verification phase uses one node. In smoke tests, none of them boots more than three instances, so we should be safe running all of them with concurrency=1. The situation right before the target smoke run might be the following: cleaning after base smoke test Ironic nodes cleaning after resource verify (KVM VMs) available
Tests Some smoke tests were skipped or worked around: - Some features are not supported by Ironic (disk-config) - Networking service ports remain in the down state when using ironic After the skips, we have 8 requests to boot an instance. For comparison, in full tempest, an instance is booted 154 times.
Ironic issues The main one was the lack of versioning of IPA (Ironic Python Agent), which led to a versioning spec proposal [1]. start_iscsi_target(..., portal_port=2222, wipe_disk_metadata=True) Ironic Old IPA conductor TypeError Upgrade testing brought some changes to devstack plugin too, as some parts of it had to be reused by two different releases of OpenStack. Backward compatibility is important! [1] https://review.openstack.org/341086
Future work: multinode is reality... Multinode DevStack networking is even more complex… But multinode grenade job will help us to: ● Test rolling upgrades! ● Test multi compute/conductor scenarios (takeover) ● Increase test concurrency
Review / Call to Action ● Help us finish the multi-node grenade (rolling upgrade testing) work for Ironic in the Ocata * cycle! ○ https://etherpad.openstack.org/p/ironic-ocata-multinode-whiteboard ○ Weekly Ironic QA meeting: https://wiki.openstack.org/wiki/Meetings/Ironic-QA ● What is Grenade ● Grenade phases ● Grenade as a plug-in ● Networking of Ironic in Devstack and Grenade ● Grenade testing difficulties for Ironic ● Future work
Thank you Q&A
https://etherpad.openstack.org/p/ironic-ocata-summit-grenade-presentation
openstack-infra/project-config job-template snippet - job-template: name: '{pipeline}- grenade -dsvm-ironic{job-suffix}' node: '{node}' - shell: | #!/bin/bash -xe export PROJECTS="openstack/ironic $PROJECTS" export PROJECTS="openstack/ironic-lib $PROJECTS" export PROJECTS="openstack/ironic-python-agent $PROJECTS" export PROJECTS="openstack/python-ironicclient $PROJECTS" export PROJECTS="openstack-dev/grenade $PROJECTS" export PYTHONUNBUFFERED=true export DEVSTACK_GATE_TEMPEST=1 export DEVSTACK_GATE_GRENADE=pullup export DEVSTACK_GATE_IRONIC=1 export DEVSTACK_GATE_NEUTRON=1 export DEVSTACK_GATE_VIRT_DRIVER=ironic export TEMPEST_CONCURRENCY=1 export DEVSTACK_GATE_OS_TEST_TIMEOUT=2400 export DEVSTACK_GATE_TEMPEST_BAREMETAL_BUILD_TIMEOUT=1200 export DEVSTACK_GATE_IRONIC_BUILD_RAMDISK=0 export GRENADE_PLUGINRC="enable_grenade_plugin ironic https://git.openstack.org/openstack/ironic" cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
Recommend
More recommend
