Deploying a baremetal cloud is hard Julia Kreger Open Source Developer Advocate IBM Twitter: @ashinclouds OpenStack Summit Sydney Email: juliaashleykreger@gmail.com November 6th, 2017
A little about me! Ironic contributor since early 2015. Ironic core and recently elected to the OpenStack Technical Committee Author of Bifrost, a set of Ansible playbooks for leveraging ironic to deploy baremetal servers. Knows the pain of deploying fleets of servers from many years of experience! Prefers purple bike sheds! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
So why baremetal? Physical Infrastructure… As in what the cloud is built on. High Performance Computing High Memory Regulatory or Compliance Production-like environments @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
What most imagine deployments are like @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
A deployment is more like... RDU -> PHL -> RDU -> ATL -> SFO -> SLC -> RDU -> SLC -> SJC -> SFO -> RDU -> ATL -> SLC -> SEA -> NRT -> LAX -> ATL -> RDU -> AUS -> RDU -> BOS -> RDU -> MSP -> RAP -> MSP -> ATL -> RDU -> PHL -> ABQ -> DEN -> SLC -> PDX -> SEA -> SJC -> SEA -> PDX -> SMF -> PSP -> LAS -> ABQ -> SLC -> LAX -> SYD … @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
A few typical realistic steps Unbox Finish base setup, i.e. add cards. Record additional information, such as MACs and WWNs Configure the Baseboard Management Controller Connect all of the cables! Verify all of the cabling! Then begins the burn-in process! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Easy? Right? @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
One would think... @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Operational constraints are a thing! What might seem like a simple step could actually take many steps. Depending on the organization specific processes may vary endlessly. Imagine a ticket per network port! At the end of the day though, self-imposed red tape can slow a deployment to a crawl… Imagine double-verification of all data! Or mandatory paper checklists per server chassis! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Common Headaches @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Architectural Mandates Photo credit: torkildr via Foter.com / CC BY-SA @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Labeling? Lack their of? Photo credit: one individual via Foter.com / CC BY-SA @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Deployment via Human Photo credit: Jemimus via Foter.com / CC BY @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Inconsistent Hardware Photo credit: Julia Kreger - @ashinclouds @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
VLANs, Inside VLANs? Photo credit: jronaldlee via Foter.com / CC BY @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Policy, More Humans? Photo credit: Foter.com @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Here be Dragons! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
A bare metal cloud is not Traditional IT! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
It is a union of self-service and raw infrastructure! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
It can support traditional workloads, but processes and workflows must adapt! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
There must be willingness for change! "Do not meddle in the affairs of dragons for you are crunchy and taste good with ketchup" -- source unknown @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
How to get it right? @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Step 0: Identify needs, not wants! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Treat it as an island not as an addition. @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Plan ahead to walk through everything! At least once! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Plan on inconsistencies in hardware! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Run current software! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Run what the community develops! Consider NOT running vendor packages! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Engage the community! We don’t read minds! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Helpful hints with Ironic @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Ironic is intended to be Admin-only @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Nova initiated and manually deployed baremetal can co-exist! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Networking will be the headache! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Ironic Networking Expectations At least one network interface must be registered as a “port” ● A cleaning network ● A deployment network ● Cleaning and deployment networks able to reach the Ironic API ● A network for the node to live on after deployment ● @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Ironic Networking Three use models “flat” - Preconfigured static networking ● “neutron” - Dynamic networking through Neutron ml2 driver controlled switches ● “noop” - Short for “no operation” used by neutron-less and stand-alone users. ● @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Ironic Networking Hardware iPXE support is not always “real” Tools like WireShark can be extremely useful for troubleshooting. ● iPXE continues to evolve and add new features. ● Some newer hardware will only boot via PXE when in UEFI mode. @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Hardware drivers often the next headache! @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Hardware Drivers Some hardware needs special drivers! To properly support, the drivers MUST be in the “deployment ramdisk” and the “instance image”. @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Ironic Drivers/Hardware Types @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Ironic Drivers and Hardware Types “Classic Drivers” pxe_ipmitool for “iscsi” based write from the conductor ● Agent_ipmitool for “direct” write to disk via the agent ● Hardware Type based drivers Driver set to “ipmi” ● deploy_interface can now be “iscsi” or “direct” ● @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Ironic’s State Machine @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Node States @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
What is cleaning? In production, never turn cleaning off and never turn off a node that is cleaning! Ironic Python Agent (IPA) will utilize shred or ATA Secure Erase to wipe the contents from disks. Custom IPA Hardware Managers can also do things like flash firmware or assert settings. @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
How do I troubleshoot? @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Troubleshooting Hints Look at the node “last_error” field `openstack baremetal node show <uuid>` If in a “wait” state? Is there connectivity? Heartbeating? ● Is there a clean_step populated or running? ● Deploy failing? Consider the [agent]/deploy_logs_collect setting in ironic.conf And feel free to ask us for help in #openstack-ironic on irc.freenode.net @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Questions? @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Thanks! https://docs.openstack.org/ironic @ashinclouds - Deploying a baremetal cloud is hard - November 6th, 2017
Recommend
More recommend
