Run Kubernetes on OpenStack and Bare Metal Fast Ramon Acedo - - PowerPoint PPT Presentation

1

Run Kubernetes on OpenStack and Bare Metal Fast

Ramon Acedo Rodriguez Senior Principal Product Manager, Red Hat

OPEN INFRASTRUCTURE SUMMIT | SHANGHAI, NOVEMBER 4-6 2019

2

OPTIONAL SECTION MARKER OR TITLE

Open Hybrid Cloud

Vision

3

BARE METAL VIRTUAL PRIVATE CLOUD

The Open Hybrid Cloud

The 4 Footprints

PHYSICAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUD APP APP APP APP

Goal: Give developers the freedom to innovate faster across on-premises and public clouds

4

BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS PHYSICAL APP APP APP APP VIRTUAL PUBLIC CLOUD

VM C C C

On-premises

Kubernetes-powered Open Hybrid Cloud

5

BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

VIRTUAL Containers | VMs | Serverless Apps BARE METAL PRIVATE CLOUD PUBLIC CLOUDS

Kubernetes on OpenStack is focused on the private cloud

6

Kubernetes on OpenStack

Introduction

7

VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

DATACENTRE

WORKLOAD DRIVEN PROGRAMMATIC API DRIVEN ACROSS INFRASTRUCTURE DEEPLY INTEGRATED

Why Kubernetes on OpenStack?

Open, scalable, managed, workloads.

SCALE OUT SOLID FOUNDATION

8

Kubernetes on OpenStack Integrations

Kubernetes and OpenStack Are Complementary

Consumption of resources

Provides the container platform layer

Exposition of resources

Provides the infrastructure layer complementary

Kubernetes on OpenStack Key Integration Points

Kubernetes on OpenStack Integration Example: Cinder

OpenShift on OpenStack Logical Architecture

13

Installation Requirements

OpenShift on OpenStack

Requirements in your OpenStack Platform

Red Hat OpenStack Platform 13 (Queens) 3 Master nodes

• At least 16 GB RAM, 4 vCPUs and 25 GB Disk

At least 2 worker nodes

• A least 8 GB RAM, 2 vCPUs and 25 GB Disk

Object Storage (Swift) CoreOS image OpenStack Resources:

• Floating IPs: 2
• Security Groups: 3
• Security Group Rules: 60
• Routers: 1
• Subnets: 1
• RAM: 112 GB
• vCPUs: 28
• Volume Storage: 175 GB
• Instances: 7
• Swift containers: 2
• Swift objects: 1
• Available space in Swift: at least 10 MB

https://github.com/openshift/installer/blob/master/docs/user/openstack/README.md

15

OpenShift Internal Load Balancing and DNS

DNS

• CoreDNS and mDNS run on all

nodes

Internal DNS and Load Balancing

Load Balancing

• HAProxy + keepalived on master nodes
• Use of keepalived VIPs for:

○ Internal API ○ Ingress traffic to workloads ○ Internal DNS requests The openshift-installer will configure internal Load Balancing & DNS

https://github.com/openshift/installer/blob/master/docs/design/openstack/networking-infrastructure.md

17

Kuryr-Kubernetes SDN

SDN for OpenShift on OpenStack

Kuryr improves the network performance

• f pods when running on OpenStack.

SDN solution using Kubernetes Container Network Interface (CNI) and OpenStack Neutron. Provides interconnectivity between Kubernetes pods and OpenStack virtual instances

Kuryr-Kubernetes Logo

Kuryr-Kubernetes CNI

18

Recommended when

• Neutron tenant networks are used for

OpenShift Application Nodes Not recommended when

• Using only OpenStack VLAN-based provider

networks (Kuryr not needed)

• Many OpenShift services and few hypervisors:

○ For each OpenShift service, Octavia will run a VM with a load balancer in it

When to use Kuryr

19

Requires

• OpenStack Queens or newer
• Octavia Load Balancer
• Neutron Trunk Ports

Kuryr Internal Architecture

20

Kuryr integrates with Neutron and Octavia Pods are connected via Kuryr CNI to Neutron Kuryr creates Load Balancers with Octavia for the Kubernetes services

21

Bandwidth tests between pods on the same hypervisor show slightly better performance

Kuryr Performance Comparison

Performance test done with OpenShift 3.11 / Kubernetes 1.11 and OpenStack Queens

22

Bandwidth tests between pods on different hypervisors show up to 9x performance improvement

Kuryr Performance Comparison

Performance test done with OpenShift 3.11 / Kubernetes 1.11 and OpenStack Queens

23

Kuryr Performance Comparison Blog Post

Performance test done with OpenShift 3.11 and OpenStack Queens

https://blog.openshift.com/accelerate-your-openshift-network-performance-on-openstack-with-kuryr

24

Reference Architecture

OpenShift on OpenStack Reference Architecture

Current: OSP 13 LTS and OCP 3.11 with Kuryr | Next: OSP 13 LTS and OCP 4.3 with Kuryr

25

https://access.redhat.com/documentation/en-us/reference_architectures/2019/html-single/deploying_red_hat_o penshift_container_platform_3.11_on_red_hat_openstack_platform_13

26

Ways to Install Kubernetes on OpenStack

Introduction

27

BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

VIRTUAL BARE METAL PRIVATE CLOUD Full Stack Automation Simplified, opinionated Installer-provisioned OpenStack resources (IPI) Pre-existing Infrastructure User-provisioned OpenStack resources (UPI)

Kubernetes Installation with OpenShift 4 on OpenStack

Installation Experiences

28

BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

VIRTUAL BARE METAL PRIVATE CLOUD PUBLIC CLOUDS Installer Provisions:

Networks Internal Load Balancers1 Internal DNS1 OpenStack Instances Red Hat CoreOS CoreOS Ignition Configs OpenShift Nodes OpenShift Cluster Resources

Full Stack Automation Installation

Deploying OpenShift on OpenStack with installer-provisioned infrastructure (IPI)

Red Hat OpenStack Platform Overcloud OpenShift Cluster . . .

• penshift-install

1External Load Balancers (routing) and external DNS servers are provided by the user

29

BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

VIRTUAL BARE METAL User Provisions:

Networks Internal Load Balancers1 Internal DNS1 OpenStack Instances Red Hat CoreOS CoreOS Ignition Configs OpenShift Nodes OpenShift Cluster Resources

Installing on Pre-existing Infrastructure

Deploying OpenShift on OpenStack with user-provisioned infrastructure (UPI)

• penshift-install

User provisioned resources

Red Hat OpenStack Platform Overcloud OpenShift Cluster . . .

30

VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

Installing on OpenStack Documentation

User-provisioned Infrastructure | Installer-provisioned Infrastructure

USER PROVISIONED INFRASTRUCTURE INSTRALLER- PROVISIONED INFRASTRUCTURE

https://github.com/openshift/installer/blob/master/docs/user/openstack/README.md https://github.com/openshift/installer/blob/master/docs/user/openstack/install_upi.md

OpenShift 4.2 on OpenStack

31

try.openshift.com

32

OpenStack Bare Metal

Ironic Introduction

Bare Metal On-Trend

OpenStack User Survey 2017

Among users who run Kubernetes on OpenStack, adoption of Ironic is even stronger with 37% relying on it.

OpenStack User Survey 2018

Popular Use Cases Kubernetes on Bare Metal High-Performance Computing Direct Access to Dedicated Hardware Devices Big Data and Scientific Applications

Bare Metal On-Trend

OpenStack Bare Metal Service - Ironic

Hardware Lifecycle Management Hardware Inspection

Servers and Network Switches (via LLDP)

OS Image Provisioning

Supporting qcow2 images

Routed Spine/Leaf Networking

Provisioning over routed networks

Multi-Tenancy

With network isolation between tenants

Node Auto-discovery Broad Power Management Support

Redfish, iDrac, iRMC, iLo, IPMI, oVirt, vBMC

36

OpenStack Bare Metal

Features

Tenant-Dedicated Networks

Instead of a shared flat network

Provisioning Over an Isolated Network Switch Ports Dynamically Configured

At deployment time and on termination

Link Aggregation L2 Switch BM

NIC NIC LAG bond

Configured by ML2 plug-in Configured by cloud-init using metadata

L2 Switch BM

NIC

VLANs set by by ML2 plug-in

BM

NIC

L2 Switch

Multi-Tenant Support with Isolation Between Tenants

Available from OpenStack Queens

Multi-Tenant Support with Isolation Between Tenants

Available from OpenStack Queens Neutron ML2 Networking-Ansible Driver Multiple Switch Platforms in a Single ML2 Driver

Leveraging the Networking Ansible modules

Available in OpenStack Queens | Red Hat OpenStack Platform 13

OSP 13 Long Life Support

Provisioning Network is configured in the switch Boot BM on Tenant Network ML2 Plug-in Configures Switch BM is Provisioned ML2 Plug-in Configures Switch Tenant Network is configured in the switch BM is ready

L2 Switch BM

NIC

BM

NIC

spine switch Bare Metal Bare Metal Bare Metal Bare Metal Bare Metal Bare Metal Bare Metal Bare Metal Bare Metal Bare Metal spine switch spine switch

L3 routed networks

ToR/leaf switch

Bare Metal Ironic Node Ironic Node Ironic Node Bare Metal

ToR/leaf switch

ToR/leaf switch

DHCP Relay DHCP Relay DHCP Relay

L3 routed networks

L3 Spine and Leaf Topologies

Ironic provisioning bare metal nodes over routed networks

DHCP Relay

Allowing PXE booting over L3 routed networks

L3 Routed Networks (Spine/Leaf Network Topologies)

Available from OpenStack Queens IPv6 Support being added to OpenStack Train and above

BIOS Configurations

docs.openstack.org/ironic/latest/admin/bios.html Get and Set BIOS Settings

Retrieve and apply BIOS settings via CLI or REST API. The desired BIOS settings are applied during manual cleaning.

Settings Applied During Node Cleaning

The desired BIOS settings are applied during manual cleaning

[{ "name": "hyper_threading_enabled”, "value": "False" }, { "name": "cpu_vt_enabled", "value": "True" }]

Ironic Inspector Nodes Auto-Discovery

Just Power On the Nodes

Nodes PXE boot from the provisioning network used by Ironic

Automatic Node Inspection

Nodes boot from the network and their hardware is inspected

Nodes Automatically Registered with Ironic

After inspection they are registered with Ironic and ready to be deployed

Use Rules to Set Node Properties

E.g. set Ironic driver (iDrac, Redfish…) based

• n inspection data, set BMC credentials,

etc.

cat > rules.json << EOF [ { "description": "Set the vendor driver for Dell hardware", "conditions": [ {"op": "eq", "field": "data://auto_discovered", "value": true}, {"op": "eq", "field": "data://inventory.system_vendor.manufacturer", "value": "Dell Inc."} ], "actions": [ {"action": "set-attribute", "path": "driver", "value": "idrac"}, {"action": "set-attribute", "path": "driver_info/drac_username", "value": "root"}, {"action": "set-attribute", "path": "driver_info/drac_password", "value": "calvin"}, {"action": "set-attribute", "path": "driver_info/drac_address", "value": "{data[inventory][bmc_address]}"} ] } ] EOF $ openstack baremetal introspection rule import rules.json

Data collected during inspection

E.g: Use the the idrac driver and its credentials if a Dell node is detected

Redfish Support in Ironic

API-driven Remote Management Platform

Manage large amounts of physical nodes via API. redfish.dmtf.org

Included in Modern BMCs

Most vendors support Redfish in the latest models

Supported in Ironic

Introduced in OpenStack Pike, along with the Sushy library

Improvements in OpenStack Train and Beyond

Out-of-band inspection of nodes, boot from virtual media (without DHCP) and BIOS configurations

• penstack baremetal node create \
• -driver redfish \
• -driver-info redfish_address=https://example.com \
• -driver-info redfish_system_id=/redfish/v1/Systems/CX34R87 \
• -driver-info redfish_username=admin \
• -driver-info redfish_password=password

43

OpenShift on OpenStack-Managed Bare Metal

Deployment of Kubernetes on the metal

Kubernetes Cluster

Kubernetes on Bare Metal

Deploy Kubernetes on OpenStack-managed bare metal nodes

OpenShift Installer

Master Node Worker Node Worker Node

Deploy OpenShift

OpenStack with Ironic https://docs.openshift.com/container-platform/4.2/installing/installing_bare_metal/installing-bare-metal.html

45

Metal3

Deployment of Kubernetes on the metal

46

BARE METAL VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

VIRTUAL Containers | Serverless Apps BARE METAL PRIVATE CLOUD PUBLIC CLOUDS

Metal3 is focused on Kubernetes on bare metal

Metal3

Metal Kubed, bare metal host provisioning for Kubernetes

Metal3 Kubernetes Operators Framework Ironic

Metal3 enables bare metal host management with Kubernetes.

48

Metal3 runs on Kubernetes. And is managed through Kubernetes interfaces.

Metal3

Metal Kubed, bare metal host provisioning for Kubernetes

49

Actuator allows Kubernetes to get Machines the same way it would in a public cloud, using the Kubernetes cluster-api Operator uses Ironic behind the scenes to manage the physical hardware represented as BareMetalHost

• bjects.

MACHINE CONTROLLER BARE-METAL ACTUATOR BARE METAL OPERATOR Bare Metal Management Pods Bare Metal Management Pods BARE-METAL MANAGEMENT PODS

Metal3

Metal Kubed, bare metal host provisioning for Kubernetes

Ironic

50

Try it: http://metal3.io/try-it.html Install OpenShift using it: https://github.com/openshift/installer/blob/master/docs/user/metal/install_ipi.md

Metal3

Metal Kubed, bare metal host provisioning for Kubernetes

51

Summary

Kubernetes on OpenStack

52

VIRTUAL PRIVATE CLOUD PUBLIC CLOUDS

Run Kubernetes on OpenStack and Bare Metal Fast

Install Kubernetes on OpenStack

https://github.com/openshift/installer/blob/master/docs/user/openstack/README.md https://github.com/openshift/installer/blob/master/docs/user/openstack/install_upi.md

Kuryr performance

https://blog.openshift.com/accelerate-your-openshift-network-performance-on-openstack-with-kuryr

Reference Architecture

https://access.redhat.com/documentation/en-us/reference_architectures/2019/html-single/deploying_r ed_hat_openshift_container_platform_3.11_on_red_hat_openstack_platform_13

Ironic Bare Metal

https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/13/html-single/bare_met al_provisioning/index

Metal3

metal3.io

53

Red Hat is the world’s leading provider of enterprise

• pen source software solutions. Award-winning

support, training, and consulting services make Red Hat a trusted adviser to the Fortune 500.

Thank you