Architecture and Implement on 3000 Nodes Bare Metal Cloud in China - PowerPoint PPT Presentation

Architecture and Implement on 3000 Nodes Bare Metal Cloud in China Mobile Yuntong Jin, Li Hao, Yao Jun yuntong.jin@intel.com lihao@cmss.chinamobile.com yaojun@cmss.chinamobile.com ecloud.10086.cn

Agenda Practice of Openstack in CMCC Architecture & Deployment Implement Detail Call for Action

Practice of Openstack in CMCC 1 Public Cloud with 2 pools, 1 Private Cloud with 2 pools IT Public cloud(GuangZhou) Public cloud(Beijing) Private cloud total: 1000 nodes total: 1000 nodes total: 6000 nodes ， 600 nova-compute 530 nova-compute 3000 nodes every pool Public cloud(GuangZhou & Beijing) is online (https://ecloud.10086.cn) Private cloud will be online in June 2017

CMCC Bare Metal Cloud in China

Bare Metal Cloud Topological Graph Public Cloud Topological Graph in CMCC Physical machine DMZ Zone CORE Zone Production Zone Haproxy 、 Keepalive 、 LVS vip ： 172.16.216.201 172.16.216.202 a b 172.16.216.1 172.16.216.1 172.16.216.8 172.16.216.8 VNCProxy Nova 、 Heat Ceilometer-API 、 Senlin Cinder-API Cinder-Volume 、 Glance vip ： 172.16.190.201 vip ： 172.16.216.201 vip ： 172.16.216.201 vip ： 172.16.216.201 vip ： 172.16.216.201 a a aa a a aa a a aa a a 172.16.216.3 172.16.216.3 172.16.216.9 172.16.216.9 172.16.216.15 aa a a 172.16.216.15 aa 172.16.191.1 172.16.191.2 172.16.191.1 172.16.191.2 172.16.191.3 172.16.191.3 172.16.216.2 172.16.216.10 172.16.216.13 172.16.216.18 172.16.216.2 172.16.216.10 172.16.216.16 172.16.216.16 172.16.216.13 172.16.216.18 172.16.216.27 172.16.216.27 172.16.172.27 172.16.172.48 172.16.172.27 172.16.172.48 172.16.172.126 172.16.172.126 a a 172.16.216.22 172.16.216.22 172.16.216.29 172.16.216.29 Gnocchi Ceilometer-Collector Keystone RabbitMQ RabbitMQ vip ： 172.16.216.202 vip ： 172.16.216.202 vip ： 172.16.216.201 vip ： 172.16.216.201 ... a a a a a a aa aa aa 172.16.216.13 172.16.216.18 172.16.216.2 172.16.216.10 172.16.216.6 172.16.216.6 172.16.216.19 172.16.216.19 172.16.216.26 172.16.216.13 172.16.216.18 172.16.216.27 172.16.216.27 172.16.216.2 172.16.216.10 172.16.216.16 172.16.216.16 172.16.216.26 172.16.216.12 172.16.216.12 172.16.216.21 172.16.216.21 N-cpu N-cpu N-cpu N-cpu N-cpu N-cpu a a a a a a aa aa aa 172.16.216.25 172.16.216.25 172.16.216.5 172.16.216.14 172.16.216.12 172.16.216.21 172.16.216.4 172.16.216.4 172.16.216.11 172.16.216.11 172.16.216.17 172.16.216.5 172.16.216.14 172.16.216.24 172.16.216.24 172.16.216.12 172.16.216.21 172.16.216.25 172.16.216.25 172.16.216.17 ... N-cpu N-cpu N-cpu N-cpu N-cpu N-cpu MYSQL Galera MYSQL Galera InfluxDB InfluxDB MYSQL Galera MYSQL Galera vip ： 172.16.216.202 vip ： 172.16.216.202 vip ： 172.16.216.201 ... 172.16.216.6 172.16.216.6 172.16.216.5 172.16.216.5 172.16.216.7 172.16.216.19 172.16.216.14 172.16.216.7 172.16.216.19 172.16.216.14 172.16.216.20 172.16.216.20 N-cpu N-cpu N-cpu N-cpu N-cpu N-cpu 172.16.216.26 172.16.216.24 172.16.216.26 172.16.216.24 172.16.216.23 172.16.216.23

Service Deployment in Harbin Pool Control Zone Service Usage Node Number Haproxy Haproxy A/P Share with coreAPI Core service Nova/Neutron/Keystone 3 api/conductor/scheduler Ceilometer Ceilomeer-api/agent-central Share with coreAPI MySQL MySQL cluster 3 RabbitMQ RabbitMQ cluseter Share with MySQL Total 6 Production Zone Service Usage Node Number Haproxy Haproxy A/P Share with Ironic Ironic Ironic-api/conductor 10 Glance Glance API Share with Ironic tftp Tftp server Share with Ironic Total 10

O Ironic + K OpenStack != OK Can’t delete instance when spawning Race in between the compute node create and initial update Need python-ironicclient 0.8.1 to support Nova configdrive

Feature Support Boot Deploy Power Manager Driver agent_ipmitool PXE Agent IPMI Support multi tenancy  local boot support Neutron network  default_network_interface=neutron for SDN integration SDN integration with Vxlan undelaying 

Feature Support - windows Image Use whole_disk_image to support windows Miss LSI SAS RAID driver

Multi-nova-compute & nova mpute HA Nova API Nova Compute …… Nova Compute Failed Ironic Node …… Ironic Node Ironic Node …… Ironic Node

Download image from Glance directly Ironic Glance Swift Ironic IPA Glance Get image Get image Expose temp URL Get Image from Swift Get Image from Glance

Specific NIC for Tenant Port choose a specific nic's mac address to create tenant port NICs on BM 1G Ethernet 1G Ethernet 10G Fiber Card 10G Fiber Card 1G Ethernet Management Network Business Network IPMI add main_nic = yes property in Ironic port. If yes, use its mac as create tenant port mac address

Port Group Walk through in 3 steps: 1. Inspection -- Get bare metal property and LLDP in up TOR 2. Provision -- Install Operating system via provision network 3. Update to Tenant network -- switch to Bare Metal business network from provision network to tenant network.

Port Group - Inspection Ironic API Neutron Conductor Tftp server BM Node Ironic node create set provision state =inspect get dhcp address Download inspect image PXE boot IPA LLDP Back to inspection server

Port Group - Provision Ironic Nova API Nova compute API/conduct Neutron BM Node or Nova boot scheduling _allocate_network_async set vif info & instance info configdrive set_provision_state active prepare_deploy boot

Provision - Ironic Ironic Ironic API Neutron Ironic Conductor do_node_deploy Create provision port Update DHCPBOOT Boot deploy Shundown

Port Group - Update to Tenant Network Nova Ironic Neutron BM Node TOR Remove provision network port Update Tenent network port LLDP reboot Cloud-Init configure tenant port

Call for Action support nova compute HA https://review.openstack.org/#/c/456865/ Ironic External display drivers https://review.openstack.org/#/c/306078/ Raid Configuration Driver Nova scheduler to support running ironc and kvm in same region expose which conductor instance is managing given Ironic node https://bugs.launchpad.net/ironic/+bug/1616878

THANKS FOR WATCHING

Intel Cloud for All