Architecture and Implement on 3000 Nodes Bare Metal Cloud in China - - PowerPoint PPT Presentation

Architecture and Implement on 3000 Nodes Bare Metal Cloud in China Mobile

Yuntong Jin, Li Hao, Yao Jun yuntong.jin@intel.com lihao@cmss.chinamobile.com yaojun@cmss.chinamobile.com

ecloud.10086.cn

Practice of Openstack in CMCC Architecture & Deployment Implement Detail Call for Action

Agenda

Practice of Openstack in CMCC

Public cloud(GuangZhou) total: 1000 nodes 600 nova-compute Public cloud(Beijing) total: 1000 nodes 530 nova-compute Private cloud total: 6000 nodes， 3000 nodes every pool

Public cloud(GuangZhou & Beijing) is online (https://ecloud.10086.cn) Private cloud will be online in June 2017 IT

1 Public Cloud with 2 pools, 1 Private Cloud with 2 pools

CMCC Bare Metal Cloud in China

Bare Metal Cloud Topological Graph

Public Cloud Topological Graph in CMCC

172.16.216.7 172.16.216.7 172.16.216.20 172.16.216.20 vip：172.16.216.201

MYSQL Galera MYSQL Galera N-cpu N-cpu

172.16.216.23 172.16.216.23 172.16.216.6 172.16.216.6 172.16.216.19 172.16.216.19 a a 172.16.216.26 172.16.216.26 aa 172.16.216.4 172.16.216.4 172.16.216.11 172.16.216.11 a a 172.16.216.17 172.16.216.17 aa

CORE Zone DMZ Zone

Keystone

Physical machine

172.16.216.6 172.16.216.6 172.16.216.19 172.16.216.19 vip：172.16.216.202

MYSQL Galera MYSQL Galera

172.16.216.26 172.16.216.26 172.16.216.12 172.16.216.12 172.16.216.21 172.16.216.21 vip：172.16.216.201

RabbitMQ RabbitMQ

172.16.216.25 172.16.216.25 172.16.216.5 172.16.216.5 172.16.216.14 172.16.216.14 vip：172.16.216.202

InfluxDB InfluxDB

172.16.216.24 172.16.216.24 172.16.216.3 172.16.216.3 172.16.216.9 172.16.216.9 a a 172.16.216.15 172.16.216.15 aa 172.16.216.22 172.16.216.22 172.16.216.29 172.16.216.29 a a

Nova、Heat

172.16.191.1 172.16.191.1 172.16.191.2 172.16.191.2 a a 172.16.191.3 172.16.191.3 aa

VNCProxy

172.16.216.1 172.16.216.1 172.16.216.8 172.16.216.8 a b

Haproxy、Keepalive、LVS

Gnocchi

Ceilometer-Collector

Ceilometer-API、Senlin

Cinder-API

Cinder-Volume、Glance

N-cpu N-cpu

N-cpu N-cpu N-cpu N-cpu N-cpu N-cpu

N-cpu N-cpu N-cpu N-cpu N-cpu N-cpu

N-cpu N-cpu

Production Zone

Service Deployment in Harbin Pool

Service Usage Node Number Haproxy Haproxy A/P Share with coreAPI Core service Nova/Neutron/Keystone api/conductor/scheduler 3 Ceilometer Ceilomeer-api/agent-central Share with coreAPI MySQL MySQL cluster 3 RabbitMQ RabbitMQ cluseter Share with MySQL Total 6 Service Usage Node Number Haproxy Haproxy A/P Share with Ironic Ironic Ironic-api/conductor 10 Glance Glance API Share with Ironic tftp Tftp server Share with Ironic Total 10 Control Zone Production Zone

O Ironic + K OpenStack != OK

Can’t delete instance when spawning Race in between the compute node create and initial update Need python-ironicclient 0.8.1 to support Nova configdrive

Feature Support



Support multi tenancy local boot support



Neutron network default_network_interface=neutron for SDN integration



SDN integration with Vxlan undelaying

Driver Boot Deploy Power Manager agent_ipmitool PXE Agent IPMI

Feature Support - windows Image

Use whole_disk_image to support windows Miss LSI SAS RAID driver

Multi-nova-compute & nova mpute HA

Nova API Nova Compute …… Nova Compute Ironic Node …… Ironic Node Ironic Node …… Ironic Node Failed

Download image from Glance directly

Ironic Glance Get image Swift Get Image from Swift Expose temp URL Ironic IPA Glance Get image Get Image from Glance

Specific NIC for Tenant Port

NICs on BM 1G Ethernet choose a specific nic's mac address to create tenant port 1G Ethernet 10G Fiber Card 10G Fiber Card Management Network Business Network

add main_nic = yes property in Ironic port. If yes, use its mac as create tenant port mac address

1G Ethernet IPMI

Port Group

Walk through in 3 steps:

• 1. Inspection -- Get bare metal property and LLDP in up TOR
• 2. Provision -- Install Operating system via provision network
• 3. Update to Tenant network -- switch to Bare Metal business

network from provision network to tenant network.

Port Group - Inspection

Ironic API Ironic node create Neutron Tftp server BM Node set provision state =inspect get dhcp address Conductor PXE boot IPA LLDP Back to inspection server Download inspect image

Port Group - Provision

Nova API Ironic API/conduct

• r

set vif info & instance info Nova compute Neutron BM Node scheduling Nova boot _allocate_network_async configdrive boot set_provision_state active prepare_deploy

Provision - Ironic

Ironic API Ironic Conductor Neutron do_node_deploy Ironic Update DHCPBOOT Boot Create provision port deploy Shundown

Port Group - Update to Tenant Network

Nova Neutron Remove provision network port Ironic BM Node Update Tenent network port Cloud-Init configure tenant port TOR LLDP reboot

Call for Action

support nova compute HA https://review.openstack.org/#/c/456865/ Ironic External display drivers https://review.openstack.org/#/c/306078/ Raid Configuration Driver Nova scheduler to support running ironc and kvm in same region expose which conductor instance is managing given Ironic node https://bugs.launchpad.net/ironic/+bug/1616878

THANKS FOR WATCHING

Intel Cloud for All