Learnings from scaling Ironic at Yahoo Arun S A G - - PowerPoint PPT Presentation

▶
learnings from scaling ironic at yahoo
SMART_READER_LITE
LIVE PREVIEW

Learnings from scaling Ironic at Yahoo Arun S A G - - PowerPoint PPT Presentation

Jan 28, 2024 159 likes •665 views

Learnings from scaling Ironic at Yahoo Arun S A G saga@yahoo-inc.com zer0c00l on freenode Yahoo Inc May 08, 2017 https://github.com/sagarun/presentations/ Background and Architecture 1 / 25 Cluster Architecture OOB - ipmi power management

slide-1
SLIDE 1

Learnings from scaling Ironic at Yahoo

Arun S A G saga@yahoo-inc.com zer0c00l on freenode

Yahoo Inc

May 08, 2017 https://github.com/sagarun/presentations/

slide-2
SLIDE 2

Background and Architecture

1 / 25

slide-3
SLIDE 3

Cluster Architecture

API Node nova-api nova-scheduler neutron-server nova-compute keystone glance-api glance-registry ironic-api Ironic Conductor OOB - ipmi power management

Openstack Mgmt Vlan Out of band network vlan

Message Queue mq[1-2].ostk zookeeper ensemble (2) API Node api[1-3].ostk nova-api nova-scheduler neutron-server nova-compute keystone glance-api glance-registry ironic-api zookeeper ensemble (3) Database Node db[1-2].ostk A T S

ats[1-2 ].ostk User

Target Nodes Ironic Conductor ic[1-2].ostk (dual connected) NFS target ITS its[1-2].ostk (10g nic) neutron-agent x ATS proxy to yapache http server dhcp (67), tftp (69) ipxe binary via tfp images + ipxe via https https/ rpc/5671,5672

Corp

nfs - ro nfs -rw Horizon https rpc/ 5671 rpc/ 5671, 5672 https/ http/ (various) mysql ro/ mysql rw/

Target inventory Nodes

Console / Tools / Bootbox Server (Existing) NFS glance nfs -rw V I P V I P V I P VIP mysql rw/ mysql ro/ VIP V I P https

2 / 25

slide-4
SLIDE 4

Migrating to Ironic

◮ Import nodes from old system into Ironic

3 / 25

slide-5
SLIDE 5

Migrating to Ironic

◮ Import nodes from old system into Ironic ◮ Create neutron port for the node

3 / 25

slide-6
SLIDE 6

Migrating to Ironic

◮ Import nodes from old system into Ironic ◮ Create neutron port for the node ◮ If the node is already active in the old system, ’fake’ boot it with fake_pxe

driver

3 / 25

slide-7
SLIDE 7

Migrating to Ironic

◮ Import nodes from old system into Ironic ◮ Create neutron port for the node ◮ If the node is already active in the old system, ’fake’ boot it with fake_pxe

driver

◮ Once everything is successful, switch to pxe_ipmitool driver

3 / 25

slide-8
SLIDE 8

Ironic

4 / 25

slide-9
SLIDE 9

Ironic Setup

◮ Ironic API runs behind Apache Server

5 / 25

slide-10
SLIDE 10

Ironic Setup

◮ Ironic API runs behind Apache Server ◮ Ironic Conductors(2)

5 / 25

slide-11
SLIDE 11

6 / 25

slide-12
SLIDE 12

7 / 25

slide-13
SLIDE 13

What could possibly go wrong?

◮ Ironic Boots started to fail

8 / 25

slide-14
SLIDE 14

What could possibly go wrong?

◮ Ironic Boots started to fail ◮ Ironic-conductor was using lot of CPU

8 / 25

slide-15
SLIDE 15

What could possibly go wrong?

◮ Ironic Boots started to fail ◮ Ironic-conductor was using lot of CPU ◮ Ironic API calls took too long

8 / 25

slide-16
SLIDE 16

Solutions

◮ Sync_Power_State periodic task

9 / 25

slide-17
SLIDE 17

Solutions

◮ Sync_Power_State periodic task ◮ Increase the number of Ironic Conductors

9 / 25

slide-18
SLIDE 18

Solutions

◮ Sync_Power_State periodic task ◮ Increase the number of Ironic Conductors ◮ Run multiple conductors on the same host

9 / 25

slide-19
SLIDE 19

Neutron

10 / 25

slide-20
SLIDE 20

Neutron setup

◮ All 3 API servers run neutron-server

11 / 25

slide-21
SLIDE 21

Neutron setup

◮ All 3 API servers run neutron-server ◮ 24 API/RPC workers

11 / 25

slide-22
SLIDE 22

Neutron setup

◮ All 3 API servers run neutron-server ◮ 24 API/RPC workers ◮ 4 neutron dhcp agents

11 / 25

slide-23
SLIDE 23

Neutron setup

◮ All 3 API servers run neutron-server ◮ 24 API/RPC workers ◮ 4 neutron dhcp agents ◮ All networks/subnets are managed by all 4 agents (HA)

11 / 25

slide-24
SLIDE 24

Neutron setup

◮ All 3 API servers run neutron-server ◮ 24 API/RPC workers ◮ 4 neutron dhcp agents ◮ All networks/subnets are managed by all 4 agents (HA) ◮ ISC DHCPD driver instead of dnsmasq

11 / 25

slide-25
SLIDE 25

What is sync state?

12 / 25

slide-26
SLIDE 26

A tale of two drivers

◮ OMShell driver

13 / 25

slide-27
SLIDE 27

A tale of two drivers

◮ OMShell driver ◮ Pypureomapi driver

13 / 25

slide-28
SLIDE 28

OMShell

  • bash-4.1$ omshell

> server 127.0.0.1 > port 7911 > key keyname secret > connect

  • bj: <null>

> new host

  • bj: host

> set hardware-address = 00:1c:1a:1d:10:54

  • bj: host

hardware-address = 00:1c:1a:1d:10:54 > open

  • bj: host

hardware-address = 00:1c:1a:1d:10:54 ip-address = 0a:d7:a6:b1 name = "hostname.yahoo.com-0" hardware-type = 00:00:00:01 >remove 14 / 25

slide-29
SLIDE 29

Sync State with OMShell

15 / 25

slide-30
SLIDE 30

Sync State with PypureOMAPI

16 / 25

slide-31
SLIDE 31

Where do we go from here?

◮ ISC DHCPD restarts are not ideal

17 / 25

slide-32
SLIDE 32

Where do we go from here?

◮ ISC DHCPD restarts are not ideal ◮ VIP thinks dhcpd is down whenever it restarts

17 / 25

slide-33
SLIDE 33

Where do we go from here?

◮ ISC DHCPD restarts are not ideal ◮ VIP thinks dhcpd is down whenever it restarts ◮ Move to Kea DHCP Server

17 / 25

slide-34
SLIDE 34

Density Test

18 / 25

slide-35
SLIDE 35

When did things started to break?

◮ At 24500 nodes, API servers started swapping

19 / 25

slide-36
SLIDE 36

Swap and memory usage on API nodes

20 / 25

slide-37
SLIDE 37

Memory usage

◮ Neutron the biggest user of memory: 1.4 GB per process

21 / 25

slide-38
SLIDE 38

Memory usage

◮ Neutron the biggest user of memory: 1.4 GB per process ◮ Subnets: 2500 Ports: 43000

21 / 25

slide-39
SLIDE 39

Memory usage

◮ Neutron the biggest user of memory: 1.4 GB per process ◮ Subnets: 2500 Ports: 43000 ◮ Easy fix: Reduce number of api_workers and rpc_workers

21 / 25

slide-40
SLIDE 40

Memory usage

◮ Neutron the biggest user of memory: 1.4 GB per process ◮ Subnets: 2500 Ports: 43000 ◮ Easy fix: Reduce number of api_workers and rpc_workers ◮ Long Term Fix: Investigate memory usage, isolate neutron

21 / 25

slide-41
SLIDE 41

Learnings

22 / 25

slide-42
SLIDE 42

Learnings

◮ Do a *density* and *scale* testing before taking on production

23 / 25

slide-43
SLIDE 43

Learnings

◮ Do a *density* and *scale* testing before taking on production ◮ Avoid spawning processes, try and use native python libraries whenever

possible

23 / 25

slide-44
SLIDE 44

Learnings

◮ Do a *density* and *scale* testing before taking on production ◮ Avoid spawning processes, try and use native python libraries whenever

possible

◮ Pay attention to periodic tasks

23 / 25

slide-45
SLIDE 45

Learnings

◮ Do a *density* and *scale* testing before taking on production ◮ Avoid spawning processes, try and use native python libraries whenever

possible

◮ Pay attention to periodic tasks ◮ Be prepared to scale horizontally

23 / 25

slide-46
SLIDE 46

Learnings

◮ Do a *density* and *scale* testing before taking on production ◮ Avoid spawning processes, try and use native python libraries whenever

possible

◮ Pay attention to periodic tasks ◮ Be prepared to scale horizontally ◮ Pay attention to number of workers,conductors,rpc_workers

23 / 25

slide-47
SLIDE 47

Learnings

◮ Do a *density* and *scale* testing before taking on production ◮ Avoid spawning processes, try and use native python libraries whenever

possible

◮ Pay attention to periodic tasks ◮ Be prepared to scale horizontally ◮ Pay attention to number of workers,conductors,rpc_workers ◮ Don’t forget to have fun :)

23 / 25

slide-48
SLIDE 48

Questions

24 / 25

slide-49
SLIDE 49

References

◮ Layout and background:

https://github.com/mtreinish/openstack-health-presentation

◮ Picture from TV show: http://www.imdb.com/title/tt4338930/ ◮ Picture of explotion: https://en.wikipedia.org/wiki/Explosion

25 / 25