DIVIDE AND CONQUER: RESOURCE SEGREGATION IN THE OPENSTACK CLOUD - - PowerPoint PPT Presentation

▶
divide and conquer resource segregation in the openstack
SMART_READER_LITE
LIVE PREVIEW

DIVIDE AND CONQUER: RESOURCE SEGREGATION IN THE OPENSTACK CLOUD - - PowerPoint PPT Presentation

Sep 21, 2023 237 likes •786 views

DIVIDE AND CONQUER: RESOURCE SEGREGATION IN THE OPENSTACK CLOUD Steve Gordon (@xsgordon) Technical Product Manager, Red Hat Why segregate resources? Infrastructure Expose logical groupings of infrastructure based on physical

slide-1
SLIDE 1

DIVIDE AND CONQUER: RESOURCE SEGREGATION IN THE OPENSTACK CLOUD

Steve Gordon (@xsgordon) Technical Product Manager, Red Hat

slide-2
SLIDE 2

Why segregate resources?

  • Infrastructure

–Expose logical groupings of infrastructure based on physical

characteristics

–Expose logical groupings of infrastructure based on some abstract

functionality/capability

–“More-massive” horizontal scalability

slide-3
SLIDE 3

Why segregate resources?

  • Infrastructure

–Expose logical groupings of infrastructure based on physical

characteristics

–Expose logical groupings of infrastructure based on some abstract

functionality/capability

–“More-massive” horizontal scalability

  • Workloads

–Ensure an even spread of a single workload –Ensure close placement of related workloads

slide-4
SLIDE 4

Segregation in datacenter virtualization

  • Infrastructure segregation:

–Logical data center constructs

  • Contain some number of logical clusters
  • Clusters typically:

–Are relatively small (0's to 00's of nodes per cluster) –Are tightly coupled to physical storage and network layout

  • Workload segregation:

–Host-level affinity/anti-affinity –CPU-level affinity/anti-affinity (pinning)

slide-5
SLIDE 5

Segregation in an elastic cloud

  • Amazon EC2:

–Infrastructure segregation:

  • Regions – Separate geographic areas (e.g. us-east-1)
  • Availability Zones – Isolated locations within a region (e.g. us-east-1a)

–Workload segregation:

  • Placement Groups – Workload affinity within an availability zone
slide-6
SLIDE 6

Segregation in an elastic cloud

  • Amazon EC2:

–Infrastructure segregation:

  • Regions – Separate geographic areas (e.g. us-east-1)
  • Availability Zones – Isolated locations within a region (e.g. us-east-1a)

–Workload segregation:

  • Placement Groups – Workload affinity within an availability zone
  • OpenStack:

–Overloads some of these terms (and more!) –Application is more flexible for deployers and operators

slide-7
SLIDE 7

Segregation in an elastic cloud

  • Wait a second...weren't we moving to the cloud to hide all this

infrastructure stuff from the user?

slide-8
SLIDE 8

Segregation in an elastic cloud

  • Wait a second...weren't we moving to the cloud to hide all this

stuff from the user?

–Yes!

  • Users and applications demand some visibility of:

–Failure domains –Premium features

  • Deployers and operators determine the level of granularity

exposed.

slide-9
SLIDE 9

Segregation in OpenStack

  • Infrastructure segregation:

–Regions –Cells –Host aggregates –Availability zones

slide-10
SLIDE 10

Segregation in OpenStack

  • Infrastructure segregation:

–Regions –Cells –Host aggregates –Availability zones

  • Workload segregation:

–Server groups

slide-11
SLIDE 11

REGIONS AND CELLS

slide-12
SLIDE 12

Regions

  • Complete OpenStack deployments

–Share at least a Keystone and Horizon installation –Implement their own targetable API endpoints

  • In default deployment all services in one region – 'RegionOne'.
  • New regions are created using Keystone:

–$ keystone endpoint-create --region “RegionTwo”

slide-13
SLIDE 13

Regions

  • Target actions at a region's endpoint (mandatory):

–CLI:

  • $ nova --os-region-name “RegionTwo” boot …

–Horizon:

slide-14
SLIDE 14

Regions

slide-15
SLIDE 15

Regions

slide-16
SLIDE 16

Cells

  • Standard (simplified) compute

deployment without Cells:

LOAD BALANCER

MESSAGE QUEUE SCHEDULER API DATABASE COMPUTE HYPERVISOR

KVM

CONDUCTOR

AMQP

OPST0007

slide-17
SLIDE 17

Cells

  • Maintains a single compute endpoint
  • Relieve pressure on queues

database at scale (000's of nodes)

  • Introduces the cells scheduler

MESSAGE QUEUE API CELL COMPUTE CELL COMPUTE CELL ...

OPST0008

slide-18
SLIDE 18

API (parent) cell

  • Adds a load balancer in front of

multiple instances of the API service

  • Has its own message queue
  • Includes a new service, nova-cells

–Handles cell scheduling –Packaged as openstack-nova-cells –Required in every cell

MESSAGE QUEUE API CELLS

nova-cells

LOAD BALANCER

nova-api

OPST0009

slide-19
SLIDE 19

Compute (child) cell

  • Each compute cell contains:

–Its own message queue and database –Its own scheduler, conductor, compute

nodes

slide-20
SLIDE 20

Common cell configuration

  • Setup database and message broker for each cell
  • Initialize cell database using nova-manage
  • Optionally:

–Modify scheduling filter/weight configuration for cells scheduler –Create cells JSON file to avoid need to avoid reloading from database

slide-21
SLIDE 21

API (parent) cell configuration

  • Nova.conf:

–Change compute_api_class –Enable cells –Name the cell –Enable and start nova-cells

slide-22
SLIDE 22

Compute (child) cell configuration

  • nova.conf

–Disable quota driver –Enable cells –Name the cell –Enable and start nova-cells

slide-23
SLIDE 23

Cells pitfalls

  • That all sounds pretty good – sign me up!
  • Lack of “cell awareness” in other projects
  • Minimal test coverage in the gate
  • Some standard functionality currently broken with cells:

–Host aggregates –Security groups

slide-24
SLIDE 24

So how do they stack up?

Regions

  • Supported by all services
  • Separate endpoints
  • Exist above scheduling
  • Linked via REST APIs

Cells

  • Supported by compute
  • Common endpoint
  • Additional scheduling layer
  • Linked via RPC
slide-25
SLIDE 25

HOST AGGREGATES AND AVAILABILITY ZONES

slide-26
SLIDE 26

Host aggregates

  • Logical groupings of hosts based on metadata
  • Typically metadata describes special capabilities hosts share:

–Fast disks for ephemeral data storage –Fast network interfaces –Etc.

  • Hosts can be in multiple host aggregates:

–“Hosts that have SSD storage and GPUs”

slide-27
SLIDE 27

Host aggregates

  • Implicitly user targetable:

–Admin defines host aggregate with metadata, and a flavor that matches it –User selects flavor with extra specifications when requesting instance –Scheduler places instance on a host in a host aggregate that matches

(extra specifications to metadata)

–User explicitly targets a capability, not an aggregate

slide-28
SLIDE 28

Host aggregates (example)

Region A Region B

Keystone Horizon Glance Cinder Nova Neutron Swift Glance Cinder Nova Neutron Swift

slide-29
SLIDE 29

Host aggregates (example)

  • Create host aggregates:

–$ nova aggregate-create storage-optimized –$ nova aggregate-create network-optimized –$ nova aggregate-create compute-optimized

slide-30
SLIDE 30

Host aggregates (example)

–$ nova aggregate-set-metadata 1 fast-storage=true –$ nova aggregate-set-metadata 2 fast-network=true –$ nova aggregate-set-metadata 3 high-freq-cpu=true

slide-31
SLIDE 31

Host aggregates (example)

  • Populate the aggregates:

–$ nova aggregate-add-host 1 host-1 –$ nova aggregate-add-host 1 host-2 –...

slide-32
SLIDE 32

Host aggregates (example)

slide-33
SLIDE 33

Host aggregates (example)

slide-34
SLIDE 34

Host aggregates (example)

Region A Region B

Keystone Horizon Glance Cinder Nova Neutron Swift Glance Cinder Nova Neutron Swift

storage-optimized

slide-35
SLIDE 35

Host aggregates (example)

Region A Region B

Keystone Horizon Glance Cinder Nova Neutron Swift Glance Cinder Nova Neutron Swift

storage-optimized network-optimized

slide-36
SLIDE 36

Host aggregates (example)

Region A Region B

Keystone Horizon Glance Cinder Nova Neutron Swift Glance Cinder Nova Neutron Swift

storage-optimized network-optimized high-freq-cpu

slide-37
SLIDE 37

Host aggregates (example)

Region A Region B

Keystone Horizon Glance Cinder Nova Neutron Swift Glance Cinder Nova Neutron Swift

storage-optimized network-optimized high-freq-cpu

slide-38
SLIDE 38

Host aggregates (example)

  • Set flavor extra specifications:

–$ nova flavor-key 1 set fast-storage=true –...

slide-39
SLIDE 39

Host aggregates (example)

  • Filter scheduler matches extra specifications of flavor to metadata
  • f aggregate.

Host 1 Host 2 Host 3

F I L T E R S W E I G H T S

Host 1 Host 3

slide-40
SLIDE 40

Availability zones

  • Logical groupings of hosts based on arbitrary factors like:

–Location (country, data center, rack, etc.) –Network layout –Power source

  • Explicitly user targetable:

–$ nova boot --availability-zone “rack-1”

  • OpenStack Block Storage (Cinder) also has availability zones
slide-41
SLIDE 41

Availability zones

  • Host aggregates are made explicitly user targetable by creating

them as an AZ:

–$ nova aggregate-create tier-1 us-east-tier-1 –tier-1 is the aggregate name, us-east-tier-1 is the AZ name

  • Host aggregate is the availability zone in this case

–Hosts can not be in multiple availability zones

  • Well...sort of.

–Hosts can be in multiple host aggregates

slide-42
SLIDE 42

Availability zones (example)

Region A Region B

Keystone Horizon Glance Cinder Nova Neutron Swift Glance Cinder Nova Neutron Swift

storage-optimized network-optimized high-freq-cpu

slide-43
SLIDE 43

Availability zones (example)

Region A Region B

Keystone Horizon

AZ 1 AZ 2

Glance Cinder Nova Neutron Swift Glance Cinder Nova Neutron Swift

AZ 3 AZ 4

storage-optimized network-optimized high-freq-cpu

slide-44
SLIDE 44

So how do they stack up?

Host Aggregates

  • Implicitly user targetable
  • Hosts can be in multiple

aggregates

  • Grouping based on common

capabilities Availability Zones

  • Explicitly user targetable
  • Hosts can not be in multiple

zones (see previous disclaimer)

  • Grouping based on arbitrary

factors such as location, power, network

slide-45
SLIDE 45

WORKLOAD SEGREGATION

slide-46
SLIDE 46

Server groups

  • Policies for defining workload placement rules for a group

–Anti-affinity filter – Grizzly –Affinity filter – Havana –API – Icehouse

  • Implemented via scheduler filters:

–ServerGroupAffinityFilter –ServerGroupAntiAffinityFilter

slide-47
SLIDE 47

Server groups

  • Affinity:

–Places instances within the group on the same host

  • Anti-affinity:

–Places instances within the group on different hosts

  • Not equivalent to AWS placement groups (host placement versus

availability zone placement)

slide-48
SLIDE 48

Server groups

  • Create the server group:

–$ nova server-group-create --policy=anti-affinity

my_group

–Really defining a policy rather than a group.

  • Specify the group UUID or name when launching instances:

–$ nova boot --image ... --flavor … --hint

group=group_id

slide-49
SLIDE 49

Server groups (affinity)

Region A Region B

Keystone Horizon

AZ 1 AZ 2

Glance Cinder Nova Neutron Swift Glance Cinder Nova Neutron Swift

AZ 3 AZ 4

storage-optimized network-optimized high-freq-cpu

slide-50
SLIDE 50

Server groups (anti-affinity)

Region A Region B

Keystone Horizon

AZ 1 AZ 2

Glance Cinder Nova Neutron Swift Glance Cinder Nova Neutron Swift

AZ 3 AZ 4

storage-optimized network-optimized high-freq-cpu

slide-51
SLIDE 51

What next?

  • Relevant design sessions:

–Simultaneous Scheduling for Server Groups

  • Friday, May 16 • 1:20pm – 2:00pm

–Scheduler hints for VM life cycle

  • Friday, May 16 • 2:10pm – 2:50pm

–Nova Dev/Ops Session

  • Friday, May 16 • 3:00pm - 3:40pm
slide-52
SLIDE 52

Resources

  • Operations Guide – Chapter 5 “Scaling”

–http://docs.openstack.org/trunk/openstack-ops/content/scaling.html

  • Configuration Reference Guide – Chapter 2 “Compute”

–http://docs.openstack.org/trunk/config-

reference/content/section_compute-cells.html

  • OpenStack in Production Blog

–http://openstack-in-production.blogspot.fr/

slide-53
SLIDE 53