What is new in Neutron QoS? Miguel Lavalle Slawek Kaplonski - - PowerPoint PPT Presentation

▶
what is new in neutron qos
SMART_READER_LITE
LIVE PREVIEW

What is new in Neutron QoS? Miguel Lavalle Slawek Kaplonski - - PowerPoint PPT Presentation

Mar 24, 2024 385 likes •850 views

What is new in Neutron QoS? Miguel Lavalle Slawek Kaplonski miguel@mlavalle.com skaplons@redhat.com IRC: mlavalle@freenode IRC: slaweq@freenode AGENDA Introduction to Neutron QoS Existing QoS features in Neutron New features

slide-1
SLIDE 1

What is new in Neutron QoS?

Miguel Lavalle miguel@mlavalle.com IRC: mlavalle@freenode Slawek Kaplonski skaplons@redhat.com IRC: slaweq@freenode

slide-2
SLIDE 2

2

AGENDA

  • Introduction to Neutron QoS
  • Existing QoS features in Neutron
  • New features added in Queens release
  • Planned New features
slide-3
SLIDE 3

Neutron QoS - basic concept

slide-4
SLIDE 4

Neutron QoS - data model

slide-5
SLIDE 5
  • penstack network qos policy create Summit_Policy

+-------------+--------------------------------------+ | Field | Value | +-------------+--------------------------------------+ | description | | | id | 8a495fea-e617-45f6-82e6-a61f7860c5cc | | is_default | False | | name | Summit_Policy | | project_id | 0783adde257d4a0280d3df1908b217c8 | | rules | [] | | shared | False | | tags | [] | +-------------+--------------------------------------+

5

Neutron QoS - usage

Create QoS policy

slide-6
SLIDE 6
  • penstack network qos rule create --type bandwidth-limit \
  • -max-kbps 1024 \
  • -max-burst-kbit 512 \

Summit_Policy +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | direction | egress | | id | f9bb1173-1bf0-4942-8932-a69b61ef5b52 | | max_burst_kbps | 512 | | max_kbps | 1024 | | name | None | | project_id | | +----------------+--------------------------------------+

6

Neutron QoS - usage

Create QoS rule in policy

slide-7
SLIDE 7
  • penstack port set --qos-policy Summit_Policy \

aa42976b-03d0-4fbb-90f8-fa7b88da8a2e

  • penstack port show aa42976b-03d0-4fbb-90f8-fa7b88da8a2e

+-----------------------+-------------------------------------------------+ | Field | Value | +-----------------------+-------------------------------------------------+ | created_at | 2018-05-07T14:51:49Z | | device_id | 80afc6fb-12a1-4bae-8cd4-6835aa819227 | | id | aa42976b-03d0-4fbb-90f8-fa7b88da8a2e | | mac_address | fa:16:3e:4e:e7:bc | | network_id | c8714b0a-f960-4db3-bbf6-f6dddb5a6843 | | project_id | 24f901074d52470da2c641e26c2753e1 | | qos_policy_id | 8a495fea-e617-45f6-82e6-a61f7860c5cc | | revision_number | 8 | | status | ACTIVE | | updated_at | 2018-05-09T20:36:11Z | +-----------------------+-------------------------------------------------+

7

Neutron QoS - usage

Attach QoS policy to port (or network)

slide-8
SLIDE 8

Open vSwitch SR-IOV Linux Bridge L3 agent Bandwidth limit

Egress / Ingress Egress (1) Egress / Ingress Egress / Ingress (3)

Minimum Bandwidth

  • Egress (2)
  • DSCP Marking

Egress

  • Egress
  • 8

Neutron QoS - available rule types

1. Max burst parameter is skipped because it's not supported by ip tool 2. Best effort - scheduler is not aware about this minimum bandwidth requirement for ports 3. Limit for Floating IP

slide-9
SLIDE 9
  • penstack network qos rule create \
  • -type bandwidth-limit \
  • -max-kbps 512 \
  • -max-burst-kbit 128 \
  • -ingress \

Summit_Policy +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | direction | ingress | | id | f5fc67f8-83a2-40be-ad12-d15fbe29381f | | max_burst_kbps | 128 | | max_kbps | 512 | | name | None | | project_id | | +----------------+--------------------------------------+

9

Neutron QoS - bandwidth limit rule

Rule API

slide-10
SLIDE 10

sudo ovs-vsctl list qos _uuid : a4857dc7-e64f-4370-9cf2-408566871bf1 external_ids : {id="tap9a566f82-0c"}

  • ther_config : {max-rate="512000"}

queues : {0=671cc958-2328-4d77-9e36-91984582e5a5} type : linux-htb sudo ovs-vsctl list queue _uuid : 671cc958-2328-4d77-9e36-91984582e5a5 dscp : [] external_ids : {id="tap9a566f82-0c", queue_type="0"}

  • ther_config : {burst="128000", max-rate="512000"}

sudo ovs-vsctl list port tap9a566f82-0c | grep qos qos : a4857dc7-e64f-4370-9cf2-408566871bf1

10

Neutron QoS - bandwidth limit rule

Backend implementation - Open vSwitch L2 agent - ingress traffic

slide-11
SLIDE 11

sudo ovs-vsctl list interface tap9a566f82-0c | grep ingress_ ingress_policing_burst: 256 ingress_policing_rate: 1024

11

Neutron QoS - bandwidth limit rule

Backend implementation - Open vSwitch L2 agent - egress traffic

slide-12
SLIDE 12

tc qdisc show dev tap2e939f9e-9e qdisc tbf 8002: root refcnt 2 rate 512Kbit burst 16Kb lat 50.0ms

12

Neutron QoS - bandwidth limit rule

Backend implementation - Linux bridge L2 agent - ingress traffic

slide-13
SLIDE 13

tc filter show dev tap2e939f9e-9e parent ffff: filter protocol all pref 49 basic chain 0 filter protocol all pref 49 basic chain 0 handle 0x1 police 0x1 rate 1024Kbit burst 32Kb mtu 64Kb action drop overhead 0b ref 1 bind 1

13

Neutron QoS - bandwidth limit rule

Backend implementation - Linux bridge L2 agent - egress traffic

slide-14
SLIDE 14

ip link show enp8s0f0 3: enp8s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 90:e2:ba:5e:a6:40 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, tx rate 1000 (Mbps), max_tx_rate 1000Mbps, spoof checking on, link-state auto vf 1 MAC 56:92:c6:c6:e5:db, tx rate 2000 (Mbps), max_tx_rate 2000Mbps, spoof checking on, link-state auto

14

Neutron QoS - bandwidth limit rule

Backend implementation - SR-IOV L2 agent - egress traffic

slide-15
SLIDE 15
  • penstack network qos rule create \
  • -type dscp-marking \
  • -dscp-mark 16 \

Summit_Policy +------------+--------------------------------------+ | Field | Value | +------------+--------------------------------------+ | dscp_mark | 16 | | id | e5256a9d-0ebd-4e45-acfe-bd2048d33d53 | | name | None | +------------+--------------------------------------+

15

Neutron QoS - DSCP marking rule

Rule API

slide-16
SLIDE 16

sudo ovs-ofctl show br-int | grep tap9a566f82-0c 141(tap9a566f82-0c): addr:fe:16:3e:47:85:27 sudo ovs-ofctl dump-flows br-int | grep in_port=141 | grep tos cookie=0x66a4b613766ef2e1, duration=70.889s, table=0, n_packets=0, n_bytes=0, idle_age=405, priority=65535,reg2=0,in_port=141 actions=mod_nw_tos:64,load:0x37->NXM_NX_REG2[0..5],resubmit(,0)

16

Neutron QoS - DSCP marking rule

Backend implementation - Open vSwitch L2 agent

slide-17
SLIDE 17

iptables -t mangle -nL neutron-linuxbri-qos-o2e939f Chain neutron-linuxbri-qos-o2e939f (1 references) target prot opt source destination DSCP all -- 0.0.0.0/0 0.0.0.0/0 DSCP set 0x10

17

Neutron QoS - DSCP marking rule

Backend implementation - Linux bridge L2 agent

slide-18
SLIDE 18
  • penstack network qos rule create \
  • -type minimum-bandwidth \
  • -min-kbps 512 \
  • -egress \

Summit_Policy +------------+--------------------------------------+ | Field | Value | +------------+--------------------------------------+ | direction | egress | | id | 5e5969f2-66f8-4bed-9905-b31b33902189 | | min_kbps | 512 | | name | None | +------------+--------------------------------------+

18

Neutron QoS - Minimum bandwidth

Rule API

slide-19
SLIDE 19

New rule types validation and reporting

slide-20
SLIDE 20

20

Rules validation - old way

Report of available rule types

Example for deployment with openvswitch and sriovnicswitch mechanism drivers

  • penstack network qos rule type list

+-----------------+ | Type | +-----------------+ | bandwidth_limit | (- supported by both drivers) +-----------------+

slide-21
SLIDE 21

21

Rules validation - old way - problems

  • There was no real validation - it was only information for

user,

  • Very limited range of rule types available in mixed

deployments,

  • Some attributes of rules have big impact of backend

implementation (e.g. direction)

slide-22
SLIDE 22

22

Rules validation - new way

Report of available rule types

Example for deployment with openvswitch and sriovnicswitch mechanism drivers:

  • penstack network qos rule type list

+-------------------+ | Type | +-------------------+ | bandwidth_limit | (- supported by both drivers) | dscp_marking | (- supported by openvswitch driver) | minimum_bandwidth | (- supported by sriovnicswitch driver) +-------------------+

slide-23
SLIDE 23

23

Rules validation - new way

Report details of rule type

Example for deployment with openvswitch and sriovnicswitch mechanism drivers

+----------------+------------------------------------------------------------------+ | Field | Value | +----------------+------------------------------------------------------------------+ | drivers | {'name': 'openvswitch', | | | 'supported_parameters': [{'parameter_values': [0, 8, .., 56], | | | 'parameter_type': 'choices', | | | 'parameter_name': 'dscp_mark'}]}] | | rule_type_name | dscp_marking | +----------------+------------------------------------------------------------------+

slide-24
SLIDE 24

24

Rules validation - new way - validation

Validation if QoS policy can be applied to port (ports) during:

  • Create/Update QoS rule,
  • Create/Update port,
  • Update network

Returns HTTP Conflict (409) error in case if rule(s) can’t be applied for at least one port

slide-25
SLIDE 25

25

Rules validation - new way - validation

  • penstack network qos policy show DSCP_MARK_Policy

+-------------+------------------------------------------+ | Field | Value | +-------------+------------------------------------------+ | name | DSCP_MARK_Policy | | rules | [{u'type': u'dscp_marking'] | +-------------+------------------------------------------+

  • penstack network qos policy show Min_BW_Policy

+-------------+------------------------------------------+ | Field | Value | +-------------+------------------------------------------+ | name | Min_BW_Policy | | rules | [{u'type': u'minimum_bandwidth'] | +-------------+------------------------------------------+

  • penstack port show 5cac029f-dbac-4c04-ab81-a6ab8faefc43 -c binding_vif_type -c id

+------------------+--------------------------------------+ | Field | Value | +------------------+--------------------------------------+ | binding_vif_type | ovs | | id | 5cac029f-dbac-4c04-ab81-a6ab8faefc43 | +------------------+--------------------------------------+

slide-26
SLIDE 26

26

Rules validation - new way - validation

  • penstack port set 5cac029f-dbac-4c04-ab81-a6ab8faefc43 --qos-policy Min_BW_Policy

ConflictException: 409

  • penstack port show 5cac029f-dbac-4c04-ab81-a6ab8faefc43 -c qos_policy_id -c id

+---------------+--------------------------------------+ | Field | Value | +---------------+--------------------------------------+ | id | 5cac029f-dbac-4c04-ab81-a6ab8faefc43 | | qos_policy_id | None | +---------------+--------------------------------------+

Apply QoS policy with minimum bandwidth rule (NOT supported by openvswitch driver) - Fails

slide-27
SLIDE 27

27

Rules validation - new way - validation

  • penstack port set 5cac029f-dbac-4c04-ab81-a6ab8faefc43 --qos-policy DSCP_MARK_Policy
  • penstack port show 5cac029f-dbac-4c04-ab81-a6ab8faefc43 -c qos_policy_id -c id

+---------------+--------------------------------------+ | Field | Value | +---------------+--------------------------------------+ | id | 5cac029f-dbac-4c04-ab81-a6ab8faefc43 | | qos_policy_id | 4d8cf13e-2a34-43a2-a642-53b5d4dd898d | +---------------+--------------------------------------+

Apply QoS policy with DSCP Marking rule (supported by

  • penvswitch driver) - OK
slide-28
SLIDE 28

New features in Queens release: bandwidth limit rules for floating IPs

slide-29
SLIDE 29

29

Overview

  • Only bandwidth limit rules for floating IPs
  • Both directions available: egress and ingress
  • Rules are implemented as tc filters applied to:

○ qg device in qr name space for legacy and HA routers ○ rfp device in qr name space for DVR local routers ○ qg device in snat name space for DVR edge routers

  • Filters are applied when floating ip is associated to a port
slide-30
SLIDE 30

30

Create egress rule

$ openstack network qos rule create --type bandwidth-limit --max-kbps 3000 --max-burst-kbits 2400 --egress Summit_Policy +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | direction | egress | | id | 152e43f2-cb81-431b-a0a5-c8b7ac060016 | | max_burst_kbps | 2400 | | max_kbps | 3000 | | name | None | | project_id | | +----------------+--------------------------------------+

slide-31
SLIDE 31

31

Create ingress rule

$ openstack network qos rule create --type bandwidth-limit --max-kbps 2100 --max-burst-kbits 1800 --ingress Summit_Policy +----------------+--------------------------------------+ | Field | Value | +----------------+--------------------------------------+ | direction | ingress | | id | cd1ac042-2e78-4565-9186-960b2ee026ad | | max_burst_kbps | 1800 | | max_kbps | 2100 | | name | None | | project_id | | +----------------+--------------------------------------+

slide-32
SLIDE 32

32

Create or update floating IP with QoS policy

$ openstack floating ip create --qos-policy Summit_Policy public +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | created_at | 2018-05-20T23:35:29Z | | description | | | dns_domain | | | dns_name | | | fixed_ip_address | None | | floating_ip_address | 172.24.4.8 | | floating_network_id | 90e59a26-8f24-4a93-b4d2-c496c2b56df5 | | id | 715bf74c-4f09-4bf2-9293-9fa9cde08716 | | name | 172.24.4.8 | | port_id | None | | project_id | d31cac1d60e34311bc47d557425fa24c | | qos_policy_id | 67293480-f179-4434-b239-eb5ae743caab | | revision_number | 1 | | router_id | None | | status | DOWN | | subnet_id | None | | tags | [] | | updated_at | 2018-05-20T23:35:29Z | +---------------------+--------------------------------------+

slide-33
SLIDE 33

33

After associating floating IP to VM port

$ sudo ip netns exec qrouter-4aa23079-dfe1-4250-a788-94777b4bb13d tc qdisc show dev \ > qg-e0da4660-c9 qdisc htb 1: root refcnt 2 r2q 10 default 0 direct_packets_stat 63 direct_qlen 1000 qdisc ingress ffff: parent ffff:fff1 ---------------- $ sudo ip netns exec qrouter-4aa23079-dfe1-4250-a788-94777b4bb13d tc -p -s -d filter show dev qg-e0da4660-c9 filter parent 1: protocol ip pref 1 u32 filter parent 1: protocol ip pref 1 u32 fh 800: ht divisor 1 filter parent 1: protocol ip pref 1 u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 match IP src 172.24.4.8/32 police 0x5 rate 3Mbit burst 300Kb mtu 2Kb action drop overhead 0b linklayer ethernet ref 1 bind 1 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

slide-34
SLIDE 34

34

After associating floating IP to VM port (cont.)

$ sudo ip netns exec qrouter-4aa23079-dfe1-4250-a788-94777b4bb13d tc -p -s -d filter show \ > dev qg-e0da4660-c9 parent ffff:fff1 prio 1 filter parent ffff: protocol ip u32 filter parent ffff: protocol ip u32 fh 800: ht divisor 1 filter parent ffff: protocol ip u32 fh 800::800 order 2048 key ht 800 bkt 0 flowid :1 match IP dst 172.24.4.8/32 police 0x4 rate 2100Kbit burst 225Kb mtu 2Kb action drop overhead 0b linklayer ethernet ref 1 bind 1 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)

slide-35
SLIDE 35

Planned new features: Strict minimum bandwidth support

slide-36
SLIDE 36

36

What do we mean by strict minimum bandwidth support?

  • Enforcement of minimum QoS policy minimum bandwidth

rules in the data plane ○ This can be done by Neutron alone ○ Egress minimum bandwidth rules are currently available for SR-IOV back-end ○ Plans to implement for ingress and for other back-ends: OVS, LB, etc.

slide-37
SLIDE 37

37

What do we mean by strict minimum bandwidth support? (cont.)

  • Avoid oversubscription of interfaces when scheduling VMs

and their ports in compute hosts ○ This will require cooperation between Neutron and Nova ○ Neutron will influence Nova’s scheduling decision ○ Nova will influence Neutron’s port binding process

slide-38
SLIDE 38

38

Limitation of first experimental release

  • Ports will have to be created and associated to QoS policy

by user and then provided in the VM boot request ○ This is because currently, when booting an instance, Nova creates its ports in the compute host, after the scheduling decision has been made ○ Plans exist to move port creation by Nova to the Conductor, i.e., before the scheduling decision

slide-39
SLIDE 39

39

Components of strict minimum bandwidth support

  • Neutron communicates to Nova the interfaces and

bandwidth inventory using the Placement API

  • User sets in Neutron the minimum bandwidth requirements
  • f a port
  • Neutron communicates to Nova the minimum bandwidth

requirements of a port

  • Nova communicates to Neutron the selected host and the

backend within that host

slide-40
SLIDE 40

40

Placement API concepts

Resource Provider

Source of resources like bandwidth

Inventories Traits

Non-consumable qualitative aspects

  • f resources

Egress bandwidth Ingress bandwidth VNIC type Physnet

slide-41
SLIDE 41

41

Placement data for each compute host

Compute RP

name = hostname

Agent RP

(OVS agent) uuid = agent_uuid Physical NIC RP uuid = uuid5(hostname:br0) traits: physnet_1, VNIC_TYPE_NORMAL inventory: {ingress_kilobits_per_sec: 10000, egrees_kilobits_per_sec: 10000} Physical NIC RP uuid = uuid5(hostname:br1) traits: physnet_2, VNIC_TYPE_NORMAL inventory: {ingress_kilobits_per_sec: 10000, egrees_kilobits_per_sec: 10000}

Agent RP

(SR-IOV agent) uuid = agent_uuid Physical NIC RP uuid = uuid5(hostname:br0) traits: physnet_3, VNIC_TYPE_DIRECT inventory: {ingress_kilobits_per_sec: 10000, egrees_kilobits_per_sec: 10000}

slide-42
SLIDE 42

Communicating to Nova interfaces and bandwidth inventories

ReST Neutron Server Placement Service Agent RPC Agent heartbeat ‘configurations’ with:

  • Bridge / interface mappings
  • Resource providers inventories
  • Resource providers traits

Config

slide-43
SLIDE 43
  • penstack network qos rule create \
  • -type minimum-bandwidth \
  • -min-kbps 512 \
  • -egress \

Summit_Policy +------------+--------------------------------------+ | Field | Value | +------------+--------------------------------------+ | direction | egress | | id | 5e5969f2-66f8-4bed-9905-b31b33902189 | | min_kbps | 512 | | name | None | +------------+--------------------------------------+

  • penstack port set --qos-policy Summit_Policy Summit_Port

43

User sets in Neutron the minimum bandwidth requirements of a port

slide-44
SLIDE 44

Neutron communicates to Nova the minimum bandwidth requirements of a port

{ “port”: { “status”: “ACTIVE”, “name”: “Summit_Port”, ... “device_id”: “5e3898d7-11be-483e-9732-b2f5eccd2b2e”, “resource_request": { "resources": { "NET_BANDWIDTH_INGRESS_KILOBITS_PER_SECOND": 1000, "NET_BANDWIDTH_EGRESS_KILOBITS_PER_SECOND": 1000, } "required": [ "CUSTOM_PHYSNET_NET0", "CUSTOM_VNIC_TYPE_NORMAL” ] } } }

slide-45
SLIDE 45

Nova communicates to Neutron the selected host and backend

{ “port”: { “binding:host_id”: “allinone”, “binding:profile”: { “allocation”: “rp uuid” } } }

slide-46
SLIDE 46

Questions?