1 Outline What is bufferbloat? Bufferbloat mitigation in - - PowerPoint PPT Presentation

1 outline
SMART_READER_LITE
LIVE PREVIEW

1 Outline What is bufferbloat? Bufferbloat mitigation in - - PowerPoint PPT Presentation

1 Outline What is bufferbloat? Bufferbloat mitigation in Linux Fixing bloat in the WiFi stack Airtime fairness and policy Status of mainline drivers Future / ongoing work - Toke Hiland-Jrgensen


slide-1
SLIDE 1

      1

slide-2
SLIDE 2

Outline

What is bufferbloat? Bufferbloat mitigation in Linux Fixing bloat in the WiFi stack Airtime fairness and policy Status of mainline drivers Future / ongoing work

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      2

slide-3
SLIDE 3

Slow internet sucks

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      3

slide-4
SLIDE 4

How does bufferbloat lead to slowness?

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      4

slide-5
SLIDE 5
  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      5

slide-6
SLIDE 6
  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      6

slide-7
SLIDE 7
  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      7

slide-8
SLIDE 8
  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      8

slide-9
SLIDE 9

This is bloat

10 20 30 40 50 60 70 Time (s) 200 400 600 800 Latency (ms)

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      9

slide-10
SLIDE 10

Fixing bufferbloat - AQM

Figure 3: A well designed servo system

Tank Bowl

Supply line Flus h valve Filler tube Inlet valve Float

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      10

slide-11
SLIDE 11

AQM - CoDel

k pkts

Timestamp

now - timestamp > 5ms?

CoDel

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      11

slide-12
SLIDE 12

Fixing bufferbloat - FQ

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      12

slide-13
SLIDE 13

FQ-CoDel sparse flow optimisation

New Old Empty

Arrival Empty or credits exhausted Empty Credits exhausted

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      13

slide-14
SLIDE 14

Results at the qdisc layer (wired links)

20 40 60 80 100 Mean induced latency (ms) 8.8 8.9 9.0 9.1 9.2 9.3 9.4 9.5 9.6 Mean TCP goodput (Mbit/s) pfifo_fast ared pie codel sfq fq_nocodel fq_codel

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      14

slide-15
SLIDE 15

A->A A->B A->C A->C B->C B->D A->A A->B A->C A->C B->C B->D A->A A->B A->C A->C B->C B->D A->A A->B A->C A->C B->C B->D A->A A->B A->C A->C B->C B->D 0.0 0.5 1.0 1.5 2.0 2.5 Mbits/s cake cake_dst cake_src cake_triple fq_codel

Pieces of CAKE

The sch_cake qdisc adds: Traffic shaping Host-based fairness queueing DiffServ handling TCP ACK filtering

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      15

slide-16
SLIDE 16

Important feature: Byte Queue Limits (BQL)

Queueing needs to happen where it can be managed There’s another queue in the hardware For Ethernet, Enabled in most drivers now Check your driver, it’s a Other technologies (DSL, etc.) need this! we have BQL pretty small patch

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      16

slide-17
SLIDE 17

What about WiFi?

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      17

slide-18
SLIDE 18

The qdisc approach doesn’t work for WiFi

50 100 150 200 250 300 350 Mean induced latency (ms) 10 20 30 40 50 60 Mean TCP goodput (Mbit/s) pfifo_fast pfifo_fast ared ared pie pie codel codel sfq sfq fq_nocodel fq_nocodel fq_codel fq_codel

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      18

slide-19
SLIDE 19

1000 *

Qdisc layer MAC layer ath9k driver

*Can be replaced with an arbitrary configuration

Per HW queue (x4) 2 aggr

FIFO

FIFO*

buf_q retry_q TID 123

Prio

buf_q retry_q

TID RR Assign TID

Retries To hardware 123

Prio

Qdisc layer (bypassed) MAC layer ath9k driver HW queue (x4)

2 aggr FIFO RR Assign TID Retries

To hardware

retry_q TID Prio Split flows 8192 (Global limit) retry_q TID

FQ-

CoDel

Prio Split flows 8192 (Global limit)

FQ-

CoDel

Fixing queueing in the WiFi stack

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      19

slide-20
SLIDE 20

Queueing latency impact

101 102 103 Latency (ms) 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative probability

With our solution Without our solution

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      20

slide-21
SLIDE 21

Throughput impact

FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime 10 20 30 40 Mbits/s Station 1 Station 2 Station 3 Average

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      21

slide-22
SLIDE 22

Airtime: fairness and policy

How can we control airtime usage of WiFi devices?

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      22

slide-23
SLIDE 23

802.11 Performance Anomaly

Effective transmission time and rate (for station ): Where is the effective rate of a station transmitting without collisions.

T(i) R(i) i ∈ I T(i) R(i) = ⎧ ⎩ ⎨ ⎪ ⎪ ⎪ ⎪

1 |I| T_data(i) ∑ _j∈IT_data(j)

with fairness

  • therwise

= T(i)R_0(i) R_0(i) =

L_i T_data(i)+T_oh

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      23

slide-24
SLIDE 24

Airtime fairness scheduler

Idea: Schedule stations to equalise airtime usage We already have per-station queueing The driver schedules stations anyway So we can simply account airtime using DRR scheduling First prototype: Change the ath9k driver

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      24

slide-25
SLIDE 25

Airtime fairness - achieved fairness (ath9k)

Fast 1 Fast 2 Slow Fast 1 Fast 2 Slow Fast 1 Fast 2 Slow Fast 1 Fast 2 Slow 0.0 0.2 0.4 0.6 0.8 1.0 Airtime share FIFO FQ-CoDel FQ-MAC Airtime fair FQ

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      25

slide-26
SLIDE 26

Airtime fairness - throughput benefit (ath9k)

FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime 10 20 30 40 Mbits/s Station 1 Station 2 Station 3 Average

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      26

slide-27
SLIDE 27

Kernel mac80211 subsystem Weighted DRR scheduler WiFi hardware Device driver Userspace Policy daemon Assoc / disassoc notifications Station queue state Set station weights Airtime usage Station

Weight Deficits

Station

Weight Deficits

Station

Weight Deficits User configuration Station state tracker

Beyond fairness: “Polifi”

Enable airtime policy enforcement, supporting: Prioritising single devices Use case, e.g.: “My TV needs more airtime” Balancing device groups Use case, e.g.: 5G network slicing Limiting groups to a maximum capacity share Use case, e.g.: Guest network

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      27

slide-28
SLIDE 28

Policy operating modes

Weights set by userspace daemon (hostapd), supporting three modes: Static mode: Specify MAC priority in config Dynamic mode: Specify weight per group (BSS) Limit mode: Like dynamic, but only limit some groups

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      28

slide-29
SLIDE 29

Policy enforcement effect

10 20 30 40 Time (s) 0.00 0.25 0.50 0.75 1.00 Airtime share 10 20 30 40 Time (s) 0.00 0.25 0.50 0.75 1.00 Airtime share

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      29

slide-30
SLIDE 30

Status in the mainline kernel

Everything presented here is upstream as of 5.1!

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      30

slide-31
SLIDE 31

Upstream feature support - kernel versions

Feature Version 3.3 3.5 4.19 4.10 4.11 5.1 BQL sch_fq_codel sch_cake WiFi queueing Airtime fairness (ath9k) Airtime fairness + policy

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      31

slide-32
SLIDE 32

Upstream feature support - WiFi drivers (5.1)

Driver WiFi-Q Airtime ath9k ✓ ✓ ath10k (✓) ✓ mt76 ✓ ✕ iwlwifi ✓ ✕

  • thers

✕ ✕ (✓): See caveat in following slides

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      32

slide-33
SLIDE 33

Future / ongoing work

Airtime Queue Limits Driver support Virtual time-based scheduler

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      33

slide-34
SLIDE 34

Airtime Queue Limits (AQL)

Just like with Ethernet drivers, there can be queueing in WiFi devices E.g., seconds of queueing in ath10k (hence the (✓)). For WiFi we know the packet duration from rate control AQL means limiting queueing based on airtime for ath10k with great effect Prototyped by Google

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      34

slide-35
SLIDE 35

Driver support

Driver support is partial. Efforts to fix this: Porting the full mac80211 TX path to TXQs A method to estimate airtime for devices w/o hw support This may tie into AQL support

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      35

slide-36
SLIDE 36

Virtual time-based scheduler

Current airtime scheduler is DRR-based Requires heavy throttling of stations to achieve fairness A virtual time-based scheduler doesn’t And may also allow MU-MIMO Prototyped in RFC patch

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      36

slide-37
SLIDE 37

Summary

Bufferbloat mitigation in Linux has come a long way! Very close to being “done” for WiFi Everything is upstream as of 5.1 Missing: AQL and full driver support Fix bufferbloat today - the network you save could be your own!

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      37

slide-38
SLIDE 38

References

Papers with more detail (from ): Bufferbloat qdisc comparison: Description of CAKE: WiFi queueing and fairness: Airtime policy: my PhD The Good, the Bad and the WiFi Piece of CAKE Ending the Anomaly PoliFi

  • Toke Høiland-Jørgensen <toke@toke.dk>
  • Toke Høiland-Jørgensen <toke@toke.dk>

      38