1
1 Outline What is bufferbloat? Bufferbloat mitigation in - - PowerPoint PPT Presentation
1 Outline What is bufferbloat? Bufferbloat mitigation in - - PowerPoint PPT Presentation
1 Outline What is bufferbloat? Bufferbloat mitigation in Linux Fixing bloat in the WiFi stack Airtime fairness and policy Status of mainline drivers Future / ongoing work - Toke Hiland-Jrgensen
Outline
What is bufferbloat? Bufferbloat mitigation in Linux Fixing bloat in the WiFi stack Airtime fairness and policy Status of mainline drivers Future / ongoing work
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
2
Slow internet sucks
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
3
How does bufferbloat lead to slowness?
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
4
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
5
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
6
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
7
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
8
This is bloat
10 20 30 40 50 60 70 Time (s) 200 400 600 800 Latency (ms)
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
9
Fixing bufferbloat - AQM
Figure 3: A well designed servo system
Tank Bowl
Supply line Flus h valve Filler tube Inlet valve Float
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
10
AQM - CoDel
k pkts
Timestamp
now - timestamp > 5ms?
CoDel
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
11
Fixing bufferbloat - FQ
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
12
FQ-CoDel sparse flow optimisation
New Old Empty
Arrival Empty or credits exhausted Empty Credits exhausted
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
13
Results at the qdisc layer (wired links)
20 40 60 80 100 Mean induced latency (ms) 8.8 8.9 9.0 9.1 9.2 9.3 9.4 9.5 9.6 Mean TCP goodput (Mbit/s) pfifo_fast ared pie codel sfq fq_nocodel fq_codel
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
14
A->A A->B A->C A->C B->C B->D A->A A->B A->C A->C B->C B->D A->A A->B A->C A->C B->C B->D A->A A->B A->C A->C B->C B->D A->A A->B A->C A->C B->C B->D 0.0 0.5 1.0 1.5 2.0 2.5 Mbits/s cake cake_dst cake_src cake_triple fq_codel
Pieces of CAKE
The sch_cake qdisc adds: Traffic shaping Host-based fairness queueing DiffServ handling TCP ACK filtering
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
15
Important feature: Byte Queue Limits (BQL)
Queueing needs to happen where it can be managed There’s another queue in the hardware For Ethernet, Enabled in most drivers now Check your driver, it’s a Other technologies (DSL, etc.) need this! we have BQL pretty small patch
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
16
What about WiFi?
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
17
The qdisc approach doesn’t work for WiFi
50 100 150 200 250 300 350 Mean induced latency (ms) 10 20 30 40 50 60 Mean TCP goodput (Mbit/s) pfifo_fast pfifo_fast ared ared pie pie codel codel sfq sfq fq_nocodel fq_nocodel fq_codel fq_codel
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
18
1000 *
Qdisc layer MAC layer ath9k driver
*Can be replaced with an arbitrary configuration
Per HW queue (x4) 2 aggr
FIFO
FIFO*
buf_q retry_q TID 123
Prio
buf_q retry_q
TID RR Assign TID
Retries To hardware 123
Prio
Qdisc layer (bypassed) MAC layer ath9k driver HW queue (x4)
2 aggr FIFO RR Assign TID Retries
To hardware
retry_q TID Prio Split flows 8192 (Global limit) retry_q TID
FQ-
CoDel
Prio Split flows 8192 (Global limit)
FQ-
CoDel
Fixing queueing in the WiFi stack
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
19
Queueing latency impact
101 102 103 Latency (ms) 0.0 0.2 0.4 0.6 0.8 1.0 Cumulative probability
With our solution Without our solution
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
20
Throughput impact
FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime 10 20 30 40 Mbits/s Station 1 Station 2 Station 3 Average
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
21
Airtime: fairness and policy
How can we control airtime usage of WiFi devices?
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
22
802.11 Performance Anomaly
Effective transmission time and rate (for station ): Where is the effective rate of a station transmitting without collisions.
T(i) R(i) i ∈ I T(i) R(i) = ⎧ ⎩ ⎨ ⎪ ⎪ ⎪ ⎪
1 |I| T_data(i) ∑ _j∈IT_data(j)
with fairness
- therwise
= T(i)R_0(i) R_0(i) =
L_i T_data(i)+T_oh
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
23
Airtime fairness scheduler
Idea: Schedule stations to equalise airtime usage We already have per-station queueing The driver schedules stations anyway So we can simply account airtime using DRR scheduling First prototype: Change the ath9k driver
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
24
Airtime fairness - achieved fairness (ath9k)
Fast 1 Fast 2 Slow Fast 1 Fast 2 Slow Fast 1 Fast 2 Slow Fast 1 Fast 2 Slow 0.0 0.2 0.4 0.6 0.8 1.0 Airtime share FIFO FQ-CoDel FQ-MAC Airtime fair FQ
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
25
Airtime fairness - throughput benefit (ath9k)
FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime FIFO FQ-CoDel FQ-MAC Airtime 10 20 30 40 Mbits/s Station 1 Station 2 Station 3 Average
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
26
Kernel mac80211 subsystem Weighted DRR scheduler WiFi hardware Device driver Userspace Policy daemon Assoc / disassoc notifications Station queue state Set station weights Airtime usage Station
Weight Deficits
Station
Weight Deficits
Station
Weight Deficits User configuration Station state tracker
Beyond fairness: “Polifi”
Enable airtime policy enforcement, supporting: Prioritising single devices Use case, e.g.: “My TV needs more airtime” Balancing device groups Use case, e.g.: 5G network slicing Limiting groups to a maximum capacity share Use case, e.g.: Guest network
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
27
Policy operating modes
Weights set by userspace daemon (hostapd), supporting three modes: Static mode: Specify MAC priority in config Dynamic mode: Specify weight per group (BSS) Limit mode: Like dynamic, but only limit some groups
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
28
Policy enforcement effect
10 20 30 40 Time (s) 0.00 0.25 0.50 0.75 1.00 Airtime share 10 20 30 40 Time (s) 0.00 0.25 0.50 0.75 1.00 Airtime share
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
29
Status in the mainline kernel
Everything presented here is upstream as of 5.1!
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
30
Upstream feature support - kernel versions
Feature Version 3.3 3.5 4.19 4.10 4.11 5.1 BQL sch_fq_codel sch_cake WiFi queueing Airtime fairness (ath9k) Airtime fairness + policy
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
31
Upstream feature support - WiFi drivers (5.1)
Driver WiFi-Q Airtime ath9k ✓ ✓ ath10k (✓) ✓ mt76 ✓ ✕ iwlwifi ✓ ✕
- thers
✕ ✕ (✓): See caveat in following slides
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
32
Future / ongoing work
Airtime Queue Limits Driver support Virtual time-based scheduler
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
33
Airtime Queue Limits (AQL)
Just like with Ethernet drivers, there can be queueing in WiFi devices E.g., seconds of queueing in ath10k (hence the (✓)). For WiFi we know the packet duration from rate control AQL means limiting queueing based on airtime for ath10k with great effect Prototyped by Google
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
34
Driver support
Driver support is partial. Efforts to fix this: Porting the full mac80211 TX path to TXQs A method to estimate airtime for devices w/o hw support This may tie into AQL support
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
35
Virtual time-based scheduler
Current airtime scheduler is DRR-based Requires heavy throttling of stations to achieve fairness A virtual time-based scheduler doesn’t And may also allow MU-MIMO Prototyped in RFC patch
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
36
Summary
Bufferbloat mitigation in Linux has come a long way! Very close to being “done” for WiFi Everything is upstream as of 5.1 Missing: AQL and full driver support Fix bufferbloat today - the network you save could be your own!
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
37
References
Papers with more detail (from ): Bufferbloat qdisc comparison: Description of CAKE: WiFi queueing and fairness: Airtime policy: my PhD The Good, the Bad and the WiFi Piece of CAKE Ending the Anomaly PoliFi
- Toke Høiland-Jørgensen <toke@toke.dk>
- Toke Høiland-Jørgensen <toke@toke.dk>
38