Perspective on Internet Congestion Control by Nathan Jay *, Noga H. - - PowerPoint PPT Presentation

▶
perspective on internet congestion
SMART_READER_LITE
LIVE PREVIEW

Perspective on Internet Congestion Control by Nathan Jay *, Noga H. - - PowerPoint PPT Presentation

Jun 28, 2023 186 likes •485 views

A Deep Reinforcement Learning Perspective on Internet Congestion Control by Nathan Jay *, Noga H. Rotman*, Brighten Godfrey, Michael Schapira, and Aviv Tamar *Equal contribution Internet Congestion Control The Internet (maybe?) End Host

slide-1
SLIDE 1

A Deep Reinforcement Learning Perspective on Internet Congestion Control

by Nathan Jay*, Noga H. Rotman*, Brighten Godfrey, Michael Schapira, and Aviv Tamar *Equal contribution

slide-2
SLIDE 2

Internet Congestion Control

End Host The Internet

(maybe?)

Server

slide-3
SLIDE 3

Internet Congestion Control

End Host Data The Internet

(maybe?)

t=1

Server

slide-4
SLIDE 4

Internet Congestion Control

End Host The Internet

(maybe?)

Server

slide-5
SLIDE 5

Internet Congestion Control

End Host The Internet

(maybe?)

Server Data

t=5.1

slide-6
SLIDE 6

Internet Congestion Control

End Host The Internet

(maybe?)

Server

t=5.2

Ack

slide-7
SLIDE 7

Internet Congestion Control

End Host The Internet

(maybe?)

Server

slide-8
SLIDE 8

Internet Congestion Control

End Host The Internet

(maybe?)

t=10.2

Ack Server

slide-9
SLIDE 9

Internet Congestion Control

End Host Data The Internet

(maybe?)

t=10.2

Ack

t=1

Server Data

t=5.1 t=5.2

Ack

slide-10
SLIDE 10

Internet Congestion Control

Latency Trace of Internet Path* Latency Time Latency

*from pantheon.stanford.edu

slide-11
SLIDE 11

Internet Congestion Control

Latency Trace of Internet Path* Latency Time Latency

*from pantheon.stanford.edu

slide-12
SLIDE 12

Internet Congestion Control

Latency Trace of Internet Path* Latency Time Latency

*from pantheon.stanford.edu

slide-13
SLIDE 13

Internet Congestion Control

Latency Trace of Internet Path* Latency Time Latency

*from pantheon.stanford.edu

Underlying Complexity:

  • Enormous, dynamic network
  • Massive agent churn
  • Very little information

~80,000 agents/second

slide-14
SLIDE 14

Revisiting Congestion Control

Congestion Control Timeline

1988 2016 2019 Flavors of TCP Congestion Control

(Tahoe, Reno, Cubic, Illinois, Vegas, …)

  • Same network model
  • Same action space
  • Slightly different control algorithms
slide-15
SLIDE 15

Revisiting Congestion Control

Congestion Control Timeline

1988 2016 2019 Flavors of TCP Congestion Control

(Tahoe, Reno, Cubic, Illinois, Vegas, …)

  • Same network model
  • Same action space
  • Slightly different control algorithms

Introduction of QUIC, replaces significant amount of Google traffic.

  • New models
  • New action space (packet pacing added to Linux)
  • Novel control algorithms and research (BBR, Copa, PCC)
slide-16
SLIDE 16

Reward-based architecture: PCC

Observations

Test Rates

Network

Performance Statistics

Monitor Interval Input Features:

1.

Send Ratio

2.

  • Lat. Ratio

3.

Lat. Inflation

slide-17
SLIDE 17

Reward-based architecture: PCC

Actions Observations

Test Rates

Network

Performance Statistics

Monitor Interval Input Features:

1.

Send Ratio

2.

  • Lat. Ratio

3.

Lat. Inflation

slide-18
SLIDE 18

Agent Architecture

Monitor Interval Send Rate Utility Throughput Latency Latency Inflation Loss Rate Monitor Interval Send Rate Utility Throughput Latency Latency Inflation Loss Rate Monitor Interval Send Rate Utility Throughput Latency Latency Inflation Loss Rate Monitor Interval Input Features: 1. Send Ratio 2.

  • Lat. Ratio

3. Lat. Inflation History Length Rate Change Factor

𝛽

New Rate = 𝛽 > 0: Old Rate x (1 + w𝛽)

𝛽 < 0: Old Rate / (1 - w𝛽)

3-Layer NN

slide-19
SLIDE 19

Agent Architecture

Monitor Interval Send Rate Utility Throughput Latency Latency Inflation Loss Rate Monitor Interval Send Rate Utility Throughput Latency Latency Inflation Loss Rate Monitor Interval Send Rate Utility Throughput Latency Latency Inflation Loss Rate Monitor Interval Input Features: 1. Send Ratio 2.

  • Lat. Ratio

3. Lat. Inflation History Length Rate Change Factor

𝛽

New Rate = 𝛽 > 0: Old Rate x (1 + w𝛽)

𝛽 < 0: Old Rate / (1 - w𝛽)

3-Layer NN Key Design Choice: Scale-free

  • bservations affect robustness
slide-20
SLIDE 20

Training/Testing Environment

Training Environment:

  • Simulated network
  • Each episode chooses link

parameters from a range:

  • Standard gym at

github.com/PCCProject/PCC-RL

Capacity Latency Loss Queue 1 - 6mbps 50 - 500ms 0 - 5% 1 - ~3000pkt

slide-21
SLIDE 21

Training/Testing Environment

Training Environment:

  • Simulated network
  • Each episode chooses link

parameters from a range:

  • Standard gym at

github.com/PCCProject/PCC-RL

Capacity Latency Loss Queue 1 - 6mbps 50 - 500ms 0 - 5% 1 - ~3000pkt

Testing Environment:

  • Real packets in Linux kernel

network emulation

  • Much wider testing range:

Capacity Latency Loss Queue 1 - 128mbps 1 - 512ms 0 - 20% 1 - 10000pkt

slide-22
SLIDE 22

State-of-the-art Results

Test Description:

  • Emulated network, with real

Linux kernel noise

  • Time-varying link

Emulated Dynamic Link Performance

slide-23
SLIDE 23

State-of-the-art Results

Test Description:

  • Emulated network, with real

Linux kernel noise

  • Time-varying link

Emulated Dynamic Link Performance

slide-24
SLIDE 24

State-of-the-art Results

Test Description:

  • Emulated network, with real

Linux kernel noise

  • Time-varying link

Emulated Dynamic Link Performance

slide-25
SLIDE 25

State-of-the-art Results

Test Description:

  • Emulated network, with real

Linux kernel noise

  • Time-varying link

Emulated Dynamic Link Performance

slide-26
SLIDE 26

State-of-the-art Results

Test Description:

  • Emulated network, with real

Linux kernel noise

  • Time-varying link

Emulated Dynamic Link Performance Aurora is on the Pareto front of state-of-the-art algorithms

slide-27
SLIDE 27

Exciting Directions

  • Multi-agent scenarios:

○ Cooperative ○ Selfish

  • Online training:

○ Few-shot training ○ Meta-learning

  • Multi-objective Learning:

○ File transfer ○ Live video

By The Opte Project - Originally from the English Wikipedia; description page is/was here., CC BY 2.5, https://commons.wikimedia.org/w/index.php?curid=1538544

slide-28
SLIDE 28

See us at:

Poster #45 6:30pm - 9:00pm Pacific Ballroom Code available at github.com/PCCProject/PCC-RL