TCP CONGESTION SIGNATURES Srikanth Sundaresan (Facebook) Amogh - - PowerPoint PPT Presentation

w w w . cai da.

• r

TCP CONGESTION SIGNATURES

1

Srikanth Sundaresan (Facebook) Amogh Dhamdhere (CAIDA/UCSD) kc Claffy (CAIDA/UCSD) Mark Allman (ICSI)

w w w . cai da.

• r

Typical Speed Tests Don’t Tell Us Much

2

w w w . cai da.

• r

Typical Speed Tests Don’t Tell Us Much

2

w w w . cai da.

• r

Typical Speed Tests Don’t Tell Us Much

2

w w w . cai da.

• r

Typical Speed Tests Don’t Tell Us Much

• Upload and download throughput measurements: no

information beyond that

2

w w w . cai da.

• r

Typical Speed Tests Don’t Tell Us Much

2

What type of congestion did the TCP flow experience?

w w w . cai da.

• r

Two Potential Sources of Congestion in the End-to-end Path

3

w w w . cai da.

• r

Two Potential Sources of Congestion in the End-to-end Path

• Self-induced congestion
• Clear path, the flow is able to saturate the bottleneck link
• eg: last-mile access link

3

w w w . cai da.

• r

Two Potential Sources of Congestion in the End-to-end Path

• Self-induced congestion
• Clear path, the flow is able to saturate the bottleneck link
• eg: last-mile access link
• External congestion
• Flow starts on an already congested path
• eg: congested interconnect

3

w w w . cai da.

• r

Two Potential Sources of Congestion in the End-to-end Path

• Self-induced congestion
• Clear path, the flow is able to saturate the bottleneck link
• eg: last-mile access link
• External congestion
• Flow starts on an already congested path
• eg: congested interconnect

3

Distinguishing the two cases has implications for users / ISPs / regulators

w w w . cai da.

• r

Does Throughput Indicate Type of Congestion?

• Cannot distinguish using just throughput numbers
• Access plan rates vary widely, and are typically not available to content /

speed test providers

• eg: Speed test reports 5 Mbps – is that the access link rate (DSL), or a

congested path?

4

w w w . cai da.

• r

Does Throughput Indicate Type of Congestion?

• Cannot distinguish using just throughput numbers
• Access plan rates vary widely, and are typically not available to content /

speed test providers

• eg: Speed test reports 5 Mbps – is that the access link rate (DSL), or a

congested path?

4

We can use the dynamics of TCP’s startup phase, i.e., Congestion Signatures

w w w . cai da.

• r

TCP’s RTT Congestion Signatures

5

w w w . cai da.

• r

TCP’s RTT Congestion Signatures

• Flows experiencing self-induced congestion fill up an

empty buffer during slow start

• Hence increase the TCP flow RTT

5

w w w . cai da.

• r

TCP’s RTT Congestion Signatures

• Flows experiencing self-induced congestion fill up an

empty buffer during slow start

• Hence increase the TCP flow RTT
• Externally congested flows encounter an already full buffer
• Less potential for RTT increases

5

w w w . cai da.

• r

TCP’s RTT Congestion Signatures

• Flows experiencing self-induced congestion fill up an

empty buffer during slow start

• Hence increase the TCP flow RTT
• Externally congested flows encounter an already full buffer
• Less potential for RTT increases
• Self-induced congestion therefore has higher RTT variance

compared to external congestion

5

w w w . cai da.

• r

TCP’s RTT Congestion Signatures

• Flows experiencing self-induced congestion fill up an

empty buffer during slow start

• Hence increase the TCP flow RTT
• Externally congested flows encounter an already full buffer
• Less potential for RTT increases
• Self-induced congestion therefore has higher RTT variance

compared to external congestion

5

We can quantify this using Max-Min and CoV of RTT

w w w . cai da.

• r

Example Controlled Experiment

• 20 Mbps “access” link

with 100 ms buffer

• 1 Gbps “interconnect”

link with 50 ms buffer

• Self-induced

congestion flows have higher values for both metrics and are clearly distinguishable

Max-Min RTT

6

CoV RTT

101 102 0.0 0.2 0.4 0.6 0.8 1.0

CDF External Self

10−2 10−1 100 0.0 0.2 0.4 0.6 0.8 1.0

CDF External Self

w w w . cai da.

• r

Example Controlled Experiment

• 20 Mbps “access” link

with 100 ms buffer

• 1 Gbps “interconnect”

link with 50 ms buffer

• Self-induced

congestion flows have higher values for both metrics and are clearly distinguishable

The two types of congestion exhibit widely contrasting behaviors

Max-Min RTT

6

CoV RTT

101 102 0.0 0.2 0.4 0.6 0.8 1.0

CDF External Self

10−2 10−1 100 0.0 0.2 0.4 0.6 0.8 1.0

CDF External Self

w w w . cai da.

• r

Model

• Max-min and CoV of RTT derived from RTT samples

during slow start

• We feed the two metrics into a simple Decision Tree
• We control the depth of the tree to a low value to minimize

complexity

• We build the decision tree classifier using controlled

experiments and apply it to real-world data

7

w w w . cai da.

• r

Validating the Method: Step 1- Controlled Experiments

8

Internet R2 R1

Server 1 Server 2 Server 3 Server 4 Pi 1 Pi 2 100 Mbps Shaped “access” 1 Gbps

w w w . cai da.

• r

Validating the Method: Step 1- Controlled Experiments

8

Internet R2 R1

Server 1 Server 2 Server 3 Server 4 Pi 1 Pi 2 100 Mbps Shaped “access” 1 Gbps

Background cross-traffic Interconnect cross-traffic

w w w . cai da.

• r

Validating the Method: Step 1- Controlled Experiments

8

Internet R2 R1

Server 1 Server 2 Server 3 Server 4 Pi 1 Pi 2 100 Mbps Shaped “access” 1 Gbps

Throughput tests Background cross-traffic Interconnect cross-traffic

w w w . cai da.

• r

Validating the Method: Step 1- Controlled Experiments

• Emulated “access” link + “core” link
• Wide range of access link throughputs, buffer sizes, loss rates, cross-

traffic (background and congestion-inducing)

• Can accurately label flows in training data as “self” or “externally”

congested

9

Internet R2 R1

Server 1 Server 2 Server 3 Server 4 Pi 1 Pi 2 100 Mbps Shaped “access” 1 Gbps

Throughput tests Background cross-traffic Interconnect cross-traffic

w w w . cai da.

• r

Validating the Method: Step 1- Controlled Experiments

10

Internet R2 R1

Server 1 Server 2 Server 3 Server 4 Pi 1 Pi 2 100 Mbps Shaped “access” 1 Gbps

Throughput tests Background cross-traffic Interconnect cross-traffic

High accuracy: precision and recall > 90% in most settings

w w w . cai da.

• r

Validating the Method: Step 2

• From Ark VP in ISP A identified congested link with ISP B using

TSLP*

11

*Dhamdhere et al. “Inferring Persistent Interdomain Congestion”, SIGCOMM 2018

ISP A ISP B

Ark VP

w w w . cai da.

• r

Validating the Method: Step 2

• From Ark VP in ISP A identified congested link with ISP B using

TSLP*

11

*Dhamdhere et al. “Inferring Persistent Interdomain Congestion”, SIGCOMM 2018

ISP A ISP B

Interdomain link

Ark VP

“near” side “far” side

w w w . cai da.

• r

Validating the Method: Step 2

• From Ark VP in ISP A identified congested link with ISP B using

TSLP*

11

*Dhamdhere et al. “Inferring Persistent Interdomain Congestion”, SIGCOMM 2018

ISP A ISP B

Interdomain link

Ark VP

Latency measurements to “near” and “far” side of interdomain link over time

“near” side “far” side

w w w . cai da.

• r

Validating the Method: Step 2

• From Ark VP in ISP A identified congested link with ISP B using

TSLP*

12

*Dhamdhere et al. “Inferring Persistent Interdomain Congestion”, SIGCOMM 2018

ISP A

Interdomain link

Ark VP

10 20 30 40 50 60 70 02/18 02/25 03/04 03/11 TSLP latency (far side)

“far” side “near” side

ISP B

w w w . cai da.

• r

Validating the Method: Step 2

• From Ark VP in ISP A identified congested link with ISP B using

TSLP*

12

*Dhamdhere et al. “Inferring Persistent Interdomain Congestion”, SIGCOMM 2018

ISP A

Interdomain link

Ark VP

10 20 30 40 50 60 70 02/18 02/25 03/04 03/11 TSLP latency (far side)

Diurnal latency elevation indicates congestion

“far” side “near” side

ISP B

w w w . cai da.

• r

Validating the Method: Step 2

13

ISP A ISP B

M-lab NDT server congested link

Ark VP

w w w . cai da.

• r

Validating the Method: Step 2

13

ISP A ISP B

M-lab NDT server congested link

Ark VP

Throughput measurements from Ark VP to M-lab NDT server traversing congested interdomain link

w w w . cai da.

• r

Validation of the Method: Step 2

14

5 10 15 20 25 30 02/18 02/25 03/04 03/11 d/l Mbps 10 20 30 40 50 60 70 02/18 02/25 03/04 03/11 TSLP latency (far side)

Strong correlation between throughput and TSLP latency: flows during elevated TSLP latency labeled as “externally” congested

w w w . cai da.

• r

Validation of the Method: Step 2

14

5 10 15 20 25 30 02/18 02/25 03/04 03/11 d/l Mbps 10 20 30 40 50 60 70 02/18 02/25 03/04 03/11 TSLP latency (far side)

Strong correlation between throughput and TSLP latency: flows during elevated TSLP latency labeled as “externally” congested “Externally” congested

w w w . cai da.

• r

Validation of the Method: Step 2

14

5 10 15 20 25 30 02/18 02/25 03/04 03/11 d/l Mbps 10 20 30 40 50 60 70 02/18 02/25 03/04 03/11 TSLP latency (far side)

Strong correlation between throughput and TSLP latency: flows during elevated TSLP latency labeled as “externally” congested “Externally” congested “self” congested

w w w . cai da.

• r

Validation of the Method: Step 2

15

5 10 15 20 25 30 02/18 02/25 03/04 03/11 d/l Mbps 10 20 30 40 50 60 70 02/18 02/25 03/04 03/11 TSLP latency (far side)

75%+ accuracy in detecting external congestion, 100% accuracy for self-induced congestion “Externally” congested “self” congested

w w w . cai da.

• r

Validation of the Method: Step 3

• M-lab’s NDT test data for real-world validation
• Cogent interconnect issue in late 2013/early 2014
• NDT tests to Cogent M-lab servers from several major U.S. ISPs saw

significantly lower throughput during peak hours: Comcast, TWC, Verizon

• Cox was notably not affected
• Underlying cause was congested interconnects

16

w w w . cai da.

• r

Using the M-lab Data

17

January 2014 April 2014

5 10 15 20

Hour of day (local)

10 20 30 40

Mbps

Comcast Cox TimeWarner Verizon 5 10 15 20

Hour of day (local)

10 20 30 40

Mbps

Comcast Cox TimeWarner Verizon

w w w . cai da.

• r

Using the M-lab Data

17

January 2014 Drop in peak-hour throughput for for Comcast, TWC, Verizon April 2014

5 10 15 20

Hour of day (local)

10 20 30 40

Mbps

Comcast Cox TimeWarner Verizon 5 10 15 20

Hour of day (local)

10 20 30 40

Mbps

Comcast Cox TimeWarner Verizon

w w w . cai da.

• r

Using the M-lab Data

17

January 2014 Cox not affected Drop in peak-hour throughput for for Comcast, TWC, Verizon April 2014

5 10 15 20

Hour of day (local)

10 20 30 40

Mbps

Comcast Cox TimeWarner Verizon 5 10 15 20

Hour of day (local)

10 20 30 40

Mbps

Comcast Cox TimeWarner Verizon

w w w . cai da.

• r

Using the M-lab Data

17

January 2014 Cox not affected Drop in peak-hour throughput for for Comcast, TWC, Verizon April 2014 Interconnection dispute resolved; no diurnal effect

5 10 15 20

Hour of day (local)

10 20 30 40

Mbps

Comcast Cox TimeWarner Verizon 5 10 15 20

Hour of day (local)

10 20 30 40

Mbps

Comcast Cox TimeWarner Verizon

w w w . cai da.

• r

Using the M-lab Data

18

Peak hour tests in Jan/Feb 2014 are likely “externally” congested Off-peak tests in Mar/Apr 2014 are likely “self” congested

5 10 15 20

Hour of day (local)

10 20 30 40

Mbps

Comcast Cox TimeWarner Verizon 5 10 15 20

Hour of day (local)

10 20 30 40

Mbps

Comcast Cox TimeWarner Verizon

w w w . cai da.

• r

Noisy Data

19

w w w . cai da.

• r

Noisy Data

• Difficult to infer interdomain congestion using throughput*

19

*Sundaresan et al. “Challenges in Inferring Interdomain Congestion using Throughput Measurements”, IMC 2017

w w w . cai da.

• r

Noisy Data

• Difficult to infer interdomain congestion using throughput*
• All tests labeled “external” may not have traversed

congested interconnects

19

*Sundaresan et al. “Challenges in Inferring Interdomain Congestion using Throughput Measurements”, IMC 2017

w w w . cai da.

• r

Noisy Data

• Difficult to infer interdomain congestion using throughput*
• All tests labeled “external” may not have traversed

congested interconnects

• We do not expect to identify all peak hour tests as

externally congested, and vice versa

• Looking for qualitative differences

19

*Sundaresan et al. “Challenges in Inferring Interdomain Congestion using Throughput Measurements”, IMC 2017

w w w . cai da.

• r

Applying the Model to M-lab data

20

Comcast TimeWarner VerizonCox Comcast TimeWarner VerizonCox Comcast TimeWarner VerizonCox 0.2 0.4 0.6 0.8 1.0

% self-induced congestion

Cogent (LAX) Cogent (LGA) Level3 (ATL)

Jan-Feb Mar-Apr

w w w . cai da.

• r

Applying the Model to M-lab data

20

Comcast TimeWarner VerizonCox Comcast TimeWarner VerizonCox Comcast TimeWarner VerizonCox 0.2 0.4 0.6 0.8 1.0

% self-induced congestion

Cogent (LAX) Cogent (LGA) Level3 (ATL)

Jan-Feb Mar-Apr

Much lower incidences of self-induced congestion for Cogent in Jan/Feb 2014 as compared to Mar/Apr

w w w . cai da.

• r

Applying the Model to M-lab data

21

Comcast TimeWarner VerizonCox Comcast TimeWarner VerizonCox Comcast TimeWarner VerizonCox 0.2 0.4 0.6 0.8 1.0

% self-induced congestion

Cogent (LAX) Cogent (LGA) Level3 (ATL)

Jan-Feb Mar-Apr

Level3 does not show significant differences, was not affected by interconnection disputes

w w w . cai da.

• r

Applying the Model to M-lab data

22

Comcast TimeWarner VerizonCox Comcast TimeWarner VerizonCox Comcast TimeWarner VerizonCox 0.2 0.4 0.6 0.8 1.0

% self-induced congestion

Cogent (LAX) Cogent (LGA) Level3 (ATL)

Jan-Feb Mar-Apr

Cox does not show significant differences, was not affected by interconnection disputes

w w w . cai da.

• r

In-band Measurements

• Can we leverage an ongoing TCP connection for path

measurements?

• e.g., where is the TCP flow bottlenecked? is the client’s

wireless access network the bottleneck? What is the capacity/available bandwidth of the path?

• Why in-band? No need to send external flows (which may

be treated differently than the application)

• TCP flow has already punched a hole in the NAT at the

client side

23

w w w . cai da.

• r

tcptrace

24

TCP

Client Server Firewall

w w w . cai da.

• r

tcptrace

24

tcp trace TCP

Client Server Firewall

w w w . cai da.

• r

tcptrace

24

tcp trace TCP

Client Server Firewall

w w w . cai da.

• r

tcptrace

24

tcp trace TCP

Client Server Firewall

In-band measurement packets

w w w . cai da.

• r

Tracetcp

• In-band high-frequency traceroute: injecting empty TTL-

limited packets in the ongoing TCP flow

• Ability to observe the buffer building up at bottleneck
• Can measure to the client past the NAT, and observe

wireless delays

• prototype at: https://github.com/ssundaresan/tracetcp

(ask me for access)

25

w w w . cai da.

• r

(more measurements in the works)

Tracetcp

• Packet-pair and packet-train techniques to measure per-

hop capacity and available bandwidth (in-band pathneck)

• Per-hop loss rate
• Main challenge: how to utilize packets from the TCP

stream, and smartly insert measurement packets without affecting the ongoing flow

26

w w w . cai da.

• r

Thanks! Questions? amogh@caida.org

27