Inferring and Debugging Path MTU Discovery Failures Matthew Luckie - - PowerPoint PPT Presentation

1

Inferring and Debugging Path MTU Discovery Failures

Matthew Luckie (U. Waikato) Kenjiro Cho (IIJ/WIDE) Bill Owens (NYSERNet)

2

The Problem

• It is desirable to send data in the fewest number of

packets possible

– Path MTU Discovery (PMTUD) – iterative process to determine the largest packet size (MTU) supported to a destination – uses feedback from ICMP Fragmentation Required / Packet Too Big (PTB) messages

• ICMP packets are not first class citizens
• PMTUD relies on these messages to work

– Unreliable at best – New PMTUD method from IETF on the way *

3

SYN ACK S Y N

• A

C K HTTP GET H T T P D a t a ICMP PTB TCP 3-Way Handshake Src Dst FW GW 1480 1500 * TU 1500 H T T P D a t a ICMP PTB

4

The Reverse-Path Problem

• A. Medina, M. Allman, S. Floyd. (2005)

Measuring the Evolution of Transport Protocols in the Internet

• 17% of 81776 targets failed at PMTUD

– 35% of 500 ‘popular’ websites failed – Did not find any which tried smaller packets – Assumed middle-boxes filtering ICMP

• 41% of targets did PMTUD successfully
• 30% did not attempt PMTUD

*

5

Contribution of this Work

• A forward-path PMTUD debugging tool

– infers the hop where large packets are discarded without the source receiving a PTB message – largest packet that can be forwarded through – uses a traceroute-like method

• A look at the problems we found when

measuring targets on networks which peer with the jumbo-capable Internet2

6

Debugging Technique: Stage 1 of 2

• Begin with a traceroute using small packets

– Infer the forward path – So we can later distinguish between all packets being silently discarded and just the large ones:

• Determine which hops will send ICMP feedback

(Time Exceeded) to small packets

• Ensure that packets can actually reach a destination

7

Debugging Technique: Stage 2 of 2

• Determine the Path-MTU

– start with the outgoing interface’s MTU – for each PTB message, reduce the working Path-MTU value until we reach the destination – we do an MTU search if

• large packets are silently discarded
• if we get a PTB message with a next-hop MTU of

zero or larger than the probe we sent

– we do a TTL search to infer the hop that we don’t receive a PTB message from

*

8

The MTU Search

• Define

– lower bound: largest packet to get a reply – upper bound: smallest packet to not get a reply

• In practice, a binary search is not suited

– MTU values tend to cluster around fairly limited numbers of media MTUs – Each probe that is silently discarded incurs two five- second timeouts (by default) – Cheaper to send a packet that gets ICMP feedback than one that does not – Use a table of MTU values to guide the search

• We use a number of heuristics to guide the search,

see the paper for complete coverage

9

Src Dst R1 R2 * R3 1500 1500 1480 1500 1. T T L 2 5 5 , S i z e 1 5 2. T T L 2 5 5 , S i z e 1 5 3. T T L 2 5 5 , S i z e 1 4 5 4 ICMP Port Unreach 4. 5. T T L 2 5 5 , S i z e 1 4 8 ICMP Port Unreach 6. 7. T T L 2 5 5 , S i z e 1 4 9 2 8. T T L 2 5 5 , S i z e 1 4 9 2 9. T T L 2 5 5 , S i z e 1 4 8 1 10. T T L 2 5 5 , S i z e 1 4 8 1 *

10

Src Dst R1 R3 1500 1500 1480 1500 11. T T L 2 , S i z e 1 5 13. T T L 3 , S i z e 1 5 * R2 * I C M P T i m e E x c e e d 12. 14. T T L 3 , S i z e 1 5

11

Methodology

• Two IPv4 hosts with 9000-byte MTU Interfaces

– connected to networks that peer with Internet2 – east.nysernet.org – nms1-chin.abilene.ucaid.edu

• 147 NLANR AMP targets

– all with 1500-byte MTU interfaces – vast majority are hosted on networks that peer with I2

• April 28th 2005, 21:50 EDT

12

Results

2 unique 2 (2) 2 (2) 2 (2) Incorrect PTB: 7 unique 4 (4) 5 (5) 6 (6) No ICMP Messages: 22 unique 13 (13) 27 (18) 26 (17) No PTB Messages: 7 unique 6 (6) 6 (6) 7 (7)

Target MTU Mismatch:

• 25

40 (30%) 41 (30%) Failures:

• 134

134 (91%) 136 (93%) Reachable:

• 147

147 147 Target Count: Total Intersection nms1-chin NYSERNet

The number on the left is the number of AMP targets on a path with this failure mode. The number in brackets is the number of unique failure points. *

13

Results: No ICMP Messages

• 7 failures (6 x 1500, 1 x 1536)
• Two were due to ingress filters

– one originated ICMP with 127.0.0.1 – another originated ICMP with RFC 1918

• Another due to an ‘Internet Free Zone’
• Another due to routing issue that allowed end-to-

end connectivity, but routers in the forward path had no route back to our source.

*

14

Results: No PTB Messages

• 22 hops sent TTL Expired, but no PTB messages

– 16 x 1500 – (4 x 4472, 2 x 4540, 1 x 4470, 1 x 2002)

• Some repetition in source of the problem, 20

distinct problem locations

– Obtained technical diagnosis for seven – Two were upgraded before diagnosis could be obtained

• Two main causes:

– no ip unreachables (does not suppress TTL Expired) – MTU Mismatches

15

Incorrect PTB Messages

• Two hops from one location sent a PTB

message with an incorrect next-hop MTU

– We sent 9000 byte probes – It said Packet Too Big: send 4586 byte packets – But the path to the next-hop could only carry packets up to 4472 bytes in size

• an MTU mismatch

16

Target MTU Mismatches

• We found 7 AMP machines were plugged

into a subnet with a router which forwarded packets larger than 1500 bytes

– An MTU mismatch with the router, as these machines (strictly speaking) can’t receive packets larger than 1500 bytes. – Two did: one managed 1506 bytes, another managed 2016 bytes.

17

An Anecdote

• A router in a commercial ISP in NYC sends

PTB messages with a next-hop MTU of 4470 bytes

– For all packets larger than 4458 bytes! – That’s a 12 byte discrepancy – Could be related to 3 4-byte MPLS labels being appended. – Could be mis-configuration – Could be a bug in the router

• would really like to know why

18

Acknowledgements

• Matt Zekauskas (Internet2) collected nms1-chin
• WIDE + CAIDA funded development of scamper
• NLANR/MNA is funded by NSF ANI-0129677

http://www.wand.net.nz/scamper/

19

MTU Mismatch Example

Router R1 Router R2

Jumbo Capable GigE Switch 9000 1500

• Router R1 thinks it can send 9000 byte packets to

R2, which can only receive 1500

• Drop happens at the switch, where no ICMP can

be sent.