SLIDE 1
Identifying Load-Balanced Backends
Ian Rodney
1
Why does it matter?
- Targeted DDoS
- Service degradation
- - -
2
Identifying Load-Balanced Backends Ian Rodney 1 Why does it - - PowerPoint PPT Presentation
Identifying Load-Balanced Backends Ian Rodney 1 Why does it - - PowerPoint PPT Presentation
Identifying Load-Balanced Backends Ian Rodney 1 Why does it matter? Targeted DDoS Service degradation - - - 2 Load Balancers Terminate & regenerate :( Pass through :) Hashing IP/Port - - - 3 Side Channels
SLIDE 2
SLIDE 3
Load Balancers
- Terminate & regenerate
:(
- Pass through
:)
- Hashing–IP/Port
3
- - -
SLIDE 4
Side Channels
- Information leaks
around shared state
- Well studied
- Setup:
- 4
SLIDE 5
IPID Mechanism
- Unique fragment ID
- 16-bit field in IPv4
- IPv6 Extension
- Counter types:
- Global
- Per-Destination
- Hybrid (2048 counters)
5
SLIDE 6
IPID Side Channel
- Global Counter
- Per-Dest
- Hybrid
6
- Covered in lecture!
SLIDE 7
IPID Side Channel
- Global Counter
- Per-Dest
- Hybrid
7
- Pretty hard to defeat
- But there is a way
SLIDE 8
IPID Side Channel
- Global Counter
- Per-Dest
- Hybrid
Xu 2018
Source: IPv6 Test; S/A RST, IPID: n Source: Victim; S/A RST, IPID: ? Source: IPv6 Test ; S/A RST, IPID: n+1
- r n+2
Global Counter with found IPv6 Address
10
SLIDE 9
Timestamps Mechanisms
Resolution Constant Drift
Kohno & Et. Al. 2005, Zander 2008, Rye 2019
HTTP 1Hz Yes ICMP 1kHz NTP adjusted TCP 1Hz – 1 kHz Yes TCP 1Hz – 1 kHz Yes ICMP 1kHz NTP adjusted HTTP 1Hz Yes
- Systems have a unique clock drift
- Is 1Hz too low?
12:00:00 12:00:01
11
NTP removes offset
SLIDE 10
Shared State Mechanisms
- Fragment reassembly buffer
- TCP SYN Cache
- Challenge ACK rate limit
13
SLIDE 11
Rate-Limit Mechanism
- Challenge ACK rate limit
- SYN or RST variants
RFC 5961
ACK: 1000 SEQ: 900 Challenge; ACK: 1000 99 left
14
RST: 1201 RST: 1000
SLIDE 12
Rate-Limit Mechanism
- Challenge ACK rate limit
- SYN or RST variants
RFC 5961
ACK: 1000 SEQ: 900 RST; SEQ: 1200 Challenge; ACK: 1000 RST; SEQ: 60,000 99 left
15
SLIDE 13
Rate-Limit Side Channel
- Infer presence of
connection
Cao & Et. Al 2016
- Seq: 1000
Source: User, SYN Seq: 500 Challenge ACK
99 left 0 left
16
SLIDE 14
Rate-Limit Side Channel
- Infer presence of
connection
Cao & Et. Al 2016
- Seq: 1000
Challenge ACK
99 left
Source: A, SYN
100x
0 left
Challenge ACK
99x
17
SLIDE 15
- Buffer
Side Channel
- Fragment buffer
& per-destination IPID
- subtle
Cao & Et. Al 2016
Source: A, Frag IPID: 20 Source: U, Frag U, IPID: 10 A, IPID: 80 Source: V, Full U, IPID: 20 A, IPID: 90
18
SLIDE 16
Source: A, New Frag
- Buffer
Side Channel
- Fragment buffer
& per-destination IPID
- subtle
Zhang 2018
U, IPID: 10 A, IPID: 80 A, IPID: 90 A, IPID: 100
19
SLIDE 17
Source: A, Remainders: 80, 90, 100
- Buffer
Side Channel
- Fragment buffer
& per-destination IPID
- subtle
Zhang 2018
U, IPID: 10 A, IPID: 80 A, IPID: 90 A, IPID: 100 Reply for 80, 90, 100
20
SLIDE 18
SYN Cache Side Channel
- Fill up cache (SYN cookies)
- Different source ports
Zhang & Et. Al 2015
- S/A
?
V, SYN S
- u
r c e : V SYN V, SYN
Exists: RST
21
SLIDE 19
SYN Cache Side Channel
- Fill up cache (SYN cookies)
- Different source ports
Zhang & Et. Al 2015
- V, SYN
Source: A SYN V, SYN SYN Cookie
22
SLIDE 20
How to leverage?
- IPID:
- Global --> straight forward
- Per-Dest/2048 --> impossible/hard
- Timestamps --> straight forward
- Shared State --> overwhelm and check
25
SLIDE 21
My contributions
- Check for side-channel presence
- Alexa Top 1000
26
SLIDE 22
My contributions
- Check for side-channel presence
- Alexa Top 1000
- ICMP/TCP/HTTP timestamps
- TCP traceroute (termination location)
27
SLIDE 23
Tools
- Scapy
- Raw pcaps
- Packet manipulation
- Requests
- HTTP
- Ray
- Distributed programming
(scanning)
28
SLIDE 24
Results
- 986 responses
- 98% had TCP responses
- 60% had TCP timestamps
- 85% had HTTP responses
- 0 ICMP
29
(a few)
SLIDE 25
Results
30
(a few)
SLIDE 26
Results
31
(a few)
SLIDE 27
Results
32
(a few)
…
ICMP TCP
SLIDE 28
Results
33
(a few)
…
ICMP TCP ICMP TCP
SLIDE 29
Results
34
(a few)
…
ICMP ICMP TCP
SLIDE 30
Results
35
(a few)
SLIDE 31
- Don't underestimate the kernel
Lessons Learned
36
NIC Kernel TCP Scapy
SLIDE 32
- Don't underestimate the kernel
- ISPs can be annoying
Lessons Learned
SLIDE 33
- Don't underestimate the kernel
- ISPs can be annoying
- I don't get IPv6
- Google IPv6 DNS + IPv6 ISP support = No connection?!
Lessons Learned
38
SLIDE 34
Experiments Next Steps
- Existence of Challenge ACKs
- IPv6 reachability
- HTTP timestamp analysis
39
SLIDE 35
Validation Next Steps
- Simple GCP Load Balanced Web Server
- Easy ground-truth
- In-the-wild validation
41
server: mw1325.eqiad.wmnet
SLIDE 36
Questions?
Thanks for listening!
42