Characterizing IPv4 Anycast Adoptjon and Deployment Dario Rossi - - PowerPoint PPT Presentation

Characterizing IPv4 Anycast Adoptjon and Deployment

Dario Rossi Professor dario.rossi@telecom-paristech.fr htup://www.enst.fr/~drossi/anycast Joint work with D.Cicalese, J. Auge, D. Joumblatu and T. Friedman

A seminal work [4] at NANOG’59 investjgates who are the IP anycasters. Our focus is instead on where they are

12

Demo shortlink: goo.gl/Ff8gdQ

Talk Teaser

Note: everything you see in this talk is available as open source at htup://www.telecom-paristech.fr/~drossi/anycast Note: everything you see in this talk is available as open source at htup://www.telecom-paristech.fr/~drossi/anycast

• Preliminaries

– Motjvatjon, defjnitjon & state of the art

• Background (wrt the anrp paper)

– Recall on Latency-based anycast geolocatjon technique [1]

• Foreground (the anrp paper)

– IPv4 anycast censuses [2] – Demo, source code, ground truth and more [3]

• Ongoing

– Study infrastructure evolutjon & usage – Applicatjon to BGP hijack detectjon

Outline

3

[1] D. Cicalese et al. A Fistgul of Pings: Accurate and Lightweight Anycast Enumeratjon and Geolocatjon, IEEE INFOCOM, Apr 2015. [2] D. Cicalese et al. Characterizing IPv4 Anycast Adoptjon and Deployment, ACM CoNEXT, Dec 2015. [3] htup://www.telecom-paristech.fr/~drossi/anycast

Motjvatjons

O(100ms) O(10ms)

400ms delay , 0.7% less searches (=less ads (=less $)) : ms (2sec) reduces by 1.2% (4.8%) 5sec speedup, +25% visit, +12% revenue

Motjvatjons

O(10ms) O(10ms)

Applicatjon-layer anycast

• How it works
• Relies on IP unicast
• Server selectjon via DNS

redirectjon, URL rewritjng

• Pros
• Ability to specify selectjon criteria
• Fine-grained control over server

load

• Maintains connectjon affjnity
• Fast failover
• Cons
• Ad hoc, per service confjguratjon
• Overhead to collect selectjon

metrics

• E.g. List of servers up, latency, load

DNS C1 X1 X2 X3 x.com? X1 IP C2 C3

IP-layer anycast

• How it works
• Shared IP address for replicate

servers

• BGP Prefjx advertjsed from

multjple points of origin

• Pros
• Natjvely supported by IP
• Transparent to upper layers
• Visibility of servers

(Global vs Local)

• Cons
• Lack of fjne-grained control
• Destjnatjon determined by

IP routjng metrics

• Prone to prefjx hijacking
• No guarantees of server affjnity

(e.g., connectjon-oriented services)

C1 X X X C2 C3 BGP announces IP forwarding

Our focus Our focus

• Problem statement: where are the anycast instances?

– E.g., Google 8.8.8.8 or CloudFlare, or EdgeCast or root servers, etc. Mountain View, CA (IP2Locatjon) New York, NY (Geobytes) United States (Maxmind)

?????

1ms ≈ 100Km

2 m s f r

• m

E U t

• U

S ? ? ? ? ? S p e e d

• f

l i g h t violaton ! !

Tools using distributed measurement aren’t betuer ! Single (tme varying) answer Unknown accuracy

Commercial databases Distributed measurement

8

Anycast geolocatjon

ICMP packets are ~3x slower than lightspeed

2 m s f r

• m

E U t

• U

S ? ? ? ? ? S p e e d

• f

l i g h t violaton ! ! Distributed measurement

• Problem statement: where are the anycast instances?

– E.g., Google 8.8.8.8 or CloudFlare, or EdgeCast or root servers, etc.

• Solutjon

– Leverage inconsistency

• f latency measurement

– This was used in NANOG’59 to detect who are the anycasters – We raise this to the next level and geolocate where they are

• Propertjes

– Lightweight: few pings – Protocol agnostc: ICMP probes – Accurate against ground truth – Fast: greedy, but as good as costly optjmum solutjon

9

Outline Background Part I: iGreedy Part II: Census Ongoing work Summary

Anycast geolocatjon

Tools using distributed measurement aren’t betuer ! But they could!

Hey, that must be anycast! Hey, that must be anycast!

Related work

Main difgerentators

• Background fjrst lightweight and protocol-agnostjc technique able to detect,

enumerate and geolocate anycast instances

• Foreground fjrst large-scale (several IPv4 censuses) geolocatjon of anycast

instances

10

BG FG FG

all IPv4

Outline Background Part I: iGreedy Part II: Census Ongoing work Summary

PlanetLab (+ RIPE Atlas)

– Broad: apply iGreedy to all IPv4 / 24 for each census – Costly: 3 Billions RIPE Atlas credits, per census; combine PlanetLab and RIPE Atlas (backup slides) – Detailed: complementary nmap portscan of detected anycast replicas

Census iGreedy

11

Workfmow

Outline Background Part I: iGreedy Part II: Census Ongoing work Summary

Census iGreedy PlanetLab and RIPE Atlas

– Lightweight: O(1) pings per target per vantage point – DNS: validate with RIPE Atlas DNS CHAOS ground truth

– PlanetLab

– CDN: ground truth with new protocol- specifjc technique (HTTP headers; see paper) – Protocols: ICMP, DNS/UDP, DNS/TCP, HTTP/TCP, etc.

iGreedy

iGreedy overview

Measure Latency

Planetlab/RIPE Atlas

Measure Latency

Planetlab/RIPE Atlas

Enumerate Solve MIS

Brute force (Optjmum) Greedy (5-approx)

Enumerate Solve MIS

Brute force (Optjmum) Greedy (5-approx)

Geolocate Classifjcatjon

Maximum likelihood pick the largest city

Geolocate Classifjcatjon

Maximum likelihood pick the largest city

Detect Speed of light violatjons Detect Speed of light violatjons

Scalability Infrastructure Lightweight

Iterate

Feedback Filter latency noise Measure Detect Enumerate Geolocate Iterate

12

Outline Background Part I: iGreedy Part II: Census Ongoing work Summary

Increase recall

iGreedy overview

Measure Latency

Planetlab/RIPE Atlas

Measure Latency

Planetlab/RIPE Atlas

Enumerate Solve MIS

Brute force (Optjmum) Greedy (5-approx)

Enumerate Solve MIS

Brute force (Optjmum) Greedy (5-approx)

Geolocate Classifjcatjon

Maximum likelihood pick the largest city

Geolocate Classifjcatjon

Maximum likelihood pick the largest city

Detect Speed of light violatjons Detect Speed of light violatjons

Scalability Infrastructure Lightweight

Iterate

Feedback Filter latency noise Measure Detect Enumerate Geolocate Iterate

13

Increase recall

1ms ≈ 100Km 10% disks < 1000Km

Median latency stretch

• ver speed-of-light Internet

Importance of side-channel infos Paris-Madrid 1053Km

iGreedy performance

14

• At a glance

Accurate enumeratjon over 75% recall Precise geolocatjon over 75% true positjves Protocol agnostjc DSN and CDN, etc. Lightweight 100x less probes than previous work Fast O(100ms) greedy vs O(1000sec) for brute force Ready Open source code [3] to measure, analyze and map (demo if tjme allows later) Reliable Open dataset [3] along the code, with latency measurement & ground truth

Census agenda

• Census preliminaries

– Confjrms iGreedy works for non-DNS services – Select a probing protocol and rate – Optjmize data storage and processing – Infrastructure consideratjon (PlanetLab vs RIPE) – Scale up the iGreedy technique (& be gentle)

• Census results

– At a glance: Geographic heatmap – Big fjshes: Top-50 deployments – Services: Complementary port-scan – Web interface: What’s available

Backup slides

Anatomy of an IPv4 census

Typical workfmow

1

• O(107) likely alive targets x O(102) actjve probes with ICMP
• O(103) targets/sensor/second , 1 census = few hours
• O(107) runs of iGreedy later….
• O(103) targets are anycast – proverbial needle in the IPv4 haystack

Anatomy of an IPv4 census

Typical workfmow

1

• O(107) likely alive targets x O(102) actjve probes with ICMP
• O(103) targets/sensor/second , 1 census = few hours
• O(107) runs of iGreedy later….
• O(103) targets are anycast – proverbial needle in the IPv4 haystack

IPv4/32 census by ANT/ISI

Anatomy of an IPv4 census

Typical workfmow

1

• O(107) likely alive targets x O(102) actve probes with ICMP
• O(103) targets/sensor/second , 1 census = few hours
• O(107) runs of iGreedy later….
• O(103) targets are anycast – proverbial needle in the IPv4 haystack

Administratjvely prohibited /24 are probed only by 1 VP Avoid

• verload

targets Greylist

Anatomy of an IPv4 census

Typical workfmow

1

• O(107) likely alive targets x O(102) actjve probes with ICMP
• O(103) targets/sensor/second , 1 census = few hours
• O(107) runs of iGreedy later….
• O(103) targets are anycast – proverbial needle in the IPv4 haystack

Service agnostjc: Higher recall Results match with difgerent protocols

Anatomy of an IPv4 census

Typical workfmow

2

• O(107) likely alive targets x O(102) actjve probes with ICMP
• O(103) targets/sensor/second , 1 census = few hours
• O(107) runs of iGreedy later….
• O(103) targets are anycast – proverbial needle in the IPv4 haystack

Avoid VP

• verload

/fjrewalls

Anatomy of an IPv4 census

Typical workfmow

2

• O(107) likely alive targets x O(102) actjve probes with ICMP
• O(103) targets/sensor/second , 1 census = few hours
• O(107) runs of iGreedy later….
• O(103) targets are anycast – proverbial needle in the IPv4 haystack

Avoid VP

• verload

/fjrewalls

• Our tool can scan more

than 10k hosts/sec/VP

• We maximize interarrival

tjme between ICMP echo request from difgerent VPs to the same target

• However, ICMP echo replies

aggregate at the VP, that receives 10k replies/sec

VP

• Some PlanetLab nodes

are fjrewalled– they drop ICMP reply traffjc

• Experimentally,

no problem with 1k req/sec

Anatomy of an IPv4 census

Typical workfmow

2

• O(107) likely alive targets x O(102) actjve probes with ICMP
• O(103) targets/sensor/second , 1 census = few hours
• O(107) runs of iGreedy later….
• O(103) targets are anycast – proverbial needle in the IPv4 haystack

100ms /run => 1wk Optjmized to 3hrs

Anatomy of an IPv4 census

Typical workfmow

2

• O(107) likely alive targets x O(102) actjve probes with ICMP
• O(103) targets/sensor/second , 1 census = few hours
• O(107) runs of iGreedy later….
• O(103) targets are anycast – proverbial needle in the IPv4 haystack

Censuses at a glance (1/2)

Conext censuses

• List of IP/24 with > 5 replicas
• Googlemap interface

PlanetLab vs RIPE Atlas

• PlanetLab: Multjple censuses,

difgerent protocols

• RIPE: Ameliorate coverage,

validate small deployments

IP/24 ASes City Country Replicas 897 100 71 38 11,598 RIPE Atlas PlanetLab

Monthly censuses Data since Dec 2015 Exported since Mai 2016

Censuses at a glance (2/2)

(Top 100 in the paper)

Top-50 IPv4 anycast deployments

26

Big fjshes! Edgecast CloudFlare Google Yahoo Microsof OVH Amazon ATT Sprint Level3 Linkedin Facebook Verisign Prolexic

Silver needle in the IPv4 haystack!

Heterogeneous IP Footprint Important ASes Popular Web content Service diversity

27

IPv4 anycast sofware census

!! ++

• Nmap census

– Stealthy scan, all ports, 1 IP per each anycast /24, from 1VP – Not only DNS! Lots of CDN/Web (++), but also e.g., Gmail & SSH (??) or >10,000 open ports on OVH (!!)

(1/2) Infrastructure evolutjon & usage

• Time evolutjon of single deployment

– Orthogonal to spatjal dimension of census – Example from historic RIPE Atlas data

28

• Anycast usage across deployments

– Orthogonal to actjve measurement – Use passive measurement at some ISP point of presence – Cross check with IP/24 anycast list

• IP anycast
• Syntactjcally equivalent to a

BGP hijack in the BGP lingo

• Only difgerence: router authorized

to advertjse the prefjx or not

• iGreedy for BGP hijack detectjon

29

(2/2) BGP hijack detectjon

Credits: renesys

Reactve scan on BGP announces

• Analyse BGP feed (eg BGPstream)

and issue iGreedy on suspicion

Problem

• BGP Hijacks are of short duratjon
• Control plane informatjon may

arrive late at some monitor

Proactve Internet-wide scan

• Scan all /24 prefjxes every ~minute

(or an IPv4 subset with opt-in/opt-out)

Problem

• Over 100x faster than current speed

(but detectjon easier than geolocatjon)

• More challenging, hence more fun!

Faculty Research Award

• IP anycast
• Syntactjcally equivalent to a

BGP hijack in the BGP lingo

• Only difgerence: router authorized

to advertjse the prefjx or not

• iGreedy for BGP hijack detectjon

30

(2/2) BGP hijack detectjon

Credits: renesys

Reactve scan on BGP announces

• Analyse BGP feed (eg BGPstream)

and issue iGreedy on suspicion

Problem

• BGP Hijacks are of short duratjon
• Control plane informatjon may

arrive late at some monitor

Proactve Internet-wide scan

• Scan all /24 prefjxes every ~minute

(or an IPv4 subset with opt-in/opt-out)

Problem

• Over 100x faster than current speed

(but detectjon easier than geolocatjon)

• More challenging, hence more fun!

Faculty Research Award Winner of the CAIDA Hackaton 2016 (HIJACK 2)

• IP anycast
• Syntactjcally equivalent to a

BGP hijack in the BGP lingo

• Only difgerence: router authorized

to advertjse the prefjx or not

• iGreedy for BGP hijack detectjon

31

(2/2) BGP hijack detectjon

Credits: renesys

Reactve scan on BGP announces

• Analyse BGP feed (eg BGPstream)

and issue iGreedy on suspicion

Problem

• BGP Hijacks are of short duratjon
• Control plane informatjon may

arrive late at some monitor

Proactve Internet-wide scan

• Scan all /24 prefjxes every ~minute

(or an IPv4 subset with opt-in/opt-out)

Problem

• Over 100x faster than current speed

(but detectjon easier than geolocatjon)

• More challenging, hence more fun!

Faculty Research Award Credits: renesys

Service soon available

store the last hour worth of censuses on demand query, in case of suspicion

Conclusions

• Perform IPv4 anycast censuses with iGreedy

– lightweight, fast and protocol agnostjc technique, implemented in open-source sofware to measure, analyze and display

• Census engineering

– Measure and analyze O(107) targets from O(102) VPs in bounded tjme, with low intrusiveness

• Census results

– Find 10,000 replicas for 1000 IP/24 spread

• ver 100 ASes, located in over 70 citjes of 30 countries

– Big fjshes, hostjng popular web content, with heterogeneous footprint and remarkable service diversity

• Montly censuses (and more)

– Avail at htup://www. telecom-paristech.fr/~drossi/anycast

32

Outline Background Part I: iGreedy Part II: Census Ongoing work Summary

References

This talk: [1] D. Cicalese , D. Joumblatu, D. Rossi, J, Auge, M.O Buob, T. Friedman. A Fistgul of Pings: Accurate and Lightweight Anycast Enumeraton and Geolocaton , IEEE INFOCOM, 2015 [2] D. Cicalese , J. Auge, D. Joumblatu, T. Friedman, D. Rossi, Characterizing IPv4 Anycast Adopton and Deployment , ACM CoNEXT, Dec 2015 [3] htup://www.telecom-paristech.fr/~drossi/anycast Related:

33

Outline Background Part I: iGreedy Part II: Census Ongoing work Summary

?? || //

34

BACKUP SLIDES

35

iGreedy peformance (details/robustness/solver) Measurement campaigns (overview/statjstjcs) Geolocatjon Validatjon for CDN Census duratjon Probing rate Protocol impact Infrastructure impact Vantage points impact

iGreedy performance

36

Accurate enumeratjon over 75% recall Precise geolocatjon over 75% true positjves Protocol agnostjc DSN and CDN, etc. Lightweight 100x less probes than previous work Consistent across measurement infrastructure Robust in spite of very noisy latency measurements! (and some odd VP geolocatjon)

iGreedy performance: robustness

37 37

Not even need for fjltering large disks, as iGreedy sorts disk by increasing size, bad disks implicitly fjltered out in the solutjon!! Not even need for fjltering large disks, as iGreedy sorts disk by increasing size, bad disks implicitly fjltered out in the solutjon!!

iGreedy performance: MIS solver

38

• MIS performance:

– In theory: Greedy = 5x-approximatjon of global optjmum – In practjce: Greedy solutjon ≈ Brute Force solutjon – Iteratjon introduces a signifjcant benefjt – O(100ms) greedy vs O(1000sec) brute force (for ~300 nodes) In practjce, greedy is good enough In practjce, greedy is good enough

Measurement campaigns

39

Experimental results in [JSAC’16] gathered with open source sofware and dataset Avail in the igreedy-v1.0 sofware at htup://www.telecom-paristech.fr/~drossi/anycast

Measurement campaign (1/2)

40

Very noisy delay measurements. Only 10% of disks are smaller than 1000km !! Very noisy delay measurements. Only 10% of disks are smaller than 1000km !!

Measurement campaigns (2/2)

41

Delay informatjon: useful for enumeratjon, bad for geolocatjon !! Delay informatjon: useful for enumeratjon, bad for geolocatjon !! In 90% of the cases, over 100 citjes in a disk In 90% of the cases, over 100 citjes in a disk Picking one city at random: less than 1/100 success in 90% of disks Picking one city at random: less than 1/100 success in 90% of disks

Validatjon for CDN

42

New technique for CDN ground truth HTTP HEAD request, manual inspectjon. CDNs encode IATA locatjons in headers

• Server: (Edgecast)
• CF-RAY: (Cloudfmare)

New technique for CDN ground truth HTTP HEAD request, manual inspectjon. CDNs encode IATA locatjons in headers

• Server: (Edgecast)
• CF-RAY: (Cloudfmare)

Geolocatjon

p

i = α

c

i

c

j

j

∑

+ (1−α) d

i

d

j

j

∑

• Classifjcatjon task

– Map each disk Dp to most likely city – Compute likelihood (p) of each city in disk based on:

• ci: Populatjon of city i
• Ai: Locatjon of ATA airport of city i
• d(x,y): Geodesic distance
• α: city vs distance weightjng
• Output policy
• Proportonal: Return all citjes in Dp with

respectjve likelihoods

• Argmax: Pick city with highest likelihood

Frankfurt p=0.30 Zurich p=0.10 Munich p=0.60 Munich

Ratjonale: users lives in densely populated area; to serve users, servers are placed close to citjes In practjce, pick the largest city is best ! (Argmax with α=1) In practjce, pick the largest city is best ! (Argmax with α=1)

Census duratjon

44

Optjmize storage (over 10x) Optjmize processing (have to open 300 fjles in parallel for sortjng due to LFSR orded to avoid overloading destjnatjon) Optjmize storage (over 10x) Optjmize processing (have to open 300 fjles in parallel for sortjng due to LFSR orded to avoid overloading destjnatjon) This is OLD. New censuses (afer Dec 2015) are faster; and we changed the format (again) This is OLD. New censuses (afer Dec 2015) are faster; and we changed the format (again)

Probing rate

• Our tool can scan more

than 10k hosts/sec per src

• We spread destjnatjons

with a linear shif register so to maximize interarrival tjme between two ICMP echo request at destjnatjon

• However, ICMP echo reply

aggregate at the source, that receives about 10k replies/sec

• Some ingress fjrewalls in PlanetLab nodes think this is an

atuack and drop ICMP traffjc

• No problems with 1k requests/sec

45

For BGP hijack detectjon, removing ICMP fjlters yields a 10x speedup ! For BGP hijack detectjon, removing ICMP fjlters yields a 10x speedup ! src

Protocol impact

• If multjple protocols answer, no notjceable

difgerence in the output, however

46

ICMP service agnostjc, maximizes (*) reply ICMP service agnostjc, maximizes (*) reply (*) CloudFlare stopped replying

• ur pings :(

Recall[%]

47

Infrastructure impact

• Combine PlanetLab andRIPE Atlas

– Ameliorate coverage – Validate small deployments

1.8M credits for the top-100 1.8M credits for the top-100 PlanetLab censuses RIPE Atlas PlanetLab 150K credits for 2-instances deployments

Vantage points impact (1/3)

• Infrastructure
• RIPE7k = full (at tjme of experiments)

– Greater coverage, but artjfacts due to geolocatjon inaccuracy

• RIPE200 = selectjon of 200 VPs at

least 100km far apart

– Betuer than PlaneLab for fewer VPs

• RIPE350 = 350VPs that have

geolocatjon tags + those VP that yielded true positjve for CDN/DNS

– Best for RIPE

• Union of RIPE and PlanetLab

– Even betuer

48

Top-100 anycast IPv4 deployments All anycast IPv4 deployment

Footprint VPs ASes Country RIPE Atlas (all) 7k 2k 150 RIPE Atlas (subset) 200 139 83 PlanetLab ~300 180 30

Union makes the force! Union makes the force!

Vantage points impact (2/3)

• Infrastructure
• RIPE7k = full (at tjme of experiments)

– Greater coverage, but artjfacts due to geolocatjon inaccuracy

• RIPE200 = selectjon of 200 VPs at least

100km far apart

– Betuer than PlaneLab for fewer VPs

• RIPE350 = 350VPs that have

geolocatjon tags + those VP that yielded true positjve for CDN/DNS

– Best for RIPE

• Union of RIPE and PlanetLab

– Even betuer

49

Footprint VPs ASes Country RIPE Atlas (all) 7k 2k 150 RIPE Atlas (subset) 200 139 83 PlanetLab ~300 180 30

Infrastructure ? (Microsof IP/24 Example)

49

Vantage point Not always :( Not always :(

• The owner of the VP

sets the geolocatjon

• 500 VP have a tag

system-auto-geoip-city

• Can we trust it?
• Only 350 are 100 km apart

Vantage points impact (3/3)

50

Spain France Middle of US Middle of Nowhere Romanian VP gone hiking

Pretuy messy. We also have picteures of PlanetLab nodes in Navajo reserves or swimming in the ocean Pretuy messy. We also have picteures of PlanetLab nodes in Navajo reserves or swimming in the ocean