Why to Study the Internet? It is there, it is a complex system, ... - PDF document

DIMES DIMES Distributed Internet MEasurement and Simulation … and what does it have to do with PlanetLab Yuval Shavitt shavitt@eng.tau.ac.il http://www.netdimes.org Why to Study the Internet? • It is there, it is a complex system, ... • Understanding its structure and dynamics – help current applications (WWW) – help designing future application (file sharing) – help improving routing – predict Internet growth • DIMES is a project set to study the Internet structure. 1

DIMES Correlating the Internet with the World: Geography, Economics, Social Sciences The Internet as a complex system: static and dynamic analysis Distributed System Design: Obtaining the Internet Structure Why DIMES? • The Internet is an engineered system, so someone must know how it is built, no? • NO! It is an uncoordinated interconnection of Autonomous Systems (ASes=networks). • No central database about Internet structure. • Several projects attempt to reveal the structure: Skitter, RouteViews, … 2

The Internet Structure routers The Internet Structure The AS graph 3

Revealing the Internet Structure Revealing the Internet Structure 4

Revealing the Internet Structure Revealing the Internet Structure Diminishing return! Diminishing return! ⇓ Deploying more boxes does not pay-off 7 new links 30 new links NO new links 5

Revealing the Internet Structure To obtain the ‘ horizontal ’ links we need strong presence in the edge What is DIMES? DIMES • Distributed Internet measurement and monitoring – Based on software agents downloaded by volunteers • Diminishing return? – Software agents ⇓ – The cost of the first agent is very high – each additional agent costs almost zero • Capabilities – Obtaining Internet maps at all granularity level • connectivity, delay, loss, bandwidth, jitter, … . – Tracking the Internet evolution in time – Monitoring the Internet in real time 6

Diminishing Return? • [Chen et al 02], [Bradford et al 01]: when you combine more and more points of view the return diminishes very fast • What have they missed? – The mass of the tail is significant No. of views Diminishing Return? • [Chen et al 02], [Bradford et al 01]: when you combine more and more points of view the return diminishes very fast • What have they missed? – The mass of the tail is significant No. of views 7

Diminish … shminimish How many ASes see an edge? ~9000/6000 are seen only by one 8

real world complex system Challenges Distributed System • It ’ s a distributed systems : – Measurement traffic looks malicious • Flying under the NOC radar screens (Agents cannot measure too much) – Optimize the architecture: • Minimize the number of measurements • Expedite the discovery rate • BUT agents are – Unreliable – Some move around real world complex system Agents Distributed System • To be able to use agents wisely we need agents profiles: – Reliablility • Daily (seen in 7 of the last 10 days) • Weekly (seen in 3 of the last 4 weeks) – Location: • Static • Bi-homed: where mostly? • Mobile: identify home base – Abilities: what type of measurements can it perform? 9

real world complex system Distributed System Static Internet Graph Analysis • Degree distribution [Faloutsos99,Lakhina03,Barford01,Chen02] • Clustering coefficient [Bar04] • Disassociativity [Vespigni] • Network motifs (ala Uri Alon) Degree Distribution DIMES+BGP (Feb 05) 12 10 Zipf plot Pr(k) 8 log(degree) 6 <k> 4 k DIMES+BGP (Feb 05) 2 14 12 0 0 2 4 6 8 10 12 14 16 log(rank) 10 log(Pr(degree)) 8 6 4 2 0 0 2 4 6 8 10 12 log(degree) 10

AS map for July 2005 BGP DIMES • 20585 nodes • 14332 nodes • 45720 edges • 60134 edges • <k> = 4.44 • <k> = 8.39 33,862 edges ⇒ DIMES has doubled the connectivity 11,858 edges AS map for July 2005 BGP DIMES • 20585 nodes • 14332 nodes • 45720 edges • 60134 edges • <k> = 4.44 • <k> = 8.39 33,862 edges 21,538 in both maps + 38,596 new edges 11,858 edges 81,672 edges <k> > 7.80 11

real world complex system Distributed System The Internet as a real world mirror • Changes in the world effect the Internet growth • To model Internet growth one needs to take into account – Geographic location – Political/caltural biases – Economic development – Human rights issues 12

real world complex system Distributed System Internet and the World • City connectivity map • Correlation between population*wealth and Internet size • Correlation between trade and Internet connectivity • PoP level map analysis Vision • A Network that optimizes itself: – every device with a measurement module. – How to concert the measurements? – How to aggregate them? – How to analyze them is a hierarchical fashion? 13

DIMES Future • DIMES as a leading research tool (6M measurements/day) – Data is available to all – Easy to run distributed experiments • Fast deploy cycle – Easy to add new capabilities • Packet Trains (joint effort with ETOMIC) • Plug-ins to improve applications – P2P communication – Web download (FireFox plug-in is available) Current Status • Over 3400 users, over 6000 agents – 80 countries – All continents – Over 570 ASes – More than 1000 are active daily • Over 6,000,000 measurements a day 14

Why PlanetLab? DIMES has more agents and better spread than PlanetLab … … But, we do not want our agents to listen on ports – Expose them to attacks – Problems with firewalls ETOMIC serves as a high precision packet sink – Only a few ETOMIC nodes are deployed – They are spread only in Europe • PlanetLab will provide sinks world wide – Challenge: Timing precision not as good as ETOMIC (null) Users by Country Albania Argentina Australia June 2005 Austria Belarus Aus Belgium Bermuda Brazil Bulgaria Canada Chile Ger China Colombia Costa Rica Croatia Cyprus Czech Republic Denmark Egypt Estonia Finland Sp France Germany Greece Guinea 15

How representative in PlanetLab? Experiment under work: Compare groups of university agents with private/business agents: • Delay distribution • Delay stability • Routing stability Who • PI: Yuval Shavitt • Ph.D. students: Eran Shir , Tomer Tankel, Amir Shay • Master ’ s student: Galit Hadad, Dima Feldman, .. • Programmers: Anat Halpern, Ohad Serfati • Undergrads: Roni Ilani, Shay • Collaborators: HUJI, ColBud 16

http://www.netdimes.org 17