An open, shared platform for developing, deploying, and accessing - - PowerPoint PPT Presentation

APAN, Cairns, 1 July 2004

An open, shared platform for developing, deploying, and accessing planetary-scale applications

Timothy Roscoe Intel Research at Berkeley Thursday, July 01, 2004

APAN, Cairns, 1 July 2004

PlanetLab is…

• Large collection of machines

spread around the world for distributed systems teaching and research

• Focus/catalyst for networking and

systems community

• Intel project ⇒ consortium of

companies and universities

APAN, Cairns, 1 July 2004

Lots of work being done in widely distributed systems…

• Students and researchers had no

vehicle to try out their next n great ideas in this space

– Lots of architectures – Lots of simulations – Lots of emulation on large clusters – Lots of folks calling their 17 friends before the next deadline

• but not the surprises and frustrations
• f experience at scale to drive

innovation

APAN, Cairns, 1 July 2004

The value proposition

• Institutions join, provide nodes

– IA32 architecture servers – Hosted outside the firewall – Provide power, cooling, & bandwidth

• In exchange, users get “slices” of

many machines worldwide.

APAN, Cairns, 1 July 2004

Origins and early progress

• “Underground” meeting March 2002
• Intel seeds

– First 100 nodes – Operational support

• First node up July 2002
• By SOSP (deadline March 2003) 25% of

accepted papers refer to PlanetLab

• Large presence at SIGCOMM
• 11 out of 27 papers in NSDI 2004
• Currently ~500 class and research

projects

APAN, Cairns, 1 July 2004

PlanetLab today

About 403 nodes, 169 sites, ~30 countries, 5 continents Universities, Labs, POPs, CoLos, DSL lines Huge presence in systems research conferences > 400 projects so far

APAN, Cairns, 1 July 2004

Network coverage

APAN, Cairns, 1 July 2004

What is PlanetLab good for?

• Planetary-Scale applications:

– Low latency to widely spread users – Span boundaries: jurisdictional and administrative – Simultaneous viewpoints: on the network or sensors – Hardware deployment is undesirable

• Long-running services, not only

experiments / exercises

APAN, Cairns, 1 July 2004

What is it used for?

• PlanetLab addresses the related

problems of:

– Deploying widely-distributed services – Evaluating competing approaches in a realistic setting – Evolving the network architecture to better support such services

• So far, PlanetLab is highly

successful at doing this.

APAN, Cairns, 1 July 2004

PlanetLab is not…

• A distributed supercomputer
• A simulation platform
• An Internet emulator
• An arena for repeatable

experiments

• Completely representative of the

current Internet

APAN, Cairns, 1 July 2004

PlanetLab is…

• An opportunity to qualitatively

validate distributed systems in a real deployment environment

• An opportunity to gain valuable

experience about what works and what doesn’t in the wide area at scale

APAN, Cairns, 1 July 2004

Why is it successful?

• Community model

– “network effects” – Lots of benefit from small entry fee

• Sliceability

– Enables multiple approaches – Sharing of the platform

• Virtual machine interface

– Emphasis on multiplexing the machine – Isolation left to the VMM

APAN, Cairns, 1 July 2004

PlanetLab relationships

• PlanetLab member institutions

– Shared control of nodes

• PlanetLab users

– Distributed virtualization, slices

• PlanetLab research builders

– Shared interfaces, unbundled mgmt

• PlanetLab rest of the Internet

– Isolation, security, packet auditing

APAN, Cairns, 1 July 2004

Architectural principles

• Application-Centric Interfaces

– Evolve an existing API rather than create a new one

• Distributed Virtualisation

– Slices and isolation

• Unbundled Management

– Community infrastructure services

• Self-obsolescence

– everything initially built should

APAN, Cairns, 1 July 2004

Application-Centric Interfaces

• Inherent problems

– Learning curve for users – stable platform versus research into platforms – integrating testbeds with desktop machines

• Approach

– take popular API (Linux), evolve implementation – later separate isolation & application interfaces – provide generic “shim” library for desktops

APAN, Cairns, 1 July 2004

Distributed Virtualization

• Services run in slices.
• Slice: set of virtual

machines (slivers)

• Created by slice

creation service acting as a broker

VMM (Linux + stuff) Service Service Service

Local admin slice

Node Manager Node structure:

APAN, Cairns, 1 July 2004

Requirements for slices

• Isolation of VMs:

– Allocate resources – Partition or contextualize namespaces – Provide stable programming base

• Isolation of PlanetLab:

– Resource accounting and limits – Auditing of slice actions (eg. packets)

• Unexpected requirement!

APAN, Cairns, 1 July 2004

Node virtualization options

• Full virtualization (VMware, etc.)

– Inadequate scaling for PlanetLab – Long-term: investigate h/w support

• Paravirtualization (Xen, Denali)

– Not yet mature, but multiple OSes – Very attractive medium-term

• Syscall-level virtualization (Vservers)

– Works today, fairly stable – Only one operating system (Linux)

APAN, Cairns, 1 July 2004

VServers

• 3rd-party Linux kernel patch

– Provides illusion of virtual Linux kernel – Much more scalable than UML

• Node manager runs in special context 0
• Linux “capabilities” for access control

– Local admin slice has extra privilege

• Slice creation provides:

– New file system (COW subset of main root) – Installs ssh keys for slice users – Empty root password

APAN, Cairns, 1 July 2004

Resource allocation

• rspecs describe resource bundles
• Node manager doles out rcaps for the

node:

rcap <- acquire(rcap)

• rcaps can be transferred or traded
• eventually bound to a slice (redeemed):

bind(slice_name, rcap)

• Binding resources to slices is a higher-

level policy

– (e.g. Globus, SHARP, Emulab, etc.)

APAN, Cairns, 1 July 2004

Dynamic slice creation

N3 N4 Nm N1 N2

. . .

Agent Broker

. . . . . . candidates reserve Description (rspec) Ticket (rcap) Description (rspec) acquire rcap lease

Application (Service Manager)

. . . . . . . . . . . .

PlanetLab Central, SHARP, Emulab, etc.

APAN, Cairns, 1 July 2004

Network Virtualization

• How to present a shared physical

network interface to VMs?

• Partition vs. Contextualization
• More than simple multiplexing:

– Raw sockets – Per-slice routing tables – GRE tunnels – Shared DNS sockets – Etc…

APAN, Cairns, 1 July 2004

Current solution: very simple

• Port reservations: included in

rspec

• “Safe raw sockets”

– Tap packets sent or received by slice’s own sockets – Send any packet that could be sent by a socket owned by the slice – E.g. user-space TCP

APAN, Cairns, 1 July 2004

Packet auditing

• “Sniffer sockets” annotate packet

trace with slice identifiers

– Used by privileged slices – Adminstrative slice logs all traffic – Web server allows remote inspection

• Enables mapping:

<ip packet, time> → <PI email addr>

APAN, Cairns, 1 July 2004

Unbundled Management

• PlanetLab management is a

Planetary-scale application

– Allow many groups to experiment with infrastructure services in parallel – Low-level, sharable interfaces – Nodes only provide local abstractions – Includes VM creation

=> Slices are defined by services

APAN, Cairns, 1 July 2004

Unbundled Management

• Partition mgmt into orthogonal services

– resource discovery – monitoring system health – topology management – manage user accounts and credentials – software distribution and updates

• Approach

– management services run in their own slice – allow competing alternatives – engineer for innovation (minimal interfaces)

APAN, Cairns, 1 July 2004

What do people use it for?

(some ones we know about)

• Overlay Networks

– RON, ROM++, ESM, XBone, ABone, etc.

• Network measurement

– Scriptroute, *Probe, I3, etc.

• Application-level multicast

– ESM, Scribe, TACT, etc.

• Wide-area distributed

storage

– Oceanstore, SFS, CFS, Palimpsest, IBP

• Resource allocation

– Sharp, Slices, XenoCorp, Automated contracts

• Distributed query

processing

– PIER, IrisLog, Sophia, etc.

• Content Dist. Networks

– CoDeeN, ESM, UltraPeer emulation, Gnutella mapping

• Management and Monitoring

– Ganglia, InfoSpect, Scout Monitor, BGP Sensors, etc.

• Distributed Hash Tables

– Chord, Tapestry, Pastry, Bamboo, etc.

• Virtualization and Isolation

– Denali, VServers, SILK, Mgmt VMs, etc.

• Router Design implications

– NetBind, Scout, NewArch, Icarus, etc.

• Testbed Federation

– NetBed, RON, XenoServers

• Etc., etc., etc.

APAN, Cairns, 1 July 2004

Example #1: OpenHash

(Brad Karp, Sylvia Ratnasamy, Sean Rhea)

• Sharable, stable DHT service

– “Turn on, put in, get out” – Accessible from any machine – “Redir” allows consistent hashing over arbitrary node sets

• Implemented over Bamboo

– Extremely robust & churn-resilient DHT implementation

• Going live this week

APAN, Cairns, 1 July 2004

Example #2: PIER

(Joe Hellerstein, Ryan Huebsch, Boon Thau Long, Timothy Roscoe, Scott Shenker, Ion Stoica, etc.)

• Pure P2P Relational Query Engine

– Join, Selection, Projection, Aggregation

• DHT (Bamboo) used for:

– Joins (rehashing / rendezvous) – Aggregation (building agg. trees)

• Used for PlanetLab monitoring &

resource discovery

APAN, Cairns, 1 July 2004

Example #3: PHI

(Brent Chun, Joe Hellerstein, Petros Maniatis, Timothy Roscoe, Nina Taft)

• “Public Health for the Internet”

– New project, in early stages

• P2P monitoring of network security and

performance

– Initial use of PlanetLab as a stable core – Built over PIER query processor – Passive network monitoring at end systems – Nodes upload data to the P2P network, and download interesting query results

APAN, Cairns, 1 July 2004

Lessons from PlanetLab

• Nothing works as expected at scale!

– Many unintended and unexpected consequences of algorithmic choices – Simulation results do not carry over well

• Simulate, deploy, measure, edit cycle
• Evaluating competing approaches “in the

wild” refines techniques

• The ability to try things out “for real”

really stimulates ideas

APAN, Cairns, 1 July 2004

Teaching with PlanetLab

• Very early stages

– Little courseware available – Ad-hoc courses (KAIST, U.C. Berkeley, U. Washington, Columbia University, etc.) – Contributions sought!

APAN, Cairns, 1 July 2004

Operational experience

• UDP replaces TCP

– N-to-N applications are different – Short-term: File Descriptor limits – Long-term: Remove abstraction barriers

• Aggressive application timeouts
• Detailed information about network perf.

– Still congestion controlled – DCCP not yet available

APAN, Cairns, 1 July 2004

Operational experience

• The Internet is extraordinarily

brittle

– Innovation resembles Denial-of- Service – IDSes default to “attack warning” – “Common cases” burned into routers

• PlanetLab now supports full packet

auditing to end-users

• Important teaching implications!

APAN, Cairns, 1 July 2004

What is PlanetLab doing to the Internet?

• PlanetLab appears to function as a

disruptive technology

– Applications use the network differently – The network sometimes reacts in a hostile manner – Leads to new requirements on infrastructure

APAN, Cairns, 1 July 2004

Long-term aims

• PlanetLab incubates the next Internet
• New networks deployed as overlays over

existing Internet

• Service-oriented network becomes the

norm

• Computation as a localizable network

resource

• Students can enjoy the opportunity to

participate in the process

APAN, Cairns, 1 July 2004

The “end-to-end” argument…

• PlanetLab changes what

constitutes an “end-system”

– Slices share physical nodes – Slices are distributed – Responsibility needs to shift to slices – TCP is no longer end-to-end

⇒ Tension between applications and “Internet architecture”

APAN, Cairns, 1 July 2004

Conclusion

• Think of PlanetLab as a communal shared

artefact

• About the experience of deploying large

systems

• Provides many diverse, overlapping projects

around the world with a stable place to stand to change things

• Forum for exchange and composition of

services and applications

• Selection environment based on real

deployment and use

• Bottom-up approach to changing the world

APAN, Cairns, 1 July 2004

Upcoming meetings

• Seoul: 17-18 September
• San Francisco: 1st week in

December

http://www.planet-lab.org/php/meetings.php

APAN, Cairns, 1 July 2004

Thanks!

Please join us!

http://www.planet-lab.org/join.php