Swarm Transparently distributed computation in the cloud Ian Clarke - - PowerPoint PPT Presentation

Swarm

Ian Clarke

ian@uprizer.com

“Transparently distributed computation in the cloud”

Swarm

Ian Clarke

ian.clarke@gmail.com

“Transparently Distributed Computation in the cloud”

About me

• Degree in AI and Comp Sci from Edinburgh

University, Scotland (1995-1999)

• Designer and co-ordinator of Freenet, the first

decentralized P2P architecture (1999-present)

• Designed P2P video streaming system that later

became part of “Joost” (2003-2004)

• Founder and Chief Scientist of Revver (2004-2006)
• CEO of Uprizer Labs (2007-present)

The Problem

Building a web-app?

You want your development process to be:

• Cheap and fast to implement
• Scalable in the event of success

Building a web-app?

You want your development process to be:

• Cheap and fast to implement
• Scalable in the event of success

Pick One!

Examples

• 22 hour outage after IPO in 1999
• Estimated cost: Over $2M

• 22 hour outage after IPO in 1999
• Estimated cost: Over $2M

• Periodic outages since it started, most

recently August ’09

• Forced fundamental rearchitecture
• Aside: Started with Ruby on Rails,

now using Scala

• 22 hour outage after IPO in 1999
• Estimated cost: Over $2M

How is this solved today?

Database Architecture

MySql WebNode Cache WebNode Cache WebNode Cache

Replicate databases

MySql WebNode Cache WebNode Cache WebNode Cache MySql MySql

Map Reduce

• Certain problems may be broken into

“map” and “reduce” operations

• Interesting because the data stays still, the

computation moves

• Good at things like distributed sort,

distributed grep, etc

• Not general-purpose

Our Proposal: Swarm

But first... Some background

Scala

• Compiles to Java bytecode
• so its fast and widely supported

Scala

• Compiles to Java bytecode
• so its fast and widely supported
• Supports closures, and type-inference
• so it solves most of Java’s problems

Scala

• Compiles to Java bytecode
• so its fast and widely supported
• Supports closures, and type-inference
• so it solves most of Java’s problems
• The upcoming Scala 2.8 supports “portable

continuations”

Scala

Continuations

What do continuations do?

• Store the state of a

computer program

• Like saving your position

in a video game

• Resume execution at

some point in the future

Scala 2.8’s continuations support

Scala 2.8’s continuations support

• “Delimited”

Scala 2.8’s continuations support

• “Delimited”
• Portable

Scala 2.8’s continuations support

• “Delimited”
• Portable
• Implemented through a

code transformation

Scala 2.8’s continuations support

• “Delimited”
• Portable
• Implemented through a

code transformation

• Complicated!

The Solution

?

What if we could distribute data and computation across multiple computers such that the programmer need not think about it?

But how?

But how?

• Move the computation, not the data

But how?

• Move the computation, not the data
• Handle this transparently within the

framework

But how?

• Move the computation, not the data
• Handle this transparently within the

framework

• Arrange the data to minimize movement of

the computation

How does it work?

Program:

• 1. print a
• 2. print b
• 3. print c

a b c

How does it work?

a b c Program:

• 1. print a
• 2. print b
• 3. print c

How does it work?

a b c Program:

• 1. print a
• 2. print b
• 3. print c

How does it work?

a b c Program:

• 1. print a
• 2. print b
• 3. print c

Arranging data with graph clustering

Forcing Swarm to migrate the continuation

Forced remote variable

What next?

• Just a simple prototype
• Many interesting sub-problems
• Open source
• Need your help!

Storage

• How do we arrange the data for optimal

efficiency?

• What about concurrency?
• Software transactional memory
• Replication and redundancy
• Garbage collection

A “universal” codebase

• Swarm requires that every node has the same

binary

• We could use the JVM’s classloader mechanism

to retrieve binaries as needed from a global namespace

• Will need to address issues of versioning and

security

“Swarm aware” libraries

• Need “Swarm” aware collections classes

like Map, List, and Set

• Develop a storage system with capabilities

similar to a relational database

• The creation of a web framework around

Swarm (similar to “Rails” or “LiftWeb”)

Swarm tools

• Continuations plugin imposes restrictions
• n the code that can be migrated
• “foreach”
• Serializable
• A Scala compiler plugin that understood

these limitations would be very useful

Interested in helping?

http://code.google.com/p/swarm-dpl/ ian@uprizer.com