Prophet a path out of the cloud http://syncwith.us - PowerPoint PPT Presentation

Prophet a path out of the cloud http://syncwith.us jesse@bestpractical.com 1

You may know me from... RT (Request Tracker) Jifty SVK Hiveminder Perl 6 Shirts 2

I’ve been hacking on an open source database called “Prophet” 3

It has an API like Amazon SimpleDB or Google App Engine’s... 4

It’s designed for “team-scale” apps 5

It’s built for P2P replication and disconnected use 6

App #1 is the canonical “offline bug tracker” 7

App #2 will probably be a BBS you can sync over sneakernet 8

But first, a brief digression... 9

...about cloud computing 10

Living in the cloud = sharecropping 11

(That’s bad) 12

This is a rant 13

The bad old days: 14

Pic of sharecroppers 15

You farmed land you didn’t own... 16

...with tools you couldn’t really afford 17

You paid for it with part of your harvest... 18

It sounded like a pretty sweet deal... 19

...until things got bad 20

(Things always got bad) 21

In a bad year, you got further in debt to the land owner 22

23

The (more recent) bad old days: 24

pic of mainframes 25

You ran code you didn’t own on hardware you didn’t own 26

Things got a little better: 27

Pic of PCs 28

Things weren’t all rosy: 29

Pic of BSOD 30

Sometimes new versions of software killed features... 31

...so you were locked in to old versions 32

pic of win 31? 33

Things got ‘better’: 34

rms che 35

Now, things are getting worse again... 36

37

What happens when your favorite service goes down? 38

pic of twitter being down 39

...or stops accepting new signups? 40

41

...or gives all your data to the secret police? 42

Pic of yahoo.cn 43

...or starts making arbitrary choices about what’s ‘safe’ content? 44

45

You don’t own the services you use 46

When the service provider cuts you off, that’s it. No recourse. 47

Not so secret shame: I’m a really bad zealot 48

My calendar lives at google.com 49

50

I make a web 2.0 tasklist service called Hiveminder.com 51

pic of hiveminder 52

Using hosted apps is going to hurt you 53

Data access is important 54

APIs are great 55

...but easy access to a service just makes it easier to get locked in 56

What about Google Gears, Adobe Air, etc? 57

Great. now you can use your word processer while you’re offline! 58

Pic of wordperfect 59

Real offline apps shouldn’t need servers 60

Real offline apps should sync like you do 61

I might be a nut job 62

...but smart people seem to agree with me 63

If we want people to have the same degree of user autonomy as we've come to expect from the world, we may have to sit down and code alternatives to Google Docs, Twitter, and EC3 that can live with us on the edge, not be run by third parties. - Danny O’Brien http://www.oblomovka.com/entries/2008/07/16 64

Back to that database thing... 65

Jesse Vincent 66

Chia-liang Kao 67

We work together 68

CL lives in Taipei Jesse lives in Boston 69

Sometimes we need to work face to face 70

TPE - BOS: 9410 mi TPE - HNL: 5,095 mi BOS - HNL: 5,069 mi 71

Our Plan Step 1: Go to Hawaii for “work” Step 2: ??? Step 3: Prophet! 72

The Plan Backfired We were there for 8 days We wrote 8000 lines of Perl We figured out step 2 73

Step 2: Build a Disconnected Syncable Database 74

Prophet 75

Getting Prophet Prophet http://syncwith.us/prophet SD http://syncwith.us/sd 76

Prophet A grounded, semirelational, peer to peer replicated, disconnected, versioned, property database with self-healing conflict resolution 77

What do all those buzzwords mean? 78

grounded Runs here 79

grounded Not here 80

grounded Runs at the edge Doesn’t need to run in the cloud Syncs with services you already use (We call the adaptors “Foreign Replicas”) 81

semirelational Joins are expensive (They’re still possible) 82

peer-to-peer replicated Update any replica Pull from any replica Push to any replica Publish a replica Changes will propagate 83

disconnected Real-time replication is hard to scale It only “works” with constant connectivity I don’t have constant connectivity Neither do you Prophet sync can happen whenever 84

versioned Every update is recorded as a change set Change sets don’t lose any data (so you can use them to go backwards) All history is introspectable Replication just replays changesets 85

property database Atomic operations CREATE, READ, UPDATE, DELETE, SEARCH Record types can have optional validation and canonicalization Records of the same type do not need to have the same properties Add and remove properties at will 86

self-healing conflict resolution Remembers all conflict resolutions Syncs all resolutions with your peers Detects identical conflicts Uses your peers’ resolutions to “vote” for the winner of a conflict 87

Working with Prophet 88

RESTy API GET /records.json GET /records/Cars.json GET /records/Cars/716499-5F9-4AC4-827.json GET /records/Cars/716499-5F9-4AC4-827/wheels.json POST /records/Cars.json POST /records/Cars/716499-5F9-4AC4-827.json POST /records/Cars/716499-5F9-4AC4-827/wheels.json 89

RESTy API Yes, we should be using PUT and DELETE Yes, you can have a commit bit and help us fix it :) 90

Native API (Yes, the core is Perl.) my $cli = Prophet::CLI->new(); my $cxn = $cli->app_handle->handle; my $record = Prophet::Record->new( handle => $cxn, type => 'Person' ); my $uuid = $record->create( props => { name => 'Jesse', age => 31 } ); $record->set_prop( name => 'age', value => 32 ); my $people = Prophet::Collection->new( handle => $cxn, type => 'Person' ); $people->matching( sub { shift->prop('species') ne 'cat' } ); 91

What could you build with Prophet? 92

sd A bug tracker: “simple defects” • id. Status, Summary • (Arbitrary other properties too) • History • Comments • Attachments 93

Create ./bin/sd ticket create -- summary="Can't sync sd with Google Code" status=new Created ticket 5 (93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4) 94

List and Search ./bin/sd ticket search --regex publish 29 } new the online help doesn't describe publish 34 } new publish a static html view of records 35 } new publish should create a static rss file 95

Updates ./bin/sd ticket update --uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 -- status=resolved 96

Bugs on my laptop aren’t interesting. 97

Sync! Jesse sd publish --to fsck.com:public_html/sd/ CL sd pull --from http://my.com/~jesse/sd 98

My project has a bug tracker 99

My project has a bug tracker Actually, mine use two: • RT • hiveminder.com 99