Prophet a path out of the cloud http://syncwith.us - - PowerPoint PPT Presentation

▶
prophet
SMART_READER_LITE
LIVE PREVIEW

Prophet a path out of the cloud http://syncwith.us - - PowerPoint PPT Presentation

Dec 10, 2023 296 likes •1.74k views

Prophet a path out of the cloud http://syncwith.us jesse@bestpractical.com 1 You may know me from... RT (Request Tracker) Jifty SVK Hiveminder Perl 6 Shirts 2 Ive been hacking on an open source database called Prophet 3 It

slide-1
SLIDE 1

Prophet

a path out of the cloud http://syncwith.us

jesse@bestpractical.com

1
slide-2
SLIDE 2

You may know me from...

RT (Request Tracker) Jifty SVK Hiveminder Perl 6 Shirts

2
slide-3
SLIDE 3

I’ve been hacking on an

  • pen source database

called “Prophet”

3
slide-4
SLIDE 4

It has an API like Amazon SimpleDB or Google App Engine’s...

4
slide-5
SLIDE 5

It’s designed for “team-scale” apps

5
slide-6
SLIDE 6

It’s built for P2P replication and disconnected use

6
slide-7
SLIDE 7

App #1 is the canonical “offline bug tracker”

7
slide-8
SLIDE 8

App #2 will probably be a BBS you can sync

  • ver sneakernet
8
slide-9
SLIDE 9

But first, a brief digression...

9
slide-10
SLIDE 10

...about cloud computing

10
slide-11
SLIDE 11

Living in the cloud = sharecropping

11
slide-12
SLIDE 12

(That’s bad)

12
slide-13
SLIDE 13

This is a rant

13
slide-14
SLIDE 14

The bad old days:

14
slide-15
SLIDE 15

Pic of sharecroppers

15
slide-16
SLIDE 16

You farmed land you didn’t own...

16
slide-17
SLIDE 17

...with tools you couldn’t really afford

17
slide-18
SLIDE 18

You paid for it with part of your harvest...

18
slide-19
SLIDE 19

It sounded like a pretty sweet deal...

19
slide-20
SLIDE 20

...until things got bad

20
slide-21
SLIDE 21

(Things always got bad)

21
slide-22
SLIDE 22

In a bad year, you got further in debt to the land owner

22
slide-23
SLIDE 23 23
slide-24
SLIDE 24

The (more recent) bad old days:

24
slide-25
SLIDE 25

pic of mainframes

25
slide-26
SLIDE 26

You ran code you didn’t

  • wn on hardware you

didn’t own

26
slide-27
SLIDE 27

Things got a little better:

27
slide-28
SLIDE 28

Pic of PCs

28
slide-29
SLIDE 29

Things weren’t all rosy:

29
slide-30
SLIDE 30

Pic of BSOD

30
slide-31
SLIDE 31

Sometimes new versions of software killed features...

31
slide-32
SLIDE 32

...so you were locked in to old versions

32
slide-33
SLIDE 33

pic of win 31?

33
slide-34
SLIDE 34

Things got ‘better’:

34
slide-35
SLIDE 35

rms che

35
slide-36
SLIDE 36

Now, things are getting worse again...

36
slide-37
SLIDE 37 37
slide-38
SLIDE 38

What happens when your favorite service goes down?

38
slide-39
SLIDE 39

pic of twitter being down

39
slide-40
SLIDE 40

...or stops accepting new signups?

40
slide-41
SLIDE 41 41
slide-42
SLIDE 42

...or gives all your data to the secret police?

42
slide-43
SLIDE 43

Pic of yahoo.cn

43
slide-44
SLIDE 44

...or starts making arbitrary choices about what’s ‘safe’ content?

44
slide-45
SLIDE 45 45
slide-46
SLIDE 46

You don’t own the services you use

46
slide-47
SLIDE 47

When the service provider cuts you off, that’s it. No recourse.

47
slide-48
SLIDE 48

Not so secret shame: I’m a really bad zealot

48
slide-49
SLIDE 49

My calendar lives at google.com

49
slide-50
SLIDE 50 50
slide-51
SLIDE 51

I make a web 2.0 tasklist service called Hiveminder.com

51
slide-52
SLIDE 52

pic of hiveminder

52
slide-53
SLIDE 53

Using hosted apps is going to hurt you

53
slide-54
SLIDE 54

Data access is important

54
slide-55
SLIDE 55

APIs are great

55
slide-56
SLIDE 56

...but easy access to a service just makes it easier to get locked in

56
slide-57
SLIDE 57

What about Google Gears, Adobe Air, etc?

57
slide-58
SLIDE 58
  • Great. now you can use

your word processer while you’re offline!

58
slide-59
SLIDE 59

Pic of wordperfect

59
slide-60
SLIDE 60

Real offline apps shouldn’t need servers

60
slide-61
SLIDE 61

Real offline apps should sync like you do

61
slide-62
SLIDE 62

I might be a nut job

62
slide-63
SLIDE 63

...but smart people seem to agree with me

63
slide-64
SLIDE 64

If we want people to have the same degree

  • f user autonomy as we've come to expect

from the world, we may have to sit down and code alternatives to Google Docs, Twitter, and EC3 that can live with us

  • n the edge, not be run by third parties.
  • Danny O’Brien

http://www.oblomovka.com/entries/2008/07/16

64
slide-65
SLIDE 65

Back to that database thing...

65
slide-66
SLIDE 66

Jesse Vincent

66
slide-67
SLIDE 67

Chia-liang Kao

67
slide-68
SLIDE 68

We work together

68
slide-69
SLIDE 69

CL lives in Taipei Jesse lives in Boston

69
slide-70
SLIDE 70

Sometimes we need to work face to face

70
slide-71
SLIDE 71

TPE - BOS: TPE - HNL: BOS - HNL: 9410 mi 5,095 mi 5,069 mi

71
slide-72
SLIDE 72

Step 1: Go to Hawaii for “work” Step 2: ??? Step 3: Prophet!

Our Plan

72
slide-73
SLIDE 73

The Plan Backfired

We were there for 8 days We wrote 8000 lines of Perl We figured out step 2

73
slide-74
SLIDE 74

Step 2: Build a Disconnected Syncable Database

74
slide-75
SLIDE 75

Prophet

75
slide-76
SLIDE 76

Prophet http://syncwith.us/prophet SD http://syncwith.us/sd

Getting Prophet

76
slide-77
SLIDE 77

A grounded, semirelational, peer to peer replicated, disconnected, versioned, property database with self-healing conflict resolution

Prophet

77
slide-78
SLIDE 78

What do all those buzzwords mean?

78
slide-79
SLIDE 79

grounded Runs here

79
slide-80
SLIDE 80

grounded Not here

80
slide-81
SLIDE 81

grounded

Runs at the edge Doesn’t need to run in the cloud Syncs with services you already use (We call the adaptors “Foreign Replicas”)

81
slide-82
SLIDE 82

Joins are expensive (They’re still possible)

semirelational

82
slide-83
SLIDE 83

Update any replica Pull from any replica Push to any replica Publish a replica Changes will propagate

peer-to-peer replicated

83
slide-84
SLIDE 84

Real-time replication is hard to scale It only “works” with constant connectivity I don’t have constant connectivity Neither do you Prophet sync can happen whenever

disconnected

84
slide-85
SLIDE 85

Every update is recorded as a change set Change sets don’t lose any data (so you can use them to go backwards) All history is introspectable Replication just replays changesets

versioned

85
slide-86
SLIDE 86

Atomic operations

CREATE, READ, UPDATE, DELETE, SEARCH

Record types can have optional validation and canonicalization Records of the same type do not need to have the same properties Add and remove properties at will

property database

86
slide-87
SLIDE 87

Remembers all conflict resolutions Syncs all resolutions with your peers Detects identical conflicts Uses your peers’ resolutions to “vote” for the winner of a conflict

self-healing conflict resolution

87
slide-88
SLIDE 88

Working with Prophet

88
slide-89
SLIDE 89

RESTy API

GET /records.json GET /records/Cars.json GET /records/Cars/716499-5F9-4AC4-827.json GET /records/Cars/716499-5F9-4AC4-827/wheels.json POST /records/Cars.json POST /records/Cars/716499-5F9-4AC4-827.json POST /records/Cars/716499-5F9-4AC4-827/wheels.json

89
slide-90
SLIDE 90

RESTy API

Yes, we should be using PUT and DELETE Yes, you can have a commit bit and help us fix it :)

90
slide-91
SLIDE 91

Native API

(Yes, the core is Perl.) my $cli = Prophet::CLI->new(); my $cxn = $cli->app_handle->handle; my $record = Prophet::Record->new( handle => $cxn, type => 'Person' ); my $uuid = $record->create( props => { name => 'Jesse', age => 31 } ); $record->set_prop( name => 'age', value => 32 ); my $people = Prophet::Collection->new( handle => $cxn, type => 'Person' ); $people->matching( sub { shift->prop('species') ne 'cat' } );

91
slide-92
SLIDE 92

What could you build with Prophet?

92
slide-93
SLIDE 93

A bug tracker: “simple defects”

  • id. Status, Summary
  • (Arbitrary other properties too)
  • History
  • Comments
  • Attachments

sd

93
slide-94
SLIDE 94

./bin/sd ticket create -- summary="Can't sync sd with Google Code" status=new Created ticket 5 (93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4)

Create

94
slide-95
SLIDE 95

./bin/sd ticket search --regex publish 29 } new the online help doesn't describe publish 34 } new publish a static html view of records 35 } new publish should create a static rss file

List and Search

95
slide-96
SLIDE 96

./bin/sd ticket update

  • -uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4
  • - status=resolved

Updates

96
slide-97
SLIDE 97

Bugs on my laptop aren’t interesting.

97
slide-98
SLIDE 98

Jesse sd publish --to fsck.com:public_html/sd/ CL sd pull --from http://my.com/~jesse/sd

Sync!

98
slide-99
SLIDE 99

My project has a bug tracker

99
slide-100
SLIDE 100

Actually, mine use two:

  • RT
  • hiveminder.com

My project has a bug tracker

99
slide-101
SLIDE 101

Foreign Replicas

Prophet makes Foreign Replicas easy SD gets them "for free"

100
slide-102
SLIDE 102

(Using only the public REST API) It took an afternoon Mirror an RT instance into SD Share it with your peers using prophet Sync changes back from your peers to RT Supports Comments and Attachments

Wrote an RT Replica for SD

101
slide-103
SLIDE 103

(Using only the public REST API)

...and one for Hiveminder

102
slide-104
SLIDE 104

I can sync my bugs with RT or Hiveminder

103
slide-105
SLIDE 105

Actually, it’s better

104
slide-106
SLIDE 106

I can sync between RT and Hiveminder

105
slide-107
SLIDE 107

I can sync between two different RTs, too

106
slide-108
SLIDE 108
  • Trac
  • Launchpad
  • Google Code
  • SourceForge
  • Bugzilla
  • Jira
  • GForge
  • debbugs
  • GNATS
  • todo.txt
  • Lighthouse
  • Redmine
  • FogBugz
  • What else?

We need more replica definitions:

107
slide-109
SLIDE 109

What else can you use Prophet for?

108
slide-110
SLIDE 110

All your “social” databases

109
slide-111
SLIDE 111
  • CRM
  • Bug tracking
  • Sales orders
  • Phone book
  • Blog
  • Trading Card

Database

  • Ideas?

All the databases you want while offline.

110
slide-112
SLIDE 112

How about a P2P BBS? Prophet doesn’t need a server. You can sync over sneakernet.

“Private” Social Networks

111
slide-113
SLIDE 113

A look inside Prophet

112
slide-114
SLIDE 114

Anatomy of a Prophet Replica

113
slide-115
SLIDE 115

The bits and pieces

Database UUID Replica UUID Record Store Changeset Store Resolution Database Configuration metadata

114
slide-116
SLIDE 116

The Record Store

Stores individual records by type Not guaranteed to have all old versions

115
slide-117
SLIDE 117

The Changeset Store

Stores every change to a set of records Guaranteed to have all old changesets Replaying all changesets will create an exact clone of the replica

116
slide-118
SLIDE 118

Replica Backends

117
slide-119
SLIDE 119

Filesystem

Readable Flat files Compact Fast (Not yet fully atomic)

118
slide-120
SLIDE 120

HTTP

Designed to let you “publish” databases Flat-files, Currently read-only. Same format as the filesystem replica type.

119
slide-121
SLIDE 121

Subversion (DEPRECATED)

Slow Steady Robust Supports remote sync Requires Subversion Perl Bindings

120
slide-122
SLIDE 122

Backends are pluggable!

The filesystem is cheap and easy The filesystem is portable Help us write new backends: CouchDB, SQLite, MySQL, Postgres, S3, AppEngine, $YOUR_FAVORITE_DB

121
slide-123
SLIDE 123

Prophet is designed to sync with “other” databases and systems They don’t need to support all of Prophet’s features - Prophet knows how to interpret mumbo-jumbo from the Cloud Foreign Replicas will usually be app specific All current examples are for SD

Foreign Replicas

122
slide-124
SLIDE 124

Synchronization

123
slide-125
SLIDE 125

Publish

Serialize and export all of a replica's resolutions and changesets

124
slide-126
SLIDE 126

Pull

Integrate unseen resolutions and then unseen changesets from a replica

125
slide-127
SLIDE 127

Push

Integrate new resolutions and changesets into a replica

126
slide-128
SLIDE 128

Conflicts

127
slide-129
SLIDE 129

Figures out the best resolution “Nullifies” the conflict so the changeset can be cleanly integrated Integrates the conflicting changeset Records the resolution as a new changeset Records the resolution decision in the resolution database

Resolving Conflicts

128
slide-130
SLIDE 130

Prophet has clever ways to figure out the best resolution. If there are previous resolutions for the same conflict and a majority agree, use that If the merger has specified a “prefer this side” choice, use that Prompt the user to make a decision, giving them info about previous decisions for this conflict

“The Best Resolution”

129
slide-131
SLIDE 131

Scaling

130
slide-132
SLIDE 132

Scaling to giant clusters is boring (Can I play the “They’re not Green” card here?) Scales to many weakly coonnected peers You are not Google. Does anyone here work for Google? Current target is databases of O(50k) records

How does it scale?

131
slide-133
SLIDE 133

We have a political agenda. Cloud computing is not Open Source. APIs for “export” are not good enough. You should always have full control. You probably don’t need to store 10 billion records in one database.

Why not, then?

132
slide-134
SLIDE 134

Do you have 10 billion bugs, customer contacts

  • r sales orders?
133
slide-135
SLIDE 135

That said, we'd love to see a scalable, high performance prophet replica store

134
slide-136
SLIDE 136

Getting Involved

135
slide-137
SLIDE 137

Project Status

Simple, well-defined Perl API RESTy web API (with microserver) Fast, lightweight backend Small, active dev community Great test coverage ...less than great documentation coverage

136
slide-138
SLIDE 138

Better ergonomics Improved search and indexing (Including full-text indexing) Client libraries for other languages Proper security model More apps

Our Plans

137
slide-139
SLIDE 139

Prophet 6937 lines of code and doc 1952 lines of tests sd 2121 lines of code and doc 973 lines of tests

Codebase

138
slide-140
SLIDE 140

Prophet is very young

Prophet designed in April Prophet core implemented in April SD designed in April SD built in June and July

139
slide-141
SLIDE 141

We need your help!

Kick-ass functional and text indexing Backend data store improvements Slick GUIs for syncing More Foreign Replicas for SD Documentation improvements A clever logo New applications

140
slide-142
SLIDE 142

Prophet http://syncwith.us/prophet/download SD http://syncwith.us/sd/download

Getting Prophet

141
slide-143
SLIDE 143

http://syncwith.us prophet-subscribe@lists.bestpractical.com #prophet on freenode IRC

Thanks!

142