Immutable data stores for safety, flexibility and profit SIDNEY - - PowerPoint PPT Presentation

▶
immutable data stores for safety flexibility and profit
SMART_READER_LITE
LIVE PREVIEW

Immutable data stores for safety, flexibility and profit SIDNEY - - PowerPoint PPT Presentation

Jan 05, 2023 348 likes •1.46k views

Immutable data stores for safety, flexibility and profit SIDNEY SHEK ARCHITECT ATLASSIAN @SIDNEYSHEK Event Sourcing: What and Why Universe of Users and Groups users users_groups GroupId UserId Id Name Username APIKey

slide-1
SLIDE 1

SIDNEY SHEK • ARCHITECT • ATLASSIAN • @SIDNEYSHEK

Immutable data stores for safety, flexibility and profit

slide-2
SLIDE 2

Event Sourcing: What and Why

slide-3
SLIDE 3

Universe of Users and Groups

Id Name Username APIKey 1 Homer homer abcd 2 Bart bart f00 users Id Name 1 Simpsons 2 Flanders groups GroupId UserId 1 1 1 2 users_groups

slide-4
SLIDE 4

Universe of Users and Groups

Id Name Username APIKey 1 Homer homer abcd 2 Bart bart f00 3 Maggie maggie baa users Id Name 1 Simpsons 2 Flanders groups GroupId UserId 1 1 1 2 1 3 users_groups

slide-5
SLIDE 5

Universe of Users and Groups

Id Name Username APIKey 1 Homer homer d0a 2 Bart bart f00 3 Maggie maggie baa users Id Name 1 Simpsons 2 Flanders groups GroupId UserId 1 1 1 2 1 3 users_groups

slide-6
SLIDE 6

Universe of Users and Groups

Id Name Username APIKey 1 Homer homer d0a 2 Bart bart f00 3 Maggie maggie baa users Id Name 1 Simpsons 2 Flanders groups GroupId UserId 1 1 1 2 1 3 users_groups

slide-7
SLIDE 7

If only there was a way to see what changed…

slide-8
SLIDE 8

Universe of Users and Groups

Id Name Username APIKey 1 Homer homer d0a 2 Bart bart f00 3 Maggie maggie baa users Id Name 1 Simpsons 2 Flanders groups GroupId UserId 1 2 1 3 users_groups Id Event Time 1 InsertUser(3, Maggie) 2 AddUserToGroup(3, 1) 10 3 SetAPIKey(1, d0a) 15 4 RemoveUserFromGroup(1, 1) 20 audit

slide-9
SLIDE 9

How many API key changes in the last 6 months?

slide-10
SLIDE 10

Universe of Users and Groups

Id Name Username APIKey 1 Homer homer d0a 2 Bart bart f00 3 Maggie maggie baa users Id Name 1 Simpsons 2 Flanders groups GroupId UserId 1 2 1 3 users_groups audit Id Event Time 1 InsertUser(3, Maggie) 2 AddUserToGroup(3, 1) 10 3 SetAPIKey(1, d0a) 15 4 RemoveUserFromGroup(1, 1) 20

slide-11
SLIDE 11

Universe of Users and Groups

Id Name Username APIKey 1 Homer homer d0a 2 Bart bart f00 3 Maggie maggie baa users Id Name 1 Simpsons 2 Flanders groups GroupId UserId 1 2 1 3 users_groups audit Id Event Time 1 InsertUser(3, Maggie) 2 AddUserToGroup(3, 1) 10 3 SetAPIKey(1, d0a) 15 4 RemoveUserFromGroup(1, 1) 20

slide-12
SLIDE 12

What if…

slide-13
SLIDE 13

Instead of audit for reporting only…

slide-14
SLIDE 14

Audit became our source of truth

slide-15
SLIDE 15

Event Sourcing!

slide-16
SLIDE 16

Universe of Users and Groups

Id Name Username APIKey 1 Homer homer d0a 2 Bart bart f00 3 Maggie maggie baa users Id Name 1 Simpsons 2 Flanders groups GroupId UserId 1 2 1 3 users_groups audit Id Event Time 1 InsertUser(3, Maggie) 2 AddUserToGroup(3, 1) 10 3 SetAPIKey(1, d0a) 15 4 RemoveUserFromGroup(1, 1) 20

slide-17
SLIDE 17

Universe of Users and Groups

Id Name Username APIKey 1 Homer homer d0a 2 Bart bart f00 3 Maggie maggie baa users Id Name 1 Simpsons 2 Flanders groups GroupId UserId 1 2 1 3 users_groups event source Id Event Time 1 InsertUser(3, Maggie) 2 AddUserToGroup(3, 1) 10 3 SetAPIKey(1, d0a) 15 4 RemoveUserFromGroup(1, 1) 20

slide-18
SLIDE 18

Universe of Users and Groups

Id Name Username APIKey 1 Homer homer d0a 2 Bart bart f00 3 Maggie maggie baa users Id Name 1 Simpsons 2 Flanders groups GroupId UserId 1 2 1 3 users_groups event source Id Event Time 1 InsertUser(3, Maggie) 2 AddUserToGroup(3, 1) 10 3 SetAPIKey(1, d0a) 15 4 RemoveUserFromGroup(1, 1) 20

slide-19
SLIDE 19

Universe of Users and Groups

Search service Remote replica Custom app

event source Id Event Time 1 InsertUser(3, Maggie) 2 AddUserToGroup(3, 1) 10 3 SetAPIKey(1, d0a) 15 4 RemoveUserFromGroup(1, 1) 20

slide-20
SLIDE 20

How do we store events?

slide-21
SLIDE 21

event source Id Event Time 1 InsertUser(3, Maggie) 2 AddUserToGroup(3, 1) 10 3 SetAPIKey(1, d0a) 15 4 RemoveUserFromGroup(1, 1) 20

Ordered ‘Sequence’ (key) Flexible Value

slide-22
SLIDE 22

We don’t need an RDBMS

slide-23
SLIDE 23

We just need a key-value store!

slide-24
SLIDE 24

Building an event sourcing library

slide-25
SLIDE 25

event sourcing lib:

bitbucket.org/atlassianlabs/eventsrc

slide-26
SLIDE 26

Step 1. Modelling the events table

slide-27
SLIDE 27

event source Id Event Time 1 InsertUser(3, Maggie) 2 AddUserToGroup(3, 1) 10 3 SetAPIKey(1, d0a) 15 4 RemoveUserFromGroup(1, 1) 20

Ordered ‘Sequence’ (S) Flexible Payload (E)

slide-28
SLIDE 28

event source Key Id Event Time 1 1 InsertUser(3, Maggie) 1 2 AddUserToGroup(3, 1) 10 1 3 SetAPIKey(1, d0a) 15 1 4 RemoveUserFromGroup(1, 1) 20

Ordered ‘Sequence’ (S) Flexible Payload (E) Stream Key (K) Event[K,S,E] Ignore this for now

slide-29
SLIDE 29

Stream of Events

trait EventStream { type E // Event payload e.g. InsertUser }

slide-30
SLIDE 30

Stream of Events

trait EventStream { type E // Event payload e.g. InsertUser type S // Sequence type e.g. Long }

slide-31
SLIDE 31

Stream of Events

trait EventStream { type E // Event payload e.g. InsertUser type S // Sequence type e.g. Long implicit def S: Sequence[S] }

slide-32
SLIDE 32

Stream of Events

trait EventStream { type E // Event payload e.g. InsertUser type S // Sequence type e.g. Long implicit def S: Sequence[S] type K // Stream key e.g. CompanyId }

slide-33
SLIDE 33

Stream of Events

case class Event[K, S, E]( payload: E)

slide-34
SLIDE 34

Stream of Events

case class Event[K, S, E]( id: EventId[K, S], payload: E) case class EventId[K, S](key: K, seq: S)

slide-35
SLIDE 35

Storing Events

trait EventStorage[F[_], K, S, E] { def put(event: Event[K, S, E]): F[Error \/ Event[K, S, E]] def get(key: K): Stream[Event[K, S, E]]? }

slide-36
SLIDE 36

Storing Events

import scalaz.stream.Process trait EventStorage[F[_], K, S, E] { def put(event: Event[K, S, E]): F[Error \/ Event[K, S, E]] def get(key: K): Process[F, Event[K, S, E]] }

slide-37
SLIDE 37

Stream of Events

trait EventStream[F[_]] { type E // Event payload e.g. InsertUser type S // Sequence type e.g. Long implicit def S: Sequence[S] type K // Stream key e.g. CompanyId type Ev = Event[K, S, E] def eventStore: EventStorage[F, K, S, E] implicit def M: Monad[F] }

slide-38
SLIDE 38

Let’s apply this to

  • ur users example
slide-39
SLIDE 39

Our User Account Events

sealed trait UserAccountEvent

slide-40
SLIDE 40

Our User Account Events

sealed trait UserAccountEvent case class InsertUser(id: UserId, name: String, username: String) extends UserAccountEvent

slide-41
SLIDE 41

Our User Account Events

sealed trait UserAccountEvent case class InsertUser(id: UserId, name: String, username: String) extends UserAccountEvent case class DeleteUser(id: UserId) extends UserAccountEvent

slide-42
SLIDE 42

Our User Account Events

sealed trait UserAccountEvent case class InsertUser(id: UserId, name: String, username: String) extends UserAccountEvent case class DeleteUser(id: UserId) extends UserAccountEvent case class SetAPIKey(id: UserId, apiKey: String) extends UserAccountEvent

slide-43
SLIDE 43

Our User Account Events

sealed trait UserAccountEvent case class InsertUser(id: UserId, name: String, username: String) extends UserAccountEvent case class DeleteUser(id: UserId) extends UserAccountEvent case class SetAPIKey(id: UserId, apiKey: String) extends UserAccountEvent case class AddUserToGroup(groupId: GroupId, userId: UserId) extends UserAccountEvent

slide-44
SLIDE 44

Our User Account Events

sealed trait UserAccountEvent case class InsertUser(id: UserId, name: String, username: String) extends UserAccountEvent case class DeleteUser(id: UserId) extends UserAccountEvent case class SetAPIKey(id: UserId, apiKey: String) extends UserAccountEvent case class AddUserToGroup(groupId: GroupId, userId: UserId) extends UserAccountEvent case class RemoveUserFromGroup(groupId: GroupId, userId: UserId) extends UserAccountEvent

slide-45
SLIDE 45

Our User Account EventStream

class UserAccountEventStream[F[_]]() extends EventStream[F] { }

slide-46
SLIDE 46

Our User Account EventStream

class UserAccountEventStream[F[_]]() extends EventStream[F] { type E = UserAccountEvent ... }

slide-47
SLIDE 47

Our User Account EventStream

class UserAccountEventStream[F[_]]() extends EventStream[F] { type E = UserAccountEvent type S = Long // Sequence[Long] already defined ... }

slide-48
SLIDE 48

Our User Account EventStream

class UserAccountEventStream[F[_]]() extends EventStream[F] { type E = UserAccountEvent type S = Long // Sequence[Long] already defined type K = CompanyId ... }

slide-49
SLIDE 49

Our User Account EventStream

class UserAccountEventStream[F[_]]( val eventStore: EventStorage[F, CompanyId, Long, UserAccountEvent]) extends EventStream[F] { type E = UserAccountEvent type S = Long // Sequence[Long] already defined type K = CompanyId ... }

slide-50
SLIDE 50

Step 2. Querying events

slide-51
SLIDE 51

Querying event streams

trait EventStream[F[_]] { ... trait QueryAPI[Key, Val] { def get(k: Key): F[Option[Val]] } }

slide-52
SLIDE 52

Querying event streams

def get(k: Key): F[Option[Val]] = eventStore.get(???)

slide-53
SLIDE 53

Querying event streams

def get(k: Key): F[Option[Val]] = streamFold(acc) { eventStore.get(???) }.map { _.value } def streamFold( f: (Snapshot[S, Val], Ev) => Snapshot[S, Val] )(stream: Process[F, Ev]): F[Snapshot[S, Val]] = ???

slide-54
SLIDE 54

Querying event streams

import scalaz.stream.process1 def get(k: Key): F[Option[Val]] = streamFold(acc) { eventStore.get(???) }.map { _.value } def streamFold( f: (Snapshot[S, Val], Ev) => Snapshot[S, Val] )(stream: Process[F, Ev]): F[Snapshot[S, Val]] = stream.pipe { process1.fold(Snapshot.zero[S, Val])(f) }.runLastOr(Snapshot.zero[S, Val])

slide-55
SLIDE 55

Querying event streams

trait QueryAPI[Key, Val] { def acc(k: Key)(s: Snapshot[S, Val], e: Ev):Snapshot[S, Val] def get(k: Key): F[Option[Val]] = streamFold(acc(k)) { eventStore.get(???) }.map { _.value } }

slide-56
SLIDE 56

Querying event streams

trait QueryAPI[Key, Val] { def toStreamKey: Key => K def acc(k: Key)(s: Snapshot[S, Val], e: Ev):Snapshot[S, Val] def get(k: Key): F[Option[Val]] = streamFold(acc) { eventStore.get(toStreamKey(k)) }.map { _.value } }

slide-57
SLIDE 57

Let’s apply this to

  • ur users example
slide-58
SLIDE 58

Get group members

class GroupMembersById extends QueryAPI[CompanyGroupId, List[UserId]] { def toStreamKey: CompanyGroupId => CompanyId = _.companyId def acc(k: CompanyGroupId)(s: Snapshot[Long, List[UserId]], e: Ev) = }

slide-59
SLIDE 59

Get group members

class GroupMembersById extends QueryAPI[CompanyGroupId, List[UserId]] { def toStreamKey: CompanyGroupId => CompanyId = _.companyId def acc(k: CompanyGroupId)(s: Snapshot[Long, List[UserId]], e: Ev) = e.payload match { case AddUserToGroup(groupId, userId) if k.groupId == groupId => } }

slide-60
SLIDE 60

Get group members

class GroupMembersById extends QueryAPI[CompanyGroupId, List[UserId]] { def toStreamKey: CompanyGroupId => CompanyId = _.companyId def acc(k: CompanyGroupId)(s: Snapshot[Long, List[UserId]], e: Ev) = e.payload match { case AddUserToGroup(groupId, userId) if k.groupId == groupId => val currentList = s.value.getOrElse(List()) } }

slide-61
SLIDE 61

Get group members

class GroupMembersById extends QueryAPI[CompanyGroupId, List[UserId]] { def toStreamKey: CompanyGroupId => CompanyId = _.companyId def acc(k: CompanyGroupId)(s: Snapshot[Long, List[UserId]], e: Ev) = e.payload match { case AddUserToGroup(groupId, userId) if k.groupId == groupId => val currentList = s.value.getOrElse(List()) val newList = userId :: currentList.filterNot { _ == userId } } }

slide-62
SLIDE 62

Get group members

class GroupMembersById extends QueryAPI[CompanyGroupId, List[UserId]] { def toStreamKey: CompanyGroupId => CompanyId = _.companyId def acc(k: CompanyGroupId)(s: Snapshot[Long, List[UserId]], e: Ev) = e.payload match { case AddUserToGroup(groupId, userId) if k.groupId == groupId => val currentList = s.value.getOrElse(List()) val newList = userId :: currentList.filterNot { _ == userId } Snapshot.value(newList, e.id.seq) } }

slide-63
SLIDE 63

Get group members

class GroupMembersById extends QueryAPI[CompanyGroupId, List[UserId]] { def toStreamKey: CompanyGroupId => CompanyId = _.companyId def acc(k: CompanyGroupId)(s: Snapshot[Long, List[UserId]], e: Ev) = e.payload match { case AddUserToGroup(groupId, userId) if k.groupId == groupId => val currentList = s.value.getOrElse(List()) val newList = userId :: currentList.filterNot { _ == userId } Snapshot.value(newList, e.id.seq) case RemoveUserFromGroup(groupId, userId) if k.groupId == groupId => } }

slide-64
SLIDE 64

Get group members

class GroupMembersById extends QueryAPI[CompanyGroupId, List[UserId]] { def toStreamKey: CompanyGroupId => CompanyId = _.companyId def acc(k: CompanyGroupId)(s: Snapshot[Long, List[UserId]], e: Ev) = e.payload match { case AddUserToGroup(groupId, userId) if k.groupId == groupId => val currentList = s.value.getOrElse(List()) val newList = userId :: currentList.filterNot { _ == userId } Snapshot.value(newList, e.id.seq) case RemoveUserFromGroup(groupId, userId) if k.groupId == groupId => val currentList = s.value.getOrElse(List()) } }

slide-65
SLIDE 65

Get group members

class GroupMembersById extends QueryAPI[CompanyGroupId, List[UserId]] { def toStreamKey: CompanyGroupId => CompanyId = _.companyId def acc(k: CompanyGroupId)(s: Snapshot[Long, List[UserId]], e: Ev) = e.payload match { case AddUserToGroup(groupId, userId) if k.groupId == groupId => val currentList = s.value.getOrElse(List()) val newList = userId :: currentList.filterNot { _ == userId } Snapshot.value(newList, e.id.seq) case RemoveUserFromGroup(groupId, userId) if k.groupId == groupId => val currentList = s.value.getOrElse(List()) val newList = currentList.filterNot { _ == userId } } }

slide-66
SLIDE 66

Get group members

class GroupMembersById extends QueryAPI[CompanyGroupId, List[UserId]] { def toStreamKey: CompanyGroupId => CompanyId = _.companyId def acc(k: CompanyGroupId)(s: Snapshot[Long, List[UserId]], e: Ev) = e.payload match { case AddUserToGroup(groupId, userId) if k.groupId == groupId => val currentList = s.value.getOrElse(List()) val newList = userId :: currentList.filterNot { _ == userId } Snapshot.value(newList, e.id.seq) case RemoveUserFromGroup(groupId, userId) if k.groupId == groupId => val currentList = s.value.getOrElse(List()) val newList = currentList.filterNot { _ == userId } Snapshot.value(newList, e.id.seq) } }

slide-67
SLIDE 67

Get group members

class GroupMembersById extends QueryAPI[CompanyGroupId, List[UserId]] { def toStreamKey: CompanyGroupId => CompanyId = _.companyId def acc(k: CompanyGroupId)(s: Snapshot[Long, List[UserId]], e: Ev) = e.payload match { case AddUserToGroup(groupId, userId) if k.groupId == groupId => val currentList = s.value.getOrElse(List()) val newList = userId :: currentList.filterNot { _ == userId } Snapshot.value(newList, e.id.seq) case RemoveUserFromGroup(groupId, userId) if k.groupId == groupId => val currentList = s.value.getOrElse(List()) val newList = currentList.filterNot { _ == userId } Snapshot.value(newList, e.id.seq) case _ => Snapshot.noop(s, e.id.seq) } }

slide-68
SLIDE 68

Bonus: Going back in time

slide-69
SLIDE 69

Querying event streams…with history

def get(k: Key): F[Option[Val]] = streamFold(acc(k)) { eventStore.get(toStreamKey(k)) }.map { _.value } def getAt(k: Key, s: S): F[Option[Val]] =

slide-70
SLIDE 70

Querying event streams…with history

def get(k: Key): F[Option[Val]] = streamFold(acc(k)) { eventStore.get(toStreamKey(k)) }.map { _.value } def getAt(k: Key, s: S): F[Option[Val]] = streamFold(acc(k)) { eventStore.get(toStreamKey(k)).takeWhile { e => S.order.lessThanOrEqual(e.id.s, s) } }.map { _.value }

slide-71
SLIDE 71

Step 3. Saving events

slide-72
SLIDE 72

trait EventStream[F[_]] { ... trait SaveAPI[Key, Val] { def save(k: Key, e: E): F[SaveResult[S, Val]] } }

slide-73
SLIDE 73

trait SaveAPI[Key, Val] { def save(k: Key, e: E): F[SaveResult[S, Val]] = for {

  • ld <- getLatestSnapshot(k)

} yield ??? }

slide-74
SLIDE 74

trait SaveAPI[Key, Val] { def save(k: Key, e: E): F[SaveResult[S, Val]] = for {

  • ld <- getLatestSnapshot(k)

putResult <- eventStore.put(k, Event.next(old.seq, e)) } yield ??? }

slide-75
SLIDE 75

trait SaveAPI[Key, Val] { def save(k: Key, e: E): F[SaveResult[S, Val]] = for {

  • ld <- getLatestSnapshot(k)

putResult <- eventStore.put(k, Event.next(old.seq, e)) saveResult <- putResult match { case \/-(ev) => SaveResult.success(newValue(old, ev)) } } yield saveResult }

slide-76
SLIDE 76

trait SaveAPI[Key, Val] { def save(k: Key, e: E): F[SaveResult[S, Val]] = for {

  • ld <- getLatestSnapshot(k)

putResult <- eventStore.put(k, Event.next(old.seq, e)) saveResult <- putResult match { case \/-(ev) => SaveResult.success(newValue(old, ev)) case -\/(Error.DuplicateEventId) => save(k, e) } } yield saveResult }

slide-77
SLIDE 77

trait SaveAPI[Key, Val] { def save(k: Key, e: E): F[SaveResult[S, Val]] = for {

  • ld <- getLatestSnapshot(k)

putResult <- eventStore.put(k, Event.next(old.seq, e)) saveResult <- putResult match { case \/-(ev) => SaveResult.success(newValue(old, ev)) case -\/(Error.DuplicateEventId) => save(k, e) case -\/(Error.Rejected(reasons)) => SaveResult.reject(reasons) } } yield saveResult }

slide-78
SLIDE 78

abstract class SaveAPI[Key, Val](query: QueryAPI[Key, Val]) { def save(k: Key, e: E): F[SaveResult[S, Val]] = for {

  • ld <- query.getLatestSnapshot(k)

putResult <- eventStore.put(k, Event.next(old.seq, e)) saveResult <- putResult match { case \/-(ev) => SaveResult.success(query.acc(old, ev).value) case -\/(Error.DuplicateEventId) => save(k, e) case -\/(Error.Rejected(reasons)) => SaveResult.reject(reasons) } } yield saveResult }

slide-79
SLIDE 79

What about data constraints?

slide-80
SLIDE 80

Operation: Constraint as a type

case class Operation[S, Val, E]( run: Snapshot[S, Val] => OpResult[E])

slide-81
SLIDE 81

Operation: Constraint as a type

case class Operation[S, Val, E]( run: Snapshot[S, Val] => OpResult[E]) sealed trait OpResult[E] case class Success[E](e: E) extends OpResult[E] case class Reject[E](reasons: List[Reason]) extends OpResult[E]

slide-82
SLIDE 82

Operation: Constraint as a type

  • bject Operation {

def ifNew(e: E): Operation[S, Val, E] = Operation { _.value match case None => Success(e) case Some(_) => Reject(List(Reason(“Duplicate value”))) } ... }

slide-83
SLIDE 83

Operation: Constraint as a type

  • bject Operation {

def ifNew(e: E): Operation[S, Val, E] = Operation { _.value match case None => Success(e) case Some(_) => Reject(List(Reason(“Duplicate value”))) } def ifSeq(seq: Option[S], e: E): Operation[S, Val, E] = Operation { s => if (s.seq == seq) Success(e) else Reject(List(Reason(“Sequence mismatch”))) } ... }

slide-84
SLIDE 84

abstract class SaveAPI[Key, Val](query: QueryAPI[Key, Val]) { def save(k: Key, e: E): F[SaveResult[S, Val]] = for {

  • ld <- query.getLatestSnapshot(k)

putResult <- eventStore.put(k, Event.next(old.seq, e)) saveResult <- putResult match { case \/-(ev) => SaveResult.success(query.acc(old, ev).value) case -\/(Error.DuplicateEventId) => save(k, e) case -\/(Error.Rejected(reasons)) => SaveResult.reject(reasons) } } yield saveResult }

Save without Constraints

slide-85
SLIDE 85

abstract class SaveAPI[Key, Val](query: QueryAPI[Key, Val]) { def save(k: Key, op: Operation[S, Val, E]): F[SaveResult[S, Val]] = for {

  • ld <- query.getLatestSnapshot(k)

saveResult <- putResult match { case \/-(ev) => SaveResult.success(query.acc(old, ev).value) case -\/(Error.DuplicateEventId) => save(k, e) case -\/(Error.Rejected(reasons)) => SaveResult.reject(reasons) } } yield saveResult }

Save with Constraints

slide-86
SLIDE 86

abstract class SaveAPI[Key, Val](query: QueryAPI[Key, Val]) { def save(k: Key, op: Operation[S, Val, E]): F[SaveResult[S, Val]] = for {

  • ld <- query.getLatestSnapshot(k)
  • pResult = op.run(old)

saveResult <- putResult match { case \/-(ev) => SaveResult.success(query.acc(old, ev).value) case -\/(Error.DuplicateEventId) => save(k, e) case -\/(Error.Rejected(reasons)) => SaveResult.reject(reasons) } } yield saveResult }

Save with Constraints

slide-87
SLIDE 87

abstract class SaveAPI[Key, Val](query: QueryAPI[Key, Val]) { def save(k: Key, op: Operation[S, Val, E]): F[SaveResult[S, Val]] = for {

  • ld <- query.getLatestSnapshot(k)
  • pResult = op.run(old)

putResult <- opResult match { case Success(e) => eventStore.put(k, Event.next(old.seq, e)) case Reject(rs) => SaveResult.reject(rs) } saveResult <- putResult match { case \/-(ev) => SaveResult.success(query.acc(old, ev).value) case -\/(Error.DuplicateEventId) => save(k, e) case -\/(Error.Rejected(reasons)) => SaveResult.reject(reasons) } } yield saveResult }

Save with Constraints

slide-88
SLIDE 88

Let’s apply this to

  • ur users example
slide-89
SLIDE 89

Saving a user record

trait DataAccess { def saveUser(u: User): F[SaveResult[Long, User]] }

slide-90
SLIDE 90

Saving a user record

def eventSourcedDataAccess(stream: UserAccountEventStream) (saveAPI: stream.SaveAPI[???, User]): DataAccess = new DataAccess { def saveUser(u: User): F[SaveResult[Long, User]] = { } }

slide-91
SLIDE 91

Saving a user record

def eventSourcedDataAccess(stream: UserAccountEventStream) (saveAPI: stream.SaveAPI[???, User]): DataAccess = new DataAccess { def saveUser(u: User): F[SaveResult[Long, User]] = { val event = InsertUser(u.id, u.name, u.username) } }

slide-92
SLIDE 92

Saving a user record

def eventSourcedDataAccess(stream: UserAccountEventStream) (saveAPI: stream.SaveAPI[???, User]): DataAccess = new DataAccess { def saveUser(u: User): F[SaveResult[Long, User]] = { val event = InsertUser(u.id, u.name, u.username) val operation = Operation[Long, User] { } saveAPI.save(???, operation) } }

slide-93
SLIDE 93

Saving a user record

def eventSourcedDataAccess(stream: UserAccountEventStream) (saveAPI: stream.SaveAPI[CompanyUsername, User]): DataAccess = new DataAccess { def saveUser(u: User): F[SaveResult[Long, User]] = { val event = InsertUser(u.id, u.name, u.username) val operation = Operation[Long, User] { } saveAPI.save((u.id.company, u.username), operation) } }

slide-94
SLIDE 94

Saving a user record

def eventSourcedDataAccess(stream: UserAccountEventStream) (saveAPI: stream.SaveAPI[CompanyUsername, User]): DataAccess = new DataAccess { def saveUser(u: User): F[SaveResult[Long, User]] = { val event = InsertUser(u.id, u.name, u.username) val operation = Operation[Long, User] { _.value match { case None => OpResult.Success(event) } } saveAPI.save((u.id.company, u.username), operation) } }

slide-95
SLIDE 95

Saving a user record

def eventSourcedDataAccess(stream: UserAccountEventStream) (saveAPI: stream.SaveAPI[CompanyUsername, User]): DataAccess = new DataAccess { def saveUser(u: User): F[SaveResult[Long, User]] = { val event = InsertUser(u.id, u.name, u.username) val operation = Operation[Long, User] { _.value match { case None => OpResult.Success(event) case Some(x) if u.id == x.id => OpResult.Success(event) } } saveAPI.save((u.id.company, u.username), operation) } }

slide-96
SLIDE 96

Saving a user record

def eventSourcedDataAccess(stream: UserAccountEventStream) (saveAPI: stream.SaveAPI[CompanyUsername, User]): DataAccess = new DataAccess { def saveUser(u: User): F[SaveResult[Long, User]] = { val event = InsertUser(u.id, u.name, u.username) val operation = Operation[Long, User] { _.value match { case None => OpResult.Success(event) case Some(x) if u.id == x.id => OpResult.Success(event) case _ => OpResult.Reject(List(Reason(“Duplicate username”))) } } saveAPI.save((u.id.company, u.username), operation) } }

slide-97
SLIDE 97

Our event sourcing library so far…

slide-98
SLIDE 98

Query API SaveAPI

User Details

EventStream Query API Query API Query API

Operation

Group Details Group Members Events

e.g. Add User

EventStorage

slide-99
SLIDE 99

Step 4. Bring it all together

slide-100
SLIDE 100

// 1. Instantiate a stream with an EventStorage val eventStore = new DynamoEventStorage(...) val stream = new UserAccountEventStream[Task](eventStore)

slide-101
SLIDE 101

// 1. Instantiate a stream with an EventStorage val eventStore = new DynamoEventStorage(...) val stream = new UserAccountEventStream[Task](eventStore) // 2. Create QueryAPIs defined for stream val userById = new stream.UserById // QueryAPI[CompanyUserId, User] val userByName = new stream.UserByName // QueryAPI[CompanyUsername, User]

slide-102
SLIDE 102

// 1. Instantiate a stream with an EventStorage val eventStore = new DynamoEventStorage(...) val stream = new UserAccountEventStream[Task](eventStore) // 2. Create QueryAPIs defined for stream val userById = new stream.UserById // QueryAPI[CompanyUserId, User] val userByName = new stream.UserByName // QueryAPI[CompanyUsername, User] // 3. Create SaveAPIs defined for stream val saveAPI = new stream.SaveAPI(userByName) // Create DataAccess with saveUser with Operation logic val dataLayer = new DataAccess(stream)(saveAPI)

slide-103
SLIDE 103

// 1. Instantiate a stream with an EventStorage val eventStore = new DynamoEventStorage(...) val stream = new UserAccountEventStream[Task](eventStore) // 2. Create QueryAPIs defined for stream val userById = new stream.UserById // QueryAPI[CompanyUserId, User] val userByName = new stream.UserByName // QueryAPI[CompanyUsername, User] // 3. Create SaveAPIs defined for stream val saveAPI = new stream.SaveAPI(userByName) // Create DataAccess with saveUser with Operation logic val dataLayer = new DataAccess(stream)(saveAPI) val saveAndGetUser: Task[Option[User]] = for { _ <- dataLayer.saveUser(User(...)) // Task[SaveResult[User]] saved <- userById.query(...) } yield saved request.run // Run it!

slide-104
SLIDE 104

Observations and next steps

slide-105
SLIDE 105

Safety?

Type-safe append-only storage Ability to query for historical values

slide-106
SLIDE 106

Flexibility?

Add QueryAPIs for new views

  • f existing data

Pluggable event storage Accumulator model supports incremental calculations e.g. hashing

slide-107
SLIDE 107

Code versus SQL…

slide-108
SLIDE 108

Performance?

We’re still tuning…

  • Delivering events to appropriate

tools e.g. ElasticSearch

  • Snapshot caching

with a few tricks up our sleeves:

  • Sharded snapshots
slide-109
SLIDE 109
slide-110
SLIDE 110

event sourcing lib:

bitbucket.org/atlassianlabs/eventsrc