Disruptive Development using Reactive Event-Sourced Systems Jan Ypma jyp@tradeshift.com 1 / 42
Agenda A bit of Tradeshift history Typical sources of failures Event sourcing Actors, Akka, and Clustering Example use case: collaboration
Frontend Backend x10 Postgresql Tradeshift in 2011 Two major software components the frontend the backend <10 developers
Payments x150 Backend SFTP Public API App Backend (old) App Backend (new) Workflow Frontend (old) Payments (new) Conversions Others... Tradeshift in 2016 30 deployed components (and growing) 150 developers 250.000 LoC in the backend
Scaling of Tradeshift's systems Moore's law applies to AWS Single point of not quite failing often enough 2016 directive: All new components must be clustered Yeah, what about the 30-ish existing ones? New architecture is needed
Scaling of Tradehift's development process 2011: "We're a Java shop"
Scaling of Tradehift's development process 2011: "We're a Java shop" 2016: Not really, at least not anymore Groovy and grails Python, go, ruby for infrastructure Crazy javascript people Scala But still, mostly Java Empower teams to pick their own language and frameworks
Frontend Backend Public API App Backend (old) Typical sources of failures We're down! We overloaded the database which caused the backend to respond slowly which caused the frontend to respond slowly which caused our users' web browsers to respond slowly which caused our users to reload their page GOTO 10
Frontend Backend Public API App Backend (old) Typical sources of failures We're down! We overloaded the database which caused the backend to respond slowly which caused the frontend to respond slowly which caused our users' web browsers to respond slowly which caused our users to reload their page GOTO 10 Enter the buzzwords Let it crash [2003, Amstrong] Micro-services [2005, Rodgers] Self-contained systems [2015, scs-architecture.org]
Service 2 Service 1 Self-contained systems No outgoing calls while handling an incoming request (except to our own databases) All inter-service communication must be asynchronous This implies data replication No single points of failure System must be clustered Design must trivially scale to 10x expected load
User #1 Created Name changed to Alice Pet added: Gary the Goldfish Pet removed: Gary the Goldfish User #2 Created Name changed to Bob Pet added: Charlie the Cat Event sourcing System considers an append-only Event journal the only source of truth Aggregate is one unit of information to which (and only which) an event atomically applies Events have a guaranteed order, but only within an aggregate
Event sourcing Nice scalability properties Each aggregate can process changes independently All information that spans >1 aggregate is materialized using event listeners Traditionally only applied inside a system Synchronous APIs only (" get customer history ") Why not expose event stream itself? Eventual consistency Latency implications Security implications
Implementation
The actor model Actor is an entity that responds only to messages by sending messages to other actors creating other (child) actors adjusting its behaviour Akka is a toolkit for writing actors in Java Actor is a normal Java class that extends UntypedActor or AbstractActor Message is an immutable, serializable, Java class Parent actor is the supervisor of its child actors. On child actor failure, parent decides what to do: Restart child Stop child Escalate
Actor ping pong public class PongActor extends UntypedActor { public void onReceive (Object message) { if (message instanceof String) { System.out.println("In PongActor - received message: " + message); getSender().tell("pong", getSelf()); } } }
Actor ping pong public class Initialize {} public class PingActor extends AbstractActor { private int counter = 0; private ActorRef pongActor = getContext().actorOf(Props.create(PongActor.class), "pongActor"); { receive(ReceiveBuilder .match(Initialize.class, msg -> { System.out.println("In PingActor - starting ping-pong"); pongActor.tell("ping", getSelf()); }) .match(String.class, msg -> { System.out.println("In PingActor - received message: " + message); counter += 1; if (counter == 3) { getContext().system().shutdown(); } else { getSender().tell("ping", getSelf()); } }) .build()); } }
Actor ping ping public static void main () { ActorSystem system = ActorSystem.create(); ActorRef pingActor = system.actorOf(Props.create(PingActor.class)); PingActor.tell( new Initialize()); } Output: In PingActor - starting ping-pong In PongActor - received message: ping In PingActor - received message: pong In PongActor - received message: ping In PingActor - received message: pong In PongActor - received message: ping In PingActor - received message: pong
Akka persistence Framework to do event sourcing using actors Persistence plugins for levelDB, cassandra, kafka, ... Each PersistentActor has a String identifier, under which events are stored public class ChatActor extends AbstractPersistentActor { private final List<String> messages = new ArrayList<>(); @Override public String persistenceId () { return "chat-1"; } private void postMessage (String msg) { persist(msg, evt -> { messages.add(msg); sender().tell(Done.getInstance(), self()); }); } private void getMessageList () { sender().tell( new ArrayList<>(messages), self()); } // ... }
Akka persistence public class ChatActor extends AbstractPersistentActor { private final List<String> messages = new ArrayList<>(); private void postMessage (String msg) { /* ... */ } private void getMessageList () { /* ... */ } @Override public String persistenceId () { return "chat-1"; } @Override public PartialFunction<Object,BoxedUnit> receiveRecover () { return ReceiveBuilder .match(String.class, messages::add) .build(); } @Override public void receiveCommand () { return ReceiveBuilder .matchEquals("/list", msg -> getMessageList()) .match(String.class, this ::postMessage) .build(); } }
Akka remoting and clustering Transparently lets actors communicate between systems ActorRef can point to a remote actor Messages must be serializable (using configurable mechanisms) akka { actor { provider = "akka.remote.RemoteActorRefProvider" } remote { enabled-transports = ["akka.remote.netty.tcp"] netty.tcp { hostname = "127.0.0.1" port = 2552 } } cluster { seed-nodes = [ "akka.tcp://ClusterSystem@127.0.0.1:2551", "akka.tcp://ClusterSystem@127.0.0.1:2552"] } }
Akka cluster sharding Dynamically distributes a group of actors across an akka cluster MessageExtractor informs cluster sharding where a particular message should go class ChatMessage { UUID conversation; String msg; } class MyMessageExtractor implements MessageExtractor { private final int numberOfShards = 256; @Override public String entityId (Object command) { return ChatMessage.cast(command).conversation.toString(); } @Override public String shardId (Object command) { return String.valueOf(entityId(command).hashCode() % numberOfShards); } @Override public Object entityMessage (Object command) { return ChatMessage.cast(command).msg; } }
Akka cluster sharding ShardRegion proxy sits between client and real (remote) persistent actor Persistent actor names will be their persistence id public class ChatActor extends AbstractPersistentActor { // ... @Override public String persistenceId () { return getSelf().path().name(); } } ActorRef proxy = ClusterSharding.get(system).start( "conversations", Props.create(ChatActor.class), ClusterShardingSettings.create(system), new MyMessageExtractor()); proxy.tell( new ChatMessage( UUID.fromString("67c67d28-4719-4bf9-bfe6-3944ed961a60"), "hello!"));
Putting it all together 2015: Let's try it out first 2016: Collaboration in production Real-time text message exchange between employees Text interface to automated travel agent In development: documents, FTP Stuff that works well for us: https://github.com/Tradeshift/ts-reaktive/ ts-reaktive-actors : Persistent actor base classes with reasonable defaults, and HTTP API for event journal ts-reaktive-marshal : Non-blocking streaming marshalling framework for akka streams ts-reaktive-replication : Master-slave replication across data centers for persistent actors
A bit of refactoring Introduce a single base class for all commands public abstract class ChatCommand {} public class GetMessageList extends ChatCommand {} public class PostMessage extends ChatCommand { private final String message; public PostMessage (String message) { this .message = message; } public String getMessage () { return message; } }
A bit of refactoring Introduce a single base class for all events public abstract class ChatEvent {} public class MessagePosted extends ChatEvent { private final String message; public MessagePosted (String message) { this .message = message; } public String getMessage () { return message; } }
Recommend
More recommend
