Opportunities & Pitfalls of Event-Driven Utopia @berndruecker
Why this talk
Why this talk
Agenda Events on the outside 3 Events on the inside 1 Service 1 Service 2
Agenda Events on the outside 3 Events on the inside 1 Service 1 Service 2 Events inside out 2
Agenda Events on the outside 3 Events on the inside 1 Service 1 Service 2 Events inside out 2
Once upon a time… Client BBC architecture Application (box - arrow – box – arrow - cylinder) Every architecture diagram you'll ever need RDMS
The great thing about this architecture Application DB gurantees RDMS (e.g. ACID)
The problem Application RDMS RDMS not webscale cloud-scale resiliency is expensive
Pat Helland https://vimeo.com/52831373
Persistent change Persistent state Append-only Log RDMS +2,500 $ … … transfered Account # Balance 12345 2,500$ bank -14.99$ account paid by created credit card Current Balance = 2,485.01 $
Persistent change Event Bank Account Created Append-only 2019/04/16 11:00 Log # 12345 +2,500 $ … … transfered Event bank -14.99$ Money T ransfer Received account paid by created credit card 2,500$ 2019/04/16 11:00 # 12345
An example from my world
Bernd Ruecker Co-founder and Chief T echnologist of Camunda mail@berndruecker.io @berndruecker
We build two different workflow engines. Why? Persistent Camunda BPM Zeebe Persistent State change
2. 1. 3. Workflow Engine 1. INSERT RDMS Workflow Current State Instance Activity Id 2. UPDAT E 2 RetrievePayment running Workflow Current State Instance Activity Id 3. UPDAT E 2 ShipGoods running Workflow Current State Instance Activity Id 2 OrderDelivered ended
2. 1. Workflow Engine Append-only 1. 2. Log create start workflow event activity lock complete activity instance occured activated created task completed … … workflow sequence task task task sequence instance flow taken created locked completed flow taken created
Event Handling, Replication & Single Writer 1 complete task send command Broker (Leader) 2 append command task completed store & replicate 3 event command store & replicate 7 5 respond event 6 append event Follower 4 process Follower Single Writer Stream (single thread) Processor
What we do different Store and replay 1 complete task send command commands Leader 2 append command task completed event store & replicate 3 command store & replicate 7 event 6 append event 5 respond Follower 4 process Persist & replicate Follower Delete records that Single Writer Stream internal state (single thread) are fully processed Processor
Consistency Availability Partition
Zeebe is CP 1 complete task send command Leader 2 append command task completed event store & replicate 3 command store & replicate 7 event 6 append event 5 respond Follower 4 process Follower Single Writer Stream (single thread) Processor
Horizontal scalability by partitioning Stream Single Writer Processor (single thread) Partition 1 instance id: 2-42 Partition 2 instance id: 3-66 Partition 3 Every workflow instance is exactly handled by one partition Partition 4
Queries and read models Zeebe ask Zeebe Broker Broker Streaming ask Exporter
@berndruecker Recap 1 – Events on the inside # Natural mechanism to build scalable services in distributed systems (with Outbox & co included) But # You have to think about reads, queries & eventual consistency # Few industry experience available
Agenda Events on the outside 3 Events on the inside 1 Service 1 Service 2 Events inside out 2
Event Store and Messaging Customer 1. Create Customer … Customer Event Store
Merge Messaging and Event Store Customer 1. Create Customer … Customer Event Store
Merge messaging and event store Customer 1. Create Customer … Shared Event Store
Enter the world of Kafka…
Merge messaging and event store Customer 1. Create Customer … Shared Event Store
Kafka as transport 1. Create Customer … Used as queue (but persistent!) Customer
Kafka as transport 1. Create Customer … Used as queue (but persistent!) Customer
Agenda Events on the outside 3 Events on the inside 1 Service 1 Service 2 Events inside out 2
Once upon a time Customer Change Address Billing
Event Notification Customer Address changed Billing
What‘s Event Notification Reverse direction direction of dependency of dependency Customer Customer Address Change changed Address Billing Billing
Event Notification Customer AdressChanged { Address customerId: 42 changed } Ask for details Billing
AddressChanged { Event-carried State T ransfer customerId: 42, oldAddress: ... newAddress: ... } AddressChanged { Customer customerId: 42, address: ... } Address changed CustomerChanged { customerId: 42, status: A, Billing address: ..., } CustomerMoved { ..., }
What‘s This decision is complex Reverse direction direction of dependency of dependency Customer Customer Address Change changed Address Billing Billing
Example Change Address Address Incoming Email From bla@company.com Submit Date 2019-04-23 09.05 To confirm your address change please click on this link: http://company.com/confirm?id=82e97d49-166c-4862- 9973-4db348e6225d
Example direction of dependency Notification http://company.com/confirm?id=82e97 Change Adress d49-166c-4862-9973-4db348e6225d Address change requested Address change Customer confirmed
Example direction of dependency Notification http://company.com/confirm?id=82e97 Change Adress d49-166c-4862-9973-4db348e6225d Send mail ‚ Confirmation ‘ ‚ Confirmation ‘ Customer approved Address changed
Challenge: Command Command vs. Event vs Event
It is NOT about communication protocols Customer Customer It can be messaging, Address Change changed REST, whatever , …. Address Billing Billing
Manifold ways of transport …
Manifold ways of transport …
Message Event ? Record Query Command Event Intend, Fact, Want s.th. to happen, happened in the past, The intention itself is a fact immutable
Message Event ? Record Query Command Event
Commands in disguise Send Wording of Message recipient The Customer Needs T o Be Sent A Message T o Confirm Address Change Wording of Sender Event
More general, Examples does not need to care about More general, who is interessted in address changes Address does not need to know Changed Order who is retrieving payments Customer Payment Order Billing Retrieve Payment Subscription Order Placed Order Notification Payment Global service Send Received Service that can Mail handle notifications Goods Notification for orders Shipped autonomously
Distributed Monoliths Authorization Service Page Document created Page Document attached moved moved Page Document Context … Context
Define stable contract/API instead Authorization Service Add auth Page Document Context … Context
Next challenge: Event chains
@berndruecker Event Chains Customer Registration Registration Address Credit Customer Event Bus requested checked checked registered Credit Check Adress Check
@berndruecker Event Chains Customer Registration Registration Address Credit Customer Event Bus requested checked checked registered How does customer Credit Check registration work? Adress Check
@berndruecker The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
@berndruecker The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
@berndruecker The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years. https://martinfowler.com/articles/201701-event-driven.html
@berndruecker Monitoring Workflows Across Microservices https://www.infoq.com/articles/monitor-workflow-collaborating-microservices
T ypical approaches Distributed T racing Data Lake / Event Monitoring Process Mining @berndruecker
@berndruecker What we currently build with customers … Camunda Optimize Elastic Registration Address Credit Customer requested checked checked registered
All great – until you have to move …
Recommend
More recommend
