Complex event flows in distributed systems @berndruecker With - - PowerPoint PPT Presentation

▶
complex event flows in
SMART_READER_LITE
LIVE PREVIEW

Complex event flows in distributed systems @berndruecker With - - PowerPoint PPT Presentation

Sep 05, 2022 228 likes •1.16k views

Complex event flows in distributed systems @berndruecker With thoughts from http://flowing.io @berndruecker | @martinschimak 3 common hypotheses I check today: # Events decrease coupling # Orchestration needs to be avoided # Workflow engines

slide-1
SLIDE 1

Complex event flows in distributed systems

@berndruecker

With thoughts from http://flowing.io @berndruecker | @martinschimak

slide-2
SLIDE 2

3 common hypotheses I check today: # Events decrease coupling # Orchestration needs to be avoided # Workflow engines are painful

slide-3
SLIDE 3

Berlin, Germany

bernd.ruecker@camunda.com @berndruecker

Bernd Ruecker

Co-founder and Developer Advocate of Camunda

slide-4
SLIDE 4
slide-5
SLIDE 5

Simplified example: dash button

Photo by 0xF2, available under Creative Commons BY-ND 2.0

  • license. https://www.flickr.com/photos/0xf2/29873149904/
slide-6
SLIDE 6

Three steps…

slide-7
SLIDE 7

Who is involved? Some bounded contexts…

Checkout Payment Inventory Shipment

slide-8
SLIDE 8

(Micro-)services

Checkout Payment Inventory Shipment

slide-9
SLIDE 9

Autonomous (micro-)services

Checkout Payment Inventory Shipment

Dedicated Application Processes Dedicated infrastructure Dedicated Development Teams

slide-10
SLIDE 10

Events decrease coupling

slide-11
SLIDE 11

Example

Checkout Payment Inventory Shipment

The button blinks if we can ship within 24 hours

slide-12
SLIDE 12

Request/response: temporal coupling

Checkout Payment Inventory Shipment

Request Response

The button blinks if we can ship within 24 hours

slide-13
SLIDE 13

T emporal decoupling with events and read models

Checkout Payment Inventory Shipment

Good Stored Read Model Good Fetched

The button blinks if we can ship within 24 hours *Events are facts about what happened (in the past)

slide-14
SLIDE 14

Events can decrease coupling*

*e.g. decentral data-management, read models, extract cross-cutting aspects

slide-15
SLIDE 15

Peer-to-peer event chains

Checkout Payment Inventory Shipment

Order placed Payment received Goods shipped Goods fetched

slide-16
SLIDE 16

Peer-to-peer event chains

Checkout Payment Inventory Shipment

Order placed Payment received Goods shipped Goods fetched

slide-17
SLIDE 17

The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.

https://martinfowler.com/articles/201701-event-driven.html

slide-18
SLIDE 18

The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.

https://martinfowler.com/articles/201701-event-driven.html

slide-19
SLIDE 19

The danger is that it's very easy to make nicely decoupled systems with event notification, without realizing that you're losing sight of that larger-scale flow, and thus set yourself up for trouble in future years.

https://martinfowler.com/articles/201701-event-driven.html

slide-20
SLIDE 20

Peer-to-peer event chains

Checkout Payment Inventory Shipment

Order placed Payment received Goods shipped Goods fetched

Fetch the goods before the payment

slide-21
SLIDE 21

Peer-to-peer event chains

Checkout Payment Inventory Shipment

Order placed Payment received Goods shipped Goods fetched

Fetch the goods before the payment Customers can pay via invoice …

slide-22
SLIDE 22

Peer-to-peer event chains

Checkout Payment Inventory Shipment

Order placed Payment received Goods shipped Goods fetched

Fetch the goods before the payment

slide-23
SLIDE 23

Peer-to-peer event chains

Checkout Payment Inventory Shipment

Fetch the goods before the payment

Goods fetched Order placed Payment received Goods shipped

slide-24
SLIDE 24

Photo by born1945, available under Creative Commons BY 2.0 license.

slide-25
SLIDE 25

Order

Extract the end-to-end responsibility

Checkout Payment Inventory Shipment

*Commands have an intent about what needs to happen in the future

Order placed Payment received Retrieve payment

slide-26
SLIDE 26

Commands help to avoid (complex) peer-to-peer event chains

slide-27
SLIDE 27

Orchestration needs to be avoided

slide-28
SLIDE 28

Smart ESB-like middleware

Checkout Payment Inventory Shipment Order

Order placed Payment received Good fetched Good shipped

slide-29
SLIDE 29

Dumb pipes

Checkout Payment Inventory Shipment Order

Smart endpoints and dumb pipes

Martin Fowler

slide-30
SLIDE 30

Danger of god services?

Checkout Order

A few smart god services tell anemic CRUD services what to do

Sam Newmann

Payment Inventory Shipment

slide-31
SLIDE 31

Danger of god services?

Checkout

Payment Inventory Shipment

Order

A few smart god services tell anemic CRUD services what to do

Sam Newmann

slide-32
SLIDE 32

A god service is only created by bad API design!

slide-33
SLIDE 33

Example

Order Payment

Retrieve Payment

slide-34
SLIDE 34

Example

Order Payment Credit Card

Retrieve Payment

slide-35
SLIDE 35

Example

Order Payment Credit Card

Retrieve Payment Rejected

slide-36
SLIDE 36

Example

Order Payment

Client of dumb endpoints easily become a god services. If the credit card was rejected, the customer can provide new details

Credit Card

Retrieve Payment Rejected Rejected

slide-37
SLIDE 37

Payment failed

Who is responsible to deal with problems?

Order Payment

If the credit card was rejected, the customer can provide new details

Credit Card

Retrieve Payment Rejected Payment received

slide-38
SLIDE 38

Payment failed

Long-running execution

Order Payment

Clients of smart endpoints remains lean. If the credit card was rejected, the customer can provide new details

Credit Card

Retrieve Payment Rejected Payment received

Smart endpoints are potentially long-running

slide-39
SLIDE 39

Persist thing (Entity, Document, Actor, …) State machine or workflow engine T ypical concerns DIY = effort, accidental complexity

Scheduling, Versioning,

  • perating, visibility,

scalability, …

Handling State

slide-40
SLIDE 40

Workflow engines are painful

Complex, proprietary, heavyweight, central, developer adverse, …

slide-41
SLIDE 41

Avoid the wrong tools!

Death by properties panel

Low-code is great! (You can get rid

  • f your developers!)

Complex, proprietary, heavyweight, central, developer adverse, …

slide-42
SLIDE 42

Workflow engines, state machines

It is

relevant

in modern architectures

slide-43
SLIDE 43

CADENCE

Silicon valley has recognized

Workflow engines, state machines

slide-44
SLIDE 44
slide-45
SLIDE 45

CADENCE

Workflow engines, state machines

slide-46
SLIDE 46

CADENCE

also at scale

Workflow engines, state machines

slide-47
SLIDE 47

Zeebe.io

slide-48
SLIDE 48

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") // .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } }

What do I mean by „leightweight?“

slide-49
SLIDE 49

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") // .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } }

Build engine in one line of code (using in- memory H2)

slide-50
SLIDE 50

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } }

Define flow e.g. in Java DSL

slide-51
SLIDE 51

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } }

Define flow e.g. in Java DSL

slide-52
SLIDE 52

BPMN

Business Process Model and Notation ISO Standard

slide-53
SLIDE 53

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } }

We can attach code…

slide-54
SLIDE 54

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } }

…that is called when workflow instances pass through

slide-55
SLIDE 55

public static void main(String[] args) { ProcessEngine engine = new StandaloneInMemProcessEngineConfiguration() .buildProcessEngine(); engine.getRepositoryService().createDeployment() // .addModelInstance("flow.bpmn", Bpmn.createExecutableProcess("flow") .startEvent() .serviceTask("Step1").camundaClass(SysoutDelegate.class) .serviceTask("Step2").camundaClass(SysoutDelegate.class) .endEvent() .done() ).deploy(); engine.getRuntimeService().startProcessInstanceByKey( "flow", Variables.putValue("city", "New York")); } public class SysoutDelegate implements JavaDelegate { public void execute(DelegateExecution execution) throws Exception { System.out.println("Hello " + execution.getVariable("city")); } }

Start instances

slide-56
SLIDE 56

Payment

Now you have a state machine!

slide-57
SLIDE 57

Payment

Easy to handle time

slide-58
SLIDE 58

martin.schimak@plexiti.com @martinschimak

slide-59
SLIDE 59

Synchronous communication

REST

Order Payment Credit Card

slide-60
SLIDE 60

Synchronous communication

REST

Order Payment Credit Card

slide-61
SLIDE 61

Synchronous communication

REST

Order Payment Credit Card

slide-62
SLIDE 62

Synchronous communication

REST

Order Payment Credit Card

Stateful Retry

slide-63
SLIDE 63

Distributed systems

slide-64
SLIDE 64

It is impossible to differentiate certain failure scenarios.

Independant of communication style!

Service Provider Client

slide-65
SLIDE 65

Distributed systems introduce complexity you have to tackle!

Credit Card Payment

REST

slide-66
SLIDE 66

Distributed systems introduce complexity you have to tackle!

Credit Card Payment

REST

slide-67
SLIDE 67

Distributed systems

slide-68
SLIDE 68

Distributed transactions using compensation *

Compensation

slide-69
SLIDE 69

Relaxed consistency

T emporarily inconsistent state But eventually consistent No Isolation (as in ACID)

slide-70
SLIDE 70

https://thenewstack.io/5-workflow-automation-use-cases-you-might-not-have-considered/

slide-71
SLIDE 71

<= milliseconds seconds minutes, weeks, … Business IT

Business processes automation Distributed T ransactions Orchestration Communication in distributed systems

long running always short running short running, but potentially long running

Use cases for workflow automation

slide-72
SLIDE 72

Biz Dev

improve communication improve communication

Ops

slide-73
SLIDE 73

Biz Dev

Leverage state machine & workflow engine Living documentation Visibility in testing

improve communication improve communication

Ops

slide-74
SLIDE 74

Visual HTML reports for test cases

slide-75
SLIDE 75

Biz Dev

Leverage state machine & workflow engine Living documentation Visibility in testing Understand and discuss business processes Evaluate optimizations in-sync with implementation

improve communication improve communication

Ops

slide-76
SLIDE 76

Living documentation for long-running behaviour

slide-77
SLIDE 77

Biz Dev

Leverage state machine & workflow engine Living documentation Visibility in testing Understand and discuss business processes Evaluate optimizations in-sync with implementation

improve communication improve communication

Ops

slide-78
SLIDE 78

Biz Dev

Leverage state machine & workflow engine Living documentation Visibility in testing Operate with visibility and context Understand and discuss business processes Evaluate optimizations in-sync with implementation

improve communication improve communication

Ops

slide-79
SLIDE 79

Proper Operations

Visibility + Context

slide-80
SLIDE 80

Before mapping processes explicitly with BPMN, the truth was buried in the code and nobody knew what was going on.

Jimmy Floyd, 24 Hour Fitnesse

„

slide-81
SLIDE 81

Workflows live inside service boundaries

slide-82
SLIDE 82

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

slide-83
SLIDE 83

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

slide-84
SLIDE 84

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

slide-85
SLIDE 85

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

slide-86
SLIDE 86

…

Manigfold architecture options

https://blog.bernd-ruecker.com/architecture-options-to-run-a-workflow-engine-6c2419902d91

slide-87
SLIDE 87

Lightweight workflow engines are great* – don‘t DIY

*e.g. enabling potentially long-running services, solving hard developer problems, can run decentralized

slide-88
SLIDE 88

Sales-Order & Order-Fulfillment via Camunda for every order worldwide (Q2 2017: 22,2 Mio)

slide-89
SLIDE 89

Code example & live demo

Inventory Payment Order Shipping Checkout Monitor

https://github.com/flowing/flowing-retail/

Human T asks

H2 H2
slide-90
SLIDE 90

# Events decrease coupling: sometimes

read-models, but no complex peer-to-peer event chains!

# Orchestration needs to be avoided: sometimes

no ESB, smart endpoints/dumb pipes, important capabilities need a home

# Workflow engines are painful: some of them

lightweight engines are easy to use and can run decentralized, they solve hard developer problems, don‘t DIY

slide-91
SLIDE 91

Thank you!

slide-92
SLIDE 92

bernd.ruecker@camunda.com @berndruecker https://bernd-ruecker.com https://blog.bernd-ruecker.com https://github.com/flowing

https://www.infoq.com/articles/events- workflow-automation

With thoughts from http://flowing.io @berndruecker | @martinschimak

Contact: Slides: Blog: Code:

https://www.infoworld.com/article/3254777/ application-development/ 3-common-pitfalls-of-microservices- integrationand-how-to-avoid-them.html https://thenewstack.io/5-workflow-automation- use-cases-you-might-not-have-considered/