The Event-Driven Enterprise: Rabobanks Data Streaming Journey - - PowerPoint PPT Presentation

in collaboration with

The Event-Driven Enterprise: Rabobank’s Data Streaming Journey

Vincent Oostindië Jeroen van Disseldorp

Who are we?

Jeroen van Disseldorp Managing Director Vincent Oostindië Business Architect

Rabobank’s mission, vision and strategy

Growing a better world together

Banking for the Netherlands Banking for Food

Excellent customer focus Rock-solid bank Empowered employees

10 Strategic Top Priorities

• 100% digital

convenience in everything

• Top customer advice

nearby

• Growth with innovation
• Top performance
• Optimal balance sheet
• Exceptionally good

execution

• Concrete socially

responsible contribution

• Involved members and

communities

• Inspired employees
• One-Rabobank culture

Meaningful cooperative

Streaming helps Rabobank fulfill its mission

Increase relevance Become real-time Be 100% digital

Goals

Support financial health and growth

What we needed

A solution where “things that happen” can be captured from anywhere and delivered to anywhere for processing, as they occur, without consumers and producers knowing about each others existence.

The One Big Constraint Requirements

Security DevOps Control 24/7

Being a bank means a lot…

• A marketplace where applications exchange Business Events at the moment they occur.
• A Business Event is something that happens or takes place to which organizational entities might want to

react.

• There are three types of interactions

§ Producers detect certain business events and post them in “business meaningful” terms. § Consumers subscribe to streams (or types of events) and handle them as they come in. § Streaming apps consume from one or more streams and produce to others.

What is a Streaming Platform?

Marketing Sales Purchasing Accounting Production Inventory Logistics Customer Service HR Asset Mgmt R&D IT

Business Event Bus

Examples of event-driven banking

Transform a youth account to a student account when turning 18 Customer birthday

Business Event Bus

Personal alert generation SMS Fraud detection Booking on a payment account Customer logging in Relevance engine Action Email Push

Choosing a streaming engine

Rabobank chose Apache Kafka

Pros

• Rich functionality
• Very good quality/cost ratio
• Popular amongst developers
• Supports on-premise & all public/private clouds
• Available as DIY, Managed Service and SaaS

Cons

• No reference architectures
• Steep learning curve

Kafka Pulsar Solace Tibco Oracle Kinesis Pub/Sub EventHub

Data Owner IT Architect Developer Operator Procurement

Shortcomings of Kafka for Enterprises

Data governance Schema management Application governance Data lineage GDPR compliance Self-service for DevOps teams HA applications Development accelerators Test frameworks CI/CD support Standardized architecture Security architecture End-to-end encryption Flexible use of stream engines Data replication Hybrid cloud Multitenancy Name spacing Monitoring Alerting Pre-defined alert rules Fool-proof operations Vendor credibility Regulatory compliance Auditability Proven solution Flexible contracts

“ Streaming Made Simple ”

We enable DevOps teams to create business value using streaming data in a matter of hours, while guarding enterprise governance and security.

Platform

General purpose Ready-to-go Secure and governed Hybrid cloud / on-premise

Tools

Development accelerators Example code Documentation Self-service

Support

DevOps team Use-case intakes Design reviews Expert consultancy

Our vision

Guiding Principles

High Availability Security Data Governance Self-Service

Guiding Principles

Business Event Bus

Client Application DC or Cloud DC or Cloud

Multi-directional message/offset replication

Apache Zookeeper

Apache Kafka

Apache Zookeeper Apache Kafka

API API

Discovery API Axual Client Library Replication within datacenters

• Each DC runs its own Kafka cluster
• Messages are replicated within a cluster
• Clusters withstand node failures

One logical platform for applications

• Axual extends Kafka across DCs
• Messages and offsets are replicated asynchronously between DCs
• Upon DC failure, apps are dynamically routed to an alternative DC

Apps’ perspective

• Applications do not know about infrastructure absolutes

§

• eg. Kafka clusters, schema registries, rest proxies, MQ endpoints
• Apps query the Cluster Discovery API to find out where they should

produce/consume

• They repeat the query every 10 minutes (supports disaster recovery

and planned maintenance windows)

Guiding Principles

Producer 1 Producer 2 Producer 3 Data Stream Data Stream Data Stream Data Stream Data Stream Data Stream Data Stream Data Stream Data Stream Consumer 1 Consumer 2 Consumer 3 Consumer n

Business Event Bus

Security Mechanisms

• All connections are secured by two-way SSL
• TLS v1.1 – v1.3 are supported, TLS v1.0 is deprecated and turned off

by default

• Certificates are used to authenticate and perform stream

authorization

• Application only gain access to streams that it has rights for

Application Catalog

• Central repository registering all known apps
• Every app must declare the SSL certificates it uses
• Certificates must be signed by a trusted Certificate Authority

Stream Access Rights Management

• Streams are secured through Access Control Lists
• Applications are assigned rights to either produce, consume or both
• Streams are physically separated from each other, stored in separate

files

Guiding Principles

Maintain control over your streaming landscape

• Assign ownership of resources like streams and schema
• Promote reuse of existing data streams
• Allow stream and schema versions to co-exist

Stream Governance

• Central repository with metadata definitions
• Administers stream properties like partitioning, retention time and

message formats

• Treat as promotable artifact, deployable to different environments

Schema Governance

• Central repository where schemas are registered and maintained
• Contains all versions of schema and allows for schema evolution
• Schema are enforced as data contracts at runtime

Guiding Principles

Guiding Principles

Keep calm and... build an enterprise-ready data streaming use case in minutes

Joris Meijer and Bogdan Sucaciu

Thursday 10:30 – 11:30 Techdemo Hall

Guiding Principles

API On-premise DC1 API On-premise DC2 API Azure Amsterdam Cluster API Azure Dublin API AWS API Google Cloud API Cluster Cluster Cluster Cluster Cluster High Availability 2.0

• Clusters can be (de)activated independently
• The Discovery API routes clients dynamically to the nearest available

cluster

• Network affinity and geolocation configurable

Multi-tenancy 2.0

• A Tenant has one or more Instances, like LST, DTA or PROD
• Any Instance can use one or more Clusters
• Clusters are independent, provide local stream access to applications
• Streams are synchronized between Clusters of an Instance
• The entire setup is software-defined

Content Delivery Network for Streams

• BEB allows transparent communication On-premise ßà Cloud
• 1H 2019: Microsoft Azure, 1H 2020: AWS and others

Rabobank on-premise

Business Event Bus hybrid cloud architecture

DMZ DC1 DC2 App 1 App 2 App 4 App 5 Cluster 1 (Single tenant)

Message/offset distribution Message/offset distribution

Cluster 2 (Single tenant)

Rabobank Azure

App 6 App 7

Axual Cloud

Amsterdam Dublin Public endpoints Cluster 3 (Multi tenant) Cluster 4 (Multi tenant)

Message/offset distribution Message/offset distribution

Trust channel Trust channel App 3

Rabobank on-premise

Business Event Bus hybrid cloud architecture

DMZ DC1 DC2 App 1 App 2 App 4 App 5 Cluster 1 (Single tenant)

Message/offset distribution Message/offset distribution

Cluster 2 (Single tenant)

Rabobank Azure

App 6 App 7

Axual Cloud

Amsterdam Dublin Public endpoints Cluster 3 (Multi tenant) Cluster 4 (Multi tenant)

Message/offset distribution Message/offset distribution

Trust channel Trust channel App 3

Deployment Architecture

DC / Cloud

Hardware, VM, OS producers/consumers

Enterprise Streaming Platform

Kafka clusters, message/offset synchronization, performance, availability

Topics + Data

Infrastructure maintenance Technical maintenance Functional maintenance

Team Team Team Team

Growth in streaming use cases

Registered Applications Topics in Production Topics in Dev / Test / Acceptance

x 2 x 3 x 4

70+

Teams supported

≈ 650M

Messages/day

Some use cases

Alerting Performance measurements Customer analytics Transactions Card activations PSD2 account management Logons

Today Tomorrow

Microservices Content publication Fraud detection Privacy dashboard Payment requests Device registrations

Contact us for more information

Jeroen van Disseldorp Managing Director Axual jeroen@axual.com Vincent Oostindië Business Architect Rabobank vincent.oostindie@rabobank.nl

www.axual.com start@axual.com