in collaboration with The Event-Driven Enterprise: Rabobank’s Data Streaming Journey Vincent Oostindië Jeroen van Disseldorp
Who are we? Vincent Oostindië Jeroen van Disseldorp Business Architect Managing Director
Rabobank’s mission, vision and strategy Growing a better world together Banking for the Netherlands Banking for Food Excellent customer Meaningful Rock-solid Empowered focus cooperative bank employees 10 Strategic Top Priorities • • • • 100% digital Concrete socially Top performance Inspired employees • • convenience in everything responsible contribution Optimal balance sheet One-Rabobank culture • • • Top customer advice Involved members and Exceptionally good nearby communities execution • Growth with innovation
Streaming helps Rabobank fulfill its mission Goals What we needed A solution where “things that happen” can Increase relevance Be 100% digital be captured from anywhere and delivered to anywhere for processing, as they occur, without consumers and producers knowing Support financial Become real-time health and growth about each others existence. The One Big Constraint Requirements DevOps Security Being a bank means a lot… 24/7 Control
What is a Streaming Platform? • A marketplace where applications exchange Business Events at the moment they occur. • A Business Event is something that happens or takes place to which organizational entities might want to react. • Marketing Sales Purchasing Accounting There are three types of interactions § Producers detect certain business events and post them in “business meaningful” HR R&D terms. Business Event Bus § Consumers subscribe to streams (or types of events) Asset and handle them as they IT Mgmt come in. § Streaming apps consume from one or more streams Customer Production Inventory Logistics and produce to others. Service
Examples of event-driven banking Customer Transform a youth account to a birthday student account when turning 18 Push Business Event Bus Personal alert generation SMS Booking on a payment account Email Fraud detection Customer Relevance engine Action logging in
Choosing a streaming engine Rabobank chose Apache Kafka Pulsar Pros Kafka • Rich functionality • Very good quality/cost ratio • Popular amongst developers • Supports on-premise & all public/private clouds • Available as DIY, Managed Service and SaaS Solace Tibco EventHub Cons • Oracle Pub/Sub No reference architectures Kinesis • Steep learning curve
Shortcomings of Kafka for Enterprises Multitenancy Name spacing Standardized architecture Vendor credibility Data governance Monitoring Self-service for DevOps teams Security architecture Regulatory compliance Schema management HA applications Alerting End-to-end encryption Auditability Application governance Pre-defined alert rules Development accelerators Flexible use of stream engines Proven solution Data lineage Fool-proof operations Test frameworks Data replication Flexible contracts GDPR compliance CI/CD support Hybrid cloud Data Owner IT Architect Developer Operator Procurement
Our vision “ Streaming Made Simple ” We enable DevOps teams to create business value using streaming data in a matter of hours, while guarding enterprise governance and security. Platform Tools Support General purpose Development accelerators DevOps team Ready-to-go Example code Use-case intakes Secure and governed Documentation Design reviews Hybrid cloud / on-premise Self-service Expert consultancy
Guiding Principles High Availability Security Data Governance Self-Service
Guiding Principles Replication within datacenters Client Application • Each DC runs its own Kafka cluster Axual Client Library • Messages are replicated within a cluster • Clusters withstand node failures One logical platform for applications • Axual extends Kafka across DCs Business Event Bus • Messages and offsets are replicated asynchronously between DCs Discovery API • Upon DC failure, apps are dynamically routed to an alternative DC Apps’ perspective API API • Applications do not know about infrastructure absolutes Apache Kafka Apache Kafka eg. Kafka clusters, schema registries, rest proxies, MQ endpoints § • Apps query the Cluster Discovery API to find out where they should Apache Zookeeper Apache Zookeeper Multi-directional produce/consume message/offset replication • They repeat the query every 10 minutes (supports disaster recovery DC or Cloud DC or Cloud and planned maintenance windows)
Guiding Principles Security Mechanisms Producer 1 Producer 2 Producer 3 • All connections are secured by two-way SSL • TLS v1.1 – v1.3 are supported, TLS v1.0 is deprecated and turned off by default • Certificates are used to authenticate and perform stream authorization • Application only gain access to streams that it has rights for Business Event Bus Application Catalog • Central repository registering all known apps Data Stream Data Stream Data Stream Data Stream Data Stream Data Stream • Every app must declare the SSL certificates it uses Data Stream Data Stream Data Stream • Certificates must be signed by a trusted Certificate Authority Stream Access Rights Management • Streams are secured through Access Control Lists • Applications are assigned rights to either produce, consume or both • Streams are physically separated from each other, stored in separate Consumer 1 Consumer 2 Consumer 3 Consumer n files
Guiding Principles Maintain control over your streaming landscape • Assign ownership of resources like streams and schema • Promote reuse of existing data streams • Allow stream and schema versions to co-exist Stream Governance • Central repository with metadata definitions • Administers stream properties like partitioning, retention time and message formats • Treat as promotable artifact, deployable to different environments Schema Governance • Central repository where schemas are registered and maintained • Contains all versions of schema and allows for schema evolution • Schema are enforced as data contracts at runtime
Guiding Principles
Guiding Principles Keep calm and... build an enterprise-ready data streaming use case in minutes Joris Meijer and Bogdan Sucaciu Thursday 10:30 – 11:30 Techdemo Hall
Guiding Principles API High Availability 2.0 API API • Clusters can be (de)activated independently • The Discovery API routes clients dynamically to the nearest available Cluster Cluster cluster • Network affinity and geolocation configurable On-premise DC1 On-premise DC2 Multi-tenancy 2.0 API API • A Tenant has one or more Instances, like LST, DTA or PROD • Any Instance can use one or more Clusters Cluster Cluster • Clusters are independent, provide local stream access to applications Azure Amsterdam Google Cloud • Streams are synchronized between Clusters of an Instance • The entire setup is software-defined API API Content Delivery Network for Streams • BEB allows transparent communication On-premise ßà Cloud Cluster Cluster • 1H 2019: Microsoft Azure, 1H 2020: AWS and others Azure Dublin AWS
Business Event Bus hybrid cloud architecture Rabobank Azure DC1 DMZ App 6 App 7 App 1 Cluster 1 (Single tenant) Message/offset Public endpoints App 2 distribution Trust Trust channel channel Amsterdam App 3 DC2 Cluster 3 Message/offset (Multi tenant) distribution Message/offset App 4 distribution Dublin Cluster 2 Cluster 4 App 5 (Single tenant) Message/offset (Multi tenant) distribution Rabobank on-premise Axual Cloud
Business Event Bus hybrid cloud architecture Rabobank Azure DC1 DMZ App 3 App 6 App 7 App 1 Cluster 1 (Single tenant) Message/offset Public endpoints App 2 distribution Trust Trust channel channel Amsterdam DC2 Cluster 3 Message/offset (Multi tenant) distribution Message/offset App 4 distribution Dublin Cluster 2 Cluster 4 App 5 (Single tenant) Message/offset (Multi tenant) distribution Rabobank on-premise Axual Cloud
Deployment Architecture Team Team Team Team producers/consumers Functional maintenance Topics + Data Technical maintenance Enterprise Streaming Platform Kafka clusters, message/offset synchronization, performance, availability Infrastructure maintenance DC / Cloud Hardware, VM, OS
Growth in streaming use cases Topics in Dev / Test / Acceptance Topics in Production x 3 x 2 Registered Applications ≈ 650M 70+ x 4 Messages/day Teams supported
Some use cases Today Tomorrow Alerting PSD2 account Privacy dashboard management Transactions Device registrations Fraud detection Card activations Content publication Payment requests Performance Customer analytics measurements Microservices Logons
Contact us for more information Vincent Oostindië Jeroen van Disseldorp Business Architect Rabobank Managing Director Axual vincent.oostindie@rabobank.nl jeroen@axual.com
www.axual.com start@axual.com
Recommend
More recommend
