Journey to a Real-Time Enterprise Neha Narkhede, Co-founder/CTO at - PowerPoint PPT Presentation

Journey to a Real-Time Enterprise Neha Narkhede, Co-founder/CTO at Confluent, Co-Creator Apache Kafka

Infrastructure Technology ? Relational Data Database Warehousing Management Systems

Adoption in Silicon Valley

Adoption in Silicon Valley 4 4

Adoption in the Enterprise 5 5

Fortune 500’s are using Apache Kafka TM Travel Companies Global Banks Insurance Telecom

Emergence of the Streaming Platform

Pre-Streaming

Request-Response Applications Deterministic Service Service Rigid App Service Tight coupling Service Service Service Service Service App

Event-Driven Applications Responsive Service Flexible App Service Extensible Developer Service APIs Service Streaming Platform Service App

Pre-Streaming -> Event-Driven Request-Response Event-Driven

Why Didn’t It Work Before? Past Solutions Are Insufficient Message-Oriented Middlewhere EAI & ESBs ETL No persistence Not event-oriented Often slow, batch oriented, and Single point of failure Fragile and bespoke non-scalable Not fault tolerant Weak transformation capabilities Point-to-point not publish subscribe Cannot order messages Not a true infrastructure platform Cannot process messaging in flight Order of magnitude lower throughput No “Replay” functionality 12 12

The World has Changed Internet of Microservices Mobile Machine Things Learning

What’s Needed? Event Centric Thinking

Events What is an event?

Events

Events A Sale An Invoice A Trade A Customer Experience

All Your Data is Streams of Events

What is a Company? A business is a series of events and reacting to those events.

Event-Driven Government Norwegian Work and Welfare Administration Life is a Stream of Events 5.2 Million Citizens

The Future of the Automotive Industry is a Real Time Data Cluster Traffic Anomaly Front Camera Front, rear and top Infrared Camera Alerts view cameras Detection MQTT MQTT MQTT MQTT MQTT MQTT Front and Rear Crash Sensors Ultrasonic Sensors Hazard Personalizatio Radar Sensors Alerts n

Royal Bank of Canada Event-Driven Banking Consumer Credit Corporate Real Investor Treasury …. Services Estate Services Services 30+ Use-cases 50+ apps 10+ different lines of businesses Digital Fraud Security Data Marketing Warehouse Microservices SaaS

Internet of Things

Banking

Retail

What is a Streaming Platform?

The Streaming Platform Technical Capabilities Publish & Subscribe Store Process

Three Lenses

Lens 01 Messaging done right.

Lens 01 Way More Than Message Queue Messaging done right. Real-time Scalability True Storage Processing

Lens 02 Hadoop made fast.

Stream Processing Lens 02

Lens 02 Applications are different Hadoop made fast.

Lens 03 ETL and Data Integration as a platform.

Lens 03 Scalable Streaming Data Pipelines

Lens 03 Stream Processing is for more than data pipelines ETL and Data Integration as a platform.

Streaming Platform

Journey to an Event-Driven Enterprise

Streaming Adoption Journey Central Nervous System Global Streaming Mission Critical, Early Production Streaming Integrated Streaming Pre-Streaming Awareness Streaming and Pilot

What does the Event-Driven Architecture look like in its end state?

An Event-Driven Enterprise What are the possibilities? ● Everything is an event ● Available instantly to all applications in a company ● Ability to query data as it arrives vs when it is too late ● Simplifying the data architecture by deploying a single platform

Search Stream Apps Processing Management Connectors DWH RDBMS HADOOP Representing Support Data K/V Real-Time Analytics Monitoring

An open streaming platform around Kafka and it’s ecosystem Search Stream Apps Processing Management Connectors DWH RDBMS HADOOP Representing Support Data K/V Real-Time Analytics Monitoring

Search Stream Apps Processing DWH RDBMS HADOOP K/V Real-Time Analytics Monitoring

Thank You