ETL is dead; long-live streams Neha Narkhede, Co-founder & - PowerPoint PPT Presentation

Oct 28, 2023 •185 likes •1.22k views

ETL is dead; long-live streams Neha Narkhede, Co-founder & CTO, Confluent Data and data systems have really changed in the past decade Old world: Two popular locations for data DB DB DWH DB DB Operational databases Relational

“ #1: Powerful and lightweight Java library; need just Kafka and your app app
“ #2: Convenient DSL with all sorts of operators: join(), map(), filter(), windowed aggregates etc
Word count program using Kafka’s streams API
“ #3: True event-at-a-time stream processing; no microbatching
“ #4: Dataflow-style windowing based on event-time; handles late-arriving data
“ #5: Out-of-the-box support for local state ; supports fast stateful processing
External state
local state
Fault-tolerant local state
“ #6: Kafka’s Streams API allows reprocessing; useful to upgrade apps or do A/B testing
reprocessing
Real-time dashboard for security monitoring
Kafka’s streams api: simple is beautiful Vision 1 Vision 2
Logs unify batch and stream processing
New shiny future of ETL: Kafka Connect API Connect API sink source Extract Load Streams API app Transforms
App App App App cache A giant mess! cache MQ MQ search monitoring security DWH Hadoop

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

THE BROWSER IS DEAD Dan North Dan North & Associates LONG LIVE THE BROWSER! Dan North

THE BROWSER IS DEAD Dan North Dan North & Associates LONG LIVE THE BROWSER! Dan North Dan North & Associates The gangs 1990: Tim Berners-Lee - (WorldWideWeb) 1993: NCSA Mosaic 1994: Netscape - Navigator 1995: Microsoft -

971 views • 21 slides

Openmoko is dead. Long live OpenPhoenux! Nikolaus Schaller, Lukas Mrdian LinuxTag, Berlin, May

Openmoko is dead. Long live OpenPhoenux! Nikolaus Schaller, Lukas Mrdian LinuxTag, Berlin, May 26 th , 2012 Agenda Part one: some history Part two: a long way home Part three: rising from the ashes Part four: flying higher Part five: use

563 views • 42 slides

OOP is dead, long live Data-oriented design Stoyan Nikolov @stoyannk CppCon 2018 | @stoyannk

OOP is dead, long live Data-oriented design Stoyan Nikolov @stoyannk CppCon 2018 | @stoyannk Who am I? In the video games industry for 10+ years Software Architect at Coherent Labs Working on game development technology

783 views • 46 slides

HMM: GUP NO MORE ! XDC 2018 Jrme Glisse HETEROGENEOUS COMPUTING CPU is dead, long live the

HMM: GUP NO MORE ! XDC 2018 Jrme Glisse HETEROGENEOUS COMPUTING CPU is dead, long live the CPU Heterogeneous computing is back, one device for each workload: GPUs for massively parallel workload Accelerators for specifjc workload

272 views • 26 slides

Functional TDD with F# Mark Seemann http://blog.ploeh.dk @ploeh TDD is dead!

Look, no Mocks! Functional TDD with F# Mark Seemann http://blog.ploeh.dk @ploeh TDD is dead! http://david.heinemeierhansson.com/2014/tdd-is-dead-long-live-testing.html Is TDD dead? A system you cant test Detestable

656 views • 63 slides

Dead Code Elimination (DCE) Dead code elimination is an optimization that removes DEAD

Dead Code Elimination (DCE) Dead code elimination is an optimization that removes DEAD variables A variable that is defined and not LIVE OUT is DEAD do { computeLiveness() foreach instruction I { if (defs.contains(I) &&

599 views • 12 slides

Proposed Mayor and Council Response to County Grand Jury Report: Redevelopment Is Dead! Long Live

Office of the Independent Budget Analyst Proposed Mayor and Council Response to County Grand Jury Report: Redevelopment Is Dead! Long Live Redevelopment! Presentation for City Council September 9, 2013 Item - S401 Overview The Grand Jury

215 views • 20 slides

CSP Is Dead, Long Live Strict CSP! Lukas Weichselbaum About Us Lukas Weichselbaum Michele

CSP Is Dead, Long Live Strict CSP! Lukas Weichselbaum About Us Lukas Weichselbaum Michele Spagnuolo Senior Information Security Senior Information Security Engineer Engineer We work in a special focus area of the Google security team aimed

553 views • 37 slides

Approximate Computing Is Dead; Long Live Approximate Computing Adrian Sampson Cornell Hardware

Approximate Computing Is Dead; Long Live Approximate Computing Adrian Sampson Cornell Hardware Programming Quality Domains Hardware Programming No more approximate functional units. Quality Domains Narrower bit widths are just as

783 views • 33 slides

400 . 20 . , - , , - , . , , .

400 . 20 . , - , , - , . , , . The Royal Hotel Spa, Resort&Convention Venter 5*Dlx Isrotel Dead Sea 5* The Daniel Dead Sea 5* Le Meridien Dead Sea 5* Crowne Plaza Dead

188 views • 17 slides

Long live robustness! Michael L. Seltzer Microsoft Research REVERB 2014 | May 10, 2014

Robustness is dead! Long live robustness! Michael L. Seltzer Microsoft Research REVERB 2014 | May 10, 2014 Collaborators: Dong Yu, Yan Huang, Frank Seide, Jinyu Li, Jui-Ting Huang Golden age of speech recognition More investment, more

748 views • 39 slides

My microbes and me: A live-long relationship 1. The first come to us latest at birth (Caesarean

My gut microbes and me : Exploring this live-long and fragile relationship Peer Bork Structural and Computational Biology EMBL, Heidelberg Aiming at a functional understanding of biological systems My microbes and me: A live-long relationship

685 views • 51 slides

USB dead drop July 2015 What is it USB device installed in a public space name

USB dead drop July 2015 What is it USB device installed in a public space name (dead drop) comes from method of espionage communication not requiring to meet directly (contrast with live drop) anonymous, offline,

330 views • 12 slides

Simple for Dead load and Continuous for Live loads (SDCL)- Steel Bridges ABC Application Summary

Simple for Dead load and Continuous for Live loads (SDCL)- Steel Bridges ABC Application Summary of more than ten years of research, field application and monitoring Aaron Yakel- University of Nebraska-Lincoln Reza Farimani, Thornton Tomasetti

1.11k views • 61 slides

CSE 143 Streams as C++ Classes Streams are C++ classes Streams have lots of built-in

CSE 143 Streams as C++ Classes Streams are C++ classes Streams have lots of built-in methods We use the . syntax to access member More Stream I/O functions, as usual. inFile.get(ch); // get a character outFile.put(ch); //

209 views • 3 slides

Vantage Optimizing video upload for time-shifted viewing of social live streams (SIGCOMM 2019)

Vantage Optimizing video upload for time-shifted viewing of social live streams (SIGCOMM 2019) Devdeep Ray , Jack Kosaian, Rashmi Vinayak, Srinivasan Seshan 1 Social live video streaming (SLVS) e m i t - l y a r e e R v i l e

882 views • 50 slides

10 Years of waveform analysis from 40 MSa/s to 40 GSa/s Aug. 2018 PICOSEC Test Beam ~2010 ATLAS

W orking W ith W aveforms Sebastian White, CERN/U.Virginia Sept. 12, 2018 ULTIMA 2018 Argonne National Lab HL-LHC upgrade program has renewed interest in Charged Particle timing* at << 100

748 views • 32 slides

43 rd Global Monitoring Annual Conference Boulder, Colorado May 19-20, 2015 ESRL GMAC 2015

43 rd Global Monitoring Annual Conference Boulder, Colorado May 19-20, 2015 ESRL GMAC 2015 Welcome May 19 20, 2015 JH Butler Housekeeping Poster session, 5:00-8:00 pm today! Lunches Delivered here. Posters are being

390 views • 9 slides

Spring 2021 Planning Discussion Suzy Nelson October 20, 2020 AILG Spring 2021 questions being

Spring 2021 Planning Discussion Suzy Nelson October 20, 2020 AILG Spring 2021 questions being considered Can MIT safely bring 1st, 2nd and 3rd year students (and SHARP and graduate students ) to MIT in the spring? What are the major

145 views • 13 slides

It is pitch black. You are likely to be eaten by a Grue. CS 152: Programming Language Paradigms

It is pitch black. You are likely to be eaten by a Grue. CS 152: Programming Language Paradigms Inform 7 Prof. Tom Austin San Jos State University The Hall is a room. Inform 7 Logic based programming Similar to Prolog Inform 6

540 views • 25 slides

Thank you 2019 CRA Sponsors! AGENDA What Really Works? Education, Enforcement & the ROI

Thank you 2019 CRA Sponsors! AGENDA What Really Works? Education, Enforcement & the ROI Mary McClellan Special Thanks to The Recycling Partnership for generating this information and allowing us to share their data! Communicating to

394 views • 22 slides

Josh Bloch Charlie Garrod 17-214 1 Administrivia HW 5b due today, 11:59EDT (framework &

Principles of Software Construction: Objects, Design, and Concurrency Part 3: Concurrency Introduction to concurrency, part 3 Concurrency primitives, libraries, and design patterns Josh Bloch Charlie Garrod 17-214 1 Administrivia HW 5b

806 views • 51 slides

Principles of Software Construction: Concurrency, Pt. 3 java.util.concurrent Josh Bloch

Principles of Software Construction: Concurrency, Pt. 3 java.util.concurrent Josh Bloch Charlie Garrod School of Computer Science 15-214 1 Administrivia Homework 5b due Tuesday 11:59 p.m. Turn in your work by Wednesday 9 a.m. to

991 views • 33 slides

Read-Copy Update (RCU) Don Porter CSE 506 RCU in a nutshell Think about data structures

Read-Copy Update (RCU) Don Porter CSE 506 RCU in a nutshell Think about data structures that are mostly read, occasionally written Like the Linux dcache RW locks allow concurrent reads Still require an atomic decrement

366 views • 18 slides