i logs
play

I Logs Apache Kafka, Stream Processing, and Real-time Data Jay - PowerPoint PPT Presentation

Nov 05, 2023 •231 likes •661 views

I Logs Apache Kafka, Stream Processing, and Real-time Data Jay Kreps The Plan 1. What is Data Integration? 2. What is Apache Kafka? 3. Logs and Distributed Systems 4. Logs and Data Integration 5. Logs and Stream Processing Data Integration

  1. I ♥ Logs Apache Kafka, Stream Processing, and Real-time Data Jay Kreps

  2. The Plan 1. What is Data Integration? 2. What is Apache Kafka? 3. Logs and Distributed Systems 4. Logs and Data Integration 5. Logs and Stream Processing

  3. Data Integration

  4. Maslow’s Hierarchy Self-Actualization Esteem Love & Belonging Safety Physiological

  5. For Data Automation Understanding Semantics Acquisition/Collection

  6. New Types of Data • Database data – Users, products, orders, etc • Events – Clicks, Impressions, Pageviews, etc • Application metrics – CPU usage, requests/sec • Application logs – Service calls, errors

  7. New Types of Systems • Live Stores – Voldemort – Espresso – Graph – OLAP – Search – InGraphs • Offline – Hadoop – Teradata

  8. Bad Espresso Operational Operational Voldemort Oracle User Tracking Espresso Voldemort Oracle Logs Metrics Espresso Voldemort Oracle Log Data Social Rec. ... Hadoop Monitoring Search Security Email Search Warehouse Graph Engine Production Services

  9. Good Espresso Operational Operational Voldemort Oracle User Tracking Espresso Voldemort Oracle Logs Metrics Espresso Voldemort Oracle Log Log Data Social Rec ... Hadoop Monitoring Search Security Email Search Warehouse Graph Engine Production Services

  10. The Plan 1. What is Data Integration? 2. What is Apache Kafka? 3. Logs and Distributed Systems 4. Logs and Data Integration 5. Logs and Stream Processing

  11. Apache Kafka producer producer producer producer producer producer producer producer producer kafka cluster consumer consumer consumer consumer consumer consumer consumer consumer consumer

  13. A � brief � history � of � Kafka

  14. Three design principles 1. One pipeline to rule them all 2. Stream processing >> messaging 3. Clusters not servers

  15. Characteristics • Scalability of a filesystem – Hundreds of MB/sec/server throughput – Many TB per server • Guarantees of a database – Messages strictly ordered – All data persistent • Distributed by default – Replication – Partitioning model

  16. Kafka At LinkedIn • 175 TB of in-flight log data per colo • Low-latency: ~1.5 ms • Replicated to each datacenter • Tens of thousands of data producers • Thousands of consumers • 7 million messages written/sec • 35 million messages read/sec • Hadoop integration

  17. The Plan 1. What is Data Integration? 2. What is Apache Kafka? 3. Logs and Distributed Systems 4. Logs and Data Integration 5. Logs and Stream Processing

  18. Kafka is about logs

  19. What is a log?

  21. Next Record 1st Record Written 0 1 2 3 4 5 6 7 8 9 10 11 12

  22. Partitioning 1 1 1 Partition 0 0 1 2 3 4 5 6 7 8 9 0 1 2 Partition 1 0 1 2 3 4 5 6 7 8 9 1 1 Partition 2 0 1 2 3 4 5 6 7 8 9 0 1

  23. Logs: pub/sub done right Data Source writes 1 1 1 0 1 2 3 4 5 6 7 8 9 Log 0 1 2 reads reads Destination Destination System A System B (time = 7) (time = 11)

  24. Logs And Distributed Systems

  25. Example: � A Fault-tolerant CEO Hash Table

  26. Operations Final State PUT('microsoft', 'bill gates') { PUT('apple', 'steve jobs') PUT('microsoft', 'steve ballmer') 'microsoft': 'satya nadella', PUT('google', 'larry page') PUT('yahoo', 'terry semel') 'apple': 'tim cook', PUT('google', 'eric schmidt') PUT('yahoo', 'jerry yang') 'google': 'larry page', PUT('yahoo', 'carol bartz') PUT('apple', 'tim cook') 'yahoo': 'marissa mayer' PUT('google', 'larry page') PUT('yahoo', 'scott thompson') } PUT('yahoo', 'marissa mayer') PUT('microsoft', 'satya nadella') Replica 1 Replica 2

  27. 0 PUT(microsoft, bill gates) PUT(apple, steve jobs) 1 PUT(microsoft, steve ballmer) 2 PUT(google, larry page) 3 PUT(yahoo, terry semel) 4 PUT(google, eric schmidt) 5 (offset=10) Replica 1 PUT(yahoo, jerry yang) 6 PUT(yahoo, carol bartz) 7 PUT(apple, tim cook) 8 PUT(google, larry page) 9 (offset=12) Replica 2 10 PUT(yahoo, scott thompson) 11 PUT(yahoo, marissa mayer) 12 PUT(microsoft, satya nadella)

  28. Two System Design Styles State-machine Primary-backup Replication Reads Writes Requests Master Slave Slave The Log state changes Peer Peer Peer The Log

  29. The Plan 1. What is Data Integration? 2. What is Apache Kafka? 3. Logs and Distributed Systems 4. Logs and Data Integration 5. Logs and Stream Processing

  30. Example: User views job Jobs Frontend Job Views Kafka Job Views Job Poster Rec. Hadoop Security Monitoring Analytics Engine

  31. It’s all one big distributed system Espresso Operational Operational Voldemort Oracle User Tracking Espresso Voldemort Oracle Logs Metrics Espresso Voldemort Oracle Log Log Data Social Rec ... Hadoop Monitoring Search Security Email Search Warehouse Graph Engine Production Services

  32. Comparing Data Transfer Mechanisms

  33. The Plan 1. What is Data Integration? 2. What is Apache Kafka? 3. Logs and Distributed Systems 4. Logs and Data Integration 5. Logs and Stream Processing

  34. Stream Processing

  36. Stream Processing = Logs + Jobs Log A Log B Log C Job 1 Job 2 Log D Log E Job 3 Log F

  37. Stream processing is a � generalization � of batch processing

  38. Examples • Monitoring • Security • Content processing • Recommendations • Newsfeed • ETL

  39. Systems Can Help

  40. Samza Architecture Job Job Job Job Samza Kafka YARN

  41. Log-centric Architecture Graph DB, Key-Value Search Query OLAP Store, Query Layer Layer Etc Monitoring Stream & Proces Log Graphs sing Hadoop

  42. � � � � Kafka � http://kafka.apache.org � Samza � http://samza.incubator.apache.org � Log Blog � http://linkd.in/199iMwY � Me � http://www.linkedin.com/in/jaykreps � @jaykreps �

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Logs on Logs on Logs No More Append Atomic & Remap Eric Mackay Venkatesh Srinivas Basics

Logs on Logs on Logs No More Append Atomic & Remap Eric Mackay Venkatesh Srinivas Basics

Logs on Logs on Logs No More Append Atomic & Remap Eric Mackay Venkatesh Srinivas Basics of Block Device Interfaces I/O is done in granularity of blocks 512 bytes is pretty standard Writing is slooooooow Data is

898 views • 15 slides
Why are UI Logs Important? UI logs will help you identify Trends and Patterns that need to be

Why are UI Logs Important? UI logs will help you identify Trends and Patterns that need to be

2/17/2015 Unusual Incident Log Reviews January 23, 2015 Why are UI Logs Important? UI logs will help you identify Trends and Patterns that need to be addressed to ensure the Health and Welfare of those you serve. To ensure that sound

589 views • 24 slides
Harvesting Logs and Events Using MetaCentrum Virtualization Services Radoslav Bod, Daniel

Harvesting Logs and Events Using MetaCentrum Virtualization Services Radoslav Bod, Daniel

Harvesting Logs and Events Using MetaCentrum Virtualization Services Radoslav Bod, Daniel Kouil CESNET EGI Community Forum, April 2013 Agenda Introduction Collecting logs Log Processing Advanced analysis Resume

601 views • 44 slides
It Can Understand the Logs, Literally Aidi Pi , Wei Chen, Will Zeller and Xiaobo Zhou IPDPSW19

It Can Understand the Logs, Literally Aidi Pi , Wei Chen, Will Zeller and Xiaobo Zhou IPDPSW19

It Can Understand the Logs, Literally Aidi Pi , Wei Chen, Will Zeller and Xiaobo Zhou IPDPSW19 @ Rio de Janeiro Outline Introduction to distributed system logs Challenges NLog: A NLP based log analysis approach Evaluation

197 views • 18 slides
Mining Invariants from Logs for System Problem Detection Jian-Guang LOU, Qiang FU Software

Mining Invariants from Logs for System Problem Detection Jian-Guang LOU, Qiang FU Software

Mining Invariants from Logs for System Problem Detection Jian-Guang LOU, Qiang FU Software Analytics Microsoft Research Asia Background Many systems produce log messages for trouble-shooting. Logs usually record important system actions

706 views • 31 slides
Exponential and Logarithm Natural Logs and e Exponential Growth and Decay Functions Slide 3 /

Exponential and Logarithm Natural Logs and e Exponential Growth and Decay Functions Slide 3 /

Slide 1 / 99 Slide 2 / 99 Table of Contents Exponential Functions Logarithmic Functions Properties of Logs Common Logs Exponential and Logarithm Natural Logs and e Exponential Growth and Decay Functions Slide 3 / 99 Slide 4 / 99 We have

291 views • 17 slides
Log all the things! Honza Krl @honzakral Logs? Events! Log lines Twitter feed Invoices

Log all the things! Honza Krl @honzakral Logs? Events! Log lines Twitter feed Invoices

Log all the things! Honza Krl @honzakral Logs? Events! Log lines Twitter feed Invoices Metrics Why? What happened last Tuesday? Grep? Multiple machines Multiple logs Analysis/Discovery Time period Time? Time?! Time! apache

592 views • 33 slides
DTTF/NB479: Dszquphsbqiz Day 24 Announcements: Term project groups and topics due midnight 1.

DTTF/NB479: Dszquphsbqiz Day 24 Announcements: Term project groups and topics due midnight 1.

DTTF/NB479: Dszquphsbqiz Day 24 Announcements: Term project groups and topics due midnight 1. HW6 due next Tuesday. 2. Questions? This week: Primality testing, factoring Discrete Logs, Computing Discrete Logs Discrete logs

571 views • 10 slides
Factorials: Turn product into a sum taking logs: ln(n!) = ln( 123(n 1)n ) =

Factorials: Turn product into a sum taking logs: ln(n!) = ln( 123(n 1)n ) =

Closed form for n! Mathematics for Computer Science MIT 6.042J/18.062J n n! ::= 1 2 3 (n-1) n = i i=1 Factorials: Turn product into a sum taking logs: ln(n!) = ln( 123(n 1)n ) = Stirlings

443 views • 3 slides
A Graphical representation for identifier structure in application logs Ari Rabkin, Wei Xu, Avani

A Graphical representation for identifier structure in application logs Ari Rabkin, Wei Xu, Avani

UC Berkeley A Graphical representation for identifier structure in application logs Ari Rabkin, Wei Xu, Avani Wildani, Armando Fox, David Patterson and Randy Katz SLAML October 3, 2010 Motivation & Summary Log analysis is

656 views • 27 slides
Oil Shale Formation Evaluation by Well Logs and Core Measurements Robert Kleinberg

Oil Shale Formation Evaluation by Well Logs and Core Measurements Robert Kleinberg

Oil Shale Formation Evaluation by Well Logs and Core Measurements Robert Kleinberg Schlumberger-Doll Research Cambridge, Massachuse:s AMSO: Alan Burnham, Roger Day TOTAL: Pierre Allix

291 views • 17 slides
When We Need Audit Logs Computer forensics in courts Recovering from an attack

When We Need Audit Logs Computer forensics in courts Recovering from an attack

Systematic and Practical Methods for Computer Attack Analysis and Forensics Dr. Sean Peisert UC Davis Computer Science Dept. NSF I/UCRC Meeting ~ Davis, CA June 17, 2008 1 When We Need Audit Logs Computer forensics in courts

524 views • 11 slides
Heterogeneous Logs Biplob Debnath and Will Dennis NEC Laboratories America, Inc .

Heterogeneous Logs Biplob Debnath and Will Dennis NEC Laboratories America, Inc .

Fast Log Analysis by Automatically Parsing Heterogeneous Logs Biplob Debnath and Will Dennis NEC Laboratories America, Inc . {biplob,wdennis}@nec-labs.com June 25 29, 2018 | New York City, NY www.qconnewyork.com #QConNYC Log Analysis

607 views • 60 slides
by Mining Console Logs Wei Xu* Ling Huang Armando Fox* David Patterson* Michael Jordan*

by Mining Console Logs Wei Xu* Ling Huang Armando Fox* David Patterson* Michael Jordan*

UC Berkeley Detecting Large-Scale System Problems by Mining Console Logs Wei Xu* Ling Huang Armando Fox* David Patterson* Michael Jordan* Intel Labs Berkeley *UC Berkeley 1 Why console logs? Detecting problems in large scale

323 views • 14 slides
In Log We Trust: Revealing Poor Security Practices with Certificate Transparency Logs and Internet

In Log We Trust: Revealing Poor Security Practices with Certificate Transparency Logs and Internet

Chair of Network Architectures and Services Department of Informatics Technical University of Munich In Log We Trust: Revealing Poor Security Practices with Certificate Transparency Logs and Internet Measurements Oliver Gasser, Benjamin Hof, Max

531 views • 39 slides
LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs

LogAnomaly: Unsupervised Detection of Sequential and Quantitative Anomalies in Unstructured Logs Weibin Meng , Ying Liu, Yichen Zhu, Shenglin Zhang, Dan Pei Yuqing Liu, Yihao Chen, Ruizhi Zhang, Shimin Tao, Pei Sun and Rong Zhou 2019/9/10 1

2.58k views • 24 slides
QE, main strategies of parallelization and levels of parallelisms Fabio AFFINITO SCAI - Cineca

QE, main strategies of parallelization and levels of parallelisms Fabio AFFINITO SCAI - Cineca

QE, main strategies of parallelization and levels of parallelisms Fabio AFFINITO SCAI - Cineca What I cannot compute, I do not understand. (adapted from Richard P. Feynman) Quantum ESPRESSO: introduction Quantum ESPRESSO is an

280 views • 16 slides
G o i n g b e y o n d L o c a l D e n s i t y a n d G r a d i e n

G o i n g b e y o n d L o c a l D e n s i t y a n d G r a d i e n

G o i n g b e y o n d L o c a l D e n s i t y a n d G r a d i e n t C o r r e c t e d X C f u n c t i o n a l s i n Q u a n t u m- E S P R E S S O Jacob's ladder of Density

619 views • 40 slides
New developments in the quantum ESPRESSO software distribution for quantum simulations at the

New developments in the quantum ESPRESSO software distribution for quantum simulations at the

New developments in the quantum ESPRESSO software distribution for quantum simulations at the nanoscale Paolo Giannozzi Universit` a di Udine, Italy Workshop From experiments to theory & models... Roma Tor Vergata, 2017/12/5 Typeset by

805 views • 35 slides
On Brewing Fresh Espresso: LinkedIns Distributed Data Serving Platform Thomas Marshall

On Brewing Fresh Espresso: LinkedIns Distributed Data Serving Platform Thomas Marshall

On Brewing Fresh Espresso: LinkedIns Distributed Data Serving Platform Thomas Marshall Motivation Better performance and horizontal scalability than traditional RDBMS. Better consistency, transactions, and schema support than

246 views • 13 slides
Performance of Density Functional Theory codes on Cray XE6 Zhengji Zhao, and Nicholas Wright

Performance of Density Functional Theory codes on Cray XE6 Zhengji Zhao, and Nicholas Wright

Performance of Density Functional Theory codes on Cray XE6 Zhengji Zhao, and Nicholas Wright National Energy Research Scientific Computing Center Lawrence Berkeley National Laboratory Outline Motivation Introduction to DFT codes

593 views • 25 slides
Using Space Effectively Ma Maneesh Agrawala CS 448B: Visualization Fall 2020 1 2 1 Last

Using Space Effectively Ma Maneesh Agrawala CS 448B: Visualization Fall 2020 1 2 1 Last

Using Space Effectively Ma Maneesh Agrawala CS 448B: Visualization Fall 2020 1 2 1 Last Time: EDA 3 Data Wrangling One often needs to manipulate data prior to analysis. Tasks include reformatting, cleaning, quality assessment, and

489 views • 31 slides
ESPRESSO Ana Catarina Leite In Colaboration with: Carlos Martins IA-Porto Paolo

ESPRESSO Ana Catarina Leite In Colaboration with: Carlos Martins IA-Porto Paolo

Testing the stability of fundamental couplings with ESPRESSO Ana Catarina Leite In Colaboration with: Carlos Martins IA-Porto Paolo Molaro INAF-OATS Testing Fine Structure Constant in the Sky Webb et al. 2011 UVES/VLT and

409 views • 7 slides
Limits to ILP Conflicting studies of amount Benchmarks (vectorized Fortran FP vs. integer C

Limits to ILP Conflicting studies of amount Benchmarks (vectorized Fortran FP vs. integer C

Limits to ILP Conflicting studies of amount Benchmarks (vectorized Fortran FP vs. integer C programs) Lecture 12: Limits of ILP and Hardware sophistication Pentium Processors Compiler sophistication How much ILP is available using

714 views • 6 slides

More recommend