KAFKA STREAMS CLOUD MONITORING AWS CLOUD MONITORING AWS APP - PowerPoint PPT Presentation

Jun 25, 2023 •183 likes •1.43k views

FROM USE CASE TO PRODUCTION KAFKA STREAMS CLOUD MONITORING AWS CLOUD MONITORING AWS APP CLOUD MONITORING AWS HTTP APP CLOUD MONITORING AWS METRICS NRDB HTTP APP CLOUD MONITORING SCHDLR JOBS AWS NRDB METRICS HTTP WORKER CLOUD

ENRICHMENT ▸ same partitioning key ▸ same number of partitions ▸ same hash function of producer
ENRICHMENT METRICS PO JAVA METRICS P1 APP METRICS P2 METADATA PO GO METADATA P1 APP METADATA P2
ENRICHMENT METRICS PO K:1, V:42 JAVA APP GO APP METADATA P2 K:1, V:eu
ENRICHMENT METRICS PO K:1, V:42 JAVA APP METADATA KSTREAMS APP PO P0 GO APP METADATA P2 K:1, V:eu
ENRICHMENT METRICS PO K:1, V:42 JAVA APP METADATA KSTREAMS APP PO P0 GO APP METADATA P2 K:1, V:eu
ENRICHMENT METRICS MURMUR2 PO JAVA APP METADATA KSTREAMS APP PO P0 FNV-1A GO APP METADATA P2
ENRICHMENT METRICS PO K:1, V:42 JAVA METRICS P1 APP METRICS P2 METADATA KSTREAMS APP REPARTITION PO P0’ METADATA’ METADATA PO P0’ K:1, V:eu GO METADATA’ METADATA P1 P1’ APP METADATA’ METADATA P2 P2’ K:1, V:eu
ENRICHMENT METRICS PO K:1, V:42 JAVA METRICS P1 APP METRICS P2 METADATA KSTREAMS APP REPARTITION PO P0’ METADATA’ METADATA PO P0’ K:1, V:eu GO METADATA’ METADATA P1 P1’ APP METADATA’ METADATA P2 P2’ K:1, V:eu
ENRICHMENT ▸ enrichment at scale thanks to co-partitioning ▸ in case of doubt — repartition ▸ a bit more kafka topics
ENRICHMENT 🏞 VACATIONS?
PRODUCTION READINESS 2 CPU 4096 MB APP
PRODUCTION READINESS 2 CPU 3x 4096 MB APP APP APP
PRODUCTION READINESS HEALTH-CHECK HEALTH-CHECK HEALTH-CHECK APP APP APP
PRODUCTION READINESS HEALTH-CHECK HEALTH-CHECK HEALTH-CHECK APP APP APP
PRODUCTION READINESS HEALTH-CHECK HEALTH-CHECK APP APP
PRODUCTION READINESS HEALTH-CHECK HEALTH-CHECK HEALTH-CHECK APP APP APP
PRODUCTION READINESS HEALTH-CHECK APP KSTREAMS + LIBRARY DSL
PRODUCTION READINESS HEALTH-CHECK STREAM APP THREAD
PRODUCTION READINESS HEALTH-CHECK STREAM APP THREAD
PRODUCTION READINESS HEALTH-CHECK STREAM APP THREAD IT’S FINE
PRODUCTION READINESS HEALTH-CHECK STREAM APP THREAD
ENRICHMENT 👃 LISTENERS
PRODUCTION READINESS RUNNING
PRODUCTION READINESS PENDING NOT RUNNING ERROR SHUTDOWN RUNNING
PRODUCTION READINESS RUNNING REBALANCING
PRODUCTION READINESS RUNNING REBALANCING
PRODUCTION READINESS RESTORE BATCH START RESTORED
PRODUCTION READINESS RESTORE BATCH RESTORE START RESTORED END
PRODUCTION READINESS RESTORE BATCH START RESTORED
PRODUCTION READINESS ‣ consuming/producing throughput ▸ missing joins ▸ rebalancing loops ▸ storage-specific metrics ▸ business-specific metrics
PRODUCTION READINESS
PRODUCTION READINESS
PRODUCTION READINESS

Download Presentation

Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Apache Kafka + Apache Mesos Highly Scalable Streaming Microservices with Kafka Streams Kai

Apache Kafka + Apache Mesos Highly Scalable Streaming Microservices with Kafka Streams Kai Waehner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.kai-waehner.de Confidential 1 Abstract Microservices establish many

977 views • 44 slides

Kafka Streams: Hands-on Session A.A. 2017/18 Matteo Nardelli Laurea Magistrale in Ingegneria

Universit degli Studi di Roma Tor Vergata Dipartimento di Ingegneria Civile e Ingegneria Informatica Kafka Streams: Hands-on Session A.A. 2017/18 Matteo Nardelli Laurea Magistrale in Ingegneria Informatica - II anno The reference

299 views • 13 slides

Fast Data apps with Alpakka Kafka connector and Akka Streams Sean Glover, Lightbend @seg1o Who

Fast Data apps with Alpakka Kafka connector and Akka Streams Sean Glover, Lightbend @seg1o Who am I? Im Sean Glover Principal Engineer at Lightbend Member of the Fast Data Platform team Organizer of Scala Toronto (scalator)

925 views • 69 slides

FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES MICROSERVICES Wojciech Rzsa,

7/17/2019 From HTTP to Kafka-based microservices FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES MICROSERVICES Wojciech Rzsa, FLYR Poland @wrzasa localhost:4567/index.html?print-pdf#/ 1/83 From HTTP to Kafka-based

1.15k views • 83 slides

Kafka Needs No Keeper Colin McCabe 2 Introduction Kafka has gotten its mileage out of

1 Kafka Needs No Keeper Colin McCabe 2 Introduction Kafka has gotten its mileage out of Zookeeper But it is still a second system KIP-500 has been adopted by the community This is not a 1-1 replacement Weve been

1.92k views • 124 slides

STREAM PROCESSING AT LINKEDIN: APACHE KAFKA & APACHE SAMZA Processing billions of events

STREAM PROCESSING AT LINKEDIN: APACHE KAFKA & APACHE SAMZA Processing billions of events every day Neha Narkhede Co-founder and Head of Engineering @ Stealth Startup Prior to this Lead, Streams Infrastructure @ LinkedIn

494 views • 45 slides

Introduction to Kafka Instructor: Ekpe Okorafor 1. Big Data Academy - Accenture 2. Computer

Introduction to Kafka Instructor: Ekpe Okorafor 1. Big Data Academy - Accenture 2. Computer Science - African University of Science & Technology Agenda Introduction - Messaging Basics Kafka Architecture Kafka

555 views • 26 slides

Cr Cruise Co Control: Effo l: Effortle less M Manage gement o of K f Kafka fka Clu

Cr Cruise Co Control: Effo l: Effortle less M Manage gement o of K f Kafka fka Clu Clusters Adem Efe Gencer Senior Software Engineer LinkedIn Kafka: A Distributed Stream Processing Platform : High throughput & low latency :

989 views • 50 slides

Kafka in Jail Running Kafka in container orchestrated clusters Sean Glover, Lightbend @seg1o

Kafka in Jail Running Kafka in container orchestrated clusters Sean Glover, Lightbend @seg1o Who am I? Im Sean Glover Senior Software Engineer at Lightbend Member of the Fast Data Platform team Contributor to various projects in

788 views • 60 slides

CSE 143 Streams as C++ Classes Streams are C++ classes Streams have lots of built-in

CSE 143 Streams as C++ Classes Streams are C++ classes Streams have lots of built-in methods We use the . syntax to access member More Stream I/O functions, as usual. inFile.get(ch); // get a character outFile.put(ch); //

209 views • 3 slides

Day 3 Lab1: Spark Streaming with Kafka Example Introductions In this example, we will write a

Day 3 Lab1: Spark Streaming with Kafka Example Introductions In this example, we will write a Spark Streaming program that consumes messages from Kafka. We will reuse the whole setup from the previous lab, so this lab is best done as a

233 views • 4 slides

High throughput High throughput kafka for science kafka for science Testing Kafkas limits

High throughput High throughput kafka for science kafka for science Testing Kafkas limits for science J Wyngaard, PhD wyngaard@jpl.nasa.gov UTLINE O UTLINE O Streaming Science Data Benchmark Context Tests and Results

473 views • 30 slides

Day 4 Lab1: Docker container for Kafka - Spark streaming - Cassandra This Dockerfile sets up

Day 4 Lab1: Docker container for Kafka - Spark streaming - Cassandra This Dockerfile sets up a complete streaming environment for experimenting with Kafka, Spark streaming (PySpark), and Cassandra. It installs Kafka 0.10.2.1 Spark 2.1.1

156 views • 4 slides

Apache Kafka Real-Time Data Pipelines http://kafka.apache.org/ Joe Stein Developer,

Apache Kafka Real-Time Data Pipelines http://kafka.apache.org/ Joe Stein Developer, Architect & Technologist Founder & Principal Consultant => Big Data Open Source Security LLC - http://stealth.ly Big Data Open Source

637 views • 40 slides

READING KAFKA IN QATAR Qatar-TESOL Conference, April 2011 Magdalena Rostron Academic Bridge

READING KAFKA IN QATAR Qatar-TESOL Conference, April 2011 Magdalena Rostron Academic Bridge Program Qatar Foundation, Doha, Qatar mrostron@qf.org.qa Introduction my story Reading Kafka In Qatar Reading Kafka in Qatar

562 views • 10 slides

Blockchain consensus Protocols in the Wild Tao Wang, Lihang Pan ECS 265 Apache Kafka

Blockchain consensus Protocols in the Wild Tao Wang, Lihang Pan ECS 265 Apache Kafka (CFT) PBFT (BFT) Kafka Architecture ISR (in- sync) {b1,b2,b3} b1 b2 b3 Photo credit to Apache

2.57k views • 32 slides

User Model Enrichment for Venue Recommendation AIRS 2016 Mohammad Aliannejadi, Ida Mele, and

Introduction Approach Dataset Results Conclusion User Model Enrichment for Venue Recommendation AIRS 2016 Mohammad Aliannejadi, Ida Mele, and Fabio Crestani Universit` a della Svizzera italiana (USI) Lugano, Switzerland December 2 nd 2016

350 views • 23 slides

Short Text Categorization Exploiting Contextual Enrichment and External Knowledge Stefano Mizzaro

Short Text Categorization Exploiting Contextual Enrichment and External Knowledge Stefano Mizzaro , Marco Pavan, Ivan Scagnetto, Martino Valenti University of Udine, Italy 1 Disclaimer Keep it simple, keep it short, and nobody

736 views • 37 slides

Enriched Regular Theories Giacomo Tendas Joint work with: Stephen Lack 8 July 2019 Outline 1

Enriched Regular Theories Giacomo Tendas Joint work with: Stephen Lack 8 July 2019 Outline 1 Theories 2 Regular Theories 3 Enriched Finite Limit Theories 4 Enriched Regular Theories 2 of 18 Theories Theories in Logic A theory is given by a

416 views • 25 slides

An Enriched Perspective on Differentiable Stacks Benjamin MacAdam Joint work with Jonathan

An Enriched Perspective on Differentiable Stacks Motivation An Enriched Perspective on Differentiable Stacks Benjamin MacAdam Joint work with Jonathan Gallagher July 9, 2019 An Enriched Perspective on Differentiable Stacks Motivation

323 views • 20 slides

Enriched algebraic weak factorisation systems Alexander Campbell Centre of Australian Category

Enriched algebraic weak factorisation systems Alexander Campbell Centre of Australian Category Theory Macquarie University Category Theory 2017 University of British Columbia Alexander Campbell (CoACT) Enriched awfs CT2017 UBC 1 / 30

1.28k views • 96 slides

Enriched Morita equivalence for S -sorted theories Mat ej Dost al joint work with Ji

Preliminaries General Morita theorem Examples and summary Enriched Morita equivalence for S -sorted theories Mat ej Dost al joint work with Ji r Velebil Czech Technical University in Prague WCMAT 2014 Mat ej Dost al,

167 views • 15 slides

Bruno Henriques (Zwicky Fellow, ETH Zurich) Fu et al. (2010, 2011, 2013), Yates et al (2013),

A self-consistent model of galaxy formation with resolved discs (Henriques, Yates, Fu, Kauffmann, Thomas, White, 2017, in prep.) Bruno Henriques (Zwicky Fellow, ETH Zurich) Fu et al. (2010, 2011, 2013), Yates et al (2013), Henriques et al. (2013,

804 views • 22 slides

Integrated System for Improved Student Achievement Eight-Step Process September 17-24, 2012 Dr.

WELCOME TO: Integrated System for Improved Student Achievement Eight-Step Process September 17-24, 2012 Dr. Peggy Hinckley Warrens Story 3 years of declining 70 test scores 60 Fall, 2001 Lowest 50 scores among 40 1999

1.58k views • 111 slides