from http to kafka based from http to kafka based
play

FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES - PowerPoint PPT Presentation

Sep 18, 2022 •304 likes •1.15k views

7/17/2019 From HTTP to Kafka-based microservices FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES MICROSERVICES Wojciech Rzsa, FLYR Poland @wrzasa localhost:4567/index.html?print-pdf#/ 1/83 From HTTP to Kafka-based

  1. 7/17/2019 From HTTP to Kafka-based microservices FROM HTTP TO KAFKA-BASED FROM HTTP TO KAFKA-BASED MICROSERVICES MICROSERVICES Wojciech Rząsa, FLYR Poland @wrzasa localhost:4567/index.html?print-pdf#/ 1/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  2. 7/17/2019 From HTTP to Kafka-based microservices ABOUT ME ABOUT ME Informatics specialist by passion and by profession 15 years of academic work PhD but primarily an engineer FLYR Inc. http://flyrlabs.com Distributed systems Rzeszow Ruby User Group http://rrug.pl localhost:4567/index.html?print-pdf#/ 2/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  3. 7/17/2019 From HTTP to Kafka-based microservices FLYR FLYR Revenue management system for airlines Offices in San Francisco, USA (PST) Kraków, Poland (CEST) Machine Learning Microservices Python GCloud Kubernetes localhost:4567/index.html?print-pdf#/ 3/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  4. 7/17/2019 From HTTP to Kafka-based microservices FLYR DEVS ON PYCON CZ FLYR DEVS ON PYCON CZ localhost:4567/index.html?print-pdf#/ 4/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  5. 7/17/2019 From HTTP to Kafka-based microservices IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP localhost:4567/index.html?print-pdf#/ 5/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  6. 7/17/2019 From HTTP to Kafka-based microservices IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP New requirements for eCommerce use case localhost:4567/index.html?print-pdf#/ 6/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  7. 7/17/2019 From HTTP to Kafka-based microservices FAN-OUT REQUESTS FAN-OUT REQUESTS localhost:4567/index.html?print-pdf#/ 7/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  8. 7/17/2019 From HTTP to Kafka-based microservices PARTIAL RESPONSES PARTIAL RESPONSES localhost:4567/index.html?print-pdf#/ 8/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  9. 7/17/2019 From HTTP to Kafka-based microservices PARTIAL RESPONSES PARTIAL RESPONSES localhost:4567/index.html?print-pdf#/ 9/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  10. 7/17/2019 From HTTP to Kafka-based microservices IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP New requirements for eCommerce use case partial responses localhost:4567/index.html?print-pdf#/ 10/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  11. 7/17/2019 From HTTP to Kafka-based microservices IN FLYR MICROSERVICES IN FLYR MICROSERVICES IPC based on HTTP New requirements for eCommerce use case partial responses performance localhost:4567/index.html?print-pdf#/ 11/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  12. 7/17/2019 From HTTP to Kafka-based microservices OK, LET'S SWITCH FROM HTTP OK, LET'S SWITCH FROM HTTP TO... A... MQ? TO... A... MQ? localhost:4567/index.html?print-pdf#/ 12/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  13. 7/17/2019 From HTTP to Kafka-based microservices BUT... BUT... localhost:4567/index.html?print-pdf#/ 13/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  14. 7/17/2019 From HTTP to Kafka-based microservices BUT... BUT... We have HTTP-based infrastructure localhost:4567/index.html?print-pdf#/ 14/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  15. 7/17/2019 From HTTP to Kafka-based microservices BUT... BUT... We have HTTP-based infrastructure We have HTTP developers experience and habits localhost:4567/index.html?print-pdf#/ 15/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  16. 7/17/2019 From HTTP to Kafka-based microservices BUT... BUT... We have HTTP-based infrastructure We have HTTP developers experience and habits We lack experience with MQ-based IPC localhost:4567/index.html?print-pdf#/ 16/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  17. 7/17/2019 From HTTP to Kafka-based microservices BUT... BUT... We have HTTP-based infrastructure We have HTTP developers experience and habits We lack experience with MQ-based IPC We need to do it well ;-) localhost:4567/index.html?print-pdf#/ 17/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  18. 7/17/2019 From HTTP to Kafka-based microservices WITH MQ WE GET WITH MQ WE GET Flexibility Reliability Scalability Robustness localhost:4567/index.html?print-pdf#/ 18/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  19. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... localhost:4567/index.html?print-pdf#/ 19/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  20. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES localhost:4567/index.html?print-pdf#/ 20/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  21. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues localhost:4567/index.html?print-pdf#/ 21/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  22. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions localhost:4567/index.html?print-pdf#/ 22/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  23. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice localhost:4567/index.html?print-pdf#/ 23/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  24. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker localhost:4567/index.html?print-pdf#/ 24/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  25. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver localhost:4567/index.html?print-pdf#/ 25/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  26. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker localhost:4567/index.html?print-pdf#/ 26/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  27. 7/17/2019 From HTTP to Kafka-based microservices NEW OPPORTUNITIES... NEW OPPORTUNITIES... TO MAKE TO MAKE NEW MISTAKES NEW MISTAKES Concurrency issues Race conditions Incorrect broker choice Incorrect driver for the correct broker Incorrect usage patterns for the correct driver Incorrect usage patterns for the correct broker . . . localhost:4567/index.html?print-pdf#/ 27/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  28. 7/17/2019 From HTTP to Kafka-based microservices LET'S CONTAIN THE RISKS IN LET'S CONTAIN THE RISKS IN ONE PLACE ONE PLACE localhost:4567/index.html?print-pdf#/ 28/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  29. 7/17/2019 From HTTP to Kafka-based microservices LET'S CONTAIN THE RISKS IN LET'S CONTAIN THE RISKS IN ONE PLACE ONE PLACE (A LIBRARY) (A LIBRARY) localhost:4567/index.html?print-pdf#/ 29/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  30. 7/17/2019 From HTTP to Kafka-based microservices AND CALL THIS PLACE AND CALL THIS PLACE async_calls async_calls localhost:4567/index.html?print-pdf#/ 30/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

  31. 7/17/2019 From HTTP to Kafka-based microservices AND CALL THIS PLACE AND CALL THIS PLACE async_calls async_calls (FOR THE LACK OF BETTER CONCEPT) (FOR THE LACK OF BETTER CONCEPT) localhost:4567/index.html?print-pdf#/ 31/83 From HTTP to Kafka-based microservices – Wojciech.Rzasa@flyrlabs.com – @wrzasa

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Apache Kafka Real-Time Data Pipelines http://kafka.apache.org/ Joe Stein Developer,

Apache Kafka Real-Time Data Pipelines http://kafka.apache.org/ Joe Stein Developer,

Apache Kafka Real-Time Data Pipelines http://kafka.apache.org/ Joe Stein Developer, Architect & Technologist Founder & Principal Consultant => Big Data Open Source Security LLC - http://stealth.ly Big Data Open Source

637 views • 40 slides
Kafka Needs No Keeper Colin McCabe 2 Introduction Kafka has gotten its mileage out of

Kafka Needs No Keeper Colin McCabe 2 Introduction Kafka has gotten its mileage out of

1 Kafka Needs No Keeper Colin McCabe 2 Introduction Kafka has gotten its mileage out of Zookeeper But it is still a second system KIP-500 has been adopted by the community This is not a 1-1 replacement Weve been

1.92k views • 124 slides
KAFKA STREAMS CLOUD MONITORING AWS CLOUD MONITORING AWS APP CLOUD MONITORING AWS HTTP APP

KAFKA STREAMS CLOUD MONITORING AWS CLOUD MONITORING AWS APP CLOUD MONITORING AWS HTTP APP

FROM USE CASE TO PRODUCTION KAFKA STREAMS CLOUD MONITORING AWS CLOUD MONITORING AWS APP CLOUD MONITORING AWS HTTP APP CLOUD MONITORING AWS METRICS NRDB HTTP APP CLOUD MONITORING SCHDLR JOBS AWS NRDB METRICS HTTP WORKER CLOUD

1.43k views • 124 slides
Introduction to Kafka Instructor: Ekpe Okorafor 1. Big Data Academy - Accenture 2. Computer

Introduction to Kafka Instructor: Ekpe Okorafor 1. Big Data Academy - Accenture 2. Computer

Introduction to Kafka Instructor: Ekpe Okorafor 1. Big Data Academy - Accenture 2. Computer Science - African University of Science & Technology Agenda Introduction - Messaging Basics Kafka Architecture Kafka

555 views • 26 slides
Cr Cruise Co Control: Effo l: Effortle less M Manage gement o of K f Kafka fka Clu

Cr Cruise Co Control: Effo l: Effortle less M Manage gement o of K f Kafka fka Clu

Cr Cruise Co Control: Effo l: Effortle less M Manage gement o of K f Kafka fka Clu Clusters Adem Efe Gencer Senior Software Engineer LinkedIn Kafka: A Distributed Stream Processing Platform : High throughput & low latency :

989 views • 50 slides
Kafka in Jail Running Kafka in container orchestrated clusters Sean Glover, Lightbend @seg1o

Kafka in Jail Running Kafka in container orchestrated clusters Sean Glover, Lightbend @seg1o

Kafka in Jail Running Kafka in container orchestrated clusters Sean Glover, Lightbend @seg1o Who am I? Im Sean Glover Senior Software Engineer at Lightbend Member of the Fast Data Platform team Contributor to various projects in

788 views • 60 slides
Apache Kafka + Apache Mesos Highly Scalable Streaming Microservices with Kafka Streams Kai

Apache Kafka + Apache Mesos Highly Scalable Streaming Microservices with Kafka Streams Kai

Apache Kafka + Apache Mesos Highly Scalable Streaming Microservices with Kafka Streams Kai Waehner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.kai-waehner.de Confidential 1 Abstract Microservices establish many

977 views • 44 slides
Day 3 Lab1: Spark Streaming with Kafka Example Introductions In this example, we will write a

Day 3 Lab1: Spark Streaming with Kafka Example Introductions In this example, we will write a

Day 3 Lab1: Spark Streaming with Kafka Example Introductions In this example, we will write a Spark Streaming program that consumes messages from Kafka. We will reuse the whole setup from the previous lab, so this lab is best done as a

233 views • 4 slides
High throughput High throughput kafka for science kafka for science Testing Kafkas limits

High throughput High throughput kafka for science kafka for science Testing Kafkas limits

High throughput High throughput kafka for science kafka for science Testing Kafkas limits for science J Wyngaard, PhD wyngaard@jpl.nasa.gov UTLINE O UTLINE O Streaming Science Data Benchmark Context Tests and Results

473 views • 30 slides
Day 4 Lab1: Docker container for Kafka - Spark streaming - Cassandra This Dockerfile sets up

Day 4 Lab1: Docker container for Kafka - Spark streaming - Cassandra This Dockerfile sets up

Day 4 Lab1: Docker container for Kafka - Spark streaming - Cassandra This Dockerfile sets up a complete streaming environment for experimenting with Kafka, Spark streaming (PySpark), and Cassandra. It installs Kafka 0.10.2.1 Spark 2.1.1

156 views • 4 slides
READING KAFKA IN QATAR Qatar-TESOL Conference, April 2011 Magdalena Rostron Academic Bridge

READING KAFKA IN QATAR Qatar-TESOL Conference, April 2011 Magdalena Rostron Academic Bridge

READING KAFKA IN QATAR Qatar-TESOL Conference, April 2011 Magdalena Rostron Academic Bridge Program Qatar Foundation, Doha, Qatar mrostron@qf.org.qa Introduction my story Reading Kafka In Qatar Reading Kafka in Qatar

562 views • 10 slides
Blockchain consensus Protocols in the Wild Tao Wang, Lihang Pan ECS 265 Apache Kafka

Blockchain consensus Protocols in the Wild Tao Wang, Lihang Pan ECS 265 Apache Kafka

Blockchain consensus Protocols in the Wild Tao Wang, Lihang Pan ECS 265 Apache Kafka (CFT) PBFT (BFT) Kafka Architecture ISR (in- sync) {b1,b2,b3} b1 b2 b3 Photo credit to Apache

2.57k views • 32 slides
INTEGRATING SYSTEMS INTEGRATING SYSTEMS IN THE AGE OF IN THE AGE OF QUARKUS, KNATIVE AND KAFKA

INTEGRATING SYSTEMS INTEGRATING SYSTEMS IN THE AGE OF IN THE AGE OF QUARKUS, KNATIVE AND KAFKA

Sponsorslides INTEGRATING SYSTEMS INTEGRATING SYSTEMS IN THE AGE OF IN THE AGE OF QUARKUS, KNATIVE AND KAFKA CONNECT QUARKUS, KNATIVE AND KAFKA CONNECT Peter Palaga @ppalaga The original reveal.js presentation this PDF was created from is

1.05k views • 93 slides
Cloud-Native and Scalable Kafka Allen Wang @allenxwang About Me Real Time Data

Cloud-Native and Scalable Kafka Allen Wang @allenxwang About Me Real Time Data

Cloud-Native and Scalable Kafka Allen Wang @allenxwang About Me Real Time Data Infrastructure @ Netflix Apache Kafka contributor (KIP-36 Rack Aware Assignment) NetflixOSS contributor (Archaius and Ribbon) Previously Cloud

722 views • 44 slides
BUBUKU kafka supervisor Dmitry Sorokin Zalando SE What is supervisor From wikipedia:

BUBUKU kafka supervisor Dmitry Sorokin Zalando SE What is supervisor From wikipedia:

BUBUKU kafka supervisor Dmitry Sorokin Zalando SE What is supervisor From wikipedia: blablabla controls the execution of other routines and regulates work scheduling, input/output operations, error actions, and similar functions and

133 views • 9 slides
PG Change Data Capture with Debezium Hannu Valtonen Kafka Meetup | PG CDC with Debezium |

PG Change Data Capture with Debezium Hannu Valtonen Kafka Meetup | PG CDC with Debezium |

PG Change Data Capture with Debezium Hannu Valtonen Kafka Meetup | PG CDC with Debezium | 2018-11-04 Me Aiven co-founder Maintainer of Open Source projects: PGHoard, pglookout and pgmemcache PostgreSQL user for last 18 years

894 views • 27 slides
Deploying Machine Translation in an Industrial Setting: Challenges and Opportunities

Deploying Machine Translation in an Industrial Setting: Challenges and Opportunities

Deploying Machine Translation in an Industrial Setting: Challenges and Opportunities Opportunities Opportunity 1 On-line spending power 2020 - $16 trillion English only! ( Source: PracticalEcommerce ) Industry without borders 2020 -

379 views • 19 slides
MULTINATIONAL & MULTILINGUAL REUSABLE E-COMMERCE ! PLATFORM CASE STUDY Maxime Topolov,

MULTINATIONAL & MULTILINGUAL REUSABLE E-COMMERCE ! PLATFORM CASE STUDY Maxime Topolov,

MULTINATIONAL & MULTILINGUAL REUSABLE E-COMMERCE ! PLATFORM CASE STUDY Maxime Topolov, @mtopolov ! CTO & Co-Founder of @Adyax ! 6 years of Drupal pain :) ! One of the oldest perfume brands in the world, created in 1828.

1.01k views • 63 slides
Freight Logistics eCommerce Trends Prepared for: Prepared by: January 12, 2018 Disclaimer

Freight Logistics eCommerce Trends Prepared for: Prepared by: January 12, 2018 Disclaimer

Freight Logistics eCommerce Trends Prepared for: Prepared by: January 12, 2018 Disclaimer Statement This report has been prepared by Transportation and Logistics Advisors (T&LA) for Stifel Capital Markets (Stifel). The report is based

616 views • 38 slides
Vertical integration in the e-commerce sector Claire Borsengerger (La Poste), Helmuth Cremer

Vertical integration in the e-commerce sector Claire Borsengerger (La Poste), Helmuth Cremer

Vertical integration in the e-commerce sector Claire Borsengerger (La Poste), Helmuth Cremer (IDEI), Denis Joram (La Poste) and Jean-Marie Lozachmeur (IDEI) 26th Conference on Postal and Delivery Economics Split, May 30- June 2, 2018

555 views • 16 slides
POSTAL, DELIVERY AND E-COMMERCE ECONOMICS AND POLICY SESSION 1 Pos ostal r tal ref efor orm

POSTAL, DELIVERY AND E-COMMERCE ECONOMICS AND POLICY SESSION 1 Pos ostal r tal ref efor orm

Postal web-conference | 19 May 2020 POSTAL, DELIVERY AND E-COMMERCE ECONOMICS AND POLICY SESSION 1 Pos ostal r tal ref efor orm SPYROS JEAN YVES ROBERT G. LORI RECTANUS GERMN JEAN PAUL PANTELIS MUYLLE TAUB VZQUEZ FORCEVILLE

528 views • 4 slides
eCommerce Strategies eCommerce Strategies Harvard Summer School 2009 John Paul Messina

eCommerce Strategies eCommerce Strategies Harvard Summer School 2009 John Paul Messina

eCommerce Strategies eCommerce Strategies Harvard Summer School 2009 John Paul Messina Teaching Fellow g Search Engine Optimization (SEO) is the process of setting a Web site to rank well for specific keywords within an organic search f

279 views • 9 slides
An International Perspective on Internet Legislation 1 7 th May 2 0 0 7 Richard Clayton Outline

An International Perspective on Internet Legislation 1 7 th May 2 0 0 7 Richard Clayton Outline

ECommerce Computer Science Tripos Part II An International Perspective on Internet Legislation 1 7 th May 2 0 0 7 Richard Clayton Outline IANAL! Data Protection Act 1998 US Privacy Laws Regulation of Investigatory Powers Act

601 views • 21 slides
Google Analytics Traversari Academy > GOOGLE ANALYTICS Conversion Definition A completed

Google Analytics Traversari Academy > GOOGLE ANALYTICS Conversion Definition A completed

Google Analytics Traversari Academy > GOOGLE ANALYTICS Conversion Definition A completed activity, online or offline, that is important to the success of your business. Examples include a completed sign-up for your email newsletter (a

478 views • 10 slides

More recommend