Kafla versus RabbitMQ Z. Salazar - R. Blanco 1 Introduction How - - PowerPoint PPT Presentation

▶
kafla versus rabbitmq
SMART_READER_LITE
LIVE PREVIEW

Kafla versus RabbitMQ Z. Salazar - R. Blanco 1 Introduction How - - PowerPoint PPT Presentation

Dec 14, 2023 355 likes •641 views

Kafla versus RabbitMQ Z. Salazar - R. Blanco 1 Introduction How do they compare against each other and which one to use? Figure 1. Interest over time (last 5 years). Worldwide.Google trends [2] 2 Outline Background 1 Pub/Sub systems

slide-1
SLIDE 1

Kafla versus RabbitMQ

  • Z. Salazar - R. Blanco

1

slide-2
SLIDE 2

Introduction

2

Figure 1. Interest over time (last 5 years). Worldwide.Google trends [2]

How do they compare against each

  • ther and which one to use?
slide-3
SLIDE 3

Outline

3

Background

Pub/Sub systems Quality of Service attributes

1

High-Level Description

Architectures

2

Qualitative Comparison

Time decoupling, Routing logic, Delivery Guarantees, Ordering Guarantees, Availability, Transactions, Multicast, Dynamic Scaling 3

Quantitative Comparison

Latency, Throughput

4

Distinct features

What is unique to Kafka and what is unique to RabbitMQ?

5

Conclusion

Preferred use cases

6

slide-4
SLIDE 4

Background: Pub/Sub Systems

1: Decoupling the publishers and

  • subscribers. Dimensions:
  • Entity decoupling
  • Time decoupling
  • Synchronization

decoupling

4

Distributed interaction paradigm

Not aware of each other Figure 2. Pub/Sub system scheme [3] Not actively participating

slide-5
SLIDE 5

Background: Pub/Sub Systems

2: Routing logic: Decides if and where a packet that is coming from a producer will end up at a

  • consumer. Types:
  • Topic-based

subscription

  • Content-based

subscription

5

Figure 2. Pub/Sub system scheme [3] #cats #France #weather #cats

...Setting standards for the assistance dog industry since 1987...

#dogs #birds

slide-6
SLIDE 6

Background: Quality-of-Service Guarantees

6

Correctness Efficiency Latency Throughput

6

Delivery Guarantees Ordering Guarantees partitioned

  • rdering

global order no ordering Availability BW Transactions Scalability Quality of

Service Guarantees

at most once at least once exactly once loss might

  • ccur

duplication might occur

Best Performance

time

slide-7
SLIDE 7

Description

Kafla 0.10 vs RabbitMQ 3.5

7

slide-8
SLIDE 8

High-level Description: Apache Kafka

  • Built at LinkedIn to handle:

○ Billions of messages ○ Consumers of the same stream, reading at difgerent speeds

  • Scalable system
  • Distributed commit log

(records)

8

Figure 3. Records in Kafla[4]

slide-9
SLIDE 9

High-level Description: Apache Kafka

9

Figure 4. Apache Kafla architecture[1]

Broker Leader/follower Distributed configuration service

slide-10
SLIDE 10

High-level Description: RabbitMQ

  • Scalable implementation of the AMQP (Advanced

Message Queuing Protocol): ○ Interoperability, performance, scalability and reliability ○ Modular approach: Exchanges and queues

  • RabbitMQ:

○ Effjcient acknowledgment mechanism for the publishers ○ Better defined transactional behavior ○ Better support for asynchronous batch transfer ○ Degree of coupling between producers and consumers (i.e the flow control)

10

Figure 5. RabbitMQ architecture[5]

slide-11
SLIDE 11

Qualitative Comparison

Kafla 0.10 vs RabbitMQ 3.5

11

slide-12
SLIDE 12

Qualitative comparison

12

DRAM RabbitMQ Kafla Time decoupling

Bufger a large amount of messages

DRAM as long as possible Designed with the various consumption rates Routing logic

Difgerent exchange types Topic based Content based API to create others

Topic based

slide-13
SLIDE 13

Qualitative comparison

13

DRAM RabbitMQ Kafla Delivery Guarantees

Acknowledgment behavior

t3 : ACK to producer, and producer can delete the message t4: consumer gets the message t5: the consumer sends an ACK, and the broker can delete the message t5: has no way of understanding ownership moved to the consumer. It will keep the message until a configured timeout expires

slide-14
SLIDE 14

Qualitative comparison

14

RabbitMQ Kafla Ordering Guarantees Flows Reorders retransmitted packets inside its queue Partitions No inter-batch order Availability Replication Mirrored queues ares not automatically created Replication

slide-15
SLIDE 15

Qualitative comparison

15

RabbitMQ Kafla Transactions If any message fails, the producer gets the chance to republish these messages, and RabbitMQ will insert them in the queue in order Does not support transactions Multicast

Same information to multiple destinations

Providing a dedicated queue per individual consume Handled at the consumer side Keeps the message for a configurable amount of time or size

slide-16
SLIDE 16

Qualitative comparison

16

RabbitMQ Kafla Dynamic Scaling

Adding additional nodes to running clusters or removing a node from a cluster

Additional nodes will be able to become master for newly created queues Cannot be used to redistribute master queue assignments of existing queues The user can decide to move existing partitions to the new node Not transparent for consumers

slide-17
SLIDE 17

Quantitative Comparison

Kafla 0.10.0.1 vs RabbitMQ 3.5.3

17

slide-18
SLIDE 18

Quantitative Comparison

18

  • Latency
  • Throughput
  • Delivery Guarantees
  • Availability

Measured in terms of: Dimensions:

slide-19
SLIDE 19

Quantitative Comparison: Latency

19

RabbitMQ Kafka

Serial Pipeline handling a packet

Storage access latency Three Erlang Process

Latency results

With and without replication:

  • Mean: 1 - 4 ms
  • Max: 2 - 17 ms

50 percentile 99.9 percentile

without replication 1 ms 15 ms with replication 1 ms 30 ms

At Least Once Mode At Most Once Mode Latency results

Increases in case of replication Not really impacted

slide-20
SLIDE 20

Quantitative Comparison: Throughput

20

RabbitMQ Kafka

Important factors

Record size, topic count and partition count Record size

At Least Once Mode At Most Once Mode Results

Throughput decreases by 50% to 75% Throughput drops by 50% Express performance in packets per unit of time Express performance in bytes per unit of time Figure 7. Kafla throughput as function

  • f partition count

Throughput decreases by 50% to 75%

slide-21
SLIDE 21

Distinct Features

Kafla 0.10 vs RabbitMQ 3.5

21

slide-22
SLIDE 22

Distinct features

22

RabbitMQ Kafka

  • Long Term Message Storage
  • Message Replay
  • Kafla Connect
  • Log Compaction
  • Standardized Protocol
  • Multi-protocol
  • Comprehensive Management and Monitoring Tools
  • Multi-tenancy and Isolation
  • Consumer Tracking
  • Disk-less
  • Publisher Flow Control
  • Queue Size Limits
  • Message TTL
slide-23
SLIDE 23

Preferred Use Cases

Kafla 0.10 vs RabbitMQ 3.5

23

slide-24
SLIDE 24

Preferred Use Cases

24

RabbitMQ Kafka

  • Pub/Sub Messaging
  • Scalable Ingestion System
  • Data-Layer Infrastructure
  • Capturing Change Feeds
  • Pub/Sub Messaging
  • Request-Response Messaging
  • Operational Metrics Tracking
  • Underlying Layer for IoT Applications Platform
  • Information-centric Networking
  • Stream Processing
  • RabbitMQ, followed by Kafla
  • Kafla, followed by RabbitMQ
  • Working in parallel
slide-25
SLIDE 25

Conclusions

Kafla 0.10 vs RabbitMQ 3.5

25

slide-26
SLIDE 26

References

[1] Philippe Dobbelaere and Kyumars Sheykh Esmaili, 2017, Industry Paper: Kafla versus RabbitMQ. A comparative study of two industry reference publish/subscribe implementations [2] https://trends.google.com/trends/explore?date=today%205-y&q=%2Fm%2F0bhc0tk,%2Fm%2F0zmynvd, 10/12/2019 [3] https://aws.amazon.com/pub-sub-messaging/, 10/12/2019 [4] https://www.youtube.com/watch?v=ElilYxUOjOQ, 10/12/2019

[5]

https://www.cloudamqp.com/blog/2015-05-18-part1-rabbitmq-for-beginners-what-is-rabbitmq.html, 10/12/2019

26

slide-27
SLIDE 27

Thank you! Any Question?

27