Kafla versus RabbitMQ Z. Salazar - R. Blanco 1
Introduction How do they compare against each other and which one to use? Figure 1. Interest over time (last 5 years). Worldwide.Google trends [2] 2
Outline Background 1 Pub/Sub systems Quality of Service attributes High-Level Description 2 Architectures Qualitative Comparison 3 Time decoupling, Routing logic, Delivery Guarantees, Ordering Guarantees, Availability, Transactions, Multicast, Dynamic Scaling Quantitative Comparison 4 Latency, Throughput Distinct features 5 What is unique to Kafka and what is unique to RabbitMQ? Conclusion 6 Preferred use cases 3
Background: Pub/Sub Systems Not aware of each other Not actively participating Distributed interaction paradigm 1: Decoupling the publishers and subscribers. Dimensions: Entity decoupling ● Time decoupling ● Synchronization ● decoupling Figure 2. Pub/Sub system scheme [3] 4
Background: Pub/Sub Systems #cats 2: Routing logic: Decides #cats #France if and where a packet #weather that is coming from a producer will end up at a #dogs consumer. Types: Topic-based ● #birds subscription Content-based ● ...Setting standards for the assistance dog industry subscription since 1987... Figure 2. Pub/Sub system scheme [3] 5
Background: Quality-of-Service Guarantees loss might at most once occur Delivery duplication at least once might occur Guarantees exactly once Correctness Best Ordering Performance no ordering Guarantees Quality of partitioned Service ordering Guarantees Latency time global order Efficiency BW Throughput 6 Transactions Scalability Availability 6
Description Kafla 0.10 vs RabbitMQ 3.5 7
High-level Description: Apache Kafka Built at LinkedIn to handle: ● Billions of messages ○ Consumers of the same stream, ○ reading at difgerent speeds Scalable system ● Distributed commit log ● (records) 8 Figure 3. Records in Kafla[4]
High-level Description: Apache Kafka Distributed configuration service Leader/follower Broker Figure 4. Apache Kafla architecture[1] 9
High-level Description: RabbitMQ Scalable implementation of the AMQP ( Advanced ● Message Queuing Protocol ): Interoperability, performance, scalability and reliability ○ Modular approach: Exchanges and queues ○ RabbitMQ: ● Effjcient acknowledgment mechanism for the publishers ○ Better defined transactional behavior ○ Figure 5. RabbitMQ architecture[5] Better support for asynchronous batch transfer ○ Degree of coupling between producers and consumers (i.e ○ the flow control) 10
Qualitative Comparison Kafla 0.10 vs RabbitMQ 3.5 11
Qualitative comparison RabbitMQ Kafla Time decoupling Designed with the DRAM as long as various consumption Bufger a large amount of possible rates DRAM messages Topic based Routing logic Topic based Content based Difgerent exchange types API to create others 12
Qualitative comparison RabbitMQ Kafla Delivery t3 : ACK to producer, and producer can delete the message Guarantees t4: consumer gets the message Acknowledgment t5: the consumer sends an t5: has no way of behavior DRAM ACK, and the broker can understanding ownership delete the message moved to the consumer. It will keep the message until a configured timeout expires 13
Qualitative comparison RabbitMQ Kafla Flows Partitions Ordering Reorders retransmitted Guarantees No inter-batch order packets inside its queue Replication Availability Replication Mirrored queues ares not automatically created 14
Qualitative comparison RabbitMQ Kafla If any message fails, the producer Does not support gets the chance to republish these Transactions transactions messages, and RabbitMQ will insert them in the queue in order Handled at the consumer Multicast side Same information Providing a dedicated queue per Keeps the message for a to multiple individual consume destinations configurable amount of time or size 15
Qualitative comparison RabbitMQ Kafla Additional nodes will be able to Dynamic The user can decide to become master for newly created Scaling move existing partitions queues Adding additional to the new node nodes to running Cannot be used to redistribute clusters or Not transparent for master queue assignments of removing a node consumers from a cluster existing queues 16
Quantitative Comparison Kafla 0.10.0.1 vs RabbitMQ 3.5.3 17
Quantitative Comparison Measured in terms of: Dimensions: ● Latency ● Delivery Guarantees ● Throughput ● Availability 18
Quantitative Comparison: Latency At Most Once Mode RabbitMQ Kafka Serial Pipeline Three Erlang Process Storage access latency handling a packet 50 percentile 99.9 percentile With and without replication: Latency results Mean: 1 - 4 ms ● without replication 1 ms 15 ms Max: 2 - 17 ms ● with replication 1 ms 30 ms At Least Once Mode Not really impacted Increases in case of replication Latency results 19
Quantitative Comparison: Throughput At Most Once Mode RabbitMQ Kafka Important factors Record size Record size, topic count and partition count At Least Once Mode Throughput drops by 50% Throughput decreases by 50% to 75% Results Throughput decreases by 50% to 75% Express performance in packets Express performance in bytes per per unit of time unit of time 20 Figure 7. Kafla throughput as function of partition count
Distinct Features Kafla 0.10 vs RabbitMQ 3.5 21
Distinct features RabbitMQ Kafka Long Term Message Storage ● Standardized Protocol ● Message Replay ● Multi-protocol ● Kafla Connect ● Comprehensive Management and Monitoring Tools ● Log Compaction ● Multi-tenancy and Isolation ● Consumer Tracking ● Disk-less ● Publisher Flow Control ● Queue Size Limits ● Message TTL ● 22
Preferred Use Cases Kafla 0.10 vs RabbitMQ 3.5 23
Preferred Use Cases Kafka RabbitMQ Pub/Sub Messaging Pub/Sub Messaging ● ● Request-Response Messaging Scalable Ingestion System ● ● Operational Metrics Tracking Data-Layer Infrastructure ● ● Underlying Layer for IoT Applications Platform Capturing Change Feeds ● ● Information-centric Networking Stream Processing ● ● RabbitMQ, followed by Kafla ● Kafla, followed by RabbitMQ ● Working in parallel ● 24
Conclusions Kafla 0.10 vs RabbitMQ 3.5 25
References [1] Philippe Dobbelaere and Kyumars Sheykh Esmaili, 2017, Industry Paper: Kafla versus RabbitMQ. A comparative study of two industry reference publish/subscribe implementations [2] https://trends.google.com/trends/explore?date=today%205-y&q=%2Fm%2F0bhc0tk,%2Fm%2F0zmynvd, 10/12/2019 [3] https://aws.amazon.com/pub-sub-messaging/, 10/12/2019 [4] https://www.youtube.com/watch?v=ElilYxUOjOQ, 10/12/2019 https://www.cloudamqp.com/blog/2015-05-18-part1-rabbitmq-for-beginners-what-is-rabbitmq.html, 10/12/2019 [5] 26
Thank you! Any Question? 27
Recommend
More recommend
