WSO2 Message Broker Scalable persistent Messaging System Outline - - PowerPoint PPT Presentation

WSO2 Message Broker

Scalable persistent Messaging System

Outline

• Messaging
• Scalable Messaging
• Distributed Message Brokers
• WSO2 MB Architecture
• Distributed Pub/sub architecture
• Distributed Queues architecture
• User Story
• Conclusion

What is Messaging ?

• We often program and design distributed

systems with RPC style communication

• E.g. Web Services, Thrift, REST
• RPC communication is
• Request/Response (there is always a response)
• Synchronous (client waits for response)
• Non-persistent (message is lot if something failed)
• But there are other 7 possibilities
• Under messaging we support those
• Build on top of single message, with flexibility

(users can choose) in other dimensions

WSO2 Inc. 3

Messaging Systems in Real World

• There are many types of message systems in

the real word

• Sensor networks
• Monitoring/ Surveillance
• Business Activity Monitoring
• Job Scheduling systems
• Social Networks

WSO2 Inc. 4

Why Messaging?

• More reliability
• E.g. via persistence, transactions
• Decupling
• Space
• Time
• Synchronization

WSO2 Inc. 5

Messaging Server Models

• Messaging is implemented with a broker (or

brokers in the middle)

• Participants send messages, and broker

delivers them to recipients

• There are two main models
• Queues - A message is delivered only once to a

single consumer.

• Publish/Subscribe: Broadcast a message to many

message consumers

WSO2 Inc. 6

Distributed Queues

• A queue in the “Cloud”
• Supports Operations
• Put(M) – put a message
• Get() – get a message (dqueue)
• Subscribe() – send me a message when there is one
• E.g. SQS (Amazon Queuing Service)
• Usecases
• Job Queues
• Sored and process

7

Publish/ Subscribe

• There is a topic space based on interest
• Publishers send messages to brokers
• Subscribers registers their interest
• Brokers matches events (messages) and

delivers to all interested parties

• Usecases
• Surveillance
• Monitoring

WSO2 Inc. 8

What is JMS ?

• JMS – Java Message Service
• A specification that define a standard API for java

programmer to perform messaging by interacting with a message broker

• Support both
• Distributed Queue
• Publish/Subscribe
• It does not define the message format or how java

API interacts with the message broker

WSO2 Inc. 9

What is AMQP ?

• Advanced Message Queuing Protocol (AMQP)
• Open standard for passing business messages

between applications or organizations.

• JMS does not define the message format, and

AMQP fills that gap

• AMQP let different systems (e.g. .NET and

Java) to interact with each other by agreeing the message format at the wire level just like Web Services.

WSO2 Inc. 10

Brokers

• Message broker support messaging
• Some brokers can be setup as a network or a

cluster

• Some of well known brokers
• Apache Qpid - http://qpid.apache.org/
• Storm MQ - http://stormmq.com/
• Active MQ - http://activemq.apache.org/
• HornetQ - http://www.jboss.org/hornetq
• Rabbit MQ - http://www.rabbitmq.com/
• IBM WebSphere MQ - http://www-

01.ibm.com/software/integration/wmq/

WSO2 Inc. 11

Scaling

• There a several dimensions of Scale
• Number of messages
• Number of Queues
• Size of messages
• Scaling Pub/Sub is relatively easy
• E.g. Consider cluster of brokers. If all node know about all

subscriptions, all publish messages can be delivered

• E.g. Narada Broker, Padres
• Scaling Distributed Queues is harder

WSO2 Inc. 12

Scaling Distributed Queues

WSO2 Inc. 13

Scaling Distributed Queues (Contd.)

Topology Pros Cons Supporting Systems Master Salve Support HA No Scalability Qpid, ActiveMQ, RabbitMQ Queue Distribution Scale to large number of Queues Does not scale for large number of messages for a queue RabbitMQ Cluster Connections Support HA Might not support in-order delivery Logic runs in the client side takes local decisions. HorentMQ Broker/Queue Networks Load balancing and distribution Fair load balancing is hard ActiveMQ

WSO2 Inc. 14

Alternative Message Broker Design

• Most persistent message brokers use a per-node

DB to store messages with message routing.

• But with large messages, cost of routing messages
• ver the network is very high
• With availability of scalable storage and

distributed coordination middleware we propose an alternative architecture for scalable message brokers

• Main idea
• Avoid message routing
• Use scalable storage to share messages between nodes
• Use distributed coordination to control the behavior

Cassandra and Zookeeper

• Cassandra
• NoSQL Highly scalable new data model (column

family)

• Highly scalable (multiple Nodes), available and no

Single Point of Failure.

• SQL like query language (from 0.8) and support

search through secondary indexes (well no JOINs, Group By etc. ..).

• Tunable consistency and replication
• Very high write throughput and good read
• throughput. It is pretty fast.
• Zookeeper
• Scalable, fault tolerant distributed coordination

framework

WSO2 Message Broker

 Use Apache Zookeeper for coordination when

needed

 Support for AMQP JMS and WS-Eventing while

enabling interoperability between protocols

 Built by extending Apache Qpid Code base

WSO2 MB Architecture

WSO2 Inc. 18

How Distributed Queues Works ?

WSO2 Inc. 19

How Distributed Queues Works Contd..

WSO2 Inc. 20

How Distributed Queues Works Contd..

 Each node contains a node queue. Message

meta data are stored in this queue. A Queue Delivery Worker running in each node and consume messages in the above node queue. Destination is extracted from this consumed message and delivered to the endpoint.

 MB stores message content separately  Delivery logic works with message IDs written

to queue representation in Cassandra and it

• nly reads the messages at delivery

WSO2 Inc. 21

Distributed Queues

• Strict ordering means there can be one message

being delivered at a give time.

• Say we receive messages m1, m2 for Queue Q.
• Say we deliver messages m1 and m2 to client c1 and

c2 for Queue Q in parallel

• Say m1->c1 failed, but by then m2->c2 is done.
• If there is no other subscribers, now m1 has to be

delivered out of order.

• Two implementation
• Strict ordering support - using a distributed shared

lock with Zookeeper

• Best effort implementation

How Pub/Sub Works ?

WSO2 Inc. 23

How Pub/Sub Works Contd…

 There is a node queue for each of the brokers.  When published message to a topic, broker get

the list of nodes where subscriptions available for the topic and write the message id to each

• f the node queue connected to brokers.

 A worker thread running in each of these

brokers to consume messages from the above node queue and deliver the message to subscriber.

WSO2 Inc. 24

MB2 JMS Support

Feature Yes No Pub / Sub √ Durable Subscriptions √ Hierarchical Topics √ Queues √ Message Selectors √ Transactions √

WSO2 Inc. 25

How does it Make a difference?

• Scale up in all 3 dimensions
• Create only one copy of message while delivery
• High Availability and Fault Tolerance
• Large message transfers in pub/sub

(asynchronous style)

• Let users choose between strict and best effort

messages

• Replication of stored messages in the storage

Conclusion and Future Work

• Provides an alternative architecture for scalable

message brokers using Cassandra and Zookeeper

• It provides
• A publish/subscribe model that does not need any

coordination between broker nodes

• A strict mode for distributed queues that provides in
• rder delivery
• A best-effort mode for distributed queue
• Future work
• Further Scalability Tests
• Testing with large messages
• Fault Tolerance Tests

Integrating with WSO2 ESB

• JMS Transport
• JMS endpoints and JMS proxy services
• Message Stores and Processors

WSO2 Inc. 28

Integrating with WSO2 DSS

• JMS Transport
• JMS transport enabled data services

WSO2 Inc. 29

User Story

• An SAP system need to distribute IDOCs to

their point of sales which are distributed Island

• wide. IDOCs are sending out from SAP as

batches issued within small amount of time

• period. These IDOCs need to transform in to

SOAP messages and need to inject some

• properties. Finally these messages need to

update the data bases in Point of Sales.

WSO2 Inc. 30

User Story Contd..

WSO2 Inc. 31

Questions?

Thank you.