Hybrid Messaging Solutions OpenStack Summit Boston 2017 About - - PowerPoint PPT Presentation

Hybrid Messaging Solutions

OpenStack Summit Boston 2017

About Us...

Ken Giusti (kgiusti@gmail.com) irc: kgiusti

• Apache Qpid Project
• Oslo.Messaging Developer
• Software Engineer at Red Hat

Mark Wagner (mwagner@redhat.com)

• Performance and Scale Engineering
• Performance Engineer at Red Hat

Agenda

• Oslo.messaging (RPC and Notification Service Patterns)
• Backend Scenarios (Hybrid Messaging)
• The Drivers
• Testing Methodology and Results
• Next Steps...

Oslo.Messaging

Oslo.Messaging

• Part of the OpenStack Oslo project that provides intra-service messaging

○ Remote Procedure Calls (RPC) & Notifications

• Abstraction that hides the details of the underlying messaging technology from the OpenStack

Services

• slo.messaging

Messaging Bus

• slo.messaging
• slo.messaging
• slo.messaging
• slo.messaging
• slo.messaging

Oslo.Messaging Services

Notifications

• Asynchronous exchange from Notifier to Listener
• Listener need not be present when notification is sent
• Temporally decoupled
• Requires store-and-forward capability (e.g. queue or store)

Remote Procedure Call (RPC)

• Synchronous exchange between client and server
• Temporally bracketed
• If Server not present, call should fail

Messaging System

Caller Application Server Application

Notification - Broker Backed Messaging

Topic Queue

OpenStack Service

Notify Client

• slo.messaging
• slo.messaging
• Notification API provides the ability to publish an event to a known topic
• Notification Listener(s) fetch the event when ready to consume
• Unidirectional and asynchronous - no traffic flows back to the notifier
• Queue store-and-forward capability is necessary for notify messaging pattern

Notify

Listener

OpenStack Service

Notify

Listener

OpenStack Service

. . .

RPC - Broker Backed Messaging

Request Queue Reply Queue

OpenStack Service

RPC Client RPC

Server

OpenStack Service

• slo.messaging
• slo.messaging
• RPC transaction takes place across 4 discrete message ownership transfers

○ Requires two queues with associated resource overhead

• Detachment of client and server

○ Messages are acknowledged but not guaranteed to be delivered

• Clients do not readily know when a server is “unavailable”

RPC - Direct Messaging

OpenStack Service

RPC Client RPC

Server

OpenStack Service

• slo.messaging
• slo.messaging
• End-to-end transfer of message ownership

○ No store-and-forward message queueing (stateless intermediaries)

• Logical link between clients and server
• Clients can immediately know when a server is “unavailable”

Oslo.Messaging

RPC Service Notification Service Shared Messaging Stuff Transport (driver) API API Message Bus

Oslo.Messaging

RPC Service Notification Service Shared Messaging Stuff RPC Transport API API Message Bus Notification Transport Message Bus

Oslo.Messaging

RPC Service Notification Service Shared Messaging Stuff RPC Transport API API Message Bus Notification Transport Message Bus get_notification_transport(URL) get_transport(URL) get_rpc_transport(URL) [TBD]

Service Configuration (e.g. nova.conf, etc.)

[DEFAULT] transport_url=rabbit://rpc_user:rpc_pw@rpc_host:rpc_port ... [oslo_messaging_notifications] transport_url=rabbit://notify_user:notify_pw@notify_host:notify_port

Single Backend

OpenStack Service

• slo.messaging

OpenStack Service

• slo.messaging

OpenStack Service

• slo.messaging

OpenStack Service

• slo.messaging

RPC Notify RPC Notify RPC Notify RPC Notify

Broker Cluster

Dual Backend

OpenStack Service

• slo.messaging

OpenStack Service

• slo.messaging

OpenStack Service

• slo.messaging

OpenStack Service

• slo.messaging

RPC Notify RPC Notify RPC Notify RPC Notify

Broker Cluster Broker Cluster

“Hybrid” Backend

OpenStack Service

• slo.messaging

OpenStack Service

• slo.messaging

OpenStack Service

• slo.messaging

OpenStack Service

• slo.messaging

RPC Notify RPC Notify RPC Notify RPC Notify

Direct Messaging Broker Cluster

Benefits of Hybrid Messaging

• Optimal alignment of messaging patterns to messaging backend

○ Peer-to-peer messaging for RPC services (direct) ■ “Fail fast - fail clean” ○ Store-and-Forward for Notification services (queueing) ■ Mirroring and message persistence ○ Increase scale - Increase performance

• Diverse Topologies & Alternative Messaging Technology

○ Centralized brokers (hub-n-spoke) ○ Distributed architectures ○ Messaging as a Service

Alternative Oslo Messaging Transports

• ZeroMQ Socket Library
• AMQP 1.0 Protocol Transport + Qpid Dispatch Router
• Apache Kafka (experimental)

○ Notifications only (no RPC support)

ZeroMQ Transport

• Dedicated TCP connection between client and server
• Matchmaker (Redis) - maps topics ←→ host addresses
• TCP concentrator to limit TCP resource consumption
• Deployer’s guide: https://docs.openstack.org/developer/oslo.messaging/zmq_driver.html

Server A Server B Client Client Client Client Server A Server B Client Client Client Client Proxy

AMQP 1.0 + Message Router

• Network of “message routers”
• Routers learn location of servers - optimal shortest path
• Stateless (no queueing) - end to end transfers
• Barcelona: https://www.youtube.com/watch?v=R0fwHr8XC1I
• Deployer’s guide: https://docs.openstack.org/developer/oslo.messaging/AMQP1.0.html

Server A Server B Client Client Client Client Server C Client

Oslo.messaging matrix (drivers, backend, patterns)

Driver Messaging Backend Backend Type RPC Notification Rabbit (kombu,pika) rabbitmq-server Broker AMQP 1.0 qdrouterd Direct Messaging ZMQ TCP, proxied Direct Messaging Kafka kafka server Broker-like Distributed Streaming

Test Methodology

Testing Methodology

• Objective: benchmark RPC using both direct and queued approaches

○ Observe behavior and quantify

• Tool - Oslo Messaging Benchmark Tool

○ A driver independent tool for distributed messaging load generation and measurement ○ https://github.com/kgiusti/ombt ○ Does NOT simulate Openstack project(s) traffic patterns

• Scenarios

○ Separation of RPC and Notification traffic ○ Scoped to rabbit:// and amqp:// drivers for now

• Traffic modeling assumptions

○ RPC-Notify Message ratios, Producer-Consumer Ratios, Message Payload size

• Single server deployments for this testing phase

○ Clusters and meshes planned for follow-up scale and resiliency comparisons

rabbit

Scenario 1 - Single Broker Backend

• Broker used for both RPC and Notifications - not durable
• Messaging Assumptions

○ RPC-Notify Traffic ratios - 50/50 ○ Producer/Consumer ratios - 2/1 ○ Payload size - 1K

Notify clients Notify listeners rpc clients rpc servers

• mbt

controller

rabbit

Scenario 2 - Single Broker Backend - Durable

• Broker used for both RPC and Notifications
• Messaging Assumptions

○ RPC-Notify Traffic ratios - 50/50 ○ Producer/Consumer ratios - 2/1 ○ Payload size - 1K

Notify clients Notify listeners rpc clients rpc servers

• mbt

controller

rabbit

Scenario 3 - Separate Broker Backend

• Dual rabbitmq-server backends
• Notifications measured with persistence

Notify clients Notify listeners rpc clients rpc servers

• mbt

controller

rabbit

amqp

Scenario 4 - Hybrid Messaging Backend

• Direct Messaging Backend used for RPC (amqp:// and qpid-dispatch-router)
• Broker Backend used for Notifications (rabbit:// and rabbitmq-server)

Notify clients Notify listeners rpc clients rpc servers

• mbt

controller

rabbit

Final Comparison

Next Steps

• Try it out

○ AMQP 1.0 devstack plugin supports a hybrid configuration “qpid-hybrid” ■ https://git.openstack.org/openstack/devstack-plugin-amqp1 ○ Zmq devstack plugin: https://git.openstack.org/openstack/devstack-plugin-zmq

• [developers]

○ “Messaging” != “Queueing” and oslo.messaging >= rabbitmq ○ Use “get_notification_transport” and “get_rpc_transport” - keep ‘em separated... ○ Get Involved! - oslo.messaging needs you!

• Expand hybrid messaging scenarios in gate checks
• Test and measure additional hybrid scenarios

○ Plausible driver-backend combinations for RPC and Notifications

• Improve the ease-of-use and configuration of hybrid backends

○ Make it easy for the operator to deploy and get immediate value