Firehose: a Unified Message Bus for Infra Services Jeremy Stanley - - PowerPoint PPT Presentation

Firehose: a Unified Message Bus for Infra Services

Matthew Treinish mtreinish@kortar.org mtreinish on Freenode Jeremy Stanley fungi@yuggoth.org fungi on Freenode May 11, 2017 https://github.com/mtreinish/firehose/tree/boston-summit

OpenStack Infrastructure

Imagine a big, complicated diagram here.

1 / 18

Firehose

◮ An MQTT broker for the OpenStack

community infrastructure

◮ Has anonymous, read-only access via

MQTT on 1883/tcp

◮ SSL/TLS MQTT also available on

8883/tcp

◮ Websockets supported (but

temporarily disabled)

2 / 18

MQTT

◮ Pub/sub messaging protocol ◮ Formerly MQ Telemetry Transport ◮ ISO/IEC 20922 ◮ Protocol dates back to 1999 ◮ Lightweight design, low bandwidth

3 / 18

MQTT Topics

◮ Topics are generated dynamically ◮ Topics are heirarchical ◮ Support wildcarding

4 / 18

Topic Examples

sensors/HOSTNAME/temperature/HDD_NAME

◮ sensors/sinanju/temperature/nvme0n1p1 ◮ sensors/+/temperature/+ ◮ sensors/sinanju/temperature/+ ◮ sensors/sinanju/#

5 / 18

QoS

◮ 0: The broker/client will deliver the message once, with no confirmation. ◮ 1: The broker/client will deliver the message at least once, with confirmation

required.

◮ 2: The broker/client will deliver the message exactly once by using a four step

handshake.

6 / 18

MQTT Clients

◮ Bindings available for most languages ◮ https://github.com/mqtt/mqtt.github.io/wiki/libraries ◮ Eclipse Paho project provides similar interfaces across multiple languages

7 / 18

MQTT Brokers

◮ Centralized broker ◮ Many different options: https://github.com/mqtt/mqtt.github.io/wiki/servers

8 / 18

Mosquitto

◮ MQTT broker implemented in C ◮ An Eclipse IoT project ◮ Support for MQTT v3.1 and v3.1.1

9 / 18

The Firehose

◮ Runs Mosquitto MQTT broker ◮ Single broker instance ◮ Hardware Specs:

vCPUs 2 CPU Frequency 2.6 GHz RAM 2 GB swap 0 B Disk 40 GB Bandwidth 200 Mbps

10 / 18

Services Using the Firehose

Service Base Topic Source of Messages Ansible ansible Ansible MQTT Callback Plugin Gerrit gerrit germqtt Launchpad launchpad lpmqtt Subunit Gearman Worker gearman-subunit subunit-gearman-worker

11 / 18

Typical Firehose Load

12 / 18

13 / 18

Manually Load Testing

14 / 18

CPU Usage: Memory Usage:

15 / 18

Benchmarking Performance

https://github.com/mtreinish/pymqttbench

16 / 18

Using Firehose

Example mosquitto client subscriber command line (provided in a mosquitto-clients package on many distros): mosquitto_sub -h firehose.openstack.org --topic ’#’

17 / 18

Listen for all Nova Comments in python

◮ CLI:

mosquitto_sub -h firehose.openstack.org --topic gerrit/openstack/nova/comment-added

◮ Python:

import paho . mqtt . c l i e n t as mqtt d e f

• n_connect ( c l i e n t ,

u se r d a t a , f l a g s , r c ) : p r i n t ( " Connected with r e s u l t code " + s t r ( r c ) ) c l i e n t . s u b s c r i b e ( ’ g e r r i t / openstack / nova /comment−added ’ ) d e f

• n_message ( c l i e n t ,

u se r d at a , msg ) : p r i n t ( msg . t o p i c+" "+s t r ( msg . p ayload ) ) # Create a websockets c l i e n t c l i e n t = mqtt . C l i e n t ( ) c l i e n t . on_connect = on_connect c l i e n t . on_message = on_message # Connect to the f i r e h o s e c l i e n t . connect ( ’ f i r e h o s e . openstack . org ’ , p o r t =1883) # L i s t e n f o r e v e r c l i e n t . l o o p _ f o r e v e r ( ) 18 / 18

Listen for all ansible tasks on health.openstack.org

◮ ruby:

r e q u i r e ’ rubygems ’ r e q u i r e ’ mqtt ’ c l i e n t = MQTT: : C l i e n t . new c l i e n t . host = ’ f i r e h o s e . openstack . org ’ c l i e n t . port = 1883 c l i e n t . connect () c l i e n t . s u b s c r i b e ( ’ a n s i b l e / playbook/+/task / health . openstack .

• rg/# ’ )

c l i e n t . get do | topic , message | puts message end

19 / 18

◮ go:

package main import ( " fmt " MQTT " g i t h u b . com/ e c l i p s e / paho . mqtt . golang " " os " " s t r c o n v " " time " ) func

• nMessageReceived ( c l i e n t MQTT. C l i e n t ,

msg MQTT. Message ) { fmt . P r i n t f ( "TOPIC : %s \n" , msg . Topic ( ) ) fmt . P r i n t f ( "MSG: %s \n" , msg . Payload ( ) ) } func main ( ) { hostname , _ :=

• s . Hostname ( )
• p t s

:= &

• MQTT. C l i e n t O p t i o n s {

C l i e n t I D : hostname+s t r c o n v . I t o a ( time . Now( ) . Second ( ) ) , }

• p t s . AddBroker ( " tcp : / / f i r e h o s e . openstack . org :1883 " )
• p t s . OnConnect = func ( c MQTT. C l i e n t )

{ i f token := c . S u b s c r i b e ( " a n s i b l e / playbook /+/ t a s k / h e a l t h . openstack . org/#" , 0 ,

• nMessageReceived ) ;

token . Wait ( ) && token . E r r o r ( ) != n i l { fmt . P r i n t l n ( token . E r r o r ( ) )

• s . E x i t ( 1 )

} } c l i e n t := MQTT. NewClient ( o p t s ) i f token := c l i e n t . Connect ( ) ; token . Wait ( ) && token . E r r o r ( ) != n i l { p a n i c ( token . E r r o r ( ) ) } f o r { time . S l e e p (1 ∗ time . Second ) } } 20 / 18

Potential Applications for Firehose

◮ 3rd Party CI Operators ◮ Desktop Notifications ◮ Inter Service communication

21 / 18

Future Plans

◮ Add germqtt subscription support to:

◮ Zuul v3+

◮ Add more publishers:

◮ Nodepool daemons ◮ Zuul v3+ ◮ (your favorite service here) 22 / 18

Where to get more information

◮ openstack-infra ML openstack-infra@lists.openstack.org ◮ #openstack-infra on Freenode ◮ http://docs.openstack.org/infra/system-config/firehose.html ◮ https://docs.openstack.org/infra/system-config/firehose_schema.html ◮ http://specs.openstack.org/openstack-infra/infra-specs/specs/firehose.html ◮ http://mqtt.org/ ◮ https://mosquitto.org/ ◮ https://www.eclipse.org/paho/

23 / 18