Firehose: a Unified Message Bus for Infra Services Jeremy Stanley - PowerPoint PPT Presentation

Firehose: a Unified Message Bus for Infra Services Jeremy Stanley Matthew Treinish fungi@yuggoth.org mtreinish@kortar.org fungi on Freenode mtreinish on Freenode May 11, 2017 https://github.com/mtreinish/firehose/tree/boston-summit

OpenStack Infrastructure Imagine a big, complicated diagram here. 1 / 18

Firehose ◮ An MQTT broker for the OpenStack community infrastructure ◮ Has anonymous, read-only access via MQTT on 1883/tcp ◮ SSL/TLS MQTT also available on 8883/tcp ◮ Websockets supported (but temporarily disabled) 2 / 18

MQTT ◮ Pub/sub messaging protocol ◮ Formerly MQ Telemetry Transport ◮ ISO/IEC 20922 ◮ Protocol dates back to 1999 ◮ Lightweight design, low bandwidth 3 / 18

MQTT Topics ◮ Topics are generated dynamically ◮ Topics are heirarchical ◮ Support wildcarding 4 / 18

Topic Examples sensors/ HOSTNAME /temperature/ HDD_NAME ◮ sensors/sinanju/temperature/nvme0n1p1 ◮ sensors/ + /temperature/ + ◮ sensors/sinanju/temperature/ + ◮ sensors/sinanju/# 5 / 18

QoS ◮ 0 : The broker/client will deliver the message once, with no confirmation. ◮ 1 : The broker/client will deliver the message at least once, with confirmation required. ◮ 2 : The broker/client will deliver the message exactly once by using a four step handshake. 6 / 18

MQTT Clients ◮ Bindings available for most languages ◮ https://github.com/mqtt/mqtt.github.io/wiki/libraries ◮ Eclipse Paho project provides similar interfaces across multiple languages 7 / 18

MQTT Brokers ◮ Centralized broker ◮ Many different options: https://github.com/mqtt/mqtt.github.io/wiki/servers 8 / 18

Mosquitto ◮ MQTT broker implemented in C ◮ An Eclipse IoT project ◮ Support for MQTT v3.1 and v3.1.1 9 / 18

The Firehose ◮ Runs Mosquitto MQTT broker ◮ Single broker instance ◮ Hardware Specs: vCPUs 2 CPU Frequency 2.6 GHz RAM 2 GB swap 0 B Disk 40 GB Bandwidth 200 Mbps 10 / 18

Services Using the Firehose Service Base Topic Source of Messages Ansible ansible Ansible MQTT Callback Plugin Gerrit gerrit germqtt Launchpad launchpad lpmqtt Subunit Gearman Worker gearman-subunit subunit-gearman-worker 11 / 18

Typical Firehose Load 12 / 18

13 / 18

Manually Load Testing 14 / 18

CPU Usage: Memory Usage: 15 / 18

Benchmarking Performance https://github.com/mtreinish/pymqttbench 16 / 18

Using Firehose Example mosquitto client subscriber command line (provided in a mosquitto-clients package on many distros): mosquitto_sub -h firehose.openstack.org --topic ’#’ 17 / 18

Listen for all Nova Comments in python ◮ CLI: mosquitto_sub -h firehose.openstack.org --topic gerrit/openstack/nova/comment-added ◮ Python: import paho . mqtt . c l i e n t as mqtt d e f on_connect ( c l i e n t , u se r d a t a , f l a g s , r c ) : p r i n t ( " Connected with r e s u l t code " + s t r ( r c ) ) c l i e n t . s u b s c r i b e ( ’ g e r r i t / openstack / nova /comment − added ’ ) d e f on_message ( c l i e n t , u se r d at a , msg ) : p r i n t ( msg . t o p i c+" "+s t r ( msg . p ayload ) ) # Create a websockets c l i e n t c l i e n t = mqtt . C l i e n t ( ) c l i e n t . on_connect = on_connect c l i e n t . on_message = on_message # Connect to the f i r e h o s e c l i e n t . connect ( ’ f i r e h o s e . openstack . org ’ , p o r t =1883) # L i s t e n f o r e v e r c l i e n t . l o o p _ f o r e v e r ( ) 18 / 18

Listen for all ansible tasks on health.openstack.org ◮ ruby: r e q u i r e ’ rubygems ’ r e q u i r e ’ mqtt ’ c l i e n t = MQTT: : C l i e n t . new c l i e n t . host = ’ f i r e h o s e . openstack . org ’ c l i e n t . port = 1883 c l i e n t . connect () c l i e n t . s u b s c r i b e ( ’ a n s i b l e / playbook/+/task / health . openstack . org/# ’ ) c l i e n t . get do | topic , message | puts message end 19 / 18

◮ go: package main import ( " fmt " MQTT " g i t h u b . com/ e c l i p s e / paho . mqtt . golang " " os " " s t r c o n v " " time " ) func onMessageReceived ( c l i e n t MQTT. C l i e n t , msg MQTT. Message ) { fmt . P r i n t f ( "TOPIC : %s \n" , msg . Topic ( ) ) fmt . P r i n t f ( "MSG: %s \n" , msg . Payload ( ) ) } func main ( ) { hostname , _ := os . Hostname ( ) o p t s := & MQTT. C l i e n t O p t i o n s { C l i e n t I D : hostname+s t r c o n v . I t o a ( time . Now( ) . Second ( ) ) , } o p t s . AddBroker ( " tcp : / / f i r e h o s e . openstack . org :1883 " ) o p t s . OnConnect = func ( c MQTT. C l i e n t ) { i f token := c . S u b s c r i b e ( " a n s i b l e / playbook /+/ t a s k / h e a l t h . openstack . org/#" , 0 , onMessageReceived ) ; token . Wait ( ) && token . E r r o r ( ) != n i l { fmt . P r i n t l n ( token . E r r o r ( ) ) os . E x i t ( 1 ) } } c l i e n t := MQTT. NewClient ( o p t s ) i f token := c l i e n t . Connect ( ) ; token . Wait ( ) && token . E r r o r ( ) != n i l { p a n i c ( token . E r r o r ( ) ) } f o r { time . S l e e p (1 time . Second ) ∗ } } 20 / 18

Potential Applications for Firehose ◮ 3rd Party CI Operators ◮ Desktop Notifications ◮ Inter Service communication 21 / 18

Future Plans ◮ Add germqtt subscription support to: ◮ Zuul v3+ ◮ Add more publishers: ◮ Nodepool daemons ◮ Zuul v3+ ◮ (your favorite service here) 22 / 18

Where to get more information ◮ openstack-infra ML openstack-infra@lists.openstack.org ◮ #openstack-infra on Freenode ◮ http://docs.openstack.org/infra/system-config/firehose.html ◮ https://docs.openstack.org/infra/system-config/firehose_schema.html ◮ http://specs.openstack.org/openstack-infra/infra-specs/specs/firehose.html ◮ http://mqtt.org/ ◮ https://mosquitto.org/ ◮ https://www.eclipse.org/paho/ 23 / 18