SLIDE 1
EGU Vienna 2015
Supervising simulations with the Prodiguer Messaging Platform
Mark A. Greenslade, Nicolas Carenton, Sebastien Denvil Institut Pierre Simon Laplace, Paris, France
EGU Vienna 2015 Supervising simulations with the Prodiguer Messaging - - PowerPoint PPT Presentation
EGU Vienna 2015 Supervising simulations with the Prodiguer Messaging - - PowerPoint PPT Presentation
EGU Vienna 2015 Supervising simulations with the Prodiguer Messaging Platform Mark A. Greenslade, Nicolas Carenton, Sebastien Denvil Institut Pierre Simon Laplace, Paris, France CONVERGENCE Big Data and Exascale challenges for Climate Sciences
SLIDE 2
SLIDE 3
Message Flow
SLIDE 4
MQ Cluster MQ Apps API DB’s
IPSL IPSL
IPSL User @ Browser | Command Line | Desktop
json
TGCC
SMTP Relay
IDRIS
SMTP Relay
CINES
SMTP Relay
IPSL
SMTP Relay
XXX
SMTP Relay
msg msg msg msg msg
SLIDE 5
Flow 1: CC.NODE ---> IPSL.SMTP
SMTP @ IPSL
simul
(fortran)
liblIGCM
(ksh)
SMTP client
args args
Compute Node @ Computing Centre
msg-batch
SLIDE 6
Flow 2: IPSL.SMTP ---> IPSL.MQ
SMTP @ IPSL MQ Cluster @ IPSL
(ampqs + rabbit-mq)
msg-batch
SLIDE 7
Flow 3: IPSL.MQ <---> IPSL.MQ-APP
MQ Cluster @ IPSL
(ampqs + rabbit-mq + shovel)
tcp/ip (port=5672)
MQ APPS @ IPSL
sim-mon
(python)
metrics
(python)
esg-f
(python)
es-doc
(python)
primary
api
(python)
smtp
(python)
img
(python)
secondary
msg msg
SLIDE 8
Flow 4: IPSL.MQ-APP ---> Other
MQ APPS @ IPSL
sim-mon
(python)
metrics
(python)
esg-f
(python)
es-doc
(python)
primary
api
(python)
smtp
(python)
cv
(python)
secondary
DB’s @ IPSL
(Relational, NO-SQL, Graph)
API
@ ESG-F
SMTP
@ IPSL
API
@ ES-DOC
GitHub FE @ browser tcp/ip https smtp https wss
SLIDE 9
Flow 5: IPSL.API ---> IPSL.FE
API @ IPSL
(tornado http & ws server)
sim-mon
(python)
FE @ browser
(jquery, backbone, lo-dash, bootstrap)
sim-mon
(javascript)
https / wss
json
SLIDE 10
Rabbit MQ
« Messaging that just works »
SLIDE 11
Rabbit MQ - Security
AMPQS = transport level akin to HTTPS Authentication = SASL PLAIN (other schemas supported) Authorization = user accounts are allocated permissions to MQ vhost, exchange, queue with regular expressions
SLIDE 12
Rabbit MQ – Message Structure
Header = AMPQ Basic Properties
Key / Value pairs Common headers mandated by AMPQ spec Prodiguer injects custom headers
Body
Text blob Content type / encoding specified in header Platform passes decoded messages to consumers
SLIDE 13
Rabbit MQ - Durability
Ensuring buffered messages aren’t lost if server crashes RabbitMQ persists messages to disk Messages marked as persistent and exchanges as durable Incurs performance overhead
SLIDE 14
Rabbit MQ - Reliability
Ensuring messages are reliably published / consumed Publishers:
Receive confirmation when message reaches exchange If unconfirmed then need failure strategy such as “retry 3 times and then save to local file system”
Consumers:
Acknowledge when unit of work succeeds Acknowledged messages are removed from queue Unacknowledged messages require failure strategy, e.g. retry
SLIDE 15
Rabbit MQ - Routing
MQ producers & consumers bind to exchanges Producers specify a “.” delimited routing key:
[mode].[user-id].[producer-id].[app-id].[type]
Consumers specify a routing key filter:
e.g. *.*.*.*.0000 = simulation initialisation messages
Routing objective: optimise ratio of queues to workers so that queues are as empty as possible
SLIDE 16
Summary
SLIDE 17
Summary - I
Secure Lossless Scalable Extensible Open Source Real time Non-intrusive
SLIDE 18