ARG Availability and reliability monitoring for e-Infrastructures - - PowerPoint PPT Presentation

Availability and reliability monitoring for e-Infrastructures

ARG

• C. Kanellopoulos, GRNET
• K. Kagkelidis, GRNET/AUTH

“You can not manage, what you do not monitor”

2

Status Availability

Reliability

Performance

Usage

Cost

Profit

Responsiveness

• In the last decade our notion of the e-Infrastructure has

dramatically changed

• Software Defined Networks
• Compute and data infrastructure as software

components

• The boundaries and shape of the e-Infrastructure are

dynamically changing to match the needs of the individual user

• Monitoring has to evolve in order to catch-up with the

dynamic nature of the systems

Dynamic e-Infrastructures

3

A model for monitoring large scale distributed infrastructures

4

Messaging Infrastructure

Ext. Source Ext. Source Ext. Source

Agents Agents

Agents

5

• Agents: collect data and publish them to the Messaging

Infrastructure

• Can be monitoring agents, accounting services etc

Messaging Infrastructure

Ext. Source Ext. Source Ext. Source

Agents Agents

• Nagios based monitoring
• Auto-configuration of monitored

services

• Nagios schedules and executes

probes against the monitored service end points

• Metric results are retrieved by each

Nagios instance, which in turn publishes them to the MBN

Monitoring agents

6

IT ES UK FR VO VO VO Messaging Infrastructure

Message Broker Network

7

• The distributed message broker network is the neural

network of the system (Publish - subscribe)

• Supports large number of concurrent publishers and

subscribers

Messaging Infrastructure

Ext. Source Ext. Source Ext. Source

Agents Agents

Messaging in ARGO

8

Message Broker Network

• ActiveMQ Message Brokers in Greece, Croatia and

Switzerland (now only in Greece and Croatia)

• High Availability configuration. Auto-discovery through

the Information System

External sources

9

Messaging Infrastructure

Ext. Source Ext. Source Ext. Source

Agents Agents

• External sources: can be used as source for information

in order to provide e-Infrastructure specific details

• Can be Configuration Databases, Management Systems

etc.

Subscribers

10

• Any number of concurrent subscribers.
• E.g. SLA (A/R) Monitoring, Accounting, Billing,

Operations portals etc.

Messaging Infrastructure

Ext. Source Ext. Source Ext. Source

Agents Agents

Abstract subscriber

11

Abstract subscriber

Web UI Message Broker Network Sync Components Data Store Stream Consumers Source Source Source

The ARGO Framework

12

A/R Compute Engine A/R Compute Engine A/R Compute Engine

Web Service + Distributed Data Store IT ES UK FR VO VO VO Message Broker Network Sync Components Hadoop Cluster Stream Consumers Source Source Source Prefiltering Batch Layer View Layer

ARGO A/R

13

• A/R consumer:
• tap and listen to the Message Broker

Network

• gather relevant messages
• able to manage and keep relevant fields on

each message

• deliver data to the A/R Compute Engine
• Sync components:
• topology information
• established grouping of services
• various weight information regarding grouping
• downtime information
• computation profiles
• ...also deliver data to the A/R Compute

Engine

A/R Compute Engine

14

• A/R Compute Engine:
• Picks up the data delivered
• both metrics and supplementary sync

files

• Stages the data (prefiltering process)
• Executes Computations
• Sending them as jobs to a Hadoop

Cluster

• r Locally if desired
• Results are stored to MongoDB for delivery

through the REST API

A/R Compute Engine

15

• Rest API:
• Supports a number of requests to view the

result data

• Retrieves data from the MongoDB
• Authentication
• Supports recomputation requests
• WebUI:
• Provides a UI dashboard with various views
• n the a/r report results
• Export of the results to various file formats
• Transforms User Interaction to specific API

requests

A/R Compute Engine

16

17

• Messaging Infrastructure
• Introduce a REST API for publishing and consuming messages
• Provide fine grained ACL support
• Messaging Infrastructure as a Service for end users
• ARGO
• Standalone version of ARGO
• Support for realtime monitoring at EGI scale
• Improve the visualisation support
• Custom of availability profiles for Cloud and Data Infrastructures
• Support federated access through eduGAIN

Next steps

18

• More information at:
• https://github.com/argoeu

19

Thank you