The Cluster Monitoring System of IHEP Qingbao Hu huqb@ihep.ac.cn - - PowerPoint PPT Presentation

The Cluster Monitoring System

• f IHEP

Qingbao Hu

huqb@ihep.ac.cn

Computing Center, Institute of High Energy Physics, Chinese Academy of Sciences

International Symposium on Grids and Clouds (ISGC) 2016

Qingbao Hu/CC/IHEP 2016/3/24 - 2

Content

• Overview of IHEP’s Monitor
• rin

ing g Sy System

• Op

Optim imiza ization

• n of the monito

torin ring g tools

• Logger

er-an anal alysis ysis Monitor

• rin

ing

• Su

Summary

Status of IHEP Cluster

• ~

~ 1,122 2 work nodes

• ~

~ 13 13,500 0 CPU cores

• ~

~ 5PB PB disk stora rage

• Lustre,

tre, gLuste ter, r, openAF AFS, S, etc.

• ~

~ 5PB

PB tape stora rage ge

• Two IBM 3584

3584 tape librari aries, es, LTO4 4 tape

• Modifie

fied d CERN CASTOR R 1.7

Qingbao Hu/CC/IHEP 2016/3/24 - 3

Cluster built with blades Tape libraries

Monitoring requirements

• A large number of hardware and software resources
• Cooperated in complex ways

– Large Scale ( > 2,000 nodes ) – Heterogeneous device resources – Good Scalability – Real-time information display and alarm – Combination of active detection service and passive

information receiving.

– Auto recovery of failed services.

Qingbao Hu/CC/IHEP 2016/3/24 - 4

Monitoring System Overview

• System

tem overvi view

Qingbao Hu/CC/IHEP 2016/3/24 - 5

Monitoring system of IHEP

Ganglia Recording the performance of different resource groups Icinga Monitoring the status of cluster devices and services Logger Analysis Collecting more comprehensive data & providing an overview of the whole cluster health status

• Monitoring the health of the cluster

– System load – CPU utilization – Network bandwidth and traffic – Memory usage

• Usage

– Records history status of the cluster – Helps system manager to fix problem

Ganglia

Qingbao Hu/CC/IHEP 2016/3/24 - 6

• The bottleneck of Ganglia

– High frequency: Collect 20 metrics from each

monitored node every 15 seconds.

– Pool scalability: Large number of nodes cause a

large amount of metrics data, which pulls up the server’s peak iowait and slows down the monitoring service.

• Workaround:

– Create a ram disk on the Ganglia server to save the

RRDs data.

– Improves the IO performance of the server disk

Ganglia of IHEP

Qingbao Hu/CC/IHEP 2016/3/24 - 7

– IOwait < 1%

Ganglia of IHEP

Qingbao Hu/CC/IHEP 2016/3/24 - 8

Different clusters The status of bws1 farm

• Created as a fork of the Nagios

– Plug-in design – Active check of the service – Flexible configuration by NagiosQL

• Usage

– Hardware (CPU load, disk usage, etc.) – Network connectivity (HTTP, POP3, ping, etc.) – Computing services on work nodes – Distributed file system services …

Icinga

Qingbao Hu/CC/IHEP 2016/3/24 - 9

• Polling agents we developed

– More services monitored

– Some crashed service faults can be recovered

automatically

– Critical errors are alarmed to system manager via both

email and SMS

Icinga of IHEP

Qingbao Hu/CC/IHEP 2016/3/24 - 10

• The bottleneck of Icinga

– Single collector node. – Vast amounts of the service detections increases the

server load, which reduces the efficiency of the polling.

– Many detection results are delayed.

• Workarounds:

– Distributed Nagios eXecutor. (DNX)+ Icinga Sever

» A modular extension of Nagios » DNX Worker requests jobs from the Icinga (Scheduling)

Server

» DNX Worker executes the plug-in agents and return the

results to Icinga server.

– Balance the load of servers via distribution – Decrease the latency of the polling

Icinga of IHEP

Qingbao Hu/CC/IHEP 2016/3/24 - 11

Icinga in IHEP

Qingbao Hu/CC/IHEP 2016/3/24 - 12

scale of Monitoring hosts scale

• f Monitorin

g service The average host delay The average service delay No DNX 1257 9796 251.588sec 256.930sec No DNX 1265 12222 789.429sec 789.000sec Use DNX 1343 13841 0.365sec 0.644sec

• Monitoring based on the logger Analysis

Log Log : records relating to activities occurring on system.

– The reliability of the hardware – The stability of the service – The availability of the system

• logger-analysis requirements

– Large Scale & Scalability – Real-time information display and alarm – Convenient query – Flexible configuration

• Provides a novel monitoring based on log analysis

Logger-analysis Monitoring

Qingbao Hu/CC/IHEP 2016/3/24 - 13

• Log data store

e & se search

• Elasti

ticsearch: csearch: Search h & Analyze Da Data in Re Real Time

– Distributed, scalable, and highly available – Real-time search and analytics capabilities – RESTful API

Logger-analysis

Qingbao Hu/CC/IHEP 2016/3/24 - 14

• Flume

– Distributed, reliable, and available service for

efficiently collecting, aggregating, and moving large amounts of log data.

– Simple and flexible architecture based

• n streaming data flows.
• Logstash

tash : Process ss Any Da Data, , From m Any Source

– Centralize data processing of all types – Normalize varying schema and formats – Quickly extend to custom log formats – Easily add plugins for custom data sources

Real-time Log Collection

Qingbao Hu/CC/IHEP 2016/3/24 - 15

• Thr

hree ee mo mode dels ls (th throu rough ghpu put) t)

• 1.Logst

gstash sh & Re & Redis is & & El Elasticsearch icsearch (lo low) w)

• 2.Logst

gstash sh & Ela Elasticsearch csearch (middle) e)

• 3.Flu

lume me & Ela Elasti ticse csearch arch (high) h)

Real-time Log Collection

• Flexibility
• Scalability
• Real-time

Logger-analysis

Qingbao Hu/CC/IHEP 2016/3/24 - 17

• No

No log missed

– logpath + inode + offset – Tail + awk

tmp file record the file inode and

the file offset info to guarantee the continuity of the log data collection when collect service crash.

• Logs from

m vario ious s servers ers can be coll llecte cted

– Log format defined by dedicate configure file by

administrator

Logger-collect client

Qingbao Hu/CC/IHEP 2016/3/24 - 18

Collect configure file

• Flume

e multi-agen gent t fan-in n flow w model

• Pre-pro

processing cessing log & Up Upload data real-ti time me

Logger-collect client

Qingbao Hu/CC/IHEP 2016/3/24 - 19

log_all log_all log_all

• Flume

e multi-agen gent t fan-out ut flow w model

• Separate

rate diffe ferent rent service ce log data and store re in different erent indexes. s.

Logger-collect server

Qingbao Hu/CC/IHEP 2016/3/24 - 20

collector

Function developed based on ES API

• use keywo

word rds s to locate e the service ce failure re time

• Re

Real-ti time me email alerts ts

• Di

Display y the health status tus of the wh whole cluster ter

Qingbao Hu/CC/IHEP 2016/3/24 - 21

• The number

r of monitored

• red nodes > 2,000

000

• The amount

t of logs collecte cted d per day ~ 20 20M entries es

• The interval

val betwe ween n the log produce ced d and store red d < 40 40 s

• Maximum

mum throug ughp hput ut reach 20 20,000 0 records ds per second

Log-analysis deployed at IHEP

Qingbao Hu/CC/IHEP 2016/3/24 - 22

• Re

Regular r expressi ssion n of log format mat wi will be support rted for r more re detailed d fields

• Archive

hive log data to HD HDFS

• Off

fflin line e log mining based on Ha Hadoop or Storm rm

Next step

Qingbao Hu/CC/IHEP 2016/3/24 - 23

Summary

• Ganglia

lia and Icinga ga guarante ntee e the stabili ility y of the IHE HEP P cluster ter. .

• Log

Log-an anal alys ysis is provid ides es a novel l monitor

• rin

ing. g.

• Log mining will be done next.

Qingbao Hu/CC/IHEP 2016/3/24 - 24

Qingbao Hu/CC/IHEP 2016/3/24 - 25

Thank you! Any Question?