The Cluster Monitoring System of IHEP Qingbao Hu huqb@ihep.ac.cn - PowerPoint PPT Presentation

The Cluster Monitoring System of IHEP Qingbao Hu huqb@ihep.ac.cn Computing Center, Institute of High Energy Physics, Chinese Academy of Sciences International Symposium on Grids and Clouds (ISGC) 2016

Content • Overview of IHEP’s Monitor orin ing g Sy System • Op Optim imiza ization on of the monito torin ring g tools • Logger er-an anal alysis ysis Monitor orin ing • Su Summary Qingbao Hu/CC/IHEP 2016/3/24 - 2

Status of IHEP Cluster • ~ ~ 1,122 2 work nodes  ~ ~ 13 13,500 0 CPU cores • ~ ~ 5PB PB disk stora rage  Lustre, tre, gLuste ter, r, openAF AFS, S, etc. ~ 5PB PB tape stora rage ge • ~  Two IBM 3584 3584 tape librari aries, es, LTO4 4 tape  Modifie fied d CERN CASTOR R 1.7 Cluster built with blades Tape libraries Qingbao Hu/CC/IHEP 2016/3/24 - 3

Monitoring requirements  A large number of hardware and software resources  Cooperated in complex ways – Large Scale ( > 2,000 nodes ) – Heterogeneous device resources – Good Scalability – Real-time information display and alarm – Combination of active detection service and passive information receiving. – Auto recovery of failed services. Qingbao Hu/CC/IHEP 2016/3/24 - 4

Monitoring System Overview  System tem overvi view Monitoring system of IHEP Logger Analysis Ganglia Icinga Collecting more Recording the Monitoring the comprehensive performance of status of cluster data & providing different devices and an overview of the resource groups services whole cluster health status Qingbao Hu/CC/IHEP 2016/3/24 - 5

Ganglia  Monitoring the health of the cluster – System load – CPU utilization – Network bandwidth and traffic – Memory usage  Usage – Records history status of the cluster – Helps system manager to fix problem Qingbao Hu/CC/IHEP 2016/3/24 - 6

Ganglia of IHEP  The bottleneck of Ganglia – High frequency: Collect 20 metrics from each monitored node every 15 seconds. – Pool scalability: Large number of nodes cause a large amount of metrics data, which pulls up the server’s peak iowait and slows down the monitoring service.  Workaround: – Create a ram disk on the Ganglia server to save the RRDs data. – Improves the IO performance of the server disk Qingbao Hu/CC/IHEP 2016/3/24 - 7

Ganglia of IHEP – IOwait < 1% Different clusters The status of bws1 farm Qingbao Hu/CC/IHEP 2016/3/24 - 8

Icinga  Created as a fork of the Nagios – Plug-in design – Active check of the service – Flexible configuration by NagiosQL  Usage – Hardware (CPU load, disk usage, etc.) – Network connectivity (HTTP, POP3, ping, etc.) – Computing services on work nodes – Distributed file system services … Qingbao Hu/CC/IHEP 2016/3/24 - 9

Icinga of IHEP  Polling agents we developed – More services monitored – Some crashed service faults can be recovered automatically – Critical errors are alarmed to system manager via both email and SMS Qingbao Hu/CC/IHEP 2016/3/24 - 10

Icinga of IHEP  The bottleneck of Icinga – Single collector node . – Vast amounts of the service detections increases the server load, which reduces the efficiency of the polling. – Many detection results are delayed.  Workarounds: – Distributed Nagios eXecutor. (DNX)+ Icinga Sever » A modular extension of Nagios » DNX Worker requests jobs from the Icinga (Scheduling) Server » DNX Worker executes the plug-in agents and return the results to Icinga server. – Balance the load of servers via distribution – Decrease the latency of the polling Qingbao Hu/CC/IHEP 2016/3/24 - 11

Icinga in IHEP scale of scale The average The average Monitoring of Monitorin host delay service hosts g service delay No DNX 1257 9796 251.588sec 256.930sec No DNX 1265 12222 789.429sec 789.000sec Use DNX 1343 13841 0.365sec 0.644sec Qingbao Hu/CC/IHEP 2016/3/24 - 12

Logger-analysis Monitoring  Monitoring based on the logger Analysis Log Log : records relating to activities occurring on system. – The reliability of the hardware – The stability of the service – The availability of the system  logger-analysis requirements – Large Scale & Scalability – Real-time information display and alarm – Convenient query – Flexible configuration  Provides a novel monitoring based on log analysis Qingbao Hu/CC/IHEP 2016/3/24 - 13

Logger-analysis  Log data store e & se search  Elasti ticsearch: csearch: Search h & Analyze Da Data in Re Real Time – Distributed, scalable, and highly available – Real-time search and analytics capabilities – RESTful API Qingbao Hu/CC/IHEP 2016/3/24 - 14

Real-time Log Collection  Flume – Distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. – Simple and flexible architecture based on streaming data flows.  Logstash tash : Process ss Any Da Data, , From m Any Source – Centralize data processing of all types – Normalize varying schema and formats – Quickly extend to custom log formats – Easily add plugins for custom data sources Qingbao Hu/CC/IHEP 2016/3/24 - 15

Real-time Log Collection • Thr hree ee mo mode dels ls (th throu rough ghpu put) t) • 1.Logst gstash sh & Re & Redis is & & El Elasticsearch icsearch (lo low) w) • 2.Logst gstash sh & Ela Elasticsearch csearch (middle) e) • 3.Flu lume me & Ela Elasti ticse csearch arch (high) h)

Logger-analysis  Flexibility  Scalability  Real-time Qingbao Hu/CC/IHEP 2016/3/24 - 17

Logger-collect client Collect configure file  No No log missed – logpath + inode + offset – Tail + awk tmp file record the file inode and the file offset info to guarantee the continuity of the log data collection when collect service crash.  Logs from m vario ious s servers ers can be coll llecte cted – Log format defined by dedicate configure file by administrator Qingbao Hu/CC/IHEP 2016/3/24 - 18

Logger-collect client  Flume e multi-agen gent t fan-in n flow w model  Pre-pro processing cessing log & Up Upload data real-ti time me log_all log_all log_all Qingbao Hu/CC/IHEP 2016/3/24 - 19

Logger-collect server  Flume e multi-agen gent t fan-out ut flow w model  Separate rate diffe ferent rent service ce log data and store re in different erent indexes. s. collector Qingbao Hu/CC/IHEP 2016/3/24 - 20

Function developed based on ES API  use keywo word rds s to locate e the service ce failure re time  Re Real-ti time me email alerts ts  Di Display y the health status tus of the wh whole cluster ter Qingbao Hu/CC/IHEP 2016/3/24 - 21

Log-analysis deployed at IHEP  The number r of monitored ored nodes > 2,000 000  The amount t of logs collecte cted d per day ~ 20 20M entries es  The interval val betwe ween n the log produce ced d and store red d < 40 40 s  Maximum mum throug ughp hput ut reach 20 20,000 0 records ds per second Qingbao Hu/CC/IHEP 2016/3/24 - 22

Next step  Re Regular r expressi ssion n of log format mat wi will be support rted for r more re detailed d fields  Archive hive log data to HD HDFS  Off fflin line e log mining based on Ha Hadoop or Storm rm Qingbao Hu/CC/IHEP 2016/3/24 - 23

Summary • Ganglia lia and Icinga ga guarante ntee e the stabili ility y of the IHE HEP P cluster ter. . • Log Log-an anal alys ysis is provid ides es a novel l monitor orin ing. g. • Log mining will be done next. Qingbao Hu/CC/IHEP 2016/3/24 - 24

Thank you! Any Question? Qingbao Hu/CC/IHEP 2016/3/24 - 25