ISGC - - Academia Academia Sinica Sinica ISGC 28 July 2004 28 July 2004 Iosif Legrand Legrand Iosif California Institute of Technology July 2004 Iosif Legrand
Monitoring Services � An essential part of managing a global Data Grid is a monitoring system that is able to monitor and track the many site facilities, networks, and the many tasks in progress, in real time. � System information for nodes and clusters � Network information Wan and LAN � Application monitoring � The monitoring information gathered also is essential for developing the required higher level services, and components of the Grid system that provide decision support, and eventually some degree of automated decisions, to help maintain and optimize workflow through the Grid. � The MonALISA system is designed as an ensemble of autonomous multi-threaded, self-describing agent-based subsystems which are registered as dynamic services, and are able to collaborate and cooperate in performing a wide range of monitoring tasks and decisions in large scale distributed applications. July 2004 Iosif Legrand
MonALISA is A Dynamic, Distributed Service Architecture � The MonALISA system is designed as an ensemble of autonomous multi-threaded, self-describing agent-based systems � The agent-based dynamic systems are able to carry out a wide range of monitoring tasks in the LHC Data Grid (and other Grids). Service Lookup Proxy CLIENT Any well suited protocol Service Lookup Proxy for the application Service Code Mobility Paradigm Dynamic Loading of Service � Remote Services Proxy == RMI Stub � Mobile Agents Proxy == Entire Service � “Smart Proxies” Proxy adjusts to the client An agent-based architecture provides the ability to invest the system with increasing degrees of intelligence; to reduce complexity and make global systems (Grids and networks) manageable in real time July 2004 Iosif Legrand
The Key MonALISA MonALISA Features for a Reliable and Features for a Reliable and The Key Scalable Monitoring and Management System Scalable Monitoring and Management System � � MonALISA is able to dynamically is able to dynamically MonALISA JAR register and and discover discover all the other all the other register Web Services Services Sever � � It is based on a multi multi- -threaded threaded It is based on a Code Loading Publish the services engine for global scalability services engine for global scalability Proxy (mobile Code) � � The services are self describing The services are self describing and and provide loadable proxies CLIEN provide loadable proxies Lookup � � Automatic & secure code update T Automatic & secure code update Service � � Dynamic configuration for services. Dynamic configuration for services. Secure Admin interface. Secure Admin interface. � � Active filter agents to process the to process the Active filter agents Service ID PROXY data and provided dedicated / data and provided dedicated / customized information to other customized information to other services or clients. services or clients. Lookup Register � � Mobile Agents for decision support for decision support Mobile Agents with ID Service and global optimization. and global optimization. MonALISA � � Dynamic proxies and WSDL & WAP and WSDL & WAP Dynamic proxies Service pages for services. pages for services. The Lease Protocol Fully Distributed System with no Single Point of Failure July 2004 Iosif Legrand
Monitoring: Data Collection Monitoring: Data Collection Dynamic PULL Other tools Thread Pool (Ganglia, MRT…) SNMP get & walk rsh | ssh remote scripts Configuration Control Configuration Control Farm End-To-End measurements Monitor Trap Listener PUSH WEB Server snmp trap Dynamic loading of modules or agents Trap Agent (ucd – snmp) perl July 2004 Iosif Legrand
Service Monitor UNIT & Data Handling Service Monitor UNIT & Data Handling Monitor Lookup Lookup WEB Service Data Stores Service Client Service Discovery Registration (other service) Web client WSDL Data Cache SOAP Service & DB data McKoi DB MySQL Client Predicates & Agents (other service) Farm Java Other tools Monitor Configuration Control (SSL) Configuration Control (SSL) UDP MySQL Predicates & Agents User defined loadable MDS Modules to write /sent data July 2004 Iosif Legrand
Registration / Discovery / Remote Registration / Discovery / Remote Notification Notification Registration Discovery MonALISA Service Client Lookup (other service) Service Services Proxy Multiplexer MonALISA Service Data Data Filters & Agents Filters & Agents Services Proxy Multiplexer Lookup MonALISA Service Client Service (other service) July 2004 Iosif Legrand
Secure – – Automatic Update Mechanism Automatic Update Mechanism Secure for Services, Clients & Embedded Applications for Services, Clients & Embedded Applications Lookup Service Key store Web Server Discovery MonaLisa download download Service Sign Distribution Update Signal SSL Update Signal SSL � All running services are update using the download download discovery mechanism Admin Client � At startup each service Update Signal SSL Update Signal SSL check if it an update is Priv. Key Key store done at a set of Web MonaLisa Servers Service Lookup � Clients use the Web Service Start mechanism EMBEDDED APPLICATIONS EMBEDDED APPLICATIONS Monitor, Control Execution, Update Monitor, Control Execution, Update July 2004 Iosif Legrand
Pseudo – – Clients & Dedicated Repositories Clients & Dedicated Repositories Pseudo Lookup Service MySQL Discovery MonaLisa IDB Service WAP F F i i l l t t e e r r A A g g e e n n t t s s / / D D a a t t a a TOMCAT JSP/servelts WEB Pseudo Client a a t a t a D MySQL D / / s s t t n n e e g g A A r r MySQL e e t l t i l F i F MonaLisa IDB Service Lookup Service July 2004 Iosif Legrand
Data Collection and Interfacing Data Collection and Interfacing with Other Tools with Other Tools MonALISA is interfaced with many monitoring tools and is is interfaced with many monitoring tools and is MonALISA capable to collect information from different applications: capable to collect information from different applications: Computing Nodes / Farms (system information , network Computing Nodes / Farms (system information , network traffic… … ) ) traffic � SNMP, Ganglia, dedicated scripts � SNMP, Ganglia, dedicated scripts Routers , Switches Routers , Switches � SNMP, MRTG, WS, very soon � SNMP, MRTG, WS, very soon NetFlow NetFlow End to End Network performance End to End Network performance � IPERF, Pipes, � IPERF, Pipes, Abing Abing, , ABping ABping … … Batch Queuing Systems Batch Queuing Systems � LSF, PBS, Condor, NQS, Grid Job Manager � LSF, PBS, Condor, NQS, Grid Job Manager Applications Applications � � ORCA, ORCA, GridFTP GridFTP, TMDB, Proof, VRVS, Apache, RRD , TMDB, Proof, VRVS, Apache, RRD … … July 2004 Iosif Legrand
MonALISA DEMO DEMO MonALISA July 2004 Iosif Legrand
Monitoring And Controlling Optical Switches Monitoring And Controlling Optical Switches External Optical Path Application External External Application Application Agent-to-Agent Agent-to-Agent Application Application Distributed Agent Proxies Communication Communication Monitoring Monitoring MonALISA MonALISA A Agents Agents g monitoring Agents Control e MonALISA monitoring n Real-time t Real-time s C A Agents o g monitoring n e t n r Real-time o t s l C o n t r o l h t a P l a Optical c t i p O Switch Optical Optical Path C o Switch n n e c t i o Optical M n a p Switch July 2004 Iosif Legrand
Communities using MonALISA MonALISA Communities using � Grid3 ABI LENE ABI LENE ~20 sites in US and 1 Korea � CMS-US sites It has been used for � CMS – DC04 Demonstrations at: - - We collected ~ 50 million CMS- CMS -DC04 DC04 monitoring records from the 6 T1 � CDF GRI D3 GRI D3 � SC2003 � D0 SAR � ABILENE backbone � GLORIAD � Telecom 2003 VRVS VRVS � STAR � ALICE � WSIS 2003 ALICE � VRVS System � RoEduNET backbone � INTERNET2 PIPES July 2004 Iosif Legrand
Recommend
More recommend
