Storage Information Services Ted Hesselroth Fermilab Abhishek Singh - - PowerPoint PPT Presentation

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Storage Information Services

Ted Hesselroth Fermilab

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Software Domains in OSG

 Used, served, or provided by OSG

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Information Services Axes

 Independently implemented  Similar architecture

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Information Services - Aggregation Stack

Aggregation Services

CEMon GIP

Grid Resources Collection Services Schema

Glue ReSS Information Gatherer Compute Element Storage Element BDII WSRF - Resource Properties Gratia Syslog-ng

Lower Part of Complete Stack

Worker Node Gratia probe DB schema Log format internal

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Information Collection - GIP

 Collects static and dynamic information

 Static: configuration files  Dynamic: accesses components of resources

 Supports Glue Schema  Invoked by CEMon

 CEMon forwards info to aggregators

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Information Collection – built-in

dCache Information Service

 Runs as dCache cell  Collects information via cell messages

 Control over collection scheduling  Replies are processed asynchronously

 Aggregates in memory as “tree”

 Can compute and store secondary information

 Serves serialized output through httpd or admin interface

 Whole or sub-tree  XML or flat files

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

dCache Information Service Graphic: Paul Millar

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Information Collection – spec’d

Services to respond to queries on configuration and state

 Implementation

 Adapters to helper service for handling queries

 Annotations?

 Common format for queries and responses

Software design should specify what information will be available. Glue Schema Software

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Information Services - Access Stack

Repositories Data Services Portals

schedd LDAP Matchmaker

Upper Part of Complete Stack Applications Utilities Reports

MyOSG

Dashboards Interactivity

ReSS filesystem DB Discovery: ReSS OSGMM Monitoring: RSV Accounting: Gratia Diagnostics: Syslog-ng condor_status ldapsearch xpathsearch OIM Gratia Web Pages

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Discovery Axis - Subsystem Software

 OSGMM adds info on site availability

OSGMM central GIP CEMon BDII ReSS Resource schedd Site availability info Aggregator

Resource Description classad based on GlueSchema

Aggregator

LDAP directory based on GlueSchema

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Storage Information Usage – ReSS

Submission using Job Description Language

universe=globus globusscheduler = $$(GlueCEInfoContactString) requirements = TARGET. GlueCEAccessControlBaseRule == “VO:Dzero” executable = srmcp arguments = -2 $$(GlueSEControlProtocolEndpoint)/$$(VOInfoPath)/test … queue …

Resource Description Job Execution

Job Description (snippet) using classads

ReSS schedd

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Command Line LDAP search using Xpath

OSG Storage Extensions Project

]$ bin/xpathldapsearch \ //GlueSE/GlueSA/GlueVOInfo[@GlueVOInfoAccessControlBaseRule='dzero']/\ ../../GlueSEControlProtocol/@GlueSEControlProtocolEndpoint httpg://sigmorgh.hpcc.ttu.edu:49443/srm/v2/server. httpg://se01.cmsaf.mit.edu:8443/srm/managerv2 httpg://se01.cmsaf.mit.edu:8443/srm/managerv1 httpg://cit-se.ultralight.org:8443/srm/managerv1 httpg://cit-se.ultralight.org:8443/srm/managerv2 httpg://cit-se2.ultralight.org:8443/srm/v2/server httpg://cit-se2.ultralight.org:8443/srm/v2/server httpg://fndca1.fnal.gov:8443/srm/managerv2 httpg://fndca1.fnal.gov:8443/srm/managerv1 httpg://fndca1.fnal.gov:8443/srm/managerv2 httpg://fndca1.fnal.gov:8443/srm/managerv1 httpg://gk03.atlas-swt2.org:8443/srm/managerv2 httpg://gk03.atlas-swt2.org:8443/srm/managerv1 httpg://srm.unl.edu:8443/srm/managerv1 httpg://srm.unl.edu:8443/srm/managerv2 httpg://dcsrm.usatlas.bnl.gov:8443/srm/managerv2 …

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Storage Information Usage - Discovery

transform,

• penldap

http interface SE discovery

• 1. VO or public use
• 2. Control Protocol
• 3. Access Protocol
• 4. Available Space

OK Sites Test Submission Scripts Schedd CondorG BDII proxy

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Data Integration

Data Transformation

XAware MDS4

Between Aggregation and Access

OGSA-DAI

 OGSA-DAI  MDS4  XAware  Single subsystem for top half of stack  Easier integration with other systems and technologies

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Other Info Integration Software – XAware

Composite Data Services Layer

• Aggregation
• Data Chaining
• Inbound XML
• Decomposition
• Synchronization
• Conditional Logic
• Transactions
• Error Handling
• Extension Points
• Security

Graphic: XAware Inc.

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Other Info Integration Software – XAware



XAware Designer – Eclipse plug-in



Visual drag-and-drop, wizard-based IDE that speeds the development of XAware-powered applications



XAware Engine



High-performance, scalable, dynamic data integration and data services execution engine



J2EE, Servlet, Batch, or Embedded



Adapters



Provide connectivity to requested data sources and systems; SQL, FTP, Email, SNMP, Text, RMI, Cobol Copybook, File, XSL, LDAP, Java, EJB, JMS, MQ, others…



Connectors



Provides synchronous and asynchronous connectivity from client applications to XAware-delivered services via SOAP, JMS, EJB, HTTP, HTTPS, and Java API

Graphic: XAware Inc.

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Thank you.

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

(XML in XMLDB)

State Representation Data Collection

(Probes) (RESTful POST with XML) (XSL) (ISO Schematron) (CIM Meta schema) (Glue, CIM, custom) (Schema adapter, Update)

v

Modeling Grammar Schema Data Operations Message Protocols Trigger Expression Condition Evaluation

Diagnostics Tool Architecture

Ted Hesselroth USCMS T3 Meeting 2008-03-03

Abhishek Singh Rana and Frank Wuerthwein UC San Diego

Ted Hesselroth OSG Storage Forum 2009-07-01

Possible Future for Information Services and Storage

How would we use an integrated frame work to achieve…

 Catalog Integration  Replication  Federation  Grid Buffering