iRODS Consortium 2013 Reception 5:00 pm Welcome and Overview, - - PowerPoint PPT Presentation

iRODS Consortium 2013 Reception

5:00 pm Welcome and Overview, Charles Schmitt 5:05 pm Introduction of new Executive Director, Brand Fortner 5:10 pm Update on iRODS Consortium, Reagan Moore 5:20 pm Technology Roadmap, Jason Coposky 5:40 pm Support Model – Expectations, Charles Schmitt 5:50 pm Features – Requests and Bug Process, Terrell Russell 6:00 pm Future Directions, Reagan Moore 6:10 pm Presentation, Use Case, Chris Smith, Distributed Bio 6:20 pm Presentation, Use Case, Pete Clapham, Sanger Institute 6:30 pm Presentation, Use Case, from Alan Hall, NCDC/NOAA 6:40 pm Wrap Up – Invitation to Thursday Tutorials, Charles Schmitt and Brand Fortner

3

Thank you

Chris Smith, Distributed Bio Pete Clapham, Sanger Institute iRODS Consortium Executive Committee and Advisory Committee Members DICE & RENCI staff and management that have gotten us here

iRODS Users - examples

—

Federal Users

• National Aeronautics and Space Administration
• National Oceanic and Atmospheric Administration
• National Archives and Records Adminstration
• USGS

—

Non-profit/Institutional Users

• Broad Institute
• International Neuroinformatics Coordinating Facilities
• Wellcome Trust Sanger Institute

—

Commercial Users

• DOW Chemical
• Bejing Genome Institute

—

Resellers/Redeployers

• Distributed Bio
• Computer Sciences Corporation (CSC)

—

Academic Users

• Too many too list

4

iRODS Users - examples

— Proven at scale:

• iPlant - 10k users
• French National Institute for Nuclear Physics and Plasma Physics – over 6 PB
• Wellcome Trust Sanger Institute – over 8 Pb
• Australian Research Collaboration Service - 512 storage resources
• NASA Center for Climate Simulations - 300 million attributes
• Cinegrid – sites across Japan-US-Europe

— Solid foundation:

• SRB: initial product (developed by DICE Group, owned by General Atomics) in 1997
• iRODS: rewrite of SRB by DICE Group in 2006; currently on version 3.3
• Enterprise iRODS: mission critical distribution co-developed by RENCI and DICE in 2012

— Support:

• Community of developers from groups worldwide
• Independent groups offering consulting and support and development
• iRODS Consortium offering formal support, training, involvement, and development help

5

iRODS Users - downloads

— As open-source software, we cannot fully track usage, only downloads: — 3.0 and 3.1 release:

• 1004 unique IP addresses combined (currently don’t have independent

counts)

— 3.2 release:

• 1202 unique IP addresses
• About 15% are confirmed commercial sites
• Another 15% are non-profit non-universities

6

iRODS Consortium

— Provide:

• Open source iRODS release
• Single code source
• Sustainable development
• Binary distribution

— iRODS data grid

• Organize data, information, knowledge
• Enable collaboration, publication, preservation
• Provide interoperability mechanisms
• Pluggable architecture for flexibility

Current iRODS Development

— DICE Center

• Features
• Sheau-Yen Chen
• Mike Conway
• Reagan Moore
• Arcot Rajasekar
• Wayne Schroeder
• Hao Xu

— RENCI

• Pluggable architecture
• Jason Coposky
• Zoey Greer
• Harry Johnson
• Terrell Russell
• Antoine de Torcy

Parallel Software Development

iRODS 4.0 (March 2014)

— Combine features from

• iRODS 3.3 (July 17, 2013)

– Hadoop Distributed File System driver – PAM/LDAP authentication – Workflow structured objects – Rule language extensions – Jargon Java I/O library – netCDF and OpeNDAP support

• E-iRODS 3.0.1 (October 31, 2013)

– Pluggable security (authentication, identity) – Pluggable micro-services – Pluggable storage drivers (hierarchical resources) – Pluggable network drivers – TLS security – Binary distribution – one-click installation

iRODS 4.0

— Hardened and tested code — One click installation (27 seconds) — Pluggable architecture — Federated security

iRODS Consortium

— iRODS Consortium

• Brand Fortner
• Charles Schmitt
• Arcot Rajasekar
• Reagan Moore
• Wayne Schroeder
• Jason Coposky
• Harry Johnson
• Terrell Russell
• Antoine de Torcy
• Leesa Brieger
• Michael Shoffner*
• Lisa Stillwell*

Joint development of combined release, iRODS version 4.0 by March 15, 2014

DataNet Federation Consortium

— Migrating to iRODS 4.0

– Spring 2014

• Reagan Moore
• Arcot Rajasekar
• Wayne Schroeder
• Sheau-Yen Chen
• Hao Xu
• Mike Conway
• Lisa Stillwell
• Charles Schmitt
• Michael Shoffner

Features: Workflow registration Workflow re-execution Workflow provenance OpeNDAP protocol NetCDF manipulation Time series archiving Soft links to external repositories Mediawiki integration VIVO integration AMQP messaging integration HIVE linked data idrop-Web interface In-Common logon Dataverse integration Antelope integration Openflow controller integration

iRODS Consortium Members

— RENCI — DICE Center — Max Planck Society — Data Direct Networks

Consortium Activities

— Generate a development roadmap — Prioritize development tasks — Standard iRODS release — Service level support — Matchmaking between support providers

and iRODS users

• Tier 1
• Tier 2
• Tier 3

iRODS Consortium Development Team - Supercomputing 2013

3.0 3.1 3.2 3.3 4.0 3.0 4.0+ 3.0.1 iRODS Community iRODS Enterprise iRODS Consortium 11/13 03/13 07/13 10/12 03/12 09/11 03/14 Plus, independent plugin releases

iRODS Consortium Roadmap Technical

State of the Union: E-iRODS 3.0.1 Features

— Feature compatible with 3.0 Community,

including bug fixes up to 3.3

— Up to date rule engine — Full SSL support with parallel transfer — Live hierarchy manipulation—move and rename

resources in a hierarchy with existing data on the resources

— Added Rebalancing as an operation—currently

implemented for replication resource

iRODS Consortium Development Team - Supercomputing 2013

State of the Union: E-iRODS Continuous Testing

— Code coverage at 54% for the Agent, up

from 28% originally

— Plugin coverage on average at 70%, up to

93%

— Continuously tested across 6 platforms — Full feature testing against 13 individual

Resource Hierarchies

— Topological Testing — Federation Testing, including 3.3

iRODS Consortium Development Team - Supercomputing 2013

State of the Union: E-iRODS Plugin Interfaces

Current (3.0.1):

• Microservices
• Resources
• Network
• Authentication

Coming Soon (4.0):

• Database

iRODS Consortium Development Team - Supercomputing 2013

State of the Union: Current Plugins

— Resource plugins: Replication, Round Robin,

Random, Passthru, Compound, UnivMSS, MockArchive, UFS, Non-blocking, and Structured File Object

— Network plugins: TCP and SSL — Authentication plugins: PAM, OSAuth, and

Native

iRODS Consortium Development Team - Supercomputing 2013

iRODS 4.0 ( March 2014 )

— Full merge between E-iRODS and up-to-date

Community code

— Repository migration to GitHub—transparent

development process

— Tickets, Workflows, approximately 50 updates

not already included (e.g., filesystem metadata collection, storage admin role, sha1 )

— Comprehensive feature testing during merge

Architectural Updates:

— Database plugins—Postgres, MySQL, Oracle

iRODS Consortium Development Team - Supercomputing 2013

Independently Released Plugins

— Released as separate packages — Authentication (13Q4)

• Kerberos
• GSI

— Resource (14Q1)

• S3
• HDFS
• Load Balancing
• WOS (streaming)*
• HPSS 7.3 and 7.4 (expanded platforms)*

iRODS Consortium Development Team - Supercomputing 2013

E-iRODS Plugins

— E-iRODS 3.0.1

—

Resource plugins:

• Replication
• Round Robin
• Random
• Compound
• UnivMSS
• Unix File System
• Non-blocking

—

Network plugins:

• TCP
• SSL

—

Authentication plugins:

• PAM
• OSAuth
• Native

— Independently Packaged Releases

— Resource Plugins:

• S3
• HDFS
• HPSS 7.3 and 7.4
• WOS

— Authentication:

• Kerberos
• GSI

State of the Union: E-iRODS Plugin Interfaces

— Reliability – Creates more easily tested software — Flexibility – Convert compile time options into run-time configuration — Stability – Create a hardened core and externalize rapid development — Community – open the development up to a large audience

E-iRODS 3.0.1: Microservices Resources Network Authentication iRODS 4.0: Database

iRODS Development Areas

— RESTful Interface—use the consortium as a

vehicle for standardization

— Refactoring—continue migrating toward more

abstract interfaces ( e.g., special collections )

— Dynamic API—extend iRODS API as any

• ther plugin interface

— Plugin Registry—keep track of plugins and

state in the catalog

— Plugin dependency model—describe

interdependencies between plugins, ship packages of plugins together as a collective feature

iRODS Consortium Development Team - Supercomputing 2013

Support Model

28

• Feedback thus far is that support for iRODS 3.0-3.3 has

been very good.

• “I submitted my first bug report on Saturday, the fix was

available on Monday when I got back to my office”.

• This support has been due to:
• Good faith efforts of the DICE team: Wayne Schroeder,

Mike Conway, Hao Xu, and others

• Funding from NSF SDCI grant
• The goal of the Consortium is to keep that level as good, if

not better as iRODS transitions off of NSF SDCI funding

• Negative and positive feedback will be greatly appreciated

Support Model – what will change?

29

• iRODS 3.0 through 3.3 will be supported by the

Consortium starting in early 2014.

• Wayne Schroeder at 50% time starting early 2014
• E-iRODS 3.0 and 3.0.1 will continue to be supported by

the E-iRODS team

• iRODS 4.0 will be supported by the iRODS Consortium

team

• EiRODS + iRODS team

Support Model – what will change?

30

• Continue good faith efforts
• plus community contributions
• Guaranteed support levels based on membership

class

• Support will be prioritized starting early 2014
• Higher membership classes have higher

priority

• Remaining guaranteed support has higher

priority

• Critical bugs will have priority for members

Support Model

31

• Additional support and consulting is available on a

contract basis

• Consortium will partner with 3rd party groups

for this

• Support has traditionally been tier 1, 2, and 3
• But, best faith effort and not 24/7
• We are investigating relationships with vendors for

tier 1 and 2 with guaranteed service response times

iRODS integrated Rule Oriented Data System Future Development

Reagan W. Moore University of North Carolina at Chapel Hill

Future Directions

— iRODS Consortium

• Focus on robust, secure, scalable middleware
• Enable interoperability across evolving

technologies

— Data grids in general

• Recognize emergence of knowledge

management goals

• Recognize emergence of software defined

networks

Feedback from User Communities

— Scalability

• Improve ability to apply file-level access controls
• Improve management of server side processing

— Searching

• Integrate with external indexing technology
• Support linked-data with HIVE vocabularies

— Network optimization

• Policy-based network routing
• Demonstration at SC’13, iRODS integration with

OpenFlow routing control

Outreach

— Research Data Alliance

• Practical Policy Working Group

– Share data management policies

• Data Foundation and Terminology Working

Group

– Concepts underlying data management and data access

— Documentation

• iRODS Micro-service Workbook 4.0

– Document all micro-services, policy enforcement points, state information

iRODS User Group Meeting

— Joint meeting with iRODS Consortium

• Tsai Auditorium (S010)
• Institute for Quantitative Social Science
• Harvard University
• 1737 Cambridge Street, Cambridge MA
• June 18-19, 2014
• Requesting presentations on user applications
• http://irods-consortium.org

iRODS ¡Consor,um ¡ Collec&on ¡

Defines ¡

A-ribute ¡

Has ¡ Has ¡

Digital ¡ Object ¡

Has ¡ Has ¡

Collec,on ¡ ¡ ¡ ¡ ¡

Purpose ¡

Defines ¡

Policy ¡ Property ¡

Defines ¡ Controls ¡

Updates ¡

Persistent ¡ ¡ State ¡ Informa,on ¡

Policy ¡Concept ¡Graph ¡

Purpose ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡

Procedure ¡

Completeness ¡ Correctness ¡ Isa ¡ Consensus ¡ Consistency ¡ HasFeature ¡ HasFeature ¡ HasFeature ¡ Integrity ¡ Isa ¡ Authen&city ¡ Isa ¡ Access ¡ control ¡ HasFeature ¡

Property ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡

Replica&on ¡ Policy ¡ Checksum ¡ Policy ¡ Quota ¡ Policy ¡ Data ¡Type ¡ Policy ¡ Isa ¡ Isa ¡ Isa ¡ Isa ¡

Policy ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡

Workflow ¡

Isa ¡

Func&on ¡

Chains ¡

Opera&on ¡

Isa ¡

Updates ¡ GetUserACL ¡ SetDataType ¡ SetQuota ¡ DataObjRepl ¡ SysChksumDataObj ¡ Isa ¡ Isa ¡ Isa ¡ Isa ¡ Isa ¡

Procedure ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡

Isa ¡

DATA_ID ¡ DATA_REPL_NUM ¡ DATA_CHECKSUM ¡

Isa ¡ Isa ¡ Isa ¡

Persistent ¡State ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡

Client ¡Ac&on ¡ Periodic ¡ Assessment ¡ Criteria ¡ Policy ¡ Policy ¡ Enforcement ¡ Point ¡

Invokes ¡ Has ¡ SubType ¡

Policy ¡Enforcement ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡ ¡

iRODS ¡Consor,um ¡

Policy-­‑based ¡Data ¡Management ¡– ¡Implementa&on ¡in ¡iRODS ¡

Collec&on ¡ Purpose ¡ ¡(5 ¡main ¡types) ¡

Completeness ¡ Correctness ¡ Consensus ¡ Defines ¡ Consistency ¡

A-ribute ¡

HasFeature ¡ HasFeature ¡ HasFeature ¡ Has ¡ Defines ¡

Policy ¡ ¡ (11 ¡default) ¡

Has ¡

Property ¡ ¡ ¡ (7 ¡default) ¡

Defines ¡

Procedure ¡ (11 ¡default) ¡

Controls ¡

Updates ¡

Clients ¡(50) ¡ Periodic ¡ Assessment ¡ Criteria ¡ Policy ¡ Policy ¡ Enforcement ¡ Points ¡(70) ¡ Workflow ¡

Invokes ¡ Has ¡ SubType ¡ Isa ¡

Micro-­‑service ¡ (317) ¡

Chains ¡

Opera&on ¡

Isa ¡

Persistent ¡ ¡ State ¡ Informa,on ¡ (338) ¡

Isa ¡

Digital ¡ Object ¡

Updates ¡ Has ¡ Has ¡ Replica&on ¡ Policy ¡ Checksum ¡ Policy ¡ Quota ¡ Policy ¡ Data ¡Type ¡ Policy ¡ Isa ¡ Isa ¡ Integrity ¡ Isa ¡ Authen&city ¡ Isa ¡ Access ¡ control ¡ Isa ¡ msiGetUserACL ¡ msiSetDataType ¡ msiSetQuota ¡ msiDataObjRepl ¡ msiSysChksumDataObj ¡ Isa ¡ Isa ¡ Isa ¡ Isa ¡ Isa ¡

DATA_ID ¡ DATA_REPL_NUM ¡ DATA_CHECKSUM ¡

Isa ¡ Isa ¡ Isa ¡ Isa ¡ Isa ¡ HasFeature ¡ Archive ¡ Data ¡grid ¡ Collec,on ¡ Digital ¡Library ¡ Processing ¡Pipeline ¡ SubType ¡

Open Source Software

— Available at

• http://irods.diceresearch.org

— Contact

• rwmoore@renci.org

— NSF OCI-0940841 “DataNet Federation Consortium” — NSF OCI-1032732 “Improvement of iRODS for Multi-Disciplinary

Applications”

— NSF OCI-0848296 “NARA

Transcontinental Persistent Archives Prototype”

— NSF SDCI-0721400 “Data Grids for Community Driven Applications”

Tutorials @ RENCI Booth

43

10:30 a.m. iRODS Tutorials (Microservice and resource plugins) Presenters: Jason Coposky, Leesa Brieger, Terrell Russell, RENCI 11:30 a.m. iRODS Tutorials (Microservice and resource plugins) Presenters: Jason Coposky, Leesa Brieger, Terrell Russell, RENCI 1:00 p.m. Knowledge management through policy-based data management systems 2 p.m. Data Bridge: A system for scientific data