iRODS Consortium 2013 Reception
5:00 pm Welcome and Overview, Charles Schmitt 5:05 pm Introduction of new Executive Director, Brand Fortner 5:10 pm Update on iRODS Consortium, Reagan Moore 5:20 pm Technology Roadmap, Jason Coposky 5:40 pm Support Model – Expectations, Charles Schmitt 5:50 pm Features – Requests and Bug Process, Terrell Russell 6:00 pm Future Directions, Reagan Moore 6:10 pm Presentation, Use Case, Chris Smith, Distributed Bio 6:20 pm Presentation, Use Case, Pete Clapham, Sanger Institute 6:30 pm Presentation, Use Case, from Alan Hall, NCDC/NOAA 6:40 pm Wrap Up – Invitation to Thursday Tutorials, Charles Schmitt and Brand Fortner
Thank you Chris Smith, Distributed Bio Pete Clapham, Sanger Institute iRODS Consortium Executive Committee and Advisory Committee Members DICE & RENCI staff and management that have gotten us here 3
iRODS Users - examples Federal Users ◦ National Aeronautics and Space Administration ◦ National Oceanic and Atmospheric Administration ◦ National Archives and Records Adminstration ◦ USGS Non-profit/Institutional Users ◦ Broad Institute ◦ International Neuroinformatics Coordinating Facilities ◦ Wellcome Trust Sanger Institute Commercial Users ◦ DOW Chemical ◦ Bejing Genome Institute Resellers/Redeployers ◦ Distributed Bio ◦ Computer Sciences Corporation (CSC) Academic Users ◦ Too many too list 4
iRODS Users - examples Proven at scale: ◦ iPlant - 10k users ◦ French National Institute for Nuclear Physics and Plasma Physics – over 6 PB ◦ Wellcome Trust Sanger Institute – over 8 Pb ◦ Australian Research Collaboration Service - 512 storage resources ◦ NASA Center for Climate Simulations - 300 million attributes ◦ Cinegrid – sites across Japan-US-Europe Solid foundation: ◦ SRB: initial product (developed by DICE Group, owned by General Atomics) in 1997 ◦ iRODS: rewrite of SRB by DICE Group in 2006; currently on version 3.3 ◦ Enterprise iRODS: mission critical distribution co-developed by RENCI and DICE in 2012 Support: ◦ Community of developers from groups worldwide ◦ Independent groups offering consulting and support and development ◦ iRODS Consortium offering formal support, training, involvement, and development help 5
iRODS Users - downloads As open-source software, we cannot fully track usage, only downloads: 3.0 and 3.1 release: ◦ 1004 unique IP addresses combined (currently don’t have independent counts) 3.2 release: ◦ 1202 unique IP addresses ◦ About 15% are confirmed commercial sites ◦ Another 15% are non-profit non-universities 6
iRODS Consortium Provide: ◦ Open source iRODS release ◦ Single code source ◦ Sustainable development ◦ Binary distribution iRODS data grid ◦ Organize data, information, knowledge ◦ Enable collaboration, publication, preservation ◦ Provide interoperability mechanisms ◦ Pluggable architecture for flexibility
Current iRODS Development DICE Center RENCI ◦ Features ◦ Pluggable architecture ◦ Sheau-Yen Chen ◦ Jason Coposky ◦ Mike Conway ◦ Zoey Greer ◦ Reagan Moore ◦ Harry Johnson ◦ Arcot Rajasekar ◦ Terrell Russell ◦ Wayne Schroeder ◦ Antoine de Torcy ◦ Hao Xu Parallel Software Development
iRODS 4.0 (March 2014) Combine features from ◦ iRODS 3.3 (July 17, 2013) Hadoop Distributed File System driver PAM/LDAP authentication Workflow structured objects Rule language extensions Jargon Java I/O library netCDF and OpeNDAP support ◦ E-iRODS 3.0.1 (October 31, 2013) Pluggable security (authentication, identity) Pluggable micro-services Pluggable storage drivers (hierarchical resources) Pluggable network drivers TLS security Binary distribution – one-click installation
iRODS 4.0 Hardened and tested code One click installation (27 seconds) Pluggable architecture Federated security
iRODS Consortium ◦ Harry Johnson iRODS Consortium ◦ Terrell Russell ◦ Brand Fortner ◦ Antoine de Torcy ◦ Charles Schmitt ◦ Leesa Brieger ◦ Arcot Rajasekar ◦ Michael Shoffner* ◦ Reagan Moore ◦ Lisa Stillwell* ◦ Wayne Schroeder ◦ Jason Coposky Joint development of combined release, iRODS version 4.0 by March 15, 2014
DataNet Federation Consortium Migrating to iRODS 4.0 Features: – Spring 2014 Workflow registration Workflow re-execution ◦ Reagan Moore Workflow provenance OpeNDAP protocol ◦ Arcot Rajasekar NetCDF manipulation ◦ Wayne Schroeder Time series archiving Soft links to external repositories ◦ Sheau-Yen Chen Mediawiki integration ◦ Hao Xu VIVO integration AMQP messaging integration ◦ Mike Conway HIVE linked data ◦ Lisa Stillwell idrop-Web interface In-Common logon ◦ Charles Schmitt Dataverse integration Antelope integration ◦ Michael Shoffner Openflow controller integration
iRODS Consortium Members RENCI DICE Center Max Planck Society Data Direct Networks
Consortium Activities Generate a development roadmap Prioritize development tasks Standard iRODS release Service level support Matchmaking between support providers and iRODS users ◦ Tier 1 ◦ Tier 2 ◦ Tier 3
iRODS Consortium Roadmap Technical iRODS Consortium iRODS Community 03/12 07/13 03/14 10/12 3.2 4.0+ 3.0 3.1 3.3 4.0 09/11 Plus, independent 3.0 3.0.1 plugin iRODS Enterprise 03/13 11/13 releases iRODS Consortium Development Team - Supercomputing 2013
State of the Union: E-iRODS 3.0.1 Features Feature compatible with 3.0 Community, including bug fixes up to 3.3 Up to date rule engine Full SSL support with parallel transfer Live hierarchy manipulation—move and rename resources in a hierarchy with existing data on the resources Added Rebalancing as an operation—currently implemented for replication resource iRODS Consortium Development Team - Supercomputing 2013
State of the Union: E-iRODS Continuous Testing Code coverage at 54% for the Agent, up from 28% originally Plugin coverage on average at 70%, up to 93% Continuously tested across 6 platforms Full feature testing against 13 individual Resource Hierarchies Topological Testing Federation Testing, including 3.3 iRODS Consortium Development Team - Supercomputing 2013
State of the Union: E-iRODS Plugin Interfaces Current (3.0.1): ◦ Microservices ◦ Resources ◦ Network ◦ Authentication Coming Soon (4.0): ◦ Database iRODS Consortium Development Team - Supercomputing 2013
State of the Union: Current Plugins Resource plugins: Replication, Round Robin, Random, Passthru , Compound, UnivMSS, MockArchive , UFS, Non-blocking, and Structured File Object Network plugins: TCP and SSL Authentication plugins: PAM, OSAuth, and Native iRODS Consortium Development Team - Supercomputing 2013
iRODS 4.0 ( March 2014 ) Full merge between E-iRODS and up-to-date Community code Repository migration to GitHub—transparent development process Tickets, Workflows, approximately 50 updates not already included (e.g., filesystem metadata collection, storage admin role, sha1 ) Comprehensive feature testing during merge Architectural Updates: Database plugins—Postgres, MySQL, Oracle iRODS Consortium Development Team - Supercomputing 2013
Independently Released Plugins Released as separate packages Authentication (13Q4) ◦ Kerberos ◦ GSI Resource (14Q1) ◦ S3 ◦ HDFS ◦ Load Balancing ◦ WOS (streaming)* ◦ HPSS 7.3 and 7.4 (expanded platforms)* iRODS Consortium Development Team - Supercomputing 2013
E-iRODS Plugins E-iRODS 3.0.1 Independently Packaged Releases Resource plugins: Resource Plugins: ◦ Replication ◦ Round Robin ◦ S3 ◦ Random ◦ Compound ◦ HDFS ◦ UnivMSS ◦ Unix File System ◦ HPSS 7.3 and 7.4 ◦ Non-blocking Network plugins: ◦ WOS ◦ TCP ◦ SSL Authentication: Authentication plugins: ◦ PAM ◦ ◦ Kerberos OSAuth ◦ Native ◦ GSI
State of the Union: E-iRODS Plugin Interfaces Reliability – Creates more easily tested software Flexibility – Convert compile time options into run-time configuration Stability – Create a hardened core and externalize rapid development Community – open the development up to a large audience E-iRODS 3.0.1: iRODS 4.0: Microservices Database Resources Network Authentication
iRODS Development Areas RESTful Interface—use the consortium as a vehicle for standardization Refactoring—continue migrating toward more abstract interfaces ( e.g., special collections ) Dynamic API—extend iRODS API as any other plugin interface Plugin Registry—keep track of plugins and state in the catalog Plugin dependency model—describe interdependencies between plugins, ship packages of plugins together as a collective feature iRODS Consortium Development Team - Supercomputing 2013
Recommend
More recommend
