ARCS Data Fabric Pauline Mak pauline.mak@arcs.org.au ARCS Data - PowerPoint PPT Presentation

ARCS Data Fabric Pauline Mak pauline.mak@arcs.org.au ARCS Data Services Pauline Mak

Outline • Introduction to the ARCS Data Fabric • Migration from SRB to iRODS • Davis • Hermes Pauline Mak

ARCS Data Fabric Pauline Mak

Why SRB? • In June 2008, iRODS lacked - Full GSI Authentication - Federation • Now migrating to iRODS 2.0 - Supports both GSI and federation - Open-sourced - Microservices and Rules Engine Pauline Mak

Migration • Modified migration script handles: - Users - SRB Objects and Collections - Metadata - Groups • Does not migrate - Permissions - User DNs Pauline Mak

Migration • Moving other processes from SRB to iRODS - Automatic account creation - Usage statistics - User sync Pauline Mak

Automatic Account (SRB) Checks MCAT for user MCAT with the same DN srb.tpac.org.au (patched SRB server) If no account matches, User attempts to calls external scripts to login with a valid generate account certificate Pauline Mak

Automatic Accounts (iRODS) 5. The server retrieves new username from script output (or query ICAT by DN) and logs in iRODS Server ICAT 4. CreateUser 1. Client connects 2. Fires the rule if the generates a new using a certificate DN is unknown account acGetUserByDN 3. Executes external script (CreateUser) Pauline Mak

Shibboleth & SLCS 2. Forwards user to institution's Shibboleth login page Short Lived TPAC Identify Certificate Server Provider (IdP) (SLCS) 4. Forwards result back to SLCS server 1. Connects to SLCS server to 5. (If valid) 3. User enters select IdP Creates a IdP username certificate and and password sends it back to the user Pauline Mak

Usage Scripts (SRB) Each zone runs a srb.tpac.org.au script daily that will srb.sapac.edu.au query MCAT for usage info Uploads usage XML file to a central location in SRB Use Scommands to collect Central collection zone users and resource info, srb.hpsc.uq.edu.au then ingest XML data into DB status.arcs.or MySQL g.au DB Pauline Mak

Usage Scripts (iRODS) Each zone runs a srb.tpac.org.au script daily that will srb.sapac.edu.au query ICAT for usage info Uploads usage XML file to a central location in SRB Central collection zone Ingests XML data into DB srb.hpsc.uq.edu.au status.arcs.or MySQL g.au DB Pauline Mak

status.arcs.org.au Pauline Mak

Migration – Zone sync • We only sync users • SRB - Szonesync: executed hourly - Federation can lag for up to an hour • iRODS - Executing rules immediately when a user is added Pauline Mak

Davis • WebDAV • Supports SRB and iRODS • Browser and client mode • Supports Shibboleth • Multi-threaded and resumable downloads • Simplifies access Pauline Mak

Davis Architecture SRB/iRODS Jargon API Browser WebDAV Authentication SLCS Mode Handler Processor Server Handler WebDAV IdP Web Browser Client Pauline Mak

Davis – Modes Pauline Mak

Davis - Permissions Pauline Mak

Davis - Metadata Pauline Mak

Hermes/commons-vfs-grid • Originally from JCU (ARCHER project) – Mathew Wyatt • Commons-vfs - local, SFTP, WebDAV • Commons-vfs-grid: - grid-related protocols - GridFTP contributed by David Meredith Pauline Mak

Hermes Architecture S/FTP SRBiRODS GridFTP SLCS + Local JSCH Jargon Globus MyProxy Shib proxy Protocol specific libraries GSIProxyManager commons-vfs(-grid) GSI Based Credentials Account FileObject Connectors Pauline Mak

Hermes – Browser Pauline Mak

Hermes - Permission Pauline Mak

Hermes - Metadata Pauline Mak

Hermes – metadata search Pauline Mak

Questions? Pauline Mak

Links • http://www.arcs.org.au (ARCS) • http://www.tpac.org.au (TPAC) • http://projects.arcs.org.au/trac/davis/ (Davis) • http://projects.arcs.org.au/trac/commons-vfs-grid/ (Hermes) • http://projects.arcs.org.au/trac/systems/wiki/DataS ervices/SRB • http://projects.arcs.org.au/trac/systems/wiki/DataS ervices/iRODS Pauline Mak