GFAL 2.0 Devresse Adrien CERN lcgutil team lcgutil-support@cern.ch
What is GFAL 2.0 ? → One and only one Toolkit for all the Grid and Cloud data operations : → Client data access : → support for all common protocols → support for all common storage systems → Data management → One API for all data management → Data transfers → 3rd party transfer support E M I N I F CERN, IT-GT-DMS 2 S O R -
Goal of GFAL 2.0 ? → Hide the Grid complexity from the client side. → Simplify the data access world → One API. → Reduce the number of dependencies. → Plugin system → Makes application independent of the technology / protocols. E M I N I F CERN, IT-GT-DMS 3 S O R -
The complexity of a random read file operation gfal2_open gfal2_pread Client Client LFC LFC lfc_stat get file replicas list GFAL 2.0 GFAL 2.0 SRM GET SRM DPM SRM DPM Server Head Node Server Head Node SRM surl → turl DPM rfio open DPM Disk node Disk node rfio lseek rfio read E M I N I F CERN, IT-GT-DMS 4 S O R -
An old-style Grid application : Grid Application X Core logic File access layer dCache, Globus Xrootd SRM Http lib DPM, framework framework client Castor lib E M I N I F CERN, IT-GT-DMS 5 S O R -
A GFAL 2.0 Grid application : Grid Application X Core logic GFAL 2.0 API E M I N I F CERN, IT-GT-DMS 6 S O R -
GFAL 2.0 architecture : API POSIX File API Transfer API POSIX File API Transfer GFAL 2 Plugin layer Plugin layer Plugin Plugin Plugin Plugin Plugin Plugin Plugin experience LFC SRM RFIO Dcap Http/dav S3 ( ? ) xrootd catalogue plugin (???) Any Cloud DPM Castor ( Http/Dav ) Dropbox E M Extensible... I N I EOS F dCache 7 S O R -
What GFAL 2.0 can do ? ➔ Meta-data operations : Remote I/O in any protocol → stat, rm, mkdir, → open/read/write/close → mv, rmdir, etc... → pread/pwrite → list directory, xattr, etc... → etc.. In any protocol : → SRM → DCAP / GSIDCAP → GRIDFTP → XROOTD → RFIO-Castor → HTTP(S) * → RFIO-DPM → WebDav(s) * → LFN → local file → GUID E M I N I F CERN, IT-GT-DMS 8 S O R -
Feature : Protocol generic filecopy Third party transfer copy : gfalt_copyfile(context, NULL, « srm://monfichier », « srm://madest/higgs ») gfalt_copyfile(context, NULL, « srm://monfichier », « gsiftp://madest/chuck_norris ») Automatic protocol resolution : gfalt_copyfile(context, NULL, « guid:1233 », «root://xrootd_dpm/curiosity») gfalt_copyfile(context, NULL, « file:///tmp/super_file», «gsidcap://42») E M I N I F CERN, IT-GT-DMS 9 S O R -
Really protocol generic : Take a, b in srm, guid, lfn, , rfio, dcap , gsidcap, gsiftp, file, xrootd, (http, dav ) gfalt_copyfile(context, NULL, « a://src/file », «b://dst/file») FINISHED ! E M I N I F CERN, IT-GT-DMS 10 S O R -
Feature : Transparent parallel operations Destination File copy operation : exist ? SRM GET Destination SRM GET exist ? SRM PUT SRM PUT transfer transfer PUTDONE PUTDONE Speed Up x2-x4 E M Old tools... GFAL 2.0 for short transfers I N I F 11 S O R -
Feature : Transparent thread-safe session reuse Transfer 1 End Pool Create session of Connections SRM, Transfer 2 GSIFTP, HTTP, Etc,.. End E M One server connection I N I F 12 S O R -
Feature : extended attributes GFAL 2.0 supports POSIX extended attributes ( getxattr, setxattr, etc...) → Clean way to uses protocol specific functionalities. → Simple get/list/set API → Easy to use. E M I N I F 13 S O R -
NO ENVIRONMENT VAR ! Environment Variable User E M I N I F 14 S O R -
Feature : Works out of the box : NO ENV VAR ! → Every parameter is in a configuration file : → /etc/gfal2.d/*.conf → No environment variable needed. → Tune it once, re-use everywhere. → Still compatible with classical environment variable of GFAL 1.0, for an easy transition. E M I N I F 15 S O R -
Missing Feature : Does not make the coffee Coming soon …. ( RFC-2324 compliant ! ) E M I N I F 16 S O R -
How to use GFAL 2.0 ? : → libgfal2 : → a C Library → a set of independent plugins → gfal2-python : → simple and pythonic python bindings → gfalFS fuse module for gfal 2 → allows to mount any grid endpoint in a local folder → gfal-tools : experimental command line tools E → gfal-ls, gfal-copy, gfal-mkdir, gfal-cat, …. M I N I F CERN, IT-GT-DMS 17 S O R -
How to get GFAL 2.0 ? : Already in production state ! → Part of EMI 2 release: → EMI 2 repository : www.eu-emi.eu → Packaged on EPEL : → sudo yum install gfal2-all gfal2-doc gfalFS → is going to be packaged on Debian ( thx Mathias ) → Source available, designed to be portable. E M https://svnweb.cern.ch/trac/lcgutil/wiki/gfal2 I N I F CERN, IT-GT-DMS 18 S O R -
Support of GFAL 2.0 : → Part of EMI ➔ will be supported after the end of EMI → GFAL 2.0 is the core of FTS 3.0 ➔ will be supported in any case. → Long term successor to : ➔ lcg clients : lcg-cp, lcg-cr, lcg util python ➔ gfal 1.0, etc... → Development Open to everybody E M I N I F CERN, IT-GT-DMS 19 S O R -
From gfal1.0 to GFAL 2.0 : → GFAL 2.0 does a lot more than GFAL 1.0 : GFAL 2.0 => gfal 1.0 + lcg-util (lcg-cp, .. ) + is-interface + gridftp-ifce + …. → a lot of Major changes : ➔ Not 100% backward compatible. E M I N I F CERN, IT-GT-DMS 20 S O R -
From gfal1.0 to GFAL 2.0 : ➔ But, We provides facilities for the transition ➔ The complete POSIX API is backward compatible ➔ We provide examples for common operations ➔ Doxygen documentation of the API. ➔ Any question is Welcome : lcgutil-support@cern.ch E M I N I F CERN, IT-GT-DMS 21 S O R -
gfal-tools ( ? ) : Lcg-util command line tools successor ? →Users wants command line tools → lcg-util (lcg-cp, lcg-cr ) are heavily used. → lcg-util suffers from his age... → outdated features → hard to maintain → lots of dependencies → → a GFAL 2.0 based -tools ? E M I N I F 22 S O R -
gfal-tools : → Experimental tools for lcg-util replacement on the long term. → We need feedback and informations from the experiments : → Which part of lcg-util is needed : → lcg-cp, lcg-ls, lcg-cr, ( and ? ) → What can be improved in their point of view ?: → It's time for suggestion ! → How can we help to a smooth transition ? E M I N I F 23 S O R -
Gfal-tools : Current status → A simple set of generic command line file utils based on GFAL 2: → gfal-copy → gfal-ls → gfal-rm → gfal-cat → gfal-save → etc.... → Prototype already available for comments ... → Still under development ! E → It's the right time for comments... M I N I F 24 S O R -
https://svnweb.cern.ch/trac/lcgutil/wiki/gfal2 <[:isPlaceholder:]> E M I N I F 25 S O R -
Recommend
More recommend
