GFAL 2.0 Devresse Adrien CERN lcgutil team lcgutil-support@cern.ch - - PowerPoint PPT Presentation
GFAL 2.0 Devresse Adrien CERN lcgutil team lcgutil-support@cern.ch What is GFAL 2.0 ? One and only one Toolkit for all the Grid and Cloud data operations : Client data access : support for all common protocols support for all
Devresse Adrien CERN lcgutil team
lcgutil-support@cern.ch
E M I I N F S O
2
CERN, IT-GT-DMS
→ One and only one Toolkit for all the Grid and Cloud data operations : → Client data access :
→ support for all common protocols → support for all common storage systems
→ Data management
→ One API for all data management
→ Data transfers
→ 3rd party transfer support
E M I I N F S O
3
CERN, IT-GT-DMS
→ Hide the Grid complexity from the client side. → Simplify the data access world → One API. → Reduce the number of dependencies. → Plugin system → Makes application independent of the technology / protocols.
E M I I N F S O
4
CERN, IT-GT-DMS
LFC LFC
get file replicas list
SRM Server SRM Server DPM Head Node DPM Head Node DPM Disk node DPM Disk node
SRM GET rfio open rfio lseek rfio read lfc_stat
GFAL 2.0 GFAL 2.0 gfal2_open gfal2_pread
SRM surl → turl
Client Client
E M I I N F S O
5
CERN, IT-GT-DMS
Grid Application X
Core logic File access layer Globus framework Xrootd framework SRM client Http lib dCache, DPM, Castor lib
E M I I N F S O
6
CERN, IT-GT-DMS
Grid Application X
Core logic GFAL 2.0 API
E M I I N F S O
7
EOS
Extensible...
API POSIX API POSIX GFAL 2 Plugin layer Plugin layer
Plugin LFC Plugin SRM Plugin RFIO Plugin Dcap Plugin xrootd Plugin Http/dav Plugin S3 ( ? )
experience catalogue plugin (???)
API Transfer API Transfer File File
Any Cloud ( Http/Dav ) DPM dCache Castor Dropbox
E M I I N F S O
8
CERN, IT-GT-DMS
➔ Meta-data operations :
→ stat, rm, mkdir, → mv, rmdir, etc... → list directory, xattr, etc... Remote I/O in any protocol → open/read/write/close → pread/pwrite → etc..
→ SRM → GRIDFTP → RFIO-Castor → RFIO-DPM → LFN → GUID → DCAP / GSIDCAP → XROOTD → HTTP(S) * → WebDav(s) * → local file
E M I I N F S O
9
CERN, IT-GT-DMS
gfalt_copyfile(context, NULL, « srm://monfichier », « srm://madest/higgs ») gfalt_copyfile(context, NULL, « srm://monfichier », « gsiftp://madest/chuck_norris »)
Third party transfer copy : Automatic protocol resolution :
gfalt_copyfile(context, NULL, « guid:1233 », «root://xrootd_dpm/curiosity») gfalt_copyfile(context, NULL, « file:///tmp/super_file», «gsidcap://42»)
E M I I N F S O
10
CERN, IT-GT-DMS
gfalt_copyfile(context, NULL, « a://src/file », «b://dst/file»)
Take a, b in srm, guid, lfn, , rfio, dcap , gsidcap, gsiftp, file, xrootd, (http, dav )
FINISHED !
E M I I N F S O
11
Destination exist ? SRM GET SRM PUT transfer PUTDONE
Old tools...
Destination exist ? SRM GET SRM PUT transfer PUTDONE
GFAL 2.0
File copy operation :
Speed Up x2-x4 for short transfers
E M I I N F S O
12
Pool
SRM, GSIFTP, HTTP, Etc,.. Create session Transfer 1 End Transfer 2 End One server connection
E M I I N F S O
13
GFAL 2.0 supports POSIX extended attributes ( getxattr, setxattr, etc...) → Clean way to uses protocol specific functionalities. → Simple get/list/set API → Easy to use.
E M I I N F S O
14
Environment Variable User
E M I I N F S O
15
→ Every parameter is in a configuration file : → /etc/gfal2.d/*.conf → No environment variable needed. → Tune it once, re-use everywhere. → Still compatible with classical environment variable of GFAL 1.0, for an easy transition.
E M I I N F S O
16
( RFC-2324 compliant ! )
E M I I N F S O
17
CERN, IT-GT-DMS
→ libgfal2 :
→ a C Library → a set of independent plugins
→ gfal2-python : → simple and pythonic python bindings → gfalFS fuse module for gfal 2
→ allows to mount any grid endpoint in a local folder
→ gfal-tools : experimental command line tools
→ gfal-ls, gfal-copy, gfal-mkdir, gfal-cat, ….
E M I I N F S O
18
CERN, IT-GT-DMS
Already in production state ! → Part of EMI 2 release:
→ EMI 2 repository : www.eu-emi.eu
→ Packaged on EPEL : → sudo yum install gfal2-all gfal2-doc gfalFS → is going to be packaged on Debian ( thx Mathias ) → Source available, designed to be portable.
https://svnweb.cern.ch/trac/lcgutil/wiki/gfal2
E M I I N F S O
19
CERN, IT-GT-DMS
→ Part of EMI
➔ will be supported after the end of EMI
→ GFAL 2.0 is the core of FTS 3.0
➔ will be supported in any case.
→ Long term successor to :
➔ lcg clients : lcg-cp, lcg-cr, lcg util python ➔ gfal 1.0, etc...
→ Development Open to everybody
E M I I N F S O
20
CERN, IT-GT-DMS
→ GFAL 2.0 does a lot more than GFAL 1.0 : GFAL 2.0 => → a lot of Major changes :
➔ Not 100% backward compatible.
gfal 1.0 + lcg-util (lcg-cp, .. ) + is-interface + gridftp-ifce + ….
E M I I N F S O
21
CERN, IT-GT-DMS
➔ But, We provides facilities for the
transition
➔ The complete POSIX API is backward
compatible
➔ We provide examples for common
➔ Doxygen documentation of the API. ➔ Any question is Welcome :
lcgutil-support@cern.ch
E M I I N F S O
22
→Users wants command line tools → lcg-util (lcg-cp, lcg-cr ) are heavily used. → lcg-util suffers from his age...
→ outdated features → hard to maintain → lots of dependencies →
E M I I N F S O
23
→ Experimental tools for lcg-util replacement on the long term. → We need feedback and informations from the experiments :
→ Which part of lcg-util is needed : → lcg-cp, lcg-ls, lcg-cr, ( and ? ) → What can be improved in their point of view ?: → It's time for suggestion ! → How can we help to a smooth transition ?
E M I I N F S O
24
→ A simple set of generic command line file utils based on GFAL 2: → gfal-copy → gfal-ls → gfal-rm → gfal-cat → gfal-save → etc.... → Prototype already available for comments ... → Still under development !
→ It's the right time for comments...
E M I I N F S O
25
https://svnweb.cern.ch/trac/lcgutil/wiki/gfal2
<[:isPlaceholder:]>