Dynamic Storage Dynamic Storage Federation Federation based on - PowerPoint PPT Presentation

EGI Community Forum 2013 Dynamic Storage Dynamic Storage Federation Federation based on open protocols based on open protocols Oliver Keeble Adrien Devresse Ricardo Brito da Rocha (presenter) Alejandro Alvarez Fabrizio Furano Patrick Fuhrmann Credits to ShuTing Liao (ASGC) Paul Millar Daniel Becker CERN IT Department CH-1211 Genève 23 Switzerland

Motivations for Federations (1) Current situation: → Data are in groups of distributed of Storage Systems at differents places → The location of data is managed by → Experiment framework → Meta-data catalogs → Jobs are ( supposed to be ) placed close to the data Dynamic Storage Federation- 2

Motivations for Federations (2) Can be improved ! → What happens to your workflow if → A storage system is offline ? → A file is missing ? → The meta-data catalog is overloaded ? → Listing and/or browsing are expensive/slow operations → The file access pattern is clearly not optimal Dynamic Storage Federation- 3

Dynamic Federation Goal ? Simple Idea : Federate on the fly Grid Distributed Storage, Cloud Storage, Any Existing meta-data Catalog, In a unique namespace Dynamic Storage Federation- 4

Basic Idea of the federation This is what Aggregation we want to see as users /dir1 With 2 /dir1/file1 Sites remain replicas /dir1/file2 independent and /dir1/ participate to a global view All the EMI INFSO-RI- Storage/MD Storage/MD metadata 261611 endpoint 1 endpoint 2 interactions are hidden /dir1/file1 /dir1/file2 and done on /dir1/file2 /dir1/file3 the fly Dynamic Storage Federation- 5

N ew possibilities Open a lot of new possibilities ! → Reliability, failover → Detect offline storage element → Detect network problems → World wide replica discovery → Smart Redirection → Geographical redirection → Network optimizations → Transparent caching Dynamic Storage Federation- 6

N ew possibilities (2) → Easy data Migration →Transparent data migration →Merge Grid and Cloud Storage namespace → Scalability → Aggressive caching → Protection of your name servers → Allow horizontal deployement → Load balancing Dynamic Storage Federation- 7

N ew possibilities (3) → Performance → Multi-stream download ( metalink ) → Reduce latency ( GeoIP, local Fed ) → Low response time ( in memory ) → Flexibility → Algorithmic filename translation → Merge any type of Storage ( Grid, Cloud ) → Use / integrate your own catalog → Allow local federations Dynamic Storage Federation- 8

How does it work ? The cache remembers what happened The next metadata Frontend interactions (Apache2+DMLite) will very likely be Metadata Federator cache cached Plugin Plugin PluginPlugin Plugin Plugin Plugi n Catalog Catalog e.g. LFC/Rucio EMI INFSO-RI- e.g. LFC/Rucio Catalog 261611 Catalog e.g. LFC e.g. LFC EMI INFSO-RI- SE SE 261611 SE SE SE 30 Oct F.Furano - Dynamic federations 2012

How is it working ? → No Database → Fetch the meta-data on demand from the Storage endpoints ( SE ) → Aggregate the meta-data on the fly → Distributed System Dynamic Storage Federation- 10

Scalable Architecture Dynamic Federation Architecture Front-end ( Dmlite + Webdav ) → In Memory Only Ugr_connector.so ( Entry point ) → several layer of Descision process plugin 2 plugin 1 cache 1st layer cache ( RAM process ) → No Database 2nd layer cache ( Memcached ) Plugin layer → Meta-data fetched dynamically from the Dav plugin LFC plugin S3 plugin Http plugin Storage Elements. Storage Elements Storage Elements Dynamic Storage Federation- 11

Deployement possibilities Global federation Dynamic federation 1 Dynamic federation 3 Dynamic federation 2 LHCb LFC Meta-data catalog GPFS EOS Dynamic Storage Federation- 12

Performances test • Two storage endpoints: DESY and CERN (poor VM) • One UGR federator at DESY, clients at CERN • 10K files are interleaved in a 4-levels deep directory Oddly-numbered files are at CERN Evenly-numbered files are at Desy • The test (written in C++) invokes Stat only once per file, using many parallel clients doing stat() at the maximum pace from 3 machines Dynamic Storage Federation- 13

Performances results Dynamic Storage Federation- 14

Dynamic Storage Federation Status • Currently available! • Technically TODAY we can dynamically aggregate: – dCache DAV/HTTP instances – DPM DAV/HTTP instances – LFC DAV/HTTP and old Cns API instances – Cloud DAV/HTTP services – Anything that can be plugged into DMLite (the new architecture for DPM/LFC) – Can be extended to other metadata sources Dynamic Storage Federation- 15

Dynamic Storage Federation Status → The system also can load a “Geo” plugin → Gives a geographical location to replicas and clients → Allows the core to choose the replica that is closer to the client → The one that’s available uses GeoIP (free) Dynamic Storage Federation- 16

Dynamic Storage Federation Demo We have a stable demo testbed, using HTTP/DAV http://federation.desy.de/myfed Dynamic Storage Federation- 17

Dynamic Storage Federation Demo • It is actually 2 demos in one – An ATLAS demo, federating 8 sites, plus LFC as name translator • Note that this is not the full ATLAS repo, it’s just 8 sites. • DESY, KIT, SARA, WUPPERTAL, NDGF, Muenchen, Prague, ASGC – A fully dynamic catalogue-free demo with the EMI testbed • Federating three endpoints. – a DPM instance at CERN – a dCache instance in DESY – one endpoint in LBNL Dynamic Storage Federation- 18

Dynamic Storage Federation News • Very stable, installable from the wiki • Recent improvement: in the case we federate catalogues, the replicas they give can be checked on the fly Use the catalogue as name translator Use the catalogue as source of file listings Check the replicas in the moment they are requested • Next item: ATLAS and Rucio We have a nice testbed, federating many ATLAS SEs We want to federate the Rucio services and the LFC(s) seamlessly together Dynamic Storage Federation- 19

Dynamic Storage Federation Links Documentation and source code – https://svnweb.cern.ch/trac/lcgdm/wiki/Dynafeds Description of the demo http://federation.desy.de/DynaFeds/The_Dynamic_Feder Power users wanted Helping in getting the best out of the system. Your cooperation is very appreciated. Dynamic Storage Federation- 20

Questions ? Dynamic Storage Federation- 21