Dynamic Storage Dynamic Storage Federation Federation based on - - PowerPoint PPT Presentation

CERN IT Department CH-1211 Genève 23 Switzerland

Dynamic Storage Dynamic Storage Federation Federation based on open protocols based on open protocols

Adrien Devresse (presenter) Fabrizio Furano Patrick Fuhrmann Paul Millar Daniel Becker

EGI Community Forum 2013

Oliver Keeble Ricardo Brito da Rocha Alejandro Alvarez Credits to ShuTing Liao (ASGC)

Dynamic Storage Federation- 2

Motivations for Federations (1)

→ Data are in groups of distributed of Storage Systems at differents places → The location of data is managed by

→ Experiment framework → Meta-data catalogs

→ Jobs are ( supposed to be ) placed close to the data

Current situation:

Dynamic Storage Federation- 3

Motivations for Federations (2)

→ What happens to your workflow if

Can be improved !

→ A storage system is offline ? → A file is missing ? → The meta-data catalog is overloaded ?

→ Listing and/or browsing are expensive/slow

• perations

→ The file access pattern is clearly not optimal

Dynamic Storage Federation- 4

Dynamic Federation Goal ?

Simple Idea :

Federate on the fly Grid Distributed Storage, Cloud Storage, Any Existing meta-data Catalog, In a unique namespace

Dynamic Storage Federation- 5

Aggregation

/dir1 /dir1/file1 /dir1/file2 /dir1/

EMI INFSO-RI- 261611

/dir1/file1 /dir1/file2 Storage/MD endpoint 1 /dir1/file2 /dir1/file3 Storage/MD endpoint 2 This is what we want to see as users Sites remain independent and participate to a global view All the metadata interactions are hidden and done on the fly

With 2 replicas

Basic Idea of the federation

Dynamic Storage Federation- 6

New possibilities

Open a lot of new possibilities ! → Reliability, failover

→ Detect offline storage element → Detect network problems → World wide replica discovery

→ Smart Redirection

→ Geographical redirection → Network optimizations → Transparent caching

Dynamic Storage Federation- 7

New possibilities (2)

→ Easy data Migration

→Transparent data migration →Merge Grid and Cloud Storage namespace

→ Scalability

→ Aggressive caching → Protection of your name servers → Allow horizontal deployement → Load balancing

Dynamic Storage Federation- 8

New possibilities (3)

→ Performance

→ Multi-stream download ( metalink ) → Reduce latency ( GeoIP, local Fed ) → Low response time ( in memory )

→ Flexibility

→ Algorithmic filename translation → Merge any type of Storage ( Grid, Cloud ) → Use / integrate your own catalog → Allow local federations

EMI INFSO-RI- 261611 EMI INFSO-RI- 261611 30 Oct 2012 F.Furano - Dynamic federations Federator

Plugin Frontend (Apache2+DMLite) Plugin PluginPlugin Plugin Plugin Plugi n

SE SE SE

Metadata cache

SE SE

Catalog e.g. LFC/Rucio Catalog e.g. LFC/Rucio

Catalog e.g. LFC Catalog e.g. LFC The cache remembers what happened The next metadata interactions will very likely be cached

How does it work ?

Dynamic Storage Federation- 10

How is it working ?

→ No Database → Fetch the meta-data on demand from the Storage endpoints ( SE ) → Aggregate the meta-data on the fly → Distributed System

Dynamic Storage Federation- 11

Scalable Architecture

Front-end ( Dmlite + Webdav ) Ugr_connector.so ( Entry point ) Descision process 1st layer cache ( RAM process ) Plugin layer Dav plugin LFC plugin S3 plugin Http plugin

plugin 1

Dynamic Federation Architecture

2nd layer cache ( Memcached )

→ In Memory Only → several layer of cache → No Database → Meta-data fetched dynamically from the Storage Elements.

plugin 2

Storage Elements Storage Elements

Dynamic Storage Federation- 12

Deployement possibilities

EOS LHCb LFC Meta-data catalog Dynamic federation 1 Dynamic federation 2 Dynamic federation 3 GPFS Global federation

Dynamic Storage Federation- 13

Performances test

• Two storage endpoints: DESY and CERN (poor VM)
• One UGR federator at DESY, clients at CERN
• 10K files are interleaved in a 4-levels deep directory

Oddly-numbered files are at CERN Evenly-numbered files are at Desy

• The test (written in C++) invokes Stat only once per

file, using many parallel clients doing stat() at the maximum pace from 3 machines

Dynamic Storage Federation- 14

Performances results

Dynamic Storage Federation- 15

Dynamic Storage Federation Status

• Currently available!
• Technically TODAY we can dynamically aggregate:

– dCache DAV/HTTP instances – DPM DAV/HTTP instances – LFC DAV/HTTP and old Cns API instances – Cloud DAV/HTTP services – Anything that can be plugged into DMLite (the new architecture for DPM/LFC) – Can be extended to other metadata sources

Dynamic Storage Federation- 16

Dynamic Storage Federation Status

→ The system also can load a “Geo” plugin

→ Gives a geographical location to replicas and clients → Allows the core to choose the replica that is closer to the client

→ The one that’s available uses GeoIP (free)

Dynamic Storage Federation- 17

Dynamic Storage Federation Demo

We have a stable demo testbed, using HTTP/DAV http://federation.desy.de/myfed

Dynamic Storage Federation- 18

Dynamic Storage Federation Demo

• It is actually 2 demos in one

– An ATLAS demo, federating 8 sites, plus LFC as name translator

• Note that this is not the full ATLAS repo, it’s just

8 sites.

• DESY, KIT, SARA, WUPPERTAL, NDGF,

Muenchen, Prague, ASGC – A fully dynamic catalogue-free demo with the EMI testbed

• Federating three endpoints.

– a DPM instance at CERN – a dCache instance in DESY – one endpoint in LBNL

Dynamic Storage Federation- 19

Dynamic Storage Federation News

• Very stable, installable from the wiki
• Recent improvement: in the case we federate

catalogues, the replicas they give can be checked on the fly

Use the catalogue as name translator Use the catalogue as source of file listings Check the replicas in the moment they are requested

• Next item: ATLAS and Rucio

We have a nice testbed, federating many ATLAS SEs We want to federate the Rucio services and the LFC(s) seamlessly together

Dynamic Storage Federation- 20

Dynamic Storage Federation Links

Documentation and source code – https://svnweb.cern.ch/trac/lcgdm/wiki/Dynafeds Description of the demo http://federation.desy.de/DynaFeds/The_Dynamic_Feder

Power users wanted

Helping in getting the best out of the system. Your cooperation is very appreciated.

Dynamic Storage Federation- 21

Questions ?