Dune computing Workshop Cdric Serfon Cedric.Serfon@cern.ch On - - PowerPoint PPT Presentation

Dune computing Workshop

Cédric Serfon

Cedric.Serfon@cern.ch On behalf of the Rucio team

2018-10-08 Rucio - Dune Workshop

Rucio in a nutshell (1)

• Rucio is a Distributed Data Management system built initially for the ATLAS

experiment

○ It allows to federate the data located on different heterogenous sites (small/big, grid/cloud, Tape/Disk) under a common namespace and hide the complexity of the underlying storage layer ○ It provides tools to manage efficiently the data according to the policies defined by the collaboration,

• ptimize the space, etc.

○ It provides tools for the the end-users to interact with the data

• It was built using more than 10 years of experience in Data Management:

○ Designed from experience from the previous ATLAS data management system DQ2 ○ Integrate new features and technologies

• Modular, highly scalable, well supported

2018-10-08 Rucio - Dune Workshop

Rucio in a nutshell (2)

• Who is using Rucio ?

○ Used by ATLAS, AMS and Xenon1T . CMS just choose to move to Rucio for LHC Run3 ○ Being evaluated by other small/medium/big HEP/Astro experiments:

• 1st Rucio community workshop was held on March 1st-2nd 2018 to present Rucio to

more collaboration/scientific communities

○ Attended by more than 80 people ○ A new workshop will be organized in spring (3 instituts already candidated to host the meeting)

2018-10-08 Rucio - Dune Workshop

Rucio main functionalities

• It provides many features (you are not forced to enable all) :

○ File and dataset catalog (logical definition and replicas) ○ Transfers between sites and staging capabilities ○ User Interface and Command Line Interface to allow user to download/upload/transfer their data ○ Extensive monitoring ○ Powerful policy engines (rules and subscriptions) ○ Bad file identification and recovery ○ Dataset popularity based replication ○ …

• Rucio can be easily integrated with Workload and Workflow Management System

○ Already supporting PanDA (ATLAS WFMS) ○ Possibilities of integration with other like Dirac More advanced features

2018-10-08 Rucio - Dune Workshop

Rucio in ATLAS

• Rucio stores all ATLAS data :

○ Data coming from the detector ○ Monte Carlo data ○ User data

• Rucio takes care of :

○ Ensuring the replication of files according to the replication policy specified by ATLAS ○ Replicate the data for other applications (e.g. panda) and for the end-users ○ Ensure file recovery ○ Staging data from TAPE ○ And plenty other things

2018-10-08 Rucio - Dune Workshop

Rucio in ATLAS

• ATLAS instance in a few numbers

○ More than 1B files, ~0.4 EB ○ Up to 4M files/2.5 PB transferred per day ○ More than 1000 active users

• Expect to gain one order of magnitude for Run4

2018-10-08 Rucio - Dune Workshop

Rucio main concepts - Accounts

• Each user who wants to use Rucio needs an account
• There are different types of accounts : user, service,

group

• One user can have different accounts and can use

different credentials (X509, kerberos token, userpass, ssh) to connect to Rucio

• Permissions and quotas are based on accounts

2018-10-08 Rucio - Dune Workshop

Rucio main concepts - DIDs

• The data stored in Rucio are identified by a

Data IDentifier (DID)

• There are different types of DIDs :

○ Files ○ Datasets : collection of files ○ Container : collection of dataset and/or container

• Each DID is composed of :

○ A scope : 25 characters to partition your data, e.g. data17, mc17 ○ A name (up to 255 character)

• A name is unique within a scope
• DIDs hold a set of basic metadata e.g.

○ Bytes ○ Checksum (for files) ○ Number of events ○ Datatype

2018-10-08 Rucio - Dune Workshop

Rucio main concepts - RSEs

• Rucio Storage Elements (RSEs) are logical entities of space

○ No software needed to run at the site ○ RSE names are arbitrary (e.g., "CERN-PROD_DATADISK", "AWS_REGION_USEAST", … ) ○ Usually one RSE per site and storage data class

• RSEs collect all necessary metadata for a storage

○ protocols, hostnames, ports, prefixes, paths, implementations, … ○ data access priorities can be set (e.g., to prefer a protocol for LAN access)

• RSEs can be tagged

○ Key/Value pairs (e.g., country=UK, type=TAPE, support=brian@unl.edu) ○ You can use RSE expressions to describe a list of RSEs (e.g. country=UK&type=TAPE)

2018-10-08 Rucio - Dune Workshop

Rucio main concepts - Rules and subscriptions

• Replication rules :

○ Describe how a Data IDentifier (DID) must be replicated on a list of Rucio Storage Elements (RSE) ○ e.g. : Make 2 replicas of dataset data15_13TeV:mydatasetname on tier=1&disk=1 ○ Rucio will create the minimum number of replicas to optimise storage space, minimise the number of transfers and automate data distribution

• Subscriptions :

○ Replication policies based on Data IDentifiers metadata, for Data IDentifiers that will be produced in the future ○ e.g. : Make 2 replicas of datasets with scope=data15_13TeV and datatype=AOD on tier=1&disk=1

2018-10-08 Rucio - Dune Workshop

Rucio future developments

• New functionalities keeps on being added. Most recent ones

○ Support for generic metadata added this summer ■ I.e. now Rucio can support whatever key:value pairs on DIDs. This was a requests of many collaborations during the 1st Rucio workshop ○ Support for archive files ■ This new feature allow to registers the constituents of an archive file ■ Rucio can extract automatically the constituent of the archive

• More to come soon :

○ New authentication/authorization based on Macaroon/SciTokens ○ Support of different QoS for the storages

2018-10-08 Rucio - Dune Workshop

Conclusion

• Rucio is an open source project, everybody can contribute.

○ Right now the core development team is composed from people from ATLAS and CMS

• All the members of the team also provide support
• Rucio is a Data Management solution that demonstrated that it can scale for large

collaboration O(1 EB)

• We’re very glad that CMS recently choose Rucio as their next Data Management

system and hope that more collaborations will follow this path

• As mentioned in previous talk, DUNE has already a Rucio prototype

2017-01-01

More information

13

http://rucio.cern.ch https://rucio.readthedocs.io https://github.com/rucio/ https://travis-ci.org/rucio/ https://hub.docker.com/r/rucio/ https://rucio.slack.com/messages/#support/ rucio-dev@cern.ch Website Documentation Repository Continuous Integration Images Online support Developer contact