PhEDEx and CMS Data Transfers Paul Rossman Fermilab Global CMS - - PowerPoint PPT Presentation

▶
phedex and cms data transfers
SMART_READER_LITE
LIVE PREVIEW

PhEDEx and CMS Data Transfers Paul Rossman Fermilab Global CMS - - PowerPoint PPT Presentation

Jun 12, 2023 178 likes •460 views

PhEDEx and CMS Data Transfers Paul Rossman Fermilab Global CMS Data Network Paul Rossman Fermilab PhEDEx and CMS Data Transfers 2 Dec 8, 2010 Global CMS Data Network Paul Rossman Fermilab PhEDEx and CMS Data Transfers 3 Dec 8, 2010

slide-1
SLIDE 1

PhEDEx and CMS Data Transfers

Paul Rossman Fermilab

slide-2
SLIDE 2

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Global CMS Data Network

2

slide-3
SLIDE 3

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Global CMS Data Network

3

slide-4
SLIDE 4

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Global CMS Data Network

4

slide-5
SLIDE 5

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

What is PhEDEx?

➨ PhEDEx is short for Physics Experiment Data Export ➨ PhEDEx was developed to manage the movement of data within CMS

  • Centralized system for making global data movement decisions
  • Provides site and data managers and users a realtime view of the

global CMS data transfer state ➨ PhEDEx automates for CMS many of low level tasks typically found in HEP experiments such as

  • large-scale data replication
  • tape migration
  • consistency

➨ In production since mid-2004

5

slide-6
SLIDE 6

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

What is a PhEDEx agent?

➨ PhEDEx is composed of a series of autonomous, robust, persistent, stateless processes or in PhEDEx terms agents ➨ These agents share information about replica and transfer state through a database and other agents at CERN ➨ There are several agents, for example

  • FileDownload
  • FileExport
  • FileRemove
  • BlockDownloadVerify

➨ Not all sites require all agents

  • For instance, sites without a tape system do not require the

FileStager agent

6

slide-7
SLIDE 7

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

What is a PhEDEx instance?

➨ Agents when combined form an instance ➨ There are currently 3 instances in use today

  • Production
  • Debug
  • Development

➨ Instances are independent and do not share information between each

  • ther

7

slide-8
SLIDE 8

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Instances in-depth

➨ Production instance (Prod)

  • All CMS experiment data is transferred using this instance

➨ Debug instance (Debug)

  • Critical instance used for site commissioning and readiness
  • Tier-1 and Tier-2 sites transfer links can be decommissioned if

transfer problems in this instance are not resolved

  • The primary focus of this instance was until recently to debug

transfer agents and links ➨ Development instance (Dev)

  • Was historically used during testing and validation of PhEDEx

software releases

  • Plans are being developed to re-task this instance to debugging

8

slide-9
SLIDE 9

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Putting it all together

9

Dev Debug Dev Debug Dev Debug Production TMDB FileRouter BlockAllocator FilePump +

  • thers

Production FileDownload(s) FileExport FileRemove FileStager Production FileDownload(s) FileExport FileRemove

Tier-2/3 Tier-1

CERN

agents instances

  • racle database
slide-10
SLIDE 10

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

So you’re probably wondering...

10

DATA

slide-11
SLIDE 11

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Transfer Request Process

11

determine where data exists submit transfer request submit Savannah request link exists? approved? monitor transfer problem? begin analysis

no no

debug using PhEDEx tools fixed?

yes no yes

slide-12
SLIDE 12

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Submit a transfer request

➨ https://cmsweb.cern.ch/phedex/prod/Request::Create ➨ Transfer and delete requests follow similar procedures

12

submit transfer request

slide-13
SLIDE 13

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Submit a request

13

submit transfer request

slide-14
SLIDE 14

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Find your request

➨ https://cmsweb.cern.ch/phedex/prod/Request::View ➨ You’ll find additional notes from approvers here

14

monitor transfer

slide-15
SLIDE 15

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Monitoring your transfer

➨ Web-based

  • https://cmsweb.cern.ch/phedex/

15

monitor transfer

slide-16
SLIDE 16

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Monitoring your transfer

➨ https://cmsweb.cern.ch/phedex/prod/Data::Subscriptions

16

monitor transfer

slide-17
SLIDE 17

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Monitoring your transfer

➨ You can also find info on the replicas page ➨ https://cmsweb.cern.ch/phedex/prod/Data::Replicas

17

monitor transfer

slide-18
SLIDE 18

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Overall transfer quality

18

monitor transfer

slide-19
SLIDE 19

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Watching your rate

19

monitor transfer

slide-20
SLIDE 20

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Watching your rate

20

monitor transfer

slide-21
SLIDE 21

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Debugging your transfer

➨ https://cmsweb.cern.ch/phedex/prod/Activity::ErrorInfo

21

debug using PhEDEx tools

slide-22
SLIDE 22

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Monitoring your transfer

➨ Data Service

  • http://cmsweb.cern.ch/phedex/datasvc/doc
  • Programmatically query PhEDEx and retrieve data in xml, json or

perl formats

22

monitor transfer

slide-23
SLIDE 23

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Using the data service

➨ Shell Example

curl http://cmsweb.cern.ch/phedex/datasvc/xml/prod/transferqueuefiles ?from=T1_US_FNAL_Buffer&to=T1_US_FNAL_MSS&priority=high&state=transferring

➨ Perl Example

#!/usr/bin/perl -w use XML::Simple; use LWP::Simple; use Data::Dumper; $xml = new XML::Simple; $data = $xml->XMLin( get("http://cmsweb.cern.ch/phedex/datasvc/xml/prod/transferqueuefiles to=T1_US_FNAL_Buffer&priority=normal&state=transferring"), forcearray => ["block"] ); print Dumper($data);

23

format instance api call

  • ptions

monitor transfer

slide-24
SLIDE 24

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

PhEDEx support resources

➨ PhEDEx documentation and other documents can be found at

  • https://twiki.cern.ch/twiki/bin/view/CMS/PhEDEx

➨ Questions regarding PhEDEx can be posted to one of these HyperNews forums

  • hn-cms-phedex@cern.ch
  • hn-cms-osg-tier3@cern.ch

➨ Other transfer or site related questions can be posted to these

  • hn-cms-dataops@cern.ch
  • hn-cms-facilitiesOps@cern.ch

➨ Transfer problems should be posted to Savannah

  • https://savannah.cern.ch/projects/cmscompinfrasup/

➨ Bug reports and feature requests should be posted to Savannah

  • https://savannah.cern.ch/projects/phedex/

➨ Please make sure you have both a HyperNews and Savannah account before posting (please don’t post anonymously to Savannah)

24

slide-25
SLIDE 25

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Additional resources and links

➨ Very good how-to links

  • http://hep-t3.physics.umd.edu/HowToForAdmins.html
  • http://hep.phys.utk.edu/wiki/PhEDEx_Installation

➨ Example configurations

  • http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/SITECONF/

T3_US_UMD/PhEDEx/ ➨ https://cmsweb.cern.ch/phedex/datasvc/doc ➨ https://cmsweb.cern.ch/overview/session ➨ There are many others, apologies for the omission!

25

slide-26
SLIDE 26

Paul Rossman Fermilab PhEDEx and CMS Data Transfers Dec 8, 2010

Questions?

26