phedex and cms data transfers
play

PhEDEx and CMS Data Transfers Paul Rossman Fermilab Global CMS - PowerPoint PPT Presentation

Jun 12, 2023 •178 likes •454 views

PhEDEx and CMS Data Transfers Paul Rossman Fermilab Global CMS Data Network Paul Rossman Fermilab PhEDEx and CMS Data Transfers 2 Dec 8, 2010 Global CMS Data Network Paul Rossman Fermilab PhEDEx and CMS Data Transfers 3 Dec 8, 2010

  1. PhEDEx and CMS Data Transfers Paul Rossman Fermilab

  2. Global CMS Data Network Paul Rossman Fermilab PhEDEx and CMS Data Transfers 2 Dec 8, 2010

  3. Global CMS Data Network Paul Rossman Fermilab PhEDEx and CMS Data Transfers 3 Dec 8, 2010

  4. Global CMS Data Network Paul Rossman Fermilab PhEDEx and CMS Data Transfers 4 Dec 8, 2010

  5. What is PhEDEx? ➨ PhEDEx is short for Ph ysics E xperiment D ata Ex port ➨ PhEDEx was developed to manage the movement of data within CMS • Centralized system for making global data movement decisions • Provides site and data managers and users a realtime view of the global CMS data transfer state ➨ PhEDEx automates for CMS many of low level tasks typically found in HEP experiments such as • large-scale data replication • tape migration • consistency ➨ In production since mid-2004 Paul Rossman Fermilab PhEDEx and CMS Data Transfers 5 Dec 8, 2010

  6. What is a PhEDEx agent? ➨ PhEDEx is composed of a series of autonomous, robust, persistent, stateless processes or in PhEDEx terms agents ➨ These agents share information about replica and transfer state through a database and other agents at CERN ➨ There are several agents, for example • FileDownload • FileExport • FileRemove • BlockDownloadVerify ➨ Not all sites require all agents • For instance, sites without a tape system do not require the FileStager agent Paul Rossman Fermilab PhEDEx and CMS Data Transfers 6 Dec 8, 2010

  7. What is a PhEDEx instance? ➨ Agents when combined form an instance ➨ There are currently 3 instances in use today • Production • Debug • Development ➨ Instances are independent and do not share information between each other Paul Rossman Fermilab PhEDEx and CMS Data Transfers 7 Dec 8, 2010

  8. Instances in-depth ➨ Production instance (Prod) • All CMS experiment data is transferred using this instance ➨ Debug instance (Debug) • Critical instance used for site commissioning and readiness • Tier-1 and Tier-2 sites transfer links can be decommissioned if transfer problems in this instance are not resolved • The primary focus of this instance was until recently to debug transfer agents and links ➨ Development instance (Dev) • Was historically used during testing and validation of PhEDEx software releases • Plans are being developed to re-task this instance to debugging Paul Rossman Fermilab PhEDEx and CMS Data Transfers 8 Dec 8, 2010

  9. Putting it all together CERN Dev Debug instances Production agents oracle database FileRouter Tier-1 Tier-2/3 BlockAllocator Dev TMDB Debug Dev FilePump + Production Debug others Production FileDownload(s) FileDownload(s) FileExport FileExport FileRemove FileRemove FileStager Paul Rossman Fermilab PhEDEx and CMS Data Transfers 9 Dec 8, 2010

  10. So you’re probably wondering... DATA Paul Rossman Fermilab PhEDEx and CMS Data Transfers 10 Dec 8, 2010

  11. Transfer Request Process determine where data exists submit Savannah no link exists? request submit transfer request no approved? no yes monitor transfer fixed? debug using yes problem? PhEDEx tools begin analysis Paul Rossman Fermilab PhEDEx and CMS Data Transfers 11 Dec 8, 2010

  12. Submit a transfer request ➨ https://cmsweb.cern.ch/phedex/prod/Request::Create submit transfer request ➨ Transfer and delete requests follow similar procedures Paul Rossman Fermilab PhEDEx and CMS Data Transfers 12 Dec 8, 2010

  13. Submit a request submit transfer request Paul Rossman Fermilab PhEDEx and CMS Data Transfers 13 Dec 8, 2010

  14. Find your request ➨ https://cmsweb.cern.ch/phedex/prod/Request::View monitor transfer ➨ You’ll find additional notes from approvers here Paul Rossman Fermilab PhEDEx and CMS Data Transfers 14 Dec 8, 2010

  15. Monitoring your transfer ➨ Web-based monitor transfer • https://cmsweb.cern.ch/phedex/ Paul Rossman Fermilab PhEDEx and CMS Data Transfers 15 Dec 8, 2010

  16. Monitoring your transfer ➨ https://cmsweb.cern.ch/phedex/prod/Data::Subscriptions monitor transfer Paul Rossman Fermilab PhEDEx and CMS Data Transfers 16 Dec 8, 2010

  17. Monitoring your transfer ➨ You can also find info on the replicas page monitor transfer ➨ https://cmsweb.cern.ch/phedex/prod/Data::Replicas Paul Rossman Fermilab PhEDEx and CMS Data Transfers 17 Dec 8, 2010

  18. Overall transfer quality monitor transfer Paul Rossman Fermilab PhEDEx and CMS Data Transfers 18 Dec 8, 2010

  19. Watching your rate monitor transfer Paul Rossman Fermilab PhEDEx and CMS Data Transfers 19 Dec 8, 2010

  20. Watching your rate monitor transfer Paul Rossman Fermilab PhEDEx and CMS Data Transfers 20 Dec 8, 2010

  21. Debugging your transfer ➨ https://cmsweb.cern.ch/phedex/prod/Activity::ErrorInfo debug using PhEDEx tools Paul Rossman Fermilab PhEDEx and CMS Data Transfers 21 Dec 8, 2010

  22. Monitoring your transfer ➨ Data Service monitor transfer • http://cmsweb.cern.ch/phedex/datasvc/doc • Programmatically query PhEDEx and retrieve data in xml, json or perl formats Paul Rossman Fermilab PhEDEx and CMS Data Transfers 22 Dec 8, 2010

  23. Using the data service ➨ Shell Example monitor transfer format instance api call curl http://cmsweb.cern.ch/phedex/datasvc/xml/prod/transferqueuefiles ?from=T1_US_FNAL_Buffer&to=T1_US_FNAL_MSS&priority=high&state=transferring options ➨ Perl Example #!/usr/bin/perl -w use XML::Simple; use LWP::Simple; use Data::Dumper; $xml = new XML::Simple; $data = $xml->XMLin( get("http://cmsweb.cern.ch/phedex/datasvc/xml/prod/transferqueuefiles to=T1_US_FNAL_Buffer&priority=normal&state=transferring"), forcearray => ["block"] ); print Dumper($data); Paul Rossman Fermilab PhEDEx and CMS Data Transfers 23 Dec 8, 2010

  24. PhEDEx support resources ➨ PhEDEx documentation and other documents can be found at • https://twiki.cern.ch/twiki/bin/view/CMS/PhEDEx ➨ Questions regarding PhEDEx can be posted to one of these HyperNews forums • hn-cms-phedex@cern.ch • hn-cms-osg-tier3@cern.ch ➨ Other transfer or site related questions can be posted to these • hn-cms-dataops@cern.ch • hn-cms-facilitiesOps@cern.ch ➨ Transfer problems should be posted to Savannah • https://savannah.cern.ch/projects/cmscompinfrasup/ ➨ Bug reports and feature requests should be posted to Savannah • https://savannah.cern.ch/projects/phedex/ ➨ Please make sure you have both a HyperNews and Savannah account before posting (please don’t post anonymously to Savannah) Paul Rossman Fermilab PhEDEx and CMS Data Transfers 24 Dec 8, 2010

  25. Additional resources and links ➨ Very good how-to links • http://hep-t3.physics.umd.edu/HowToForAdmins.html • http://hep.phys.utk.edu/wiki/PhEDEx_Installation ➨ Example configurations • http://cmssw.cvs.cern.ch/cgi-bin/cmssw.cgi/COMP/SITECONF/ T3_US_UMD/PhEDEx/ ➨ https://cmsweb.cern.ch/phedex/datasvc/doc ➨ https://cmsweb.cern.ch/overview/session ➨ There are many others, apologies for the omission! Paul Rossman Fermilab PhEDEx and CMS Data Transfers 25 Dec 8, 2010

  26. Questions? Paul Rossman Fermilab PhEDEx and CMS Data Transfers 26 Dec 8, 2010

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Data Management Network transfers Network data transfers Not everyone needs to transfer large

Data Management Network transfers Network data transfers Not everyone needs to transfer large

Data Management Network transfers Network data transfers Not everyone needs to transfer large amounts of data on and off a HPC service Sometimes data is created and consumed on the same service. If you do need to move large amounts of

336 views • 32 slides
Optimizing Explicit Data Transfers for Data Parallel Applications on Heterogeneous Multi-core

Optimizing Explicit Data Transfers for Data Parallel Applications on Heterogeneous Multi-core

Optimizing Explicit Data Transfers for Data Parallel Applications on Heterogeneous Multi-core Platforms S. Saidi 1 , 2 P.Tendulkar 1 T. Lepley 2 O. Maler 1 1 Verimag 2 STMicroelectronics Hipeac 2012 Saidi, Tendulkar, Lepley, Maler Data Transfers

1.15k views • 81 slides
Address Transfers in APNIC George Michaelson (Geoffs Data)

Address Transfers in APNIC George Michaelson (Geoffs Data)

Address Transfers in APNIC George Michaelson (Geoffs Data) History of Transfers: how many /24s History of Transfers - AcFvity ObservaFons AcFvity

318 views • 6 slides
PRACTITIONERS DEEDS AND TRANSFERS UNDER THE RPA Fraud in conveyances and transfers of property

PRACTITIONERS DEEDS AND TRANSFERS UNDER THE RPA Fraud in conveyances and transfers of property

TIPS FOR PRACTITIONERS DEEDS AND TRANSFERS UNDER THE RPA Fraud in conveyances and transfers of property under the rpo is increasing and attorneys should be cautious in reviewing deeds and transfers which constitute the title to a property

403 views • 39 slides
Data Transfers in the Grid: Data Transfers in the Grid: Workload Analysis of Globus Globus

Data Transfers in the Grid: Data Transfers in the Grid: Workload Analysis of Globus Globus

Data Transfers in the Grid: Data Transfers in the Grid: Workload Analysis of Globus Globus GridFTP Workload Analysis of GridFTP Nicolas Kourtellis, Lydia Prieto, Gustavo Zarrate, Adriana Iamnitchi Adriana Iamnitchi Nicolas Kourtellis, Lydia

965 views • 27 slides
Serial Peripheral Interface (SPI) Synchronous serial data transfers Multipoint serial

Serial Peripheral Interface (SPI) Synchronous serial data transfers Multipoint serial

Serial Peripheral Interface (SPI) Synchronous serial data transfers Multipoint serial communication between a master and a slave device Clock permits faster data rates than async communications (framing unnecessary)

337 views • 21 slides
Towards Optimizing Large-Scale Data Transfers with End-to-End Integrity Verification Raj

Towards Optimizing Large-Scale Data Transfers with End-to-End Integrity Verification Raj

Towards Optimizing Large-Scale Data Transfers with End-to-End Integrity Verification Raj Ke'muthu Argonne Na2onal Laboratory & University of Chicago Si Liu, Xian-He Sun, Illinois Ins2tute of Technology Eun-Sung Jung, Jongik University

215 views • 8 slides
Delay Tolerant Bulk Data Transfers on the Internet by N. Laoutaris et al., SIGMETRICS09 Ilias

Delay Tolerant Bulk Data Transfers on the Internet by N. Laoutaris et al., SIGMETRICS09 Ilias

Delay Tolerant Bulk Data Transfers on the Internet by N. Laoutaris et al., SIGMETRICS09 Ilias Giechaskiel Cambridge University, R212 ig305@cam.ac.uk March 4, 2014 Conclusions Takeaway Messages Need to transfer multiple terabytes daily

441 views • 21 slides
DATA TRANSFERS Maximillian Schrems v. Data Protection Commissioner Court of Justice of the

DATA TRANSFERS Maximillian Schrems v. Data Protection Commissioner Court of Justice of the

DATA TRANSFERS Maximillian Schrems v. Data Protection Commissioner Court of Justice of the European Union, C-362/14, 6 October 2015 In a landmark judgment the Court of Justice of the European Union (CJEU) followed the Attorney Generals

413 views • 3 slides
The Missing Transfers: Estimating Mis-reporting in Dyadic Data Margherita Comola Marcel

The Missing Transfers: Estimating Mis-reporting in Dyadic Data Margherita Comola Marcel

The Missing Transfers: Estimating Mis-reporting in Dyadic Data Margherita Comola Marcel Fafchamps Paris School of Economics Stanford University Comola and Fafchamps () Misreporting 1 / 23 Idea We have data on link ij = { 0 , 1 } between

364 views • 23 slides
Barriers in business transfers Intra-family and external transfers - Technical and non-technical

Barriers in business transfers Intra-family and external transfers - Technical and non-technical

Business Transfer Conference, Malta, March 17 2017 Barriers in business transfers Intra-family and external transfers - Technical and non-technical barriers - Leif Melin Most likely future owner in order of likeliness 1. Another operating

335 views • 16 slides
Nuclear Material Transfers Nuclear Material Transfers Project Four Waste streams

Nuclear Material Transfers Nuclear Material Transfers Project Four Waste streams

Harwell Nuclear Material Transfers Nuclear Material Transfers Project Four Waste streams Significant Lifetime cost savings Scope To Sellafield to NDA estate. Consolidation of NM on a Category 1 site (Sellafield) and 251

322 views • 7 slides
Shuffle Phase Executed only in the case of one or more reducers Transfers data between the

Shuffle Phase Executed only in the case of one or more reducers Transfers data between the

Shuffle Phase Executed only in the case of one or more reducers Transfers data between the mappers and reducers Groups records by their keys to ensure local processing in the reduce phase 01/23/2018 15 Shuffle Phase Map 1 Map 2 Map 3

102 views • 8 slides
Delay Tolerant Bulk Data Transfers on the Internet Nikolaos Laoutaris, Georgios Smaragdakis,

Delay Tolerant Bulk Data Transfers on the Internet Nikolaos Laoutaris, Georgios Smaragdakis,

Delay Tolerant Bulk Data Transfers on the Internet Nikolaos Laoutaris, Georgios Smaragdakis, Pablo Rodriguez, Ravi Sundaram Presented by Petko Georgiev 5 March 2013 Motivation There are important applications requiring exchange of Delay

367 views • 23 slides
Automatically Encapsulating HPC Best Practices Into Data Transfers Paul Z. Kolano NASA

Automatically Encapsulating HPC Best Practices Into Data Transfers Paul Z. Kolano NASA

National Aeronautics and Space Administration Automatically Encapsulating HPC Best Practices Into Data Transfers Paul Z. Kolano NASA Advanced Supercomputing Division paul.kolano@nasa.gov www.nasa.gov Outline of Presentation

423 views • 40 slides
Understanding JESD204B High-speed inter-device data transfers for SDR Lars-Peter Clausen

Understanding JESD204B High-speed inter-device data transfers for SDR Lars-Peter Clausen

Understanding JESD204B High-speed inter-device data transfers for SDR Lars-Peter Clausen Introduction JESD204 Standard Designed as high-speed serial data link between converter (ADC, DAC) and logic device Up to 32 lanes per link Up

617 views • 37 slides
Multiclass object recognition Sharing parts and transfer learning Sharat Chikkerur Outline

Multiclass object recognition Sharing parts and transfer learning Sharat Chikkerur Outline

Multiclass object recognition Sharing parts and transfer learning Sharat Chikkerur Outline Historical perspective and motivation Discriminative approach A. Torralba, K. Murphy, W. Freeman, Sharing visual features for multiclass and

456 views • 44 slides
Multi-Task Learning & Transfer Learning Basics CS 330 1 Logistics Homework 1 posted Monday

Multi-Task Learning & Transfer Learning Basics CS 330 1 Logistics Homework 1 posted Monday

Multi-Task Learning & Transfer Learning Basics CS 330 1 Logistics Homework 1 posted Monday 9/21 , due Wednesday 9/30 at midnight. TensorFlow review session tomorrow at 6:00 pm PT. Project guidelines posted early next week. 2 Plan for Today

996 views • 38 slides
globus online Simplify big data sharing with Globus Online Steve Tuecke Computation Institute

globus online Simplify big data sharing with Globus Online Steve Tuecke Computation Institute

globus online Simplify big data sharing with Globus Online Steve Tuecke Computation Institute University of Chicago and Argonne National Laboratory The Challenge: Moving and Sharing Big Data Easily What should be trivial I need my

459 views • 14 slides
Campaigning in Britain Justin Fisher (Brunel University) PARTY SYSTEMS IN THE UK Calculation of

Campaigning in Britain Justin Fisher (Brunel University) PARTY SYSTEMS IN THE UK Calculation of

Party Systems, Voters and Campaigning in Britain Justin Fisher (Brunel University) PARTY SYSTEMS IN THE UK Calculation of ENEP 1 p i 2 ENEP 2010 1 (.36 + .29 + .23 + .12) 2 = 3.55 BACK TO TWO DOMINANT PARTIES? IMPACT OF DEVOLUTION

816 views • 39 slides
pab stakeholder discussion January 30 and January 31 pab stakeholder goals and timeline

pab stakeholder discussion January 30 and January 31 pab stakeholder goals and timeline

pab stakeholder discussion January 30 and January 31 pab stakeholder goals and timeline Provide current CHFA PAB summary data Share 2020 PAB priorities Share ideas and solicit feedback from our partners and stakeholders related to

510 views • 16 slides
CS11-737: Multilingual Natural Language Processing Translation Yulia Tsvetkov Translation Mr.

CS11-737: Multilingual Natural Language Processing Translation Yulia Tsvetkov Translation Mr.

CS11-737: Multilingual Natural Language Processing Translation Yulia Tsvetkov Translation Mr. and Mrs. Dursley, who lived at El seor y la seora Dursley, que number 4 on Privet Drive, were proud vivan en el nmero 4 de Privet Drive,

771 views • 42 slides
Deep Transfer Mapping for Unsupervised Writer Adaptation Hong-Ming Yang 1,2 , Xu-Yao Zhang 1,2 ,

Deep Transfer Mapping for Unsupervised Writer Adaptation Hong-Ming Yang 1,2 , Xu-Yao Zhang 1,2 ,

Deep Transfer Mapping for Unsupervised Writer Adaptation Hong-Ming Yang 1,2 , Xu-Yao Zhang 1,2 , Fei Yin 1,2 , Jun Sun 4 , Cheng-Lin Liu 1,2,3 1 NLPR, Institute of Automation, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences 3

486 views • 19 slides
2018 U.S. Utility-Scale Photovoltaics-Plus-Energy Storage System Costs Benchmark Ran Fu, Timothy

2018 U.S. Utility-Scale Photovoltaics-Plus-Energy Storage System Costs Benchmark Ran Fu, Timothy

2018 U.S. Utility-Scale Photovoltaics-Plus-Energy Storage System Costs Benchmark Ran Fu, Timothy Remo, and Robert Margolis October 2018 NREL/PR-6A20-72401 1 Contents Introduction Standalone Energy Storage Cost Model PV Plus Energy

476 views • 24 slides

More recommend