CMS Plans and Strategy for Physics Analysis on the Grids Lothar A. - - PowerPoint PPT Presentation
CMS CMS CMS Plans and Strategy for Physics Analysis on the Grids Lothar A. T. Bauerdick/Fermilab Invited Talk at the International Symposium for Grid Computing 2006 Academia Sinica, Taipei, May 2006 LATBauerdick Fermilab ISGC 2006 CMS
f
ISGC 2006 — CMS Analysis May 2, 2006 Contents of Talk ✦ Introduction: ★ Worldwide LHC Computing Grid should be ready for Physics Soon! ✦ CMS Computing and Data Model ★ computing tiers, data tiers, data structures ✦ Data Analysis Strategy ★ Analysis Process and Model ✦ First experience ★ CMS data analysis on the grid using the CRAB system ✦ first attempts and many open questions ✦ Acknowledgment ★ many slides lifted from CMS talks at the recent CHEP06 in Mumbai! 2 f
ISGC 2006 — CMS Analysis May 2, 2006 3 LHC startup plan L=3x1028 - 2x1031 Stage 1 Initial commissioning 43x43 to 156x156, N=3x1010 Zero to partial squeeze Stage 2 75 ns operation 936x936, N=3-4x1010 partial squeeze L=7x1032 - 2x1033 L=1032 - 4x1032 Stage 3 25 ns operation 2808x2808, N=3-5x1010 partial to near full squeeze f
ISGC 2006 — CMS Analysis May 2, 2006 LHC First Physics Run in 2008 4 Integrated luminosity with the current LHC plans Run 2008 1.E-01 1.E+00 1.E+01 1.E+02 1.E+03 1.E+04 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 weeks luminosity (10**30 cm-2 sec-1) integrated luminosity (pb-1) events/crossing 1031 Lumi (cm-2s-1) 1032 1033 1.9 fb 1.9 fb-1 f
ISGC 2006 — CMS Analysis May 2, 2006 LHC First Physics Run in 2008 4 Integrated luminosity with the current LHC plans Run 2008 1.E-01 1.E+00 1.E+01 1.E+02 1.E+03 1.E+04 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 weeks luminosity (10**30 cm-2 sec-1) integrated luminosity (pb-1) events/crossing 1031 Lumi (cm-2s-1) 1032 1033 1.9 fb 1.9 fb-1 f
ISGC 2006 — CMS Analysis May 2, 2006 Pilot Run 5 ✦ 30 days, maybe less (?); 43 x 43 bunches, then 156 x 156 bunches ✦ jets and IVB production — 15 pb-1 ==> 30K W’s and 4K Zs into leptons ✦ Measure cross sections, W and Z charge asymmetry (pdfs; IVB+jet production; top!) PILOT RUN 1.E-04 1.E-03 1.E-02 1.E-01 1.E+00 1.E+01 1.E+02 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 DAYS luminosity (10**30 cm-2 sec-1) integrated luminosity (pb-1)" events/crossing 1029 1030 1028 1031 Lumi (cm-2s-1) Pile-up f
ISGC 2006 — CMS Analysis May 2, 2006 6 Distributed Computing Better Be Delivering on Time! ✦ Last year of preparation for Grid computing for LHC to work ★ computing resources are geographically distributed, interconnected via high throughput networks and operated by means of Grid software ★ WLCG systems is still very fragile, but it is functional and being used ★ Tier-0, Tier-1, Tier-2 and CAFs all are essential for success ✦ Large Aggregate Computing Resources Required: ★ in 2008 CMS requests total of 45 MSI2k CPU, 14 PB disk, 24 PB tape ➡ CMS computing model document (CERN-LHCC-2004-035) ➡ CMS computing Technical Design Report (CERN-LHCC-2005-023) f
ISGC 2006 — CMS Analysis May 2, 2006 Large Computing Resource Pledges 7 ✦ Seven Tier-1 centers catering to CMS ★ ASGC amongst them! ★ Still, CMS did not get sufficient pledges for data storage at Tier-1 centers ➡ not enough tape library space to store all 2008 event and simulated data ✦ Most of the CMS Data Analysis will happen at Tier-2 centers ★ Tier-2 resource situation looks good! 50 MSI2k ~ 10,000 nodes! ★ some 30 Tier-2 sites are offering computing resources to CMS ➡ most of them listed in the WLCG-MoU ★ some eleven Tier-2s already working actively f
ISGC 2006 — CMS Analysis May 2, 2006 Principles of CMS Computing Model ✦ Emphasizing the inherent structure of CMS data and data access ★ Structured Data and Structured Grids ✦ Data Granularity and Data Tiers ★ optimize sequential data access to well-defined Data Tiers ➡ eliminate object database philosophy from Event Data Model ★ Data always needs to be considered in its trigger context -> trigger paths ➡ O(2PB)/yr raw data split into O(50) (40TB) trigger-determined datasets ✦ Computing Tiers and hierarchical Data Grid ★ map data flow and data handling functions to a hierarchical structure ➡ event data flows Tier-0 —> Tier-1 —> Tier-2, data being analyzed at Tier-2 ★ facilitates well-defined roles and visible responsibilities for centers ✦ Building ability to prioritize is very important ★ In 2007/8, computing system efficiency may not be 100%.... 8 f
ISGC 2006 — CMS Analysis May 2, 2006 9 Computing Tiers Tier-0: ✦ Accepts data from DAQ ✦ Prompt reconstruction ✦ Data archive and distribution to Tier-1’s Tier-1’s: f
ISGC 2006 — CMS Analysis May 2, 2006 10 Computing Tiers CMS-CAF Analysis Facility at CERN f
ISGC 2006 — CMS Analysis May 2, 2006 11 CMS Data Analysis at CMS-CAF ✦ LHC running time is precious ★ Require short latency feedback and fast turnaround: hours, not days ✦ fast, efficient monitoring of data quality, trigger quality ★ With ad-hoc study of detector data (special data streams) ★ With a few critical analysis that verify physics (masses, cross sections) ✦ Calibration and Alignment ★ Require fast turn-around for Tier-0 and (potentially) the online filter farm ✦ Physics Assurance and Analysis ★ Are we seeing something unexpected (background or signal) that calls for trigger adjustment now ? Rapid analysis of ‘express-line’ physics without initially having to rely on a fully functional and perfectly operating Grid. ✦ As the experiment matures, in 2010 and beyond, some CAF responsibilities can be farmed out to T1s or T2s, but not during the startup phase. f
ISGC 2006 — CMS Analysis May 2, 2006 12 Data Tiers and Data Volume for 2008 ✦ RAW ➡ Detector data + L1, HLT results after online formatting ➡ Includes factors for poor understanding of detector, compression, etc ➡ 1.5MB/evt @ ~150 Hz; ~ 4.5 PB/year (two copies) ✦ RECO ➡ Reconstructed objects with their associated hits ➡ 250kB/evt; ~2.1 PB/year (incl. 3 reprocessing versions) ➡ Supports pattern recognition, recalibration, Root-browsable, for interactive analysis ✦ FEVT=RAW+RECO ➡ ~1.75MB/event, to keep RAW and RECO together for data handling ➡ 1 copy at Tier-0 and one spread over all Tier-1’s ✦ AOD ➡ The main analysis format; fragment of RECO for analysis: objects + minimal hit info ➡ 50kB/evt; ~2.6PB/year - whole copy at each Tier-1 ➡ shipped out to all T1s and on demand to T2 and laptops ✦ Should be inclusive so that all groups can use it. ✦ may allow some re-calibration and re-alignment (refit) ✦ Plus MC in ~ 1:1 ratio with data f
ISGC 2006 — CMS Analysis May 2, 2006 13 Event Data Model RECO/contains AOD ✦ “Event” holds all data taken during triggered physics event ✦ Provenance of each Event Data Product is being tracked ✦ Persistent objects designed such that they provide useful analysis information without needing external information ✦ Event Tiers: FEVT contains RAW and RECO which includes AOD Tracks TracksExtra TracksHits TrackDigis TrackRaw Electrons BasicClusters Cells EcalDigis EcalRaw SuperClusters Photons KtJets ConeJets … CaloTowers HcalDigis HcalRaw For Tracking for E/Gamma For 4 Jets RAW (includes Digi f
ISGC 2006 — CMS Analysis May 2, 2006 14 CMS Data in ROOT Format ✦ Goal: one format, one program for all (reconstruction, analysis) ★ Store “simple” structures that are browse-able by plain ROOT; ★ And then: load CMSSW classes and act on data as in a “batch”/”reconstruction” job ➡ Same jet-finding; muon-matching code; cluster corrections ➡ Issue is what data is available (RAW, RECO, AOD) gSystem>Load("libPhysicsToolsFWLite") AutoLibraryLoader::enable() TFile f("reco.root") Events.Draw("Tracks.phi()- TrackExtra.outerPhi(): Tracks.pt()", "Tracks.pt()<10", "box") f
ISGC 2006 — CMS Analysis May 2, 2006 15 CMS has Re-done Event Data Model f
ISGC 2006 — CMS Analysis May 2, 2006 16 Data Structured For Sequential Access: Primary Datasets ✦ membership of event to a Primary Datasets determined immutably by “trigger path” — real-time decision during data taking ★ allows to classify events into separate datasets ✦ Trigger Path is the “name” of the sequence of requirements that resulted in an event being recorded: ★ Level-1 Trigger accept ★ Higher Level Trigger confirmation ✦ All data analysis based on these trigger paths! ✦ We foresee about 50 primary datasets to be identified during the HLT processing, each following a strict trigger path ★ How many will CMS need? ★ Serious Physics Coordination issue! f
ISGC 2006 — CMS Analysis May 2, 2006 Using The System 17 14 CHEP06, Mumbai, India f
ISGC 2006 — CMS Analysis May 2, 2006 18 Analysis Process RECO/AOD Datasets AOD Preselection Plot Official CMS format tracks, muons, electrons, … Official CMS analysis format Complete datasets Specific content for different f
ISGC 2006 — CMS Analysis May 2, 2006 19 Another Example RECO/AOD Datasets AOD pre Cand, User Data AOD Signal dataset Background dataset(s) pre Cand, User Data At Tier 1/ Tier2 At Tier 0/ Tier1 AOD, Cand AOD, Cand pre pre At Tier 2 Laptop ? 500 GB 50 GB Guessed numbers at start-up 10 times more already possible f
ISGC 2006 — CMS Analysis May 2, 2006 20 How Will The User Interact? ★ Setup: authenticate to system, which creates the analysis context ensure the correct installation of software packages ★ Determine data sets and eventually event components ➡ Input data are selected via a query to a metadata catalogue ★ Perform iterative analysis activity ➡ pass selection and algorithm together with spec of the execution environment to a workload management system, ➡ Algorithms are executed on one or many nodes ➡ User monitors progress of job execution ➡ Results are gathered together and passed back to the job owner ➡ Resulting datasets can be published to be accessible to other users ARDA Scenario ca 2003 f
ISGC 2006 — CMS Analysis May 2, 2006 21 Distributed Analysis Scenarios ★ System selects dataset(s) from the CMS Dataset Bookkeeping System using metadata conditions provided by the user. ★ System splits the tasks and sends to the location of data sets. ➡ DM system decides data replication to avoid hot-spots ★ Spawn sub-jobs and submit to Workload Management with precise job descriptions ➡ User can control the results while and after data are processed ★ Collect and Merge available results from all terminated sub-jobs on request ★ Analysis objects associated with the analysis task remains persistent in the Grid environment so the user can go offline and reload an analysis task at a later date, check the status, merge current results or resubmit the same task with modified analysis code ARDA Scenario ca 2003 f
ISGC 2006 — CMS Analysis May 2, 2006 Data Analysis on the Grids: CRAB ✦ CRAB “CMS Remote Analysis Builder” ★ CMS Application Service Layer interfacing to Grid Middleware ★ CRAB encapsulates data discovery, job creation and submission ✦ Makes it easy to create large number of user analysis job ★ Assume all jobs are the same except for some parameters (event number to be accessed, output file name…) ✦ Allows to access distributed data efficiently ★ Hiding WLCG middleware complications. All interactions are transparent for the end user ✦ Manages job submission, tracking, monitoring, output harvesting ★ User doesn’t have to take care about how to interact with sometimes complicated grid commands ✦ Uses BOSS as Grid independent submission engine 22 f
ISGC 2006 — CMS Analysis May 2, 2006 23 ✦ Data discovery ★ Data are distributed, so need to know where data is interacting with DM ✦ Job creation ★ Both .sh (wrapper script for the real executable) and .jdl (a script which drives the real job towards the “grid”) ★ User parameters passed via config file ✦ Job submission ★ Scripts are ready to be sent to those sites which host data ★ Submitting jobs to the Resource Broker or Condor-G(on OSG) via BOSS ✦ CRAB monitors, via BOSS, the status of the whole submission ★ The user has to ask for jobs status ✦ When jobs finish CRAB retrieves all output ★ Both standard output/error and relevant files produced by analysis code ★ Either the job copies the output to the SE ★ Or it takes it back to the UI Main CRAB functionalities f
ISGC 2006 — CMS Analysis May 2, 2006 24 Most accessed sites since July 05 ✦ The Systems is already providing analysis capabilities to CMS ★ many ten thousands of pTDR analysis jobs run successfully ★ Tier-1 centers gain operational experience (not yet so much Tier-2s) ➡ subset of really reliable and responsive sites used in pTDR production ★ good fraction of Tier-2s acting fast to become analysis sites for CMS Physics Users are using the Grid! f
ISGC 2006 — CMS Analysis May 2, 2006 25 Distributed Analysis Scenarios ★ Communities of Scientists Using the Grid for Distributed Analysis ★ Infrastructure for sharing, consistency of physics and calibration data, software f
ISGC 2006 — CMS Analysis May 2, 2006 26 Future Distributed Analysis Scenarios ★ Persistency of the (virtual) analysis workspace that a user can later re-connect to, re-submit the analysis with modified parameters or code, check the status, merge results between analyses, share datasets with f
ISGC 2006 — CMS Analysis May 2, 2006 27 ✦ Focusing on data analysis requires a significant paradigm shift ★ from well-defined production jobs —> to interactive user analysis ★ from DAGs of process —> to “Sessions” and state-full environments ★ from producing “sets of files” —> to accessing massive amounts of data ★ from files —> data sets and collection of objects ★ from using essentially “raw data” —> to complex layers of event data tiers ★ from “assignments” from a central repository —> to Grid-wide queries ★ from “user registration” —> to enabling sharing and building communities ✦ Are the (Grid) technologies ready for this? ★ currently, Grid middleware functionality usage is basic — and WLCG focus is on robustness and performance, not functionality ✦ What will the “new paradigms” be that are exposed to the user? ★ local user “data analysis session” transparently extended to Grid? ➡ but requires a more prescriptive and declarative approach to analysis ★ set of services for “collaborative” work? ➡ new paradigms beyond “analysis” Distributed Analysis: Still Requires a Shift of Focus