McM Monte-Carlo Management Service Jean-Roch Vlimant for PdmV ( * ) - - PowerPoint PPT Presentation

McM Monte-Carlo Management Service

Jean-Roch Vlimant for PdmV(*) & Generator Groups

CMSDAS @ FNAL, January 2014 https://twiki.cern.ch/twiki/bin/view/CMS/SWGuideCMSDataAnalysisSchool2014GeneratorExerciseatFNAL https://twiki.cern.ch/twiki/bin/view/CMS/PdmVMcM https://cms-pdmv.cern.ch/mcm/ (production) https://cms-pdmv-dev.cern.ch/mcm/ (test instance)

(*) PdmV : Physics Data-MC Validation

1/5/14 McM @ CMSDAS, Jean-Roch Vlimant 2

Global Picture

Generator Contacts

Prepare requests Report special needs Keep an eye on production Operate at various steps

Production Manager

Operate McM Guide preparation of

requests Computing Operation

Receive well formatted

requests

Dispatch production to sites Follow through with sites

issues

Manage central space

McM

Provide validation Provide book-keeping Guide production

1/5/14 McM @ CMSDAS, Jean-Roch Vlimant 3

The Path To An Analysis Dataset

• Figure out the relevant generator parameters
• Possible modifications in simulation itself

➢ for better data/MC agreement

• Figure out the requirements in terms of digitization and reconstruction to match the

analyzed data

• All steps can be done in the same workflow, however

➢ Computing requirements  Generation and simulation mostly done at T2s  Digitization and reconstruction mostly done at T1s ➢ Chronology of production  Generation and simulation may start before digi-reco  Several digi-reco might be required for a given sample of generated events ➔ Steps are split in several processing workflows ➔ Each group of steps is organized in a campaign

1/5/14 McM @ CMSDAS, Jean-Roch Vlimant 4

The Concept of Chained Campaigns

1/5/14 McM @ CMSDAS, Jean-Roch Vlimant 5

Terminology

• Campaign

Campaign is a group of requests that share the same of very similar configuration, energy, software release series, physics performance …

✔ Example : Summer12 is for all gen-sim at 8TeV using the 5.3 release serie

• Request

Request is a specific member of a campaign for the production of a given dataset, being intermediate or for analysis (AOD)

✔ Example : TOP-Summer12-00177 is the request for production gen-sim dataset of

top pair production and decay in leptons, with top mass hypothesis of 173.5 GeV

• Flow

Flow is a connector between campaigns, it defines specific operation that needs to be done while using the output of one request as input to its next step

✔ Example : flowS12to53RD defines the necessary ingredients to have a “run

dependent” digi-reco of Summer12 gen-sim samples within Summer12DR53X campaign

• Chained Campaign

Chained Campaign is constructed from campaigns that are connected together with flows

✔ Example : chain_Summer12pLHE_flowLHE2S12_flowS12to53RD (aliased to

pLHE12DR53RD) is the chain of steps required to produce a run-dependent analysis dataset from a privately produced set of .lhe files

• Chained Requests

Chained Requests is a specific member of a chained campaign, made from existing requests.

✔ Example : HIG-chain_Summer12pLHE_flowLHE2S12_flowS12to53RD-00001 is

the chain of request required to produced the analysis dataset for a Higgs+top production with decay in a pair of photons

1/5/14 McM @ CMSDAS, Jean-Roch Vlimant 6

Terminology

• Campaign

Campaign

• Request

Request

• Flow

Flow

• Chained Campaign

Chained Campaign

• Chained Requests

Chained Requests

1/5/14 McM @ CMSDAS, Jean-Roch Vlimant 7

Why All This

• Flows and chained campaigns are overkills if there is only one way of producing

samples

✔ It used to be the case ✗ It is far from being true anymore (RD digi-reco, several pile-up, …)

• In the context of knowing what should happen for the production of a given dataset,

flows, chained campaigns and chained requests allow

✔ The production manager to know what is next the next step to be done ✔ The generator contact to keep an eye on the evolution of the production through

the different steps

✔ The user to have at a glance the history for the production of an analysis sample

• Not only McM is a production management tool, it also allows for book-keeping and

documentation

✔ Generator parameters can be updated at anytime ✔ Notes should contain details of the content of the sample if there are some

specificity

✔ The configuration files used for production is accessible ✔ The “cmsDriver” command is accessible

• McM offers extra protection to computing operation and the user themselves

✔ Runtest of the request ensures run-ability ✔ Validation step allows to scrutinize the generator content of a request

1/5/14 McM @ CMSDAS, Jean-Roch Vlimant 8

Evolution of a Chain of Requests

1/5/14 McM @ CMSDAS, Jean-Roch Vlimant 9

Practical Exercise : Request 1/2

• Required ingredients

 dsn : https://twiki.cern.ch/twiki/bin/view/CMS/ProductionDataSetNames ➔ Necessary to label the output datasets

/<dsn>/<some string>/<data tier>

 Time/event ➔ Mandatory to be accurate so as to not exceed job time-out in production  Size/event ➔ Required to be accurate to allocate enough space for the output dataset  Efficiencies ➔ Mandatory to be accurate so that the exact number of events get produced  Extension number ➔ Mandatory to guide the user in what need to be combined, and prevent dataset

• ver-writting in production

 Process string ➔ Mandatory when the request is slightly changed from the campaign by the user,

and need to be properly label /<dsn>/<some string containing the request process string>/<data tier>

• Insert a request in McM

➔ Create a new request and insert the gen-fragment from previous exercise ➔ Select to be provided with validation plots ➔ Then run the local test : to verify that everything is in order

• Verify requests values

➔ Edit with correct values : to have all ingredients accurately inserted

• Toggle “validation” (and move on to next step) : triggers the runtest to be

performed under McM

1/5/14 McM @ CMSDAS, Jean-Roch Vlimant 10

Practical Exercise : Request 2/2

• Prepare a request for processing the request towards an analysis dataset

➔ Insert an MccM document for the request : simplifies the report at MC coordination

meeting with exact ingredients

➔ Do not select the block # until really sure of the rest : to prevent the chains to be

generated automatically as part of the exercise

• When validation is finished (watch for notification and change of status)

➔ Inspect validation (requires cmsweb authentication) : verify that the generator

fragment is creating what is expected

➔ Toggle “define” : status defined means that it's read and validated

• Watch the requests getting automatically processed as part of the exercise

✔ Approved : the generator conveners has looked at the request and things seem to

be in order

✔ Chained : the production manager has acted on the MccM document to create the

required chains, and reserved the necessary requests

✔ Submitted : the production manager manually submitted the request or McM

submitted automatically once chained and approved

✔ Done : after processing by comp-ops, the output dataset is set to VALID and the

request is “done”. At warp speed for the purpose of the exercise, there is no real processing

1/5/14 McM @ CMSDAS, Jean-Roch Vlimant 11

Practical Exercise : Find Information

• Use the production instance https://cms-pdmv.cern.ch/mcm/

✔ Find the status of certain requests

➔ /TTJets_MSDecays_matchingdown_TuneZ2star_8TeV-madgraph-

tauola/Summer12_DR53X-PU_S10_START53_V19-v2/AODSIM

➔ /TTJets_MSDecays_mass178_5_TuneZ2star_8TeV-madgraph-

tauola/Summer12_DR53X-PU_S10_START53_V19-v1/AODSIM

✔ Find the Xsec of a given dataset

➔ /TTJets_SemiLeptDecays_8TeV-sherpa/Summer12_DR53X-

PU_S10_START53_V19-v1/AODSIM

✔ Find the generator fragment used to produce a given dataset

➔ /TprimeTprimeToTHBW_HToGammaGamma_M-900_TuneZ2star_8TeV-

madgraph/Summer12_DR53X-PU_S10_START53_V19-v1/AODSIM

✔ Find the filter efficiency of the physics process in a give dataset

➔ /Xibstar0ToXibPi_8TeV-pythia6-evtgen/Summer12_DR53X-

PU_S10_START53_V19-v1/AODSIM

✔ Find the “cmsDriver” command used to produce a give dataset

➔ /TTToHcWb_HToGG_8TeV-madgraph-pythia6/Summer12_DR53X-

PU_RD1_START53_V7N-v2/AODSIM

• Find the same info through DAS

➔ All but config file and cmsDriver can easily be obtained by having DAS

show the information stored in McM

1/5/14 McM @ CMSDAS, Jean-Roch Vlimant 12

Documentation

Monte Carlo Coordination Meeting https://twiki.cern.ch/twiki/bin/view/CMS/PdmVMccM Main Twiki https://twiki.cern.ch/twiki/bin/view/CMS/PdmVMcM Presentation and Tutorials https://twiki.cern.ch/twiki/bin/view/CMS/PdmVMcMTutorials

1/5/14 McM @ CMSDAS, Jean-Roch Vlimant 13

Summary of Exercise

• McM allows for submission of request to

central production under central prioritization

• McM provides means of validation of the

request content

• McM keeps track of the various required

step of production

• McM provides book-keeping and

documentation of the content of the analysis datasets

• McM information is aggregated and

accessible through DAS