CDF Data production model CDF Data production model S. Hou S. Hou - - PowerPoint PPT Presentation

▶
cdf data production model cdf data production model
SMART_READER_LITE
LIVE PREVIEW

CDF Data production model CDF Data production model S. Hou S. Hou - - PowerPoint PPT Presentation

Aug 28, 2022 429 likes •762 views

CDF Data production model CDF Data production model S. Hou S. Hou for the CDF data production team for the CDF data production team 02 May 2006 02 May 2006 CDF data production model 2 Outline Outline Data streams - trigger, streaming,

slide-1
SLIDE 1

CDF Data production model CDF Data production model

  • S. Hou
  • S. Hou

for the CDF data production team for the CDF data production team 02 May 2006 02 May 2006

slide-2
SLIDE 2

CDF data production model

2

Outline Outline

Data streams

  • trigger, streaming, data logging

Computing model

  • architecture, CAF linux farms, SAM data-handling

Production tasks

  • submission, concatenation

Monitoring and Bookkeeping

  • resource, file counting, recovery

Scalability

  • capacity, limits, scaling options
slide-3
SLIDE 3

CDF data production model

3

CDF CDF collaboration

collaboration

Collider Detector experiment at the Fermilab Tevatron collider

  • Study proton-antiproton collisions at CM energy ~ 2 TeV
  • Large data volume, computing load
slide-4
SLIDE 4

CDF data production model

4

Trigger, detector data flow Trigger, detector data flow

3 Level trigger/data buffer 52 physics triggers CDF detector data taking capacity

2006 upgrade 2005 Achieved

40 MB/s 20 MB/s to Tape storage rate :

150 Hz 110 Hz Level-3 acceptance : 500 MB/s 850X0.2 MB/s Event Builder (EVB) : 1 kHz 850 Hz Level-2 acceptance : 40 kHz 27 kHz Level-1 acceptance : 3x1032 cm-2s-1 1.8x1032 cm-2s-1 Tevatron luminosity :

Event size : ~140 kByte ‘06 data taking rate ~ 5 M events/day Upgrade to improve DAQ efficiency

slide-5
SLIDE 5

CDF data production model

5

Streams, data logging Streams, data logging

Consumers 8 Streams:

A B C,D E,J G H

Consumer server/Logger (CSL)

  • receive physics events
  • write to disks in 8 streams
  • distribute to online consumers

An event may have multiple triggers, Stream overlap ~ 5% increase with Tevatron luminosity

Data in 52 triggers

slide-6
SLIDE 6

CDF data production model

6

Data logging rate Data logging rate

Data logging rate up to Nov 2005 1.3 fb-1 of data written to tape Data logging rate increase w. Tevatron luminosity Good-run physics data

Feb 2002 - Dec 2004 1040 M events = 210 k files = 188 TByte Dec 2004 - Feb 2006 1270 M events = 172 k files = 159 TByte

1.6 fb-1 delivered by Tevatron 1.3 fb-1 in tape!

slide-7
SLIDE 7

CDF data production model

7

CDF CDF computing model,

computing model, ‘ ‘06 06

CDF DAQ Production farm Enstore CDF Analysis Farm remote remote remote CAFs CAFs CAFs

User desk top

dCache raw raw datasets datasets production production datasets datasets

slide-8
SLIDE 8

CDF data production model

8

Computing network, Computing network, ‘ ‘06 06

dCache

file-servers

10Gbit

2Gbit

Remote sites Analysis Analysis farm farm

Production Production farm farm

Enstore

tape library File-servers Servers

CDF Online DAQ

2Gbit Oracle DB

  • ffline users
slide-9
SLIDE 9

CDF data production model

9

Production data flow Production data flow

Level-3 farm

Level-1,2 Trigger, DAQ

sub-detector

DataBase

Calibration

8 raw-datasets 52 physics datasets

Run splitter File catalog

Split data in production 8 raw data streams 52 physics datasets Final storage Enstore tape library STK 9940B drives 200 GB/tape 30 MByte/s read/write Steady R/W rate ~1TByte/drive/day

dCache CAF, fileservers

slide-10
SLIDE 10

CDF data production model

10

In service 2000-2004 Direct I/O to Enstore tape

  • Custom I/O node to Enstore

FBS batch system

  • dfarm collection of all worker IDE

buffer of input and output files

Farm Processing system

  • MySQL for bookkeeping
  • Concatenation in rigid event order
  • utput truncated to 1 GB files

Performance

  • Peak rate at 1 TB input/day

Data production, 1 Data production, 1st

st model

model

dfarm

network MySQL,DB run-splitter

calibration Register concatenated

5 5 1 1 2 2 3 3

worker stager concatenator

4 4 6 6

Register

  • utput

Register input

slide-11
SLIDE 11

CDF data production model

11

SAM SAM-

  • farm upgrade,

farm upgrade, ‘ ‘05 05

to CAF & SAM Data Handling to CAF & SAM Data Handling

Toward a distributed computing infrastructure CAF (CDF Analysis Farm) Condor system with CAF interface for job submission and monitoring Advantage:

  • uniform platform to other CDF computing facilities
  • compatible to distributed computing development

SAM Data handling system SAM (Sequential Access via Metadata) file delivery and DB service dCache virtualizes disk usage

slide-12
SLIDE 12

CDF data production model

12

SAM production farm SAM production farm

  • utput

merged

4 4

fileserver network SAM,DB input-URL run-splitter calibration Declare/update metadata worker

2 2 1 1 3 3 5 5

dCache

CAF/SAM in parallel :

  • SAM Project
  • Activating file delivery of

an assigned SAM dataset

  • Tracking file consumption status
  • Condor batch Job
  • Consume files in SAM project
  • update/declare SAM metadata

for bookkeeping

Concatenation of output

Merge output files sorted in run sequence

Store to Enstore via SAM

Declare metadata, update file parentage for bookkeeping

slide-13
SLIDE 13

CDF data production model

13

P Production challenge

roduction challenge

Operation

Resource monitoring Automatic submission and monitoring

  • 1. binary jobs of SAM projects on CAF farms
  • 2. concatenation on Fileservers
  • 3. store to SAM/Enstore

Service interface

Network, Enstore tape I/O dCache, SAM Data handling, DB service CDF online, calibration DB, software

Timely process every event collected Interface to Data-handing, DataBase, multiple CAF’s Precision bookkeeping on millions of files zero tolerance to error, every event is counted

slide-14
SLIDE 14

CDF data production model

14

Use cases in production Use cases in production

Fast beam-line calibration :

immediately after data is available on Enstore Raw-data Histograms concatenation

Detector Monitoring :

quick detector feedback and good-run definition immediate after beam-line is available Raw-data production/Enstore Histograms

Physics calibration :

statistics required for chosen events Raw-data Histograms

Physics production :

  • Raw-data Multiple outputs concatenation Enstore
  • Production files Single output concatenation Enstore

Multiple outputs concatenation Enstore

slide-15
SLIDE 15

CDF data production model

15

SAM projects in production SAM projects in production

Cron jobs accessing SAM DB

  • 1. Check online DB, make same SAM input data sets
  • 2. Submit SAM projects to condor CAF
  • 3. Merge output files and samStore on fileserver

ProExe merge

SAM

  • peration

node

query nextfile declare declare

raw merged reco-children

samStore Input dataset

Online DB good_runs

1. 3.

gphysr_… gphysr_runXX gphysr_runXX gX… gXjs00 gXcrs0

reco.gphysr_..

Control metadata Input datasets physics-datasets

2.

slide-16
SLIDE 16

CDF data production model

16

Data handling in production Data handling in production

Independent cron jobs on operation node/ fileservers

  • 1. Submit a SAM project / CAF job,

fetch files in input dataset

  • 2. Concatenation on fileserver
  • 3. samStore to Enstore
  • 1. ProExe
  • 2. merge

condor CAF

R/W dCache

reco merged

  • 3. samStore

/pnfs /dCache /samcache /pnfs

slide-17
SLIDE 17

CDF data production model

17

Each CPU take one job CAF headnode dispatch ProdExe tarball (self contained with all libs, 140 MB) Production script

  • 1. Fetch one input file in assigned SAM dataset
  • 2. Binary execution split table

calibration

  • 3. Declear split outputs
  • 4. Copy to concatenation area
  • 5. Update bookkeeping
  • 6. Cleanup

~4 hours per file (1 ~4 hours per file (1 GByte GByte) )

  • n 1 GHz P3
  • n 1 GHz P3

Binary jobs on Condor worker Binary jobs on Condor worker

CAF headnode Unpack Unpack tarball tarball Worker Worker Scratch Scratch area area SAM DB dCache input Calibration DB Output to Concatenation

slide-18
SLIDE 18

CDF data production model

18

Local on fileservers to reduce IDE, network bandwidth Independent from production submission

  • 1. SAM DB query, make files lists in order
  • f a dataset, size varies 5MB to 1GB,
  • utput size ~1 GByte
  • 2. Merge, rootd binary, ~3 min per GByte
  • 3. SAM DB update, declared merged files

SAMstore merged files

  • Directly to Enstore
  • SAM DB update file parentage

Challenge is on Bookkeeping

  • Plural SAM DB query
  • No data loss
  • No duplication

100% exact in produciton Easy recovery

Concatenation / SAMstore Concatenation / SAMstore

Production Production 52 datasets 52 datasets 8 streams Concatenation Concatenation SAMstore SAMstore

slide-19
SLIDE 19

CDF data production model

19

R Resource monitoring

esource monitoring

CDF DB, SAM DB, Data-Handling CAF condor batch system Fileserver storage Prohibited jobs missing required services

slide-20
SLIDE 20

CDF data production model

20

CAF farm monitoring CAF farm monitoring

Worker CPUs (Ganglia)

& input (rcp) waiting

Traffic to fileserver (xfs)

Bandwidth limit :

Input: Enstore loading to dCache Output: multiple workers to fileservers 1Gbit network port to IDE: 40 MB/s 1output dataset to Enstore: 30 MB/s

slide-21
SLIDE 21

CDF data production model

21

CAF condor monitoring CAF condor monitoring

Tarball (archived execution binary file) distributed to worker CPUs Input files copied via SAM from dCache End of job, output files are copied to assigned fileserver

CPU engagement is monitore CPU engagement is monitored d

Commnads Commnads

  • executed now

executed now

  • CPU of a section

CPU of a section

  • CPU

CPU’ ’s of a CAF job s of a CAF job

slide-22
SLIDE 22

CDF data production model

22

SAM project monitor SAM project monitor

Input is delivered by SAM Data-Handling system

  • Input files are organized in datasets
  • Each data-set is submitted to a SAM project
  • Each project is associated with a CAF condor job

List of projects List of projects

  • Cumulative

Cumulative file consumption file consumption

slide-23
SLIDE 23

CDF data production model

23

Monitor a SAM project Monitor a SAM project

Consumption of a dataset is monitored

  • File delivery by SAM from registered locations (dCache, samCache, Enstore etc)
  • Consumption by CAF worker is monitored
  • status of

status of file consumption file consumption

  • f a project
  • f a project

List of files/ List of files/ parantage parantage in a dataset in a dataset

slide-24
SLIDE 24

CDF data production model

24

Bookkeeping / Bookkeeping / SAM metadata

SAM metadata

Each file created in production has a metadata Parent-daughter is updated after concatenation and SAMstore Metadata is used for bookkeeping SAM query on metadata tabulated and counted

slide-25
SLIDE 25

CDF data production model

25

Recovery Recovery

SAM input datasets are tabulated

  • 1. SAM/CAF submit:

automatic for datasets with incomplete daughters are

  • 2. Updated after concatenation / SAMstore

to complete the production tasks of a input dataset

slide-26
SLIDE 26

CDF data production model

26

P Production rate

roduction rate

SAM farm peak performance: Jobs distributed to two CAF’s (Analysis & Production farm) use 540 CPU for 6 physics streams 8 dCache input file servers, 6 output fileservers stable processing speed at 25 M events/day

(~5 time CDF DAQ ‘05)

3 TB input, 4 TB output /day

(output has 20% overlap in 52 datasets, 15% compressed H,J streams) Integrated Output event logging Daily file consumption

slide-27
SLIDE 27

CDF data production model

27

Stability in production Stability in production

CAF condor is very reliable worker hardware failure occasional RAID down-graded occasional Service 24x7 Oracle, Enstore service SAM, dCache shift support Produciton in parallel 6 streams, output to 6 Fileserver Rougher CPU usage at the end as streams were finishing up

CAF+Farm max=540 jobs Farm CPU

Traffic to/from Production farm

GREEN In bits/sec BLUE Out bits/sec DARK Peak In bits/sec PINK Peak Out bits/sec

slide-28
SLIDE 28

CDF data production model

28

Bottlenecks Bottlenecks

reco merged

Production farm

dCache 10Gbit

2Gbit 1Gbit each

Server Port and IDE speed : 1 Gbit peak ~ 50 MB/s IDE peak ~ 40 MB/s matching to ~ 100 CPU max Enstore single dataset write :

single mover, 30 MB/s instant, ~ 1 TB/day

dual P3 server : 2 TB

network av. 50 MB/s I+O concatenation CPU : 1GB/3min/1CPU ~ 1 TB/day

dual Xeon server : 8 TB

network av. 100 MB/s I+O concatenation CPU : 1GB/1.5min/1CPU ~ 2 TB/day Unmatched port ratio

slide-29
SLIDE 29

CDF data production model

29

Scaling capacity Scaling capacity

At peak performance of 3 TB input, 4 TB output /day

farm switch (2Gbit capacity) average load is 800 Mbit/s limited by the fileserver Gbit links (40 MB/s each)

Scaling on CPU

More CPU in a CAF

  • r more CAF’s

Scaling on network I/O

more streams in parallel production more fileservers (more Gbit links) Eventual limit is the tape drives

slide-30
SLIDE 30

CDF data production model

30

Upgrade to GRID Upgrade to GRID

Little changes required for submission to GRID Operation

Bookkeeping stay with SAM Production scripts are portable can be multiplied Binary Tarball self-sustained, grid compatible

Services

Data-handling SAMGrid modify dCache copy CAF OSG-CAF modify batch-submit

slide-31
SLIDE 31

CDF data production model

31

Summary Summary

CDF data production is stable, currently at 3 TByte/day Capacity is scalable by increasing CPU, I/O ports, storage Zero data loss tolerant to error, and easy recovery