dCache @ Fermilab Dmitry Litvintsev Dmitry Litvintsev, 6th - - PowerPoint PPT Presentation

▶
dcache fermilab
SMART_READER_LITE
LIVE PREVIEW

dCache @ Fermilab Dmitry Litvintsev Dmitry Litvintsev, 6th - - PowerPoint PPT Presentation

Nov 25, 2022 29 likes •209 views

dCache @ Fermilab Dmitry Litvintsev Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen Wednesday, April 18, 2012 Science at Fermilab Fermilab: Home of the Tevatron and buffalos Energy frontier: BSM,

slide-1
SLIDE 1

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

dCache @ Fermilab

Dmitry Litvintsev

Wednesday, April 18, 2012

slide-2
SLIDE 2

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

Science at Fermilab

Fermilab: Home of the Tevatron and buffalos

  • Energy frontier: BSM,

precision EWK, Higgs, HF, QCD

  • CDF, D0 (ramping

down), CMS T1

  • Intensity Frontier: BSM,

neutrino physics, rare decays

  • Astrophysics: dark matter

search, high energy cosmic rays, galaxies surveys

Wednesday, April 18, 2012

slide-3
SLIDE 3

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

dCache @ Fermilab

total online

  • n tape

version public CDF CMS

0.1 PiB 2.5 PiB 1.9.5-28 1.5 PiB 10 PiB 1.9.5-22 13.5 PiB 17 PiB 1.9.5-23 httpdDoors: http://cdfdca.fnal.gov:2288/ http://fndca.fnal.gov:2288/ http://cmsdcam.fnal.gov:2288/

Size

Wednesday, April 18, 2012

slide-4
SLIDE 4

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

Tape Backend

  • 7 x SL8500 tapr robot libraries managed by

Enstore HSM

  • http://www-ccf.fnal.gov/enstore/
  • LTO4 and T10K tape drives
  • Total data on tape - 38 PB

Wednesday, April 18, 2012

slide-5
SLIDE 5

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

Tape Backend

  • 7 x SL8500 tapr robot libraries managed by

Enstore HSM

  • http://www-ccf.fnal.gov/enstore/
  • LTO4 and T10K tape drives
  • Total data on tape - 38 PB

Wednesday, April 18, 2012

slide-6
SLIDE 6

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

public dCache

4x dcap 2x gsi dcap 2x Kdcap 2x K FTP 14x GFTP WFTP

SRM

PnfsManager PoolManager httpd admin gPlazma1

dCache Enstore 7xSL8500 GUMS

XACML

w-pools r-pools rw-pools v-pools

movers movers movers

HSM

Wednesday, April 18, 2012

slide-7
SLIDE 7

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

Pool Setup

  • Experiments buy pool hardware
  • Their data directed to their pools by regex match between ugroup store units and file

storageclass extracted from layer 4 (storage_group + file_family combination + HSM name)

psu set regex on psu create unit -store ktev.ktev@enstore psu create unit -store des.*@enstore ... psu create ugroup DESSelGrp psu addto ugroup DESSelGrp des.*@enstore psu create ugroup KTeVReadSelGrp psu addto ugroup KTeVReadSelGrp ktev.ktev@enstore ...

Wednesday, April 18, 2012

slide-8
SLIDE 8

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

transfer rates @ public dCache

  • 1.5e+13
  • 1e+13
  • 5e+12

5e+12 1e+13 1.5e+13 11-03-01 11-05-01 11-07-01 11-09-01 11-11-01 12-01-01 12-03-01 12-05-0 Bytes Date (Year-month-day) Bytes Written Sun Apr 15 01:47:31 2012 dCache Enstore

  • 1.5e+13
  • 1e+13
  • 5e+12

5e+12 1e+13 1.5e+13 2e+13 2.5e+13 11-03-01 11-05-01 11-07-01 11-09-01 11-11-01 12-01-01 12-03-01 12-05-0 Bytes Date (Year-month-day) Bytes Read Sun Apr 15 01:47:25 2012 dCache Enstore

Bytes / day

10 TB/day 10 TB/day

Bytes / day Reads Writes

Wednesday, April 18, 2012

slide-9
SLIDE 9

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

transfers by experiment @ public dCache

theory 0% e906 0% des 1% e907 1% lqcd 21% minos 22% cdms 24% exp-db 32%

theory e906 des e907 lqcd minos cdms exp-db

Wednesday, April 18, 2012

slide-10
SLIDE 10

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

transfers by country

AU 0% GB 0% JP 0% MX 0% CH 0% SI 0% CO 0% CZ 0% BR 0% PT 1% FR 1% IT 2% ES 2% NL 2% DE 6% US 85% AU GB JP MX CH SI CO CZ BR PT FR IT ES NL DE US

Wednesday, April 18, 2012

slide-11
SLIDE 11

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

pnfs->chimera

  • required to develop chimera aware Enstore client (encp v3_10e)
  • to support direct encp required customized DB triggers to fill location information

from layer1 and layer4 and enstore2chimera migration stored procedure. See https://srm.fnal.gov/twiki/bin/view/DcacheCorner/ChimeraEnstore

  • converted pnfs -> chimera on Feb 23rd, 2012
  • 14718667 files

pnfsDump 6h52m SQL import 9h47m enstore2chimera 1h8m import of companion 17m md5sum verification 113h24m

Wednesday, April 18, 2012

slide-12
SLIDE 12

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

Reliability

  • All planned downtimes included, public

dCache uptime is 99.2%

Wednesday, April 18, 2012

slide-13
SLIDE 13

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

plans

  • migrate pnfs -> chimera @ CDF and D0 (May 1st)
  • upgrade public dCache and CDF dCache to 1.9.12-x
  • provide nfsv4.1 server for IF experiments @ public

dCache

Wednesday, April 18, 2012

slide-14
SLIDE 14

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

public dCache for IF

  • Intensity Frontier (IF) experiment require POSIX I/O
  • => use dCache as backup storage. Use srmcp to copy

data to BlueArc for access via NFS.

  • Concerns of long term scalability of this solution
  • We offered test dCache install with nfsV4.1 server for

IF use case studies.

Wednesday, April 18, 2012

slide-15
SLIDE 15

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

nfsV4.1 for IF

Client side:

  • SLF6 VM @ Fermicloud
  • 2.6.40 kernel

Server side:

  • dCache 1.9.12. 1 head node, 2 pool nodes.
  • pool nodes - 2 RAID6 w/ 4x250GB SATA drives

/dev/sdb1 /dev/sdb2

Nexsan SATABlade dCache Pool Node (dmsdca09)

2Gb FC

dCache Head Node (dmsdca08)

/dev/sdb1 /dev/sdb2

Nexsan SATABlade dCache Pool Node (dmsdca10)

2Gb FC

VM Client VM Client VM Client VM Client

Fermicloud Test dCache System

1GigE exports nfsv41 filesystem pNFS/nfsv41 - enabled Linux clients

Wednesday, April 18, 2012

slide-16
SLIDE 16

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

Iozone in cluster mode with sequential r/w tests

One 4GB file per client (twice the VM memory), one client per VM Multiple clients per VM

N umber of c l ients 2 4 6 8 10 MB /sec 5 0 100 15 0 200 25 0 3 00

IOzone test Sequential write Sequential read

Number of clients 20 40 60 80 100 120 140 160 180 200 MB/sec 50 100 150 200 250 300

IOzone connectivity test Sequential read Wednesday, April 18, 2012

slide-17
SLIDE 17

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

mdtest

mdtest w/ multiple MPI tasks create/stat/remove 100 directories w/ 100 zero length files (stats performed with read-your-neighbor option to avoid reading locally cached files)

Tree creation

Number of clients 2 4 6 8 10 12 14 16 18 20 Operations/sec 50 100 150 200 250 300

dCache mdtest ops Dir creation Dir removal File creation File removal Tree creation Tree removal

Number of clients 2 4 6 8 10 12 14 16 18 20 Operations/sec 500 1000 1500 2000 2500 3000 3500

dCache mdtest ops Dir stat File stat

Wednesday, April 18, 2012

slide-18
SLIDE 18

Dmitry Litvintsev, 6th International dCache Workshop, 17-19 April 2012, DESY Zeuthen

nfsV4.1 for IF

  • nfsV4.1 performance is adequate to meet IF

experiments’ requirements

  • further tests with real data and

experiments workflows are foreseen

Wednesday, April 18, 2012