CMS TriDAS project Infrastructure issues for the CMS online farm - - PowerPoint PPT Presentation

▶
cms tridas project
SMART_READER_LITE
LIVE PREVIEW

CMS TriDAS project Infrastructure issues for the CMS online farm - - PowerPoint PPT Presentation

May 11, 2023 369 likes •627 views

CMS TriDAS project Infrastructure issues for the CMS online farm attila.racz@cern.ch Outline Introduction Bill of Material Cooling issues Fire protection Power supplies Networking What next ? / Conclusion July

slide-1
SLIDE 1

CMS TriDAS project

Infrastructure issues for the CMS

  • nline farm

attila.racz@cern.ch

slide-2
SLIDE 2

July 8, 2003 Attila RACZ CERN/EP-CMD 2

Outline

  • Introduction
  • Bill of Material
  • Cooling issues
  • Fire protection
  • Power supplies
  • Networking
  • What next ? / Conclusion
slide-3
SLIDE 3

July 8, 2003 Attila RACZ CERN/EP-CMD 3

General view

slide-4
SLIDE 4

July 8, 2003 Attila RACZ CERN/EP-CMD 4

One column… out of 512 !

slide-5
SLIDE 5

July 8, 2003 Attila RACZ CERN/EP-CMD 5

Building names…

slide-6
SLIDE 6

July 8, 2003 Attila RACZ CERN/EP-CMD 6

DAQ building (SCX)

Computer rooms Conference rooms / labs False floor

Data fibers and commodity networks from the pit 352 m2, ~6m height 165 m2, ~3m height Overall dimension: ~ 14 x 30 m2

slide-7
SLIDE 7

July 8, 2003 Attila RACZ CERN/EP-CMD 7

Timescales

  • Buildings available Q2-Q3 2005
  • Start of Detector-DAQ integration: Q2 2006
  • First LHC beams: April 1, 2007
  • Computing farm is deployed by “slices”

– Processing power will follow beam luminosity ramping-up and budget time-profile… – We want to profit from Moore’s law…

slide-8
SLIDE 8

July 8, 2003 Attila RACZ CERN/EP-CMD 8

SCX BOM…

  • FEDB switch:

16 x 128 port switches (2.5 Gb/s)

  • RUs:

256 PCs

  • RUB switch:

8 x 128 port switches (2.5 Gb/s)

  • BUs:

256 PCs

  • FUs:

2048 PCs, dual processor

– Other form factor: blades (see next slide)

  • Misc:

assuming equivalent of 4 racks

  • FU PCs:

1U, 12 kg, 250 W

  • RU/BU PCs:

3U, 20 kg, 300 W

  • 2.5 Gb/s Switches: 9U, 31 kg, 960 W
  • Rack:

42U, 170 kg

slide-9
SLIDE 9

July 8, 2003 Attila RACZ CERN/EP-CMD 9

1U PC versus Blades

  • Computer blades allow up to ~170 processors per rack vs

~80 proc. with 1U dual CPU PCs

  • With such form factor, we could reduce the number of FU

racks from 64/52 down to 25

  • Hardware maintenance operations are optimized
  • But…power dissipation can go up to ~18kW per rack…
  • No interchangeability from vendor to vendor as with 1U PCs
  • Proprietary management software
  • Farm upgrade/renewal more “problematic”
  • 1U PC for FUs is our baseline today
slide-10
SLIDE 10

July 8, 2003 Attila RACZ CERN/EP-CMD 10

DAQ racks (1)…

  • We don’t fill the racks to the max. for keeping the power

density to a minimum

– And some contingency is a good practice…

  • A total of 138 racks:

– 4 FEDB racks: 4 * 9U 128 port switch – 32 RU racks: 8 * 3U PCs + commodity GB switch – 2 RUB racks: 4 * 9U 128 port switch – 32 BU racks: 8 * 3U PCs + commodity GB switch – 64 FU racks: 32 * 1U PCs + commodity GB switch – 4 racks for misc. (servers, local storage, …)

slide-11
SLIDE 11

July 8, 2003 Attila RACZ CERN/EP-CMD 11

DAQ racks (2)…

~738 16 535.6 81.4 7.68 81.4 15.36 Total power (kW) ~385 100% 50% 50% 50% 50% 100% At Startup (50 kHz) 64/52 8.3/10.3 560/655 Filter Unit

2k*1U PCs @ 250W

4 4 331/411 Computing services 32/22 2.5/3.7 331/411 Builder Unit

256*3U PCs @ 300W

2 3.84 294 Readout Unit builder 138/106 Total 32/22 2.5/3.7 331/411 Readout Unit

256*3U PCs @ 300W

4 3.84 294 FED builder Rack Quantity Power per rack (kW) Weight (kg) Rack type

slide-12
SLIDE 12

July 8, 2003 Attila RACZ CERN/EP-CMD 12

Rack layout (low packing)

slide-13
SLIDE 13

July 8, 2003 Attila RACZ CERN/EP-CMD 13

3D rack layout

slide-14
SLIDE 14

July 8, 2003 Attila RACZ CERN/EP-CMD 14

Cooling issues (1)…

  • Typical power per rack is 3-4 kW max. and

typical power density is 0.5-0.8kW/m2

  • We have ~8 kW/rack and 2 kW/m2

– Traditional plenum floor is not adapted (floor surface, air-flow, efficiency…)

  • We intend to use water-cooled racks
slide-15
SLIDE 15

July 8, 2003 Attila RACZ CERN/EP-CMD 15

Cooling issues (2)…

  • We don’t want to have water in the PCs

– Restriction on the PC choice, delicate operations when maintenance or upgrade

  • Up to now, very few products on the market
  • Water cooling seems to be a solution for

future high power density data centers

– Keep an eye opened on the market…

slide-16
SLIDE 16

July 8, 2003 Attila RACZ CERN/EP-CMD 16

Rack cooling products…

Ecobay: 10.5 kW/cab, Internal cooling module, closed air-circuit RackCooler: 8 kW, cooling module mounted

  • n standard racks

Knurr: 10 kW Internal cooling module, closed air-circuit

slide-17
SLIDE 17

July 8, 2003 Attila RACZ CERN/EP-CMD 17

Fire protection

  • The experimental zone is at ~30 min

away from the fire brigade…

– Need for a fire inertion system

  • Rack type has big influence

– With open racks, full volume protection (human presence may be a problem…) – With closed racks, built-in inertion system and environmental control

  • Need for a comparative cost analysis !
slide-18
SLIDE 18

July 8, 2003 Attila RACZ CERN/EP-CMD 18

Power supplies

  • The idea is to have a main switching unit

for each rack. Then, within a rack, have a remote controlled Power Distribution Unit

  • UPS for strategic machines only
  • Too early to go further…
slide-19
SLIDE 19

July 8, 2003 Attila RACZ CERN/EP-CMD 19

Networking…

  • Ideally, separated network for:

– Data transport – Control messages – Monitoring and services

  • The topology should

– Minimize the cabling in the building (stay within a rack as much as possible, BU+FUs) – Allow easy reconfiguration: no direct intervention on the machines -> usage of starpoints

  • Again, too early to go further…
slide-20
SLIDE 20

July 8, 2003 Attila RACZ CERN/EP-CMD 20

What next ?

  • Preserie (1/8th) assembled in 2004 for gaining

experience on HW and SW aspects

  • Continue to optimize the hardware elements

w.r.t. computing requirements

  • Look at the technological evolution for

Computers/telecom market and its impact on the infrastructures

slide-21
SLIDE 21

July 8, 2003 Attila RACZ CERN/EP-CMD 21

Conclusion

  • Most of the DAQ must stay modular…

– Parallel DAQ (partitions) – Sliced deployment – Usage of new/appearing technologies

  • Mixing of different platforms and different

vendors is very likely

slide-22
SLIDE 22

July 8, 2003 Attila RACZ CERN/EP-CMD 22

DAQ Opticable

200m long 72 fibers, 50/125um multimode, LC-LC connector 16 FRLs per cable, 30 + 2 cables for whole DAQ (8 spare cables and 8 spare fibers per cable)

Polyester Tape Aramid Yarn PVC Coated Aramid C.S.M. Aramid Yarn

Tight Buffer Optical Fiber

PVC Sub-unit Jacket * Sub- unit HFFR Jacket * Rip Cord.

slide-23
SLIDE 23

July 8, 2003 Attila RACZ CERN/EP-CMD 23

DAQ rack layout in USC

Air guide 2U Fan tray 1U Chassis 6U Heat ex. 1U

  • Tang. Fans 4U

1U PCs 5U 9U 17U 25U 33U 41U 46U 52U Cable feed-through 1U Filling plate 4U 56U Heat exchanger + Extraction fans Mounted on door

Front view

0U

Side view

Room air Rear door Smoke detection device

slide-24
SLIDE 24

July 8, 2003 Attila RACZ CERN/EP-CMD 24

FE racks and DAQ racks (1)

slide-25
SLIDE 25

July 8, 2003 Attila RACZ CERN/EP-CMD 25

FE racks and DAQ racks (2)