DAQ architecture - networking J.MARTEAU IPNL, Universit de Lyon, - - PowerPoint PPT Presentation

▶
daq architecture networking
SMART_READER_LITE
LIVE PREVIEW

DAQ architecture - networking J.MARTEAU IPNL, Universit de Lyon, - - PowerPoint PPT Presentation

Oct 24, 2022 253 likes •375 views

DAQ architecture - networking J.MARTEAU IPNL, Universit de Lyon, CNRS-IN2P3, UMR 5822 DUNE DAQ WG October 19, 2015 WA105 data network processing) Storage : Processing : (storage/ 15 disks servers R730 16 lames M630 B/E 2 metadata

slide-1
SLIDE 1

DAQ architecture - networking

J.MARTEAU IPNL, Université de Lyon, CNRS-IN2P3, UMR 5822 DUNE DAQ WG October 19, 2015

slide-2
SLIDE 2

light

WA105 data network

F/E-out : Charge + PMT 10 F/E-in

6+1 = 70 Gbps max

B/E (sorting/ Filtering/ Clock)

Raw data : charge

B/E (storage/ processing)

Event building workstations Storage : 15 disks servers R730 2 metadata servers R630 1 config. server R430 6 = 60 Gbps Triggers : Beam Counters

charge

Compressed data / ZS data

charge

4 = 160 Gbps

15 3

10 Gbps Processing : 16 lames M630 16x24 = 384 cores Master switch MasterCLK

PC : WR slave Trigger board

Raw data : light

CERN C.C. LAr Top of cryostat C.R. C.R. C.C.

  • C.R.

10 Gbps 10 Gbps 10 Gbps 10 Gbps 10 Gbps 40 Gbps

slide-3
SLIDE 3

DUNE proposed scheme

Surface Underground LAr

Charge R/O ASIC Light R/O µTCA Q-crates (240) µTCA L-crates (20)

Top

LV power supply HV power supply Signal Signal 240 x 10GbE 20 x 10GbE

10/40 GbE switch (15)

60 x 40GbE

10/40 GbE switch (2)

5 x 40GbE

B/E cards*/** (20) B/E cards* (2)

(* Could fit In 3 teraboxes) (** 3 x 40GbE in 1 x 40 GbE out) 20 x 40GbE 2 x 40GbE

10/40 GbE switch (5) 10/40 GbE switch (1) Storage servers (30)

(Reduction factor = 6) 20 x 10GbE 8 x 10GbE

150 kW

WR Switch (15) WR Switch (2) WR Master Switch (1) Master Clock (1)

slide-4
SLIDE 4

DUNE proposed scheme

Surface Underground LAr

Charge R/O ASIC Light R/O µTCA Q-crates (240) µTCA L-crates (20)

Top

LV power supply HV power supply Signal Signal 240 x 10GbE 20 x 10GbE 26 x 100GbE

B/E cards (8+1)

9 x 40GbE

10/40 GbE switch (2) Storage servers (30)

(Reduction factor = 6) 30 x 10GbE

150 kW

WR Switch (15) WR Switch (2) WR Master Switch (1) Master Clock (1) 10/100 GbE switch 260 ports 10 26 ports 100

B/E boards :

  • 3 inputs 100 Gbps
  • 1 output 40 Gbps

1 box with 8 boards 1 separated board (light)

slide-5
SLIDE 5

10/40 switches specs.

16 k$

slide-6
SLIDE 6

10/40 switches specs.

OPTION 1 : Nexus 3172TQ : 48 ports 1/10G baseT, 6 ports 40G QSFP+ N3K-C3172TQ-10GT Nexus 3172T 48 x 1/10GBase-T and 6 QSFP+ ports OPTION 2 : Nexus 9372TX : 48 ports 1/10G baseT, 6 ports 40G QSFP+ N9K-C9372TX Nexus 9300 with 48p 1/10G-T and 6p 40G QSFP+ OPTION 3 : Nexus 9396TX : 48 ports 1/10G baseT, 12 ports 40G QSFP+ N9K-C9396TX Nexus 9300 48p 1/10GBASE-T & additional uplink module req. N9K-M12PQ ACI Uplink Module for Nexus 9300 12p 40G QSFP

10 to 12 k€

  • w. academic

discount 14 k€

slide-7
SLIDE 7

10/100 switches specs

DUNE dual phase :

  • 240 10Gbps links

for charge

  • 20 10Gbps links

for light (7 kW)

slide-8
SLIDE 8

10/100 switches specs

DUNE dual phase :

  • 240 10Gbps links

for charge

  • 20 10Gbps links

for light DUNE dual phase :

  • 240 10Gbps links

for charge

  • 20 10Gbps links

for light (12 kW)

slide-9
SLIDE 9

DUNE B/E board

Terabox

  • 24 TeraFLOPS processing: 16x Altera Arria 10 or Stratix V FPGAs

Up to 18 million logic elements (Arria 10 GX) Up to 62,000 multipliers (Stratix V GS) Dual card wrt WA105 cards New FPGA-board with Stratix X available by the time of DUNE

  • 1.28 Terabits/sec I/O
  • 128x 10GigE, 32x 40GigE, 32x 100 GigE,
  • r 32x QDR Infiniband
  • 6.5 Terabits/sec memory bandwidth

Up to 64 banks DDR3-1600 (512 GBytes) DDR4, QDR-IV, QDRII+, and RLDRAM3 memory options

  • 4U or 5U Rackmount PCIe system

(server, industrial, or expansion) Dual socket Intel Ivy Bridge with up to 12 cores Up to 768 GBytes of system memory 8 Gen3 x16 PCIe slots

  • Complete software support

Windows and Linux 64 drivers, interface libraries, and hardware management FPGA development kit for Arria 10 and Stratix V

4 fibres per board : 3 R / 1 W

slide-10
SLIDE 10

Power estimate

ITEM CONSUMPTION # of UNITS TOTAL (kW) TOTAL F/E 150 kW

  • B/E cards (terabox)
  • 1.5 kW max.

2 3.0

  • Switch (10/100 GbE)
  • 12 kW max.

1 12

  • Switch (10/40 GbE)
  • 0.35 kW max.

2 0.7

  • WR switch
  • 0.3 kW typ.

18 5.4

  • B/E storage
  • 0.7 max. / 0.35 meas.

30 10.5

  • B/E configuration
  • 0.4 kW max.

6 2.4

  • B/E online proc.
  • 3 kW max.

2 6.0 TOTAL B/E 40.0 Underground.vs.surface : (150+12+17x0.3).vs.(3.0+2x0.35+1x0.3+18.9) = 167.1.vs.22.9

slide-11
SLIDE 11

Comments

  • Simplified network architecture with 100GbE standards already available on the

market.

  • Making the exercise one goes from ~20 switches down to 1 !
  • Prices will decrease...
  • Position of the WR Master Switch may be moved underground if GPS timing is

available there.

  • Estimate of the F/E power is at the limit, very conservative as well.
  • The total power consumption is estimated below 200kW for the double phase option,

80 % of which go underground.