DUNE DAQ Brainstorming E. Hazen October 31, 2017 Slide 1/9 - - PowerPoint PPT Presentation

▶

Mar 15, 2023 494 likes •588 views

DUNE DAQ Brainstorming E. Hazen October 31, 2017 Slide 1/9 2017-10-31 E. Hazen et. al. Brief Review of the Numbers Item Channels No/APA No/TPC bits/s ea Channel 1 2,560 384,000 24 Mb/s TDC Link 32 80 1200 768 Mb/s ROB 128 20

SLIDE 1

DUNE DAQ Brainstorming

E. Hazen

October 31, 2017

Slide 1/9 2017-10-31

E. Hazen et. al.

SLIDE 2

Brief Review of the Numbers

Item Channels No/APA No/TPC bits/s ea Channel 1 2,560 384,000 24 Mb/s TDC Link 32 80 1200 768 Mb/s ROB 128 20 300 3.1 Gb/s APA 2560 1 150 61.4 Gb/s TPC 384,000 0.05 1 9.2 Tb/s No encoding overhead included The current cold elx plan suggests four 1 Gb/s links per ROB This is a total of 1200 links per TPC or 4800 for 4 TPCs.

Slide 2/9 2017-10-31

E. Hazen et. al.

SLIDE 3

Readout Schemes... some choices

APA

2560 ch 61.4 Gb/s Fiber Tx FPGA Board (ATCA?) COTS Hardware (GPU / CPU)

80 x 1Gb/s copper 80 x 1Gb/s (simplex) fiber Ethernet (duplex fibers)

APA

2560 ch 61.4 Gb/s FPGA Board (on-det.) COTS Hardware (GPU / CPU)

80 x 1Gb/s copper Ethernet 10 x 10Gb/s Or 1 x 100Gb/s (duplex fibers)

APA

2560 ch 61.4 Gb/s FPGA Board (on-det.) COTS Hardware (GPU / CPU)

80 x 1Gb/s copper

FPGA Board (ATCA?)

Ethernet (duplex fibers) Proprietary mux'd fibers (simplex) 10 x 10 or 4 x 25

I II I III

Slide 3/9 2017-10-31

E. Hazen et. al.

SLIDE 4

Scheme I

Simple WIB with 1Gb/s fiber Tx Simplex 1Gb/s (unidirectional) fibers to underground FPGA Rx 1Gb/s links with MGT or generic I/O

◮ Pros:

◮ Minimum power at cryostat (??) ◮ Minimum noise at cryostat (??) ◮ Minimum engineering required for board

design

◮ Single layer of FPGA required in data

path

◮ Cons:

◮ Lots of fibers to run ◮ FPGA board must be underground

(or run 4800 fibers up shaft)

APA

2560 ch 61.4 Gb/s Fiber Tx FPGA Board (ATCA?) COTS Hardware (GPU / CPU) 80 x 1Gb/s copper 80 x 1Gb/s (simplex) fiber Ethernet (duplex fibers)

APA

2560 ch 61.4 Gb/s FPGA Board (on-det.) COTS Hardware (GPU / CPU) 80 x 1Gb/s copper Ethernet 10 x 10Gb/s Or 1 x 100Gb/s (duplex fibers)

APA

2560 ch 61.4 Gb/s FPGA Board (on-det.) COTS Hardware (GPU / CPU) 80 x 1Gb/s copper FPGA Board (ATCA?) Ethernet (duplex fibers) Proprietary mux'd fibers (simplex) 10 x 10 or 4 x 25

I II I III

Slide 4/9 2017-10-31

E. Hazen et. al.

SLIDE 5

Scheme II

WIB with FPGA multiplexing (ala ProtoDUNE) Output 10-40 Gb/s unidirectional fibers

(could be long-haul to surface or even Lead)

Second processing FPGA (on surface)

◮ Pros:

◮ Can spy on data for debugging ◮ Reduced fiber count 8x or 20x ◮ Most processing can be moved to surface ◮ Independent of processing technology

◮ Cons:

◮ More noise and power at cryostat ◮ More engineering

APA

2560 ch 61.4 Gb/s Fiber Tx FPGA Board (ATCA?) COTS Hardware (GPU / CPU) 80 x 1Gb/s copper 80 x 1Gb/s (simplex) fiber Ethernet (duplex fibers)

APA

2560 ch 61.4 Gb/s FPGA Board (on-det.) COTS Hardware (GPU / CPU) 80 x 1Gb/s copper Ethernet 10 x 10Gb/s Or 1 x 100Gb/s (duplex fibers)

APA

2560 ch 61.4 Gb/s FPGA Board (on-det.) COTS Hardware (GPU / CPU) 80 x 1Gb/s copper FPGA Board (ATCA?) Ethernet (duplex fibers) Proprietary mux'd fibers (simplex) 10 x 10 or 4 x 25

I II I III

Slide 5/9 2017-10-31

E. Hazen et. al.

SLIDE 6

Scheme III

WIB with FPGA multiplexing and Ethernet Output 10-40 Gb/s unidirectional fibers

(could be long-haul to surface or even Lead)

Std. network protocol (Ethernet) TCP/IP (simplified) or UDP

◮ Pros:

◮ Can spy on data for debugging ◮ Reduced fiber count 8x or 20x ◮ Most processing can be moved to surface ◮ All hardware past WIB is COTS ◮ One board to design

◮ Cons:

◮ All hardware past WIB is COTS

limits FPGA processing options

◮ More noise and power at cryostat ◮ More engineering

APA

2560 ch 61.4 Gb/s Fiber Tx FPGA Board (ATCA?) COTS Hardware (GPU / CPU) 80 x 1Gb/s copper 80 x 1Gb/s (simplex) fiber Ethernet (duplex fibers)

APA

2560 ch 61.4 Gb/s FPGA Board (on-det.) COTS Hardware (GPU / CPU) 80 x 1Gb/s copper Ethernet 10 x 10Gb/s Or 1 x 100Gb/s (duplex fibers)

APA

2560 ch 61.4 Gb/s FPGA Board (on-det.) COTS Hardware (GPU / CPU) 80 x 1Gb/s copper FPGA Board (ATCA?) Ethernet (duplex fibers) Proprietary mux'd fibers (simplex) 10 x 10 or 4 x 25

I II I III

Slide 6/9 2017-10-31

E. Hazen et. al.

SLIDE 7

Scheme III Feasibility

Case study: AMC13XG MicroTCA Module [1]

◮ Developed originally for CMS (100’s installed) ◮ Two firmware builds:

◮ Original CMS version (10 Gb/s proprietary protocol) ◮ Muon g-2 version (10 Gb Ethernet TCP/IP)

◮ Throughput of within a few % of line speed

Link Tx (in AMC) FIFO Event Builder TTC L1A FIFO

IPbus control / monitor / local DAQ

DDR3 SDRAM Simplified TCP/IP

GbE

10 GbE DAQ Fiber Input Links 5 Gb/s x 12 (uTCA) Simplified TCP/IP 10 GbE DAQ Fiber Simplified TCP/IP 10 GbE DAQ Fiber

1600MT/s (6.4 GB/s) [1] “The AMC13XG: a new generation clock/timing/DAQ module for CMS MicroTCA”, http://dx.doi.org/10.1088/1748-0221/8/12/C12036

Slide 7/9 2017-10-31

E. Hazen et. al.

SLIDE 8

A possible “Scheme III” WIB

Bandwidth required (1 40:10 mux): 40 Gb/s (in+out) = 80 Gb/s (2.5 GHz @ 32 bits)

This might be a bit aggressive with the DRAM, but easy to increase BW Ultrascale+ class FPGA (1 or 2) 40x 1.0 Gb/s DDR4-2666 x 32 DDR4-2666 x 32 40x 1.0 Gb/s 5x 10 Gb/s 5x 10 Gb/s

Slide 8/9 2017-10-31

E. Hazen et. al.

SLIDE 9

Summary

◮ I believe that sending the data to the surface must be cheaper

in the long run (having worked on 6 underground experiments now!)

◮ It is (I assert) feasible using existing techniques to multiplex

all data onto Ethernet in a simple way

◮ I believe this need not introduce noise into the cryostat if care

is taken at the feedthru

◮ The main point of this presentation is to provoke discussion...

Slide 9/9 2017-10-31

E. Hazen et. al.