Comments on DUNE DAQ Challenges Architecture Ba Babak Abi DUNE - - PowerPoint PPT Presentation

Comments on DUNE DAQ Challenges Architecture

Ba Babak Abi

DUNE DAQ Simulations Meeting

16 16 Oct 20 2017 17

1

Review :

• Alternative DAQ Architectures
• DUNE DAQ data fellow
• More about GPU based Architectures

DAQ Architecture; comments on note 4481

• Comments and discussion on note 4481:
• SLAC ( Ryan&Matt,..) , Rick, Dave .
• Components Life time
• Modularity and scalability
• Noise and detector performance
• Optical & network Links , data error check
• Reserve DAQ capacity (in case of bad detector performance)
• Using the commercial components
• …..

2

DUNE DAQ Architectures

• We (DUNE DAQ ) already have at least 3
• 1. RCE 4th generation upgrade for DUNE

1. SLAC and Oxford work on 4th generation DPM right now.

• 2. GPU based

1. This talk and next talk from Phil.

• 3. FPGA based (simplified RCE?) (detailed talk next week)
• 4. Other Alternatives? (and DaqD3 and 4 in note 4481)
• Major questions :
• 1. Ring Buffer
• 2. Trigger generation factors (Detector performance & algorithms )
• 3. Time and clock accuracy (1uS tolerance time accuracy! & No need to phase

locked Clock to each APA!)

• 4. Dual-phase merge& compatibility
• 5. What more ?

3

DUNE DAQ data fellow

• 4

APA

2560 channels

Global

DAQ

FPGA + CPU 100GE Main DATA stream Master -control Timing, trigger,.. Time/Clock/ Trigger distribution Network A Network B Secondary Network Control/monitor/Data Flange Trigger Processor Farm 2 4 1 2 1 3

40 SiPMs

10 Photon Detection System per APA

SSP

Data fell llow; ; Rin ing Buffer & Tri rigg gger Pri rimitives

5

1- Lower bandwidth for triggered DATA 2- Relaxed Trigger latency -> send through TCP packets back to buffer holder 3- Global Time 4- Separate clock

• event builder can handle triggered data ?

Timing tag Trigger Processor Farm SN,Proton,Cosmics TPC Front-end Electronics Global Trigger Beam,Random,Calibration

Board-Reader Event-Builder BACK-END DAQ SSP

Trigger Primitive Generator

Photon Detector Ring Buffer

TPC Trigger Primitive Generator

Timing tag Ring Buffer Triggered DATA

DaqD1: Min inimum In In-detector architecture

6

APA 2560 channels 100Gb 80 x 1.25Gb serial Links PC- backend DAQ 1x fiber pairs 2Km to surface Network

Detector side Off-Detector side at Surface 10x10Gb/S

20:1 MUX CWDM

Processor unit PCI + GPU Buffer + Tagging + compression GPU Processor unit PCI +GPU Buffer + Tagging + compression GPU

FPGA/ASIC MUX 10 x 10Gb serial Links APA 2560 channels 100Gb 80 x 1.25Gb serial Links FPGA/ASIC MUX 10 x 10Gb serial Links 50MHz Clock distribution Global Time PTP network CAT6

10x10Gb/S 10x10Gb/S

Global Trigger CAT6

Another alternatives like : It is possible to send DATA through TCP/UDP Packets through optical fibre but constrains on NIC and FPGA!

DaqD1: : Min inimum In In-detector architecture

7

1 APA requires : 1 Sever, 1 PCI card, 2/4 GPU, 2x10GE NIC 200GB DDR4 RAM as Ring Buffer

• PCI card
• PCI 5 ready in 2019, 63 GB/s(x16)
• Commercially availed BUT an RCE

with PCI extension can do the job!

• Minimum Firmware R&D
• Ring Buffer time stamp: PTP Time
• All CPU/NIC clock accuracy 100-

500nS

• All server NICs are PTP enables and has at

least two NIC port , One dedicated for DATA transport and the other is for PTP, trigger, monitoring ….

• GPU, easy to program, almost zero R&D

PC- backend DAQ Network

Off-Detector side at Surface GPU

Global Time PTP network CAT6 Global Trigger CAT6

PCI GPU NIC

Global Time PTP network CAT6

Ring Buff er Time PTP

1.Most of the components are off-shell, minimum R&D and maximum flexibility 2.It is cost effective, low maintenance and rapid R&D with minimum usage of FPGA and related custom build components. 3.Commercial availability of Optical C&D/WDM Transceiver/Ethernet Optical Interfaces 100Gb/s ( to 400Gb in near future) 4.Very low power consumption ( in-detector cave )

5.Extremely modular and scalable

GPU based architectures : : proof of f concept

8

• What is the PTP accuracy :
• We investigating PTP in Oxford,

Created PTP a setup with two nodes and boundary switch

• GPU concept:
• Is it possible ?
• How many GPUs we need ?
• What is the bottle neck ?
• How difficult is GPU programming?
• …..
• Please listen to Phil’s Talk

PTP test set-up in Oxford

Back up slides

9

Architecture; Tim ime and clo lock dis istribution

• What is the clock and time distribution requirements?
• Assuming feeding 1 clock line and one time stamp for each APA
• What is clock phase difference between two APA (and SSPs)?
• What is the time (stamp) accuracy per APA and SSP?
• Sample rate is 2Ms/S (500ns), what if the 2 APA are out of phase for

clock and time stamp in ring buffer?

• “DUNE can tolerate a few tics ambiguity in clock and time stamp between

APAs“ ?

• We need a 50MHz! stable clock distribution generated by a atomic

clock.

10

Architecture; PTP Tim ime dis istribution

• Providing the time to stamp the ring buffer:

1. White Rabbit sub-nano second accuracy but very expensive. 2. Precision Time Protocol (PTP), 100ns to 500nS accuracy, almost all servers’ NIC and FPGA like ZYNQ are PTP enabled, only we need a PTP enabled switch to distribute the GPS + atomic global time.

• This can be combined with Bristol's innovative clock

distribution proposal to have simple clock and timing distribution.

• Trigger can be send through TCP/IP data stream since we have a little relaxed letancy due to

long ring buffer

11