ProtoDUNE DAQ evolution Philip Rodrigues (summarizing discussions - - PowerPoint PPT Presentation

ProtoDUNE DAQ evolution

Philip Rodrigues

(summarizing discussions with Kurt, Wes, Brett, David C, Josh)

15 Feb 2019

Motivation/strategy

• Lots of new pieces to design/implement for DUNE DAQ relative to ProtoDUNE DAQ
• One part of the strategy is to prototype what we can in ProtoDUNE. So far, we’ve

thought about overall dataflow in a self-triggering system.

• Plan to make changes incrementally: have a working system at each stage
• Examples of what we plan to learn through doing, and feed back into DUNE design:
• Which pieces are easy/hard
• Who talks to who?
• Edge cases (eg race conditions, rate limitations)
• Things this isn’t:
• A proposal for exactly what we do in the final system
• A proposal for details of data selection algorithms (just data flow)

First task in this strategy: TPC self-trigger

• Aim: trigger ProtoDUNE events based on TPC data, within artdaq,

with minimal changes to the existing system

• Trigger condition needs to be easy-to-identify, not too frequent.

Probably horizontal (multi-)cosmics

The “existing system”, high-level view

BR Data Queue

Board reader

Fragment generator

Electronics artdaq Not artdaq

BR Data Queue

Board reader

Fragment generator

Electronics

• Fragment generators are custom

code for the given electronics

• Board reader code (artdaq) calls

fragment generator in a loop to fill BR data queue

• Actually, fragment generator

could be connected to anything, as long as it passes fragments to the queue

• r, A Gentle Introduction to artdaq

The “existing system”, high-level view

BR Data Queue

Board reader

Timing frag gen

Timing board artdaq Not artdaq

BR Data Queue

Board reader

Fragment generator

Electronics Rest of artdaq (Routing Master, Event builders)

BR Data Queue

Board reader

Fragment generator

Electronics “Trigger at timestamp X” “Trigger at timestamp X”

• Timing board sends out

trigger commands on fibres, and to timing BR

• Timing BR, in “push” mode,

immediately sends data to rest of artdaq

The “existing system”, high-level view

BR Data Queue

Board reader

Timing frag gen

Timing board artdaq Not artdaq

BR Data Queue

Board reader

Fragment generator

Electronics Rest of artdaq (Routing Master, Event builders)

BR Data Queue

Board reader

Fragment generator

Electronics “Trigger at timestamp X” “Trigger at timestamp X”

• Timing board sends out

trigger commands on fibres, and to timing BR

• Timing BR, in “push” mode,

immediately sends data to rest of artdaq

• “Rest of artdaq” requests

data for the given timestamp from the other BRs

“Give me data for X”

The “existing system”, high-level view

BR Data Queue

Board reader

Timing frag gen

Timing board artdaq Not artdaq

BR Data Queue

Board reader

FELIX frag gen

FELIX Rest of artdaq (Routing Master, Event builders)

BR Data Queue

Board reader

Fragment generator

Electronics “Trigger at timestamp X” “Trigger at timestamp X”

• Timing board sends out

trigger commands on fibres, and to timing BR

• Timing BR, in “push” mode,

immediately sends data to rest of artdaq

• “Rest of artdaq” requests

data for the given timestamp from the other BRs

• Extra complication: each

trigger is passed to FELIX frag gen via ZeroMQ

“Give me data for X”

“Trigger at timestamp X”

Proposed change

BR Data Queue

Board reader

Timing frag gen

Timing board artdaq Not artdaq

BR Data Queue

Board reader

FELIX frag gen

FELIX Rest of artdaq (Routing Master, Event builders)

BR Data Queue

Board reader

Hit finding frag gen

“Trigger at timestamp X”

• Add a board reader that

produces (CPU or FPGA) hits

• Add a data selector process

that receives hits and “timestamps of interest” (from timing BR)

• Run timing BR in normal

mode (no immediate send), with high rate of timing board triggers

• Data selector uses hits to

identify which “timestamps

• f interest” to read out
• Leaves “rest of artdaq” box

almost unchanged

“Give me data for Y”

“Trigger at timestamp X”

Software data selector

“Trigger at timestamp Y”

Implementation details: board readers

subscriber

message serialize write message serialize write SPSC Queue

trigger matcher

(ext) trigger request compress memcpy frames to fragments fetch data

BR/FragGen

getNext_() BR Data Queue

BR/FragGen

(ext) data request fetch data store to EB

Diagram adapted from Kurt, showing the FELIX board reader. Each vertical line is a thread. Fragment generator has its own internal queue of all data; compresses triggered data and hands it on to artdaq

Implementation details: board readers, right now

subscriber

message serialize write message serialize write SPSC Queue

trigger matcher

(ext) trigger request compress memcpy frames and hits to fragments fetch data

BR/FragGen

getNext_() BR Data Queue

BR/FragGen

(ext) data request fetch data store to EB

hit finder thread

(pointers to) all data from SPSC queue find hits Hit Queue hits fetch hits

Implementation details: board readers, next

subscriber

message serialize write message serialize write SPSC Queue

trigger matcher

(ext) trigger request compress memcpy frames to fragments fetch data

BR/FragGen

getNext_() BR Data Queue

BR/FragGen

(ext) data request fetch data store to EB

hit finder thread

(pointers to) all data from SPSC queue find hits hits

BR/FragGen

getNext_() BR Data Queue

BR/FragGen

fetch all data store to artdaq hits from finder thread hits to BR

pass-through

Implementation details: board readers, with PBM

subscriber

message serialize write message serialize write SPSC Queue

trigger matcher

(ext) trigger request compress memcpy frames to fragments fetch data

BR/FragGen

getNext_() BR Data Queue

BR/FragGen

(ext) data request fetch data store to EB

BR/FragGen

getNext_() BR Data Queue

BR/FragGen

fetch all data store to artdaq hits from PBM hits to BR

pass-through

Data selector process

subscriber to hit broadcasts

message write, if algo is passed message

Trigger Command Queue

trigger matcher

(ext) trigger request TriggerCommands that

• verlap with external

trigger requests get sent out look for TC in queue

BR/FragGen

getNext_() fragment pushed to EB run sample DataSel algo run sample DataSel algo write, if algo is passed

Here’s an outline implementation of the data selector process implemented as an artdaq board reader, from Kurt:

Summary

• We have a concrete plan for step 1 of evolving ProtoDUNE DAQ: make

self-triggers possible

• Tasks assigned: Phil writing hit finding, Brett writing hit message API,

Pierre writing hit-finding fragment generators

• An outline plan exists for the data selector (implemented as a board

reader). Needs a bit more work

• Other TODOs: work out simple algorithm to go in data selector

process; work out details of what changes in “rest of artdaq”