ProtoDUNE DAQ evolution Philip Rodrigues (summarizing discussions - PowerPoint PPT Presentation

ProtoDUNE DAQ evolution Philip Rodrigues (summarizing discussions with Kurt, Wes, Brett, David C, Josh) 15 Feb 2019

Motivation/strategy • Lots of new pieces to design/implement for DUNE DAQ relative to ProtoDUNE DAQ • One part of the strategy is to prototype what we can in ProtoDUNE . So far, we’ve thought about overall dataflow in a self-triggering system. • Plan to make changes incrementally: have a working system at each stage • Examples of what we plan to learn through doing, and feed back into DUNE design: • Which pieces are easy/hard • Who talks to who? • Edge cases (eg race conditions, rate limitations) • Things this isn’t: • A proposal for exactly what we do in the final system • A proposal for details of data selection algorithms (just data flow)

First task in this strategy: TPC self-trigger • Aim: trigger ProtoDUNE events based on TPC data, within artdaq, with minimal changes to the existing system • Trigger condition needs to be easy-to-identify, not too frequent. Probably horizontal (multi-)cosmics

The “existing system”, high -level view or, A Gentle Introduction to artdaq • Fragment generators are custom Electronics Electronics Not artdaq code for the given electronics • Board reader code (artdaq) calls artdaq fragment generator in a loop to Fragment Fragment fill BR data queue generator generator • Actually, fragment generator BR BR could be connected to anything, Data Data as long as it passes fragments to Queue Queue the queue Board reader Board reader

The “existing system”, high -level view “Trigger at timestamp X” • Timing board sends out Electronics Timing board Electronics Not artdaq trigger commands on fibres, and to timing BR artdaq • Timing BR, in “push” mode, Fragment Timing Fragment immediately sends data to frag gen generator generator rest of artdaq BR BR BR Data Data Data Queue Queue Queue Board reader Board reader Board reader “Trigger at timestamp X” Rest of artdaq (Routing Master, Event builders)

The “existing system”, high -level view “Trigger at timestamp X” • Timing board sends out Electronics Timing board Electronics Not artdaq trigger commands on fibres, and to timing BR artdaq • Timing BR, in “push” mode, Fragment Timing Fragment immediately sends data to frag gen generator generator rest of artdaq • “Rest of artdaq ” requests BR BR BR Data Data Data data for the given timestamp Queue Queue Queue from the other BRs Board reader Board reader Board reader “Give me data for X” “Trigger at timestamp X” Rest of artdaq (Routing Master, Event builders)

The “existing system”, high -level view “Trigger at timestamp X” • Timing board sends out Electronics Timing board FELIX Not artdaq trigger commands on fibres, and to timing BR artdaq “Trigger at timestamp X” • Timing BR, in “push” mode, Fragment Timing FELIX immediately sends data to frag gen frag gen generator rest of artdaq • “Rest of artdaq ” requests BR BR BR Data Data Data data for the given timestamp Queue Queue Queue from the other BRs Board reader Board reader Board reader • Extra complication: each trigger is passed to FELIX frag “Give me data for X” “Trigger at timestamp X” gen via ZeroMQ Rest of artdaq (Routing Master, Event builders)

Proposed change • Add a board reader that produces (CPU or FPGA) hits • Add a data selector process “Trigger at timestamp X” that receives hits and “timestamps of interest” Timing board FELIX Not artdaq (from timing BR) artdaq • Run timing BR in normal “Trigger at timestamp X” mode (no immediate send), Hit finding Timing FELIX frag gen frag gen frag gen with high rate of timing board triggers BR BR BR • Data selector uses hits to Data Data Data Queue Queue Queue identify which “timestamps of interest” to read out Board reader Board reader Board reader • Leaves “rest of artdaq ” box “Give me data for Y” Software almost unchanged data selector Rest of artdaq (Routing Master, Event builders) “Trigger at timestamp Y”

Implementation details: board readers subscriber trigger matcher BR/FragGen BR/FragGen message (ext) trigger request fetch data serialize compress memcpy frames BR to fragments write Data SPSC getNext_() Queue message Queue serialize store (ext) data request write fetch data to EB Diagram adapted from Kurt, showing the FELIX board reader. Each vertical line is a thread. Fragment generator has its own internal queue of all data; compresses triggered data and hands it on to artdaq

Implementation details: board readers, right now subscriber trigger matcher BR/FragGen BR/FragGen message (ext) trigger request fetch data serialize compress memcpy frames fetch hits and hits BR write to fragments Data SPSC getNext_() Queue message Queue serialize store (ext) data request write fetch data to EB hit finder thread (pointers to) all data from SPSC queue find hits Hit hits Queue

Implementation details: board readers, next subscriber trigger matcher BR/FragGen BR/FragGen BR/FragGen BR/FragGen message (ext) trigger request fetch data serialize compress BR memcpy frames Data BR to fragments write Queue Data SPSC getNext_() Queue message Queue getNext_() serialize store store (ext) data request write fetch data fetch all data to artdaq to EB hit finder thread (pointers to) all data from SPSC queue find hits hits from finder thread hits hits to BR pass-through

Implementation details: board readers, with PBM subscriber trigger matcher BR/FragGen BR/FragGen BR/FragGen BR/FragGen message (ext) trigger request fetch data serialize compress BR memcpy frames Data BR to fragments write Queue Data SPSC getNext_() Queue message Queue getNext_() serialize store store (ext) data request write fetch data fetch all data to artdaq to EB hits from PBM hits to BR pass-through

Data selector process Here’s an outline implementation of the data selector process implemented as an artdaq board reader, from Kurt: trigger matcher BR/FragGen subscriber to hit broadcasts (ext) trigger request message look for TC in queue TriggerCommands that overlap with external run sample trigger requests get sent out write, if DataSel algo Trigger algo is Command getNext_() passed Queue message fragment pushed to EB run sample write, if DataSel algo algo is passed

Summary • We have a concrete plan for step 1 of evolving ProtoDUNE DAQ: make self-triggers possible • Tasks assigned: Phil writing hit finding, Brett writing hit message API, Pierre writing hit-finding fragment generators • An outline plan exists for the data selector (implemented as a board reader). Needs a bit more work • Other TODOs: work out simple algorithm to go in data selector process; work out details of what changes in “rest of artdaq ”