Data Selection Workshop: Welcome, Overview, Goals
Josh Klein, Penn
Data Selection Workshop: Welcome, Overview, Goals Josh Klein, Penn - - PowerPoint PPT Presentation
Data Selection Workshop: Welcome, Overview, Goals Josh Klein, Penn Welcome to Penn First University in the US, Ben Franklin, ENIAC, blah blah blah Overview Data Selection is hierarchical Trigger Primitives generated at front-end
Data Selection Workshop: Welcome, Overview, Goals
Josh Klein, Penn
Welcome to Penn
First University in the US, Ben Franklin, ENIAC, blah blah blah
Overview
APA 1 APA 150
Module 1
Trigger Primitive Generation Trigger Candidate Generation Trigger Primitive Generation Trigger Candidate Generation Module-Level Triggers
Trigger commands
Det 1 Det N
Module 4
Trigger Primitive Generation Trigger Candidate Generation Trigger Primitive Generation Trigger Candidate Generation Module-Level Triggers
Trigger commands
External Triggers
SN Trigger commands SN Trigger commands
Software or Firmware [Mostly] Software Software Software
DS Scope: 1 SP TPC
Front End System (APA level trigger primitive generation) [x150] L0 (APA level) data selection L1 (module level) data selection L2 (High-level data selection) Applied to each event independently Event Builder (per Module) Permanent storage at Fermilab Front End System (APA level trigger primitive generation) [x150] Front End System (APA level trigger primitive generation) [x150] Long-term storage
Run Control & Config Databases External & Calibration Signals
Goals for this Workshop
Goals for this Workshop
LBNC Meeting has reinforced the need for:
The era of “It could be this way or that way…” and “This will probably work…” is over.
Goals for this Workshop: Requirements
This is not so easy.
This is not so easy.
Goals for this Workshop: Requirements
Goals for this Workshop: Requirements
Data Selection Physics requirements takeaway:
Goals for this Workshop: Requirements
But nothing about:
We have our own versions of these but they are not tied to the physics requirements in any direct way. We need to have agreement on what these should be, and how the tie to physics requirements.
Goals for this Workshop: Informing Costs
Data selection scope extends from algorithms (but not hardware) for TP generation to L2 and “external triggering.”
Front End System (APA level trigger primitive generation) [x150] L0 (APA level) data selection L1 (module level) data selection L2 (High-level data selection) Applied to each event independently Event Builder (per Module) Permanent storage at Fermilab Front End System (APA level trigger primitive generation) [x150] Front End System (APA level trigger primitive generation) [x150] Long-term storage
Run Control & Config Databases External & Calibration Signals
Goals for this Workshop: Informing Costs
Basic cost model so far assumes:
hardware distinct from readout (e.g., FELIX+CPU, etc.) hardware 2. Generation of Module Level (L1) triggers done by yet more distinct hardware (perhaps one single server) and needs PDS, GPS, calibration information
and detectors
but also possibly TPs (less hardware because built event rate is low)
Goals for this Workshop: Informing Costs
Basic cost model so far assumes:
hardware distinct from readout (e.g., FELIX+CPU, etc.) hardware T
est L0 algorithms on “realistic” hardware and data samples to determine speeds and latencies
A lot of what we’ll discuss these two days is how to do this by…October.
Goals for this Workshop: Informing Costs
Basic cost model so far assumes:
(perhaps one single server) and needs PDS, GPS, calibration information T
est L1 algorithms on “realistic” hardware and data samples to determine speeds and latencies
A lot of what we’ll discuss these two days is how to do this by…October.
Goals for this Workshop: Informing Costs
Basic cost model so far assumes:
and detectors T
est external triggering on “realistic” hardware and data samples to determine speeds and latencies
A lot of what we’ll discuss these two days is how to do this by…October.
Goals for this Workshop: Informing Costs
Basic cost model so far assumes:
but also possibly TPs (less hardware because built event rate is low) T
est L2 algorithms on “realistic” hardware and data samples to determine speeds and latencies
A lot of what we’ll discuss these two days is how to do this by…October.
The Way Forward
Existing task list (next slides) needs to get applied to 3 parallel paths:
requirements
By my count, DUNE data selection has about 3-4 FTEs working on it now. That sounds…a little bit light…. (Current costing has 12.4 in CY19; max of 17.4 in CY20)
Data Selection Task List
1. Trigger primitive development and testing (SP and DP) A. Determination of primitive data 1. Summed ADC, peak, time…?) [Tyler, Phil] 2. Windowed or not (from an algorithm perspective, not implementation)? [Brett,Phil] 3. Threshold target and efficiency [Tyler, Phil, Georgia] 4. Filtering [Babak,Russel,Phil…] B. Algorithm development 1. Choice of hit-finding approach (discriminator, gauss, etc.) [Phil,Brett,?] 2. Baseline determination and consequent transfer function [Phil, Brett] C. Adding primitive generation in simulation [Tyler, Phil,Chris,?] 1. “Clean” MC waveforms+offline noise models [Phil,Chris,Miquel,Babak,Rivera…] D. Firmware tests on simulated waveforms [Russel, ?] 1. Determination of implementation constraints on algorithms [Russel,?] E. Firmware tests on (e.g.) ProtoDUNE waveforms [Russel, ?] F. Software tests on simulated waveforms [Phil, Alex, Joe, Georgia,…] 1. Determination of implementation constraints on algorithms [Phil, Brett,?] G. Software tests on (e.g.) ProtoDUNE waveforms [Rivera,?]
Data Selection Task List
A. Supernova burst algorithm [Alex,Pierre,?]
B. Beam triggering [David L, David R, ?] C. Atmospheric neutrino triggering [Olivia,?] D. Solar neutrino (?) triggering [Alex, Nuno, David R?] E. PDK triggers [?] F. “High-Level” (L2) Trigger algorithm development [Giles, Georgia,?]
3. PDS trigger development [hardware/software] A. Determination of trigger primitive data packets[Lasorak,Edinburgh] (charge, time, etc.) B. Decision on where primitive generation happens [?] C. Test of primitive generation in candidate hardware [?] D. Test of dispatch of PDS trigger primitives [?] 4. Trigger candidate algorithm testing [software] A. Define framework for software candidate generation [Weinstein, Warburton,Chris,?] B. Define hardware environment for candidate generation [Viren?,?] C. Create sample trigger primitive data streams [?] D. Implement algorithms on candidate hardware [Rodrigues,Viren,?] E. Implement High-Level Algorithms on candidate hardware [Barr, Karagiorgi,?] F. Speed tests with realistic rates [Rodrigues,?] G. Dispatch of candidates to module-level logic [?]
Data Selection Task List
5. Module-level trigger logic A. Define framework for software module trigger generation [Weinstein,?] B. Define hardware environment for module trigger generation [Viren?,Rodrigues?,?] C. Create sample trigger candidate data streams [?] D. Implement algorithms on candidate hardware [?] E. Speed and latency tests [?] F. Trigger command generation development [?] 6. “External” trigger development A. Define framework for external trigger generation [Weinstein,?] B. Define hardware environment for external trigger generation [Viren?,Rodrigues,?] C. Create sample module-level trigger data streams [?] D. Implement algorithms on candidate hardware [?] E. Speed tests with realistic rates [?] F. Development of trigger commands to send back to modules [?]
Data Selection Task List
Data Selection Task List
Summary
BACKUPS (Requirements, Costing, DAQ details)
Suggested Data Selection Requirements Discussed Previously
Rationale: Collaboration requirement
Rationale: Maintain sensitivity during downtimes as well as full 40 kt for Supernova bursts
Rationale: Reasonable limit for downstream analysis and storage Comments: “Enable” means downstream nearline can filter further
Rationale: No excuse to be otherwise; maximum livetime for follower events, supernova bursts
Rationale: Provide robustness to variations in noise; needed for detector calibrations
Rationale: Allow tests in offline analysis; coordinate with other expts; provide unique IDs
Rationale: Provide simple sorting for online and offline analyses; allow systematic checks
Rationale: Simplify tagging and run control interface
Rationale: Systematic checks of overall detector health
Data selection shall:
Rationale: May need to deal with high-rate calibration sources, or low-threshold triggers
Rationale: Allow zero suppression via location
Rationale: Detector health feedback
Rationale: Commissioning of elements that may come together at different times
Data selection shall:
Rationale: Best exploitation of beam data Efficiency needs to be defined!
Rationale: Smaller total flux in low (<100 MeV) window Comments: Change this to 50 MeV? 20 MeV…?
Rationale: Seems reasonable?
Rationale: Seems reasonable?
Data selection shall:
Georgia’s Costing Slides
DS Scope: 1 SP TPC
Front End System (APA level trigger primitive generation) [x150] L0 (APA level) data selection L1 (module level) data selection L2 (High-level data selection) Applied to each event independently Event Builder (per Module) Permanent storage at Fermilab Front End System (APA level trigger primitive generation) [x150] Front End System (APA level trigger primitive generation) [x150] Long-term storage
Run Control & Config Databases External & Calibration Signals
Front End System (APA level trigger primitive generation) [x150] L0 (APA level) data selection L1 (module level) data selection L2 (High-level data selection) Applied to each event independently Event Builder (per Module) Permanent storage at Fermilab Front End System (APA level trigger primitive generation) [x150] Front End System (APA level trigger primitive generation) [x150] Long-term storage
Run Control & Config Databases External & Calibration Signals
DS Scope: 1 SP TPC
Within DS scope is development of trigger primitive generation algorithms (that can be implemented in CPU or FPGA or GPU) but not the hardware where these algorithms reside, and physics validation studies. Notes:
software development and physics validation
development and validation on the teststand
development site: Columbia U. (also at other sites)
installation, commissioning of FE system (DS focused)
Guideline: PD=$100k/yr GS=$120k/yr
Front End System (APA level trigger primitive generation) [x150] L1 (module level) data selection L2 (High-level data selection) Applied to each event independently Event Builder (per Module) Permanent storage at Fermilab Front End System (APA level trigger primitive generation) [x150] Front End System (APA level trigger primitive generation) [x150] Long-term storage
Run Control & Config Databases External & Calibration Signals
DS Scope: 1 SP TPC
L0 (APA level) data selection
Within DS scope is development of L0 trigger algorithms (APA- level) as well as the hardware where these algorithms reside, and physics validation studies.
More specifically, within DS scope:
(CPU or GPU; possibility of FPGA?)
and physics validation
inputs
commissioning of system Example scheme 1: Each APA reports the following candidates to L1:
Example scheme 2:
Each candidate is reported with some position, time and calorimetric info (TBD)
Front End System (APA level trigger primitive generation) [x150] L1 (module level) data selection L2 (High-level data selection) Applied to each event independently Event Builder (per Module) Permanent storage at Fermilab Front End System (APA level trigger primitive generation) [x150] Front End System (APA level trigger primitive generation) [x150] Long-term storage
Run Control & Config Databases External & Calibration Signals
DS Scope: 1 SP TPC
L0 (APA level) data selection
L0 hardware scope: For a CPU approach, we can use current DUNE clustering resource use as a basis for evaluating system requirements Trigger primitives, if dominated by Ar39, amount to ~50k argon decays per drift per module * 4 x 2B words per primitive, or ~200 MB/s. This amounts to ~0.5 PB/3 months so need local(?) disk with 0.5 PB total long-term storage (disk) for trigger primitives. à Do we want these stored on-site
Front End System (APA level trigger primitive generation) [x150] L1 (module level) data selection L2 (High-level data selection) Applied to each event independently Event Builder (per Module) Permanent storage at Fermilab Front End System (APA level trigger primitive generation) [x150] Front End System (APA level trigger primitive generation) [x150] Long-term storage
Run Control & Config Databases External & Calibration Signals
DS Scope: 1 SP TPC
L0 (APA level) data selection
Clustering in DUNE:
take up to 10 sec per simulated event (1x2x6 APA x 5.4ms readout) for events with radiologicals. This is probably too large! (translates to >45k dunegpvm interactive machines in order to keep up)
tested for resource usage per APA; need to check average and max hit multiplicity per event, and different clustering algorithms Overall L0 hardware scope (CPU solution): Is 25 multi-core machines dedicated to clustering and generating APA-level trigger primitives at $10k each sufficient for 150 APAs?
Front End System (APA level trigger primitive generation) [x150] L1 (module level) data selection L2 (High-level data selection) Applied to each event independently Event Builder (per Module) Permanent storage at Fermilab Front End System (APA level trigger primitive generation) [x150] Front End System (APA level trigger primitive generation) [x150] Long-term storage
Run Control & Config Databases External & Calibration Signals
DS Scope: 1 SP TPC
L0 (APA level) data selection
Within DS scope is development of L1 trigger algorithms (that can be implemented in CPU as well as the hardware where these algorithms reside, and physics validation studies. Also included is integration with external systems (calibration, accelerator, SNEWS, and timing systems) More specifically, within DS scope:
receive timing and other (ext) trigger info, e.g. GPS card, or WD node card)
Front End System (APA level trigger primitive generation) [x150] L1 (module level) data selection L2 (High-level data selection) Applied to each event independently Event Builder (per Module) Permanent storage at Fermilab Front End System (APA level trigger primitive generation) [x150] Front End System (APA level trigger primitive generation) [x150] Long-term storage
Run Control & Config Databases External & Calibration Signals L0 (APA level) data selection
Within DS scope is development of L2 selection algorithms (that can be implemented in CPU or GPU as well as the hardware where these algorithms reside, and physics validation studies. Also included is integration with external systems (nearline processing, DQM, or any other systems which might be clients
More specifically, within DS scope:
DS Scope: 1 SP TPC
Front End System (APA level trigger primitive generation) [x150] L1 (module level) data selection L2 (High-level data selection) Applied to each event independently Event Builder (per Module) Permanent storage at Fermilab Front End System (APA level trigger primitive generation) [x150] Front End System (APA level trigger primitive generation) [x150] Long-term storage
Run Control & Config Databases External & Calibration Signals L0 (APA level) data selection
DS Scope: 1 SP TPC
GPU option: Based on CNN studies at Columbia: 6-fold classification takes 35 ms (up to 50 ms) per APA (collection only) per drift on single NVIDIA GeForce TITAN X 12 GB A single GPU card could process 1s/0.050s = 20 APA * 1 drift images in 1 second. Assuming a 1 Hz maximum trigger rate from event builder (single module) stage, we’d need (150 APA * 2 drifts /event) / (20 APA * 1 drift /card) = 15 cards to keep up GPU hardware scope could be:
per server)
Hard Drives (sized for 5 SN events 100 s each, ~150 TB x 5)
DAQ Paradigm Slides
Current DAQ Paradigm
suppression (~10 Tb/s/10 kt)
from all channels is stored
“Standard” Triggering (not supernova burst triggering!) A single “DUNE Event” is therefore 6.22 GB (uncompressed). We have a cap of 30 PB/year for all 4 modules=4.8 million events/year.
Current DAQ Paradigm
suppression (~10 Tb/s/10 kt)
equivalent (e.g., 3.5 MeV), it is passed on as a “trigger candidate” to a higher level
supernova burst is triggered
is (-10 s to +20 s) around the time at which the burst criterion was satisfied
lower threshold for a while (few minutes?)
Supernova Triggering
[TPC] Trigger Primitives
Working assumption:
above baseline
Sets of 960 collection wires/APA “streaming” primitives out 100% live. Need to:
[TPC] Trigger Primitives
For a 10 MeV threshold, maximum Dt is ~50 µs (= 3 39Ar decays/APA) More extended tracks (in time) could have primitives broken up into 50 µs chunks In this model, list of channels, times, charges, sent as primitives every 50 µs (or whatever Dt).
APA Level “L0”
Going from Trigger Primitives to “Trigger Candidates”
Trigger Candidates are based on “clustering” of hit wires, total charge of adjacent wires, and maximum time-over-threshold Trigger Candidates must include
Dominant source of background is neutrons from rock (etc.)---detector is entirely unshielded
Alex Booth
Module Level “L1”
Goal is to go from Trigger Candidates to Trigger Commands
For supernova bursts, simplest case is to count trigger candidates in a fixed time window (cf. Alex Booth) such as 10 s
Alex Booth
Tradeoff will always be between fake burst rate (=coverage) and total duration of burst
Module Level “L1”
Goal is to go from Trigger Candidates to Trigger Commands
More sophisticated approach would be to use weights Nint = S10 s(wi(Ei)) >Nburst So, e.g., wi=1.0 for Ei=18 “MeV” and wi=0.01(?) for Ei=3 “MeV”(?) Which has the nice behavior that we would see (maybe even) just two 18 MeV neutrinos that would extend our sensitivity for some fraction of more distant events (and non-standard events).
Module Level “L1”
Goal is to go from Trigger Candidates to Trigger Commands L1 is also the place where we would add in information from the photon system (cf. Pierre Lasorak). Could also help to reject “neutron bursts” from cosmics going through dead regions like rock.
High Level Trigger “L2”
Goal is to Provide Additional Filtering After L1
high rate single events that are clearly not physics (e.g., HV discharge) or fake supernova bursts
External Trigger (“L3”?)
Goal is to do Inter-Module and Inter-Detector Triggers
neutrons, etc.)
could form realtime coincidences
TP DAQ Schemes
Technical Proposal Model
The “Nominal” design
Technical Proposal Model
The “Alternate” design