S i mp l i fj e d F P G A - b a s e d s c h e me : A s c a l i n g e x e r c i s e u s i n g S B N D J o s é I . C r e s p o - A n a d ó n C o l u m b i a U n i v e r s i t y N e v i s L a b o r a t o r i e s D U N E F D D A Q “ D e s i g n ” Wo r k s h o p 3 0 – 3 1 O c t o b e r 2 0 1 7 , C o l u m b i a U n i v e r s i t y , U S A
Introduction Will cover only TPC readout. SBND DAQ will be quite similar to MicroBooNE (already in operation, see Wes' talk): This talk focuses on back-end electronics with FPGA (same functionality as ● MicroBooNE). System not optimized for DUNE scale. 2 José I. Crespo-Anadón | Simplified FPGA-...
SBND DAQ design requirements Triggered readout stream ● Trigger rate: ≤ 20 Hz (15 Hz BNB + 1.25 Hz NuMI) – ✕ Readout time window: 3 1.28 ms (drift time) = 3.84 ms – Data rate: 11,264 channels ✕ 2 MS/s ✕ 2 Bytes ✕ 3.84 ms ✕ 20 Hz ≤ 3.46 GB/s – distributed between 11 DAQ servers: 315 MB/s/server – Lossless compression (Huffman) Continuous (a. k. a. supernova) readout stream ● ✕ ✕ Data rate: 11,264 channels 2 MS/s 2 Bytes = 45 GB/s – distributed between 11 DAQ servers: 4.1 GB/s/server – Lossless compression (Huffman) and zero suppression (factor ✕ ~ 80) 3 José I. Crespo-Anadón | Simplified FPGA-...
Reminder of SBND front-end electronics Assuming cold ADC design. Very similar design to ProtoDUNE-SP. ● 11264 channels. ● 88 Front End Mother Boards (FEMB) . 128 ch/FEMB. ● Amplification and shaping in custom CMOS ASICs (180 nm). Four gains: – 4.7, 7.8, 14, 25 mV/fC. Four peaking times: 0.5, 1.0, 2.0, 3.0 μs. Two baselines: 200 mV, 900 mV. 12-bit ADC at 2 MS/s . – FPGA (Altera Cyclone IV) mezzanine sends the data out of the cryostat – over cold cable (4 links × 1.28 Gbps/link). Warm Interface Board (WIB) with an FPGA (Altera Arria V) receives the data ● from 4 FEMB (512 ch) and sends it to back-end electronics through 8 optical links (2.125 Gbps/link). SBND: arXiv:1503.01520 [physics.ins-det] 4 José I. Crespo-Anadón | Simplified FPGA-...
SBND front-end & back-end (partial) overview 5 José I. Crespo-Anadón | Simplified FPGA-...
SBND Crate, Clock board & Controller XMIT Crate 6U crate with capacity for 20 boards. ● Clock board 1 board on the back of the backplane receives 16 MHz clock from Master Clock NIM ● module. Controller Interface between DAQ server (with PCIe card via optical link) and boards in crate. ● Receives trigger from Trigger Fan Out (FO) in NIM bin. Can also generate local trigger. ● Enables slow readout for debugging. ● 6 José I. Crespo-Anadón | Simplified FPGA-...
SBND FEM Front End Module (FEM) . 64 ch/board (typically 32 induction ch + 32 collection ch). Optical receiver (ORx) receives data from WIB via 2.125 Gbps optical link. Then deserialized. 1 M × 36 bit 128 MHz SRAM as ring buffer. 8 frames in buffer (1.28 ms/frame × 8 frames = 10.24 ms). 64 MHz for writing in time-order. 64 MHz for reading by channel. No deadtime. Data processing by FPGA (Altera Stratix III). Splits data in two streams: 1) Triggered stream : reads out 4 frames, trims to 1 frame before + 2 frames after trigger. 64 ch × 3 × 1.28 ms window/ch × 2 MS/s × 2B/S = 0.98 MB/board/trigger. Applies lossless (Huffman) compression. 2) Continuous stream : read out all the time. FPGA applies zero-suppression based on dynamical estimation of baseline and amplitude threshold (next slide), then applies Huffman compression. Each stream has a DRAM to buffer the processed data until retrieved. 7 José I. Crespo-Anadón | Simplified FPGA-...
FPGA-based zero suppression for continuous stream DATA from MicroBooNE FEM at Nevis test stand SN stream Dynamic baseline (emulated) ± threshold 7 presamples 8 postsamples Trigger stream Implemented in the FEM FPGA. ● Only the waveform passing a certain amplitude threshold (configurable) with respect ● to the channel baseline is saved, plus presamples and postsamples (configurable) . The baseline is dynamically computed using 192 preceding samples. ● 8 José I. Crespo-Anadón | Simplified FPGA-...
MicroBooNE & SBND: “supernova” stream Surface detectors. Cannot self-trigger on supernova neutrinos . ● Instead, write data to disk continuously and rely on a delayed (hours) external trigger from the Supernova Early Warning System (SNEWS). Bottleneck of the stream is the disk writing speed at the DAQ server (~ 50 – 100 ● MB/s). Input: ~ 4 GB/s/server. Use zero suppression + Huffman compression to achieve compression goal (~ × 40 – 80) . Finalizing commissioning in MicroBooNE. Similar system planned for SBND. ● Michel e candidate from supernova stream Michel e candidate from trigger stream Data from region on Bremsstrahlung μ collection plane after g n i zero-suppression p p Michel e o t S Amplitude threshold: +30 ADC (noise/MIP μ separation) g n i p p o t S Michel e Bremsstrahlung 9 José I. Crespo-Anadón | Simplified FPGA-...
SBND XMIT and PCIe card XMIT Transmitter (XMIT) board 1 board/crate reads up to 16 FEMs = 2 WIBs = 8 FEMBs = 1024 ch. ● Fetching data from each FEM through backplane (512 MB/s) using token ● passing. Triggered stream prioritized over continuous stream. ● Data sent to PCIe card on DAQ server via optical links (3.125 Gbps × 2). ● Each stream has its own optical link pair. ● PCIe card Receives data from one stream from XMIT. PCIe card ● DMA using PCIe bus. 4 lanes × 2.5 Gbps = 1.25 GB/s. ● 10 José I. Crespo-Anadón | Simplified FPGA-...
SBND Trigger board Inputs: PMT trigger ● Accelerator gates (BNB, NuMI, BNB ● dump; external) Cosmic Ray Tracker trigger ● Inputs Laser (calibration) ● (12) GPS PPS ● Configurable logic and prescaling. Outputs: Outputs Global trigger. ● (10) Trigger information sent to DAQ server ● (PCIe card via optical link). DAQ-driven calibration trigger (for cold ● ASICs). Scaler outputs. ● 11 José I. Crespo-Anadón | Simplified FPGA-...
SBND back-end electronics for DUNE: deliverables 150 APA/module. 2560 ch/APA. 40 FEMs/APA. 6000 FEMs/module. ● (Assuming 16 FEMs/crate as SBND) 2.5 crates/APA. 375 crates/module. ● Each crate includes 1 Controller, 1 XMIT, 1 Clock board, 1 – Backplane, 1 Chassis. 1 power supply/crate. – 1 crate is read by 1 DAQ server with 3 PCIe cards. – SBND has +10% components as hot spares. 10% of spares for ● DUNE are enough to fully instrument 3 SBND. Assuming +2%. 12 José I. Crespo-Anadón | Simplified FPGA-...
SBND back-end electronics for DUNE: cost FEM, Crate (instrumented) and Trigger Board cost based on MicroBooNE's ● experience. Not included: ● Optical links between WIB and FEM. – Racks and optical links between XMIT and PCIe cards. – Not accounting for inflation, chip obsolescence, or cost reduction. ● Can save 4.3% (595 k$) by having 18 FEMs/crate (334 + 7 crates). ● 13 José I. Crespo-Anadón | Simplified FPGA-...
SBND back-end electronics for DUNE: power Back-end electronics: ● ~ 240 W/crate × 375 crates/module = 90 kW/module – (not including cooling, factor ~ 2 in SBND) DAQ server: ● 360 W/server × 375 servers/module = 135 kW/module – 14 José I. Crespo-Anadón | Simplified FPGA-...
Possible improvements over SBND design for DUNE After splitting the data into triggered and continuous streams, keep them physically separated. Do not use prioritization; use two different buses in the crate backplane instead. ● Do not read both streams with the same DAQ server; have different DAQ servers for ● different streams, and have one DAQ server reading more than one crate instead. Fit as many PCIe cards as possible per DAQ – server. Limited by PCIe slots in server motherboard/PCIe bus bandwidth. Disk-write limitation for continuous stream can be – avoided by keeping data in (large) circular buffer in RAM and run a software trigger (GPU-powered?) to filter what is written/further processed. TPC-based trigger fostered by having multiple ● crates converging into one DAQ server → More channels to find patterns. 15 José I. Crespo-Anadón | Simplified FPGA-...
Backup 16 José I. Crespo-Anadón | Simplified FPGA-...
SBND TPC back-end To other crates Clock Trigger FO 16 MHz Crate DAQ server Clock board Trigger Controller FEM R Backplane (512 MB/s) Frame Frame Frame Frame PCIe bus (1.25 GB/s) Frame Frame Frame Frame PCIe card W Optical link (390 MB/s × 2 ) PCIe DRAM Neutrino SRAM card ORx TPC Deserial 1 M × 36 bit 128 MHz XMIT DRAM Supernova PCIe card 17 José I. Crespo-Anadón | Simplified FPGA-...
Recommend
More recommend
