DUNE Software and Computing Incomplete Overview for NPPS Brett Viren Physics Department NPPS – 2019-06-05
Outline Experiment Online Computing (Far Detector DAQ) Offline Computing Brett Viren (BNL) DUNE S&C 05 June 2019 2 / 24
Experiment DUNE Experiment and Physics • Long-baseline neutrino beam: discover ν CP-violation, ν mass hierarchy, precision ν oscillation parameter measurements. • Nucleon decay: targeting SUSY-favored modes ( p → K + ¯ ν ) • Supernova Neutrino Burst (SNB): sensitive to the galaxy, sensitive to ν e (complementary to water-Chernkov). Brett Viren (BNL) DUNE S&C 05 June 2019 3 / 24
Experiment Brett Viren (BNL) DUNE S&C 05 June 2019 5 / 24
Experiment Brett Viren (BNL) DUNE S&C 05 June 2019 6 / 24
Experiment Brett Viren (BNL) DUNE S&C 05 June 2019 7 / 24
Experiment Brett Viren (BNL) DUNE S&C 05 June 2019 8 / 24
Experiment Brett Viren (BNL) DUNE S&C 05 June 2019 9 / 24
Online Computing (Far Detector DAQ) DUNE Far Detector DAQ Overview • Common DAQ for 4 loosely-coupled 10kton modules. ◦ LAr 3-plane wire readout, LAr/GAr 2-plane strip readout, future LAr pixel? • O ( 10 6 ) channels, 2 MHz, 12bit waveforms, 3 – 4 TB/s into DAQ. • Output to tape: 30 PB/year ≈ 3000 × reduction. • Physics drivers of DAQ design: ◦ supernova neutrino burst : 10s of pre-trigger buffer, 100s full readout ◦ natural 39 Ar decay : 10 MHz, 0.5 MeV endpoint energy, reject • Self-triggering on ionization activity (largely software-based) • Fermilab’s artDAQ used now in protoDUNE, expected to provide basis for DAQ back-end. ◦ considering to aggregate triggered data “event” via distributed file system directory (eg, glusterfs ), or as entry in key-value store (eg, DAQ-DB ), following ATLAS R&D and technology studies. Brett Viren (BNL) DUNE S&C 05 June 2019 10 / 24
Online Computing (Far Detector DAQ) FELIX as DUNE FD DAQ Input Interface FELIX collaboration: BNL, ANL, Bologna, CERN, FNAL, Irvine, Nikhef, UCL and Weizmann BNL: hardware design and co-development of firmware. • Thin custom hardware and FPGA between detector electronics and DAQ’s commodity computing. • R&D shared with ATLAS, protoDUNE, sPHENIX, Belle2, others. • Powerful FPGA, 48 optical I/O (460 Gbps), support for daughterboard. • Commodity host PC interface: PCIe gen3 x16, (16 GB/s). o goal: 75 front-end PCs, each with 2 FELIX PCIe gen3 boards (per 10kt) o stretch goals: reduce PC/board count by 2 – 4 with PCIe gen4 Brett Viren (BNL) DUNE S&C 05 June 2019 11 / 24
Offline Computing Data Transfer and File Catalog • For prototype detectors (ProtoDUNE) transfer raw data from CERN to FNAL via Fermi-FTS using SAM for data catalog. • For DUNE, expecting to transition to Rucio for online → offline and production data management. • Replacement for SAM under consideration. Brett Viren (BNL) DUNE S&C 05 June 2019 12 / 24
Offline Computing Major Offline Processing Stages Signal processing noise filters and detector response deconvolution. Heavy use of FFTs. Output signal-regions-of-interest ≈ 100 × data reduction. 3D Imaging reconstruct ionization activity patterns. Fast, compressed sensing techniques. Conventional reconstruction clustering, track/show modeling, pattern recognition. Machine learning dense and sparse CNN, Graph NN, GANS. Brett Viren (BNL) DUNE S&C 05 June 2019 13 / 24
Offline Computing Wire-Cell Toolkit • Provides leading LAr TPC signal+noise simulation, noise filtering, signal processing, 3D imaging algorithms. ◦ pattern recognition, charge/light matching in prototype • Toolkit supports data flow programming paradigm. ◦ dynamic plugin system, comprehensive configuration via Jsonnet • Abstract DFP graph execution engine, multiple implementations ◦ Default is low-memory, single-threaded ◦ Experimental multi-thread based on Intel TBB ◦ Future, multi-node engine possible • Developed and maintained by BNL. ◦ Initially for MicroBooNE, now for ProtoDUNE, ICARUS, DUNE.... • Runs stand-alone CLI or embedded in Fermilab’s art /LArSoft framework Brett Viren (BNL) DUNE S&C 05 June 2019 14 / 24
Offline Computing WCT Job Graph for ProtoDUNE 3D Imaging GridTiling [tiling-apa0-face0] 0 0 JsonClusterTap anode = AnodePlane:apa0 0 BlobSetSync BlobClustering BlobSolving ClusterSink BlobGrouping [clustertap-apa0] face = 0 [blobsetsync-apa0] 0 0 [blobclustering-apa0] 0 0 0 0 [blobsolving-apa0] 0 0 0 0 [clustersink-apa0] [blobgrouping-apa0] drift_speed = 0.0016 1 multiplicity = 2 spans = 1 threshold = 0 filename = clusters-apa-apa0-%d.dot filename = clusters-apa0-%04d.json SliceFanout 0 GridTiling 0 [slicefanout-apa0] [tiling-apa0-face1] SumSlices multiplicity = 2 1 0 0 anode = AnodePlane:apa0 [slicing-apa0] AnodePlane face = 1 0 anode = AnodePlane:apa0 0 [apa0] tag = faces = list(2) tick_span = 4 ident = 0 nimpacts = 10 wire_schema = WireSchemaFile GridTiling [tiling-apa1-face0] 0 0 anode = AnodePlane:apa1 BlobSetSync BlobClustering BlobSolving JsonClusterTap ClusterSink face = 0 0 BlobGrouping [clustertap-apa1] [blobsetsync-apa1] 0 0 [blobclustering-apa1] 0 0 0 0 [blobsolving-apa1] 0 0 0 0 [clustersink-apa1] [blobgrouping-apa1] drift_speed = 0.0016 1 multiplicity = 2 spans = 1 threshold = 0 filename = clusters-apa-apa1-%d.dot filename = clusters-apa1-%04d.json SliceFanout 0 GridTiling 0 [slicefanout-apa1] [tiling-apa1-face1] SumSlices multiplicity = 2 1 0 0 anode = AnodePlane:apa1 [slicing-apa1] AnodePlane face = 1 0 anode = AnodePlane:apa1 0 [apa1] tag = faces = list(2) tick_span = 4 ident = 1 nimpacts = 10 wire_schema = WireSchemaFile GridTiling [tiling-apa2-face0] 0 0 anode = AnodePlane:apa2 JsonClusterTap face = 0 0 BlobSetSync BlobClustering BlobSolving ClusterSink BlobGrouping [clustertap-apa2] [blobsetsync-apa2] 0 0 [blobclustering-apa2] 0 0 0 0 [blobsolving-apa2] 0 0 0 0 [clustersink-apa2] [blobgrouping-apa2] drift_speed = 0.0016 1 multiplicity = 2 spans = 1 threshold = 0 filename = clusters-apa-apa2-%d.dot SliceFanout 0 filename = clusters-apa2-%04d.json GridTiling 0 [slicefanout-apa2] [tiling-apa2-face1] SumSlices multiplicity = 2 1 0 0 anode = AnodePlane:apa2 [slicing-apa2] AnodePlane face = 1 0 anode = AnodePlane:apa2 0 [apa2] tag = faces = list(2) tick_span = 4 ident = 2 nimpacts = 10 0 wire_schema = WireSchemaFile 1 wclsCookedFrameSource ChannelSplitter 2 [sigs] [peranode] GridTiling 0 0 art_tag = xxxx anodes = list(6) 3 [tiling-apa3-face0] 0 0 frame_tags = list(0) tag_rules = list(6) SumSlices anode = AnodePlane:apa3 JsonClusterTap 4 0 BlobSetSync BlobClustering BlobSolving ClusterSink [slicing-apa3] SliceFanout face = 0 BlobGrouping [clustertap-apa3] 0 [blobsetsync-apa3] 0 0 [blobclustering-apa3] 0 0 0 0 [blobsolving-apa3] 0 0 0 0 [clustersink-apa3] 5 0 anode = AnodePlane:apa3 0 0 [slicefanout-apa3] [blobgrouping-apa3] drift_speed = 0.0016 1 multiplicity = 2 spans = 1 threshold = 0 filename = clusters-apa-apa3-%d.dot tag = multiplicity = 2 1 filename = clusters-apa3-%04d.json GridTiling tick_span = 4 [tiling-apa3-face1] 0 0 anode = AnodePlane:apa3 AnodePlane face = 1 [apa3] faces = list(2) ident = 3 nimpacts = 10 wire_schema = WireSchemaFile GridTiling [tiling-apa4-face0] 0 0 JsonClusterTap anode = AnodePlane:apa4 0 BlobSetSync BlobClustering BlobSolving ClusterSink SumSlices BlobGrouping [clustertap-apa4] face = 0 [blobsetsync-apa4] 0 0 [blobclustering-apa4] 0 0 0 0 [blobsolving-apa4] 0 0 0 0 [clustersink-apa4] [slicing-apa4] SliceFanout 0 [blobgrouping-apa4] drift_speed = 0.0016 1 multiplicity = 2 spans = 1 threshold = 0 filename = clusters-apa-apa4-%d.dot 0 anode = AnodePlane:apa4 0 0 [slicefanout-apa4] filename = clusters-apa4-%04d.json 1 tag = multiplicity = 2 GridTiling tick_span = 4 [tiling-apa4-face1] 0 0 anode = AnodePlane:apa4 AnodePlane face = 1 [apa4] faces = list(2) ident = 4 nimpacts = 10 wire_schema = WireSchemaFile GridTiling [tiling-apa5-face0] 0 0 JsonClusterTap anode = AnodePlane:apa5 BlobSetSync BlobClustering BlobSolving ClusterSink SumSlices 0 BlobGrouping [clustertap-apa5] face = 0 [blobsetsync-apa5] 0 0 [blobclustering-apa5] 0 0 0 0 [blobsolving-apa5] 0 0 0 0 [clustersink-apa5] [slicing-apa5] SliceFanout 0 [blobgrouping-apa5] drift_speed = 0.0016 1 multiplicity = 2 spans = 1 threshold = 0 filename = clusters-apa-apa5-%d.dot 0 anode = AnodePlane:apa5 0 0 [slicefanout-apa5] filename = clusters-apa5-%04d.json tag = multiplicity = 2 1 GridTiling tick_span = 4 [tiling-apa5-face1] 0 0 anode = AnodePlane:apa5 AnodePlane face = 1 [apa5] faces = list(2) ident = 5 nimpacts = 10 wire_schema = WireSchemaFile Brett Viren (BNL) DUNE S&C 05 June 2019 15 / 24
Offline Computing “Bee” BNL’s Web-based Visualization https://www.phy.bnl.gov/wire-cell/bee/set/bccdc2d7-16e1-4363-8034-032d9fe2de50/event/0/ Developed/maintained by Chao Zhang. WebGL, GPU accelerated. PC/phone browsers. JSON data, user uploads. Brett Viren (BNL) DUNE S&C 05 June 2019 16 / 24
Recommend
More recommend
