[PPT] - News and Announcements Tom Junk and Andrew Norman DUNE Core PowerPoint Presentation

SLIDE 1

News and Announcements

Tom Junk and Andrew Norman DUNE Core Computing Meeting August 29, 2017

SLIDE 2

Upcoming Dates

ProtoDUNE-SP Cold Box test: Sep-Oct 2017.
Christos: APA 1 will have cold electronics installed this week and move to

cold box week of Sep. 4

DUNE Physics Week, Nov 14—17 at Fermilab – with structured working time

("Hack Days" style)

8/8/17

T. Junk and A. Norman | News

2

SLIDE 3

Many Thanks for the Tutorial!

August 14, the day before the collaboration meeting. Thanks to:
Eileen Berman – organizer
Erica Snider – LArSoft tutorial
Marc Paterno – Gallery tutorial
Mike Kirby – FIFE howto
Ken Herner – Running jobs on the grid
Pengfei Ding – Data Storage and Access
Tom Junk – Homework and programming advice

8/8/17

T. Junk and A. Norman | News

3

SLIDE 4

3x1x1 Data Import

392 .dat files transferred by Steve, uploaded to enstore, cataloged in
SAM. 384 GB total. Thanks!
Filenames changed from

wa105_r693-0.dat to wa105_r693_s0_1491239778.dat

This fooled the input service which expected filenames of the original

format to extract the run and subrun numbers.

I wrote a script that makes a symlink with the original format file name.
Steve defined a dataset called "wa105" with these dimensions:
nline.runnumber >= 633 and online.runnumber <= 843
No data tier specified for the .dat files.

8/8/17

T. Junk and A. Norman | News

4

SLIDE 5

3x1x1 Data Import

Some bugs found:
import script edited the fcl file to point to a pedestal file but failed to do it at

all.

import script relies on specific lines of the fcl file having specific line
numbers. A sed command with hardcoded line numbers is in the import
script. Fragile! Anyone who changes the fcl file and not the script breaks it.

But it was broken anyway

Pedestal file was in eos. I made a copy in dCache and asked Alex Himmel

to put it in dune_pardata

We were unknowingly guinea-pigging a new version of FTS which put

wrong checksums in the metadata. Robert Illingworth straightened it out right away.

And a questionable setting – Compression Level 0 on the root output. No

real reason for this.

8/8/17

T. Junk and A. Norman | News

5

SLIDE 6

3x1x1 Data Import

Data compression test, single file:
Raw .dat file: wa105_r776_s5_1499939129.dat

Size: 1.07 GB Event Count: 334

Imported .root file (raw::RawDigits)

Size: 1.44 GB (uncompressed)

Compressed .root file (default compression level=4)

Size: 0.400 GB

CPU: 13 sec for the file import without compression

260 sec with default compression (on dunegpvm08)

Compression factors: 3.6 over larsoft root, 2.7 over original .dat

8/8/17

T. Junk and A. Norman | News

6

SLIDE 7

3x1x1 Data Import

Imported data uploaded to enstore, cataloged in SAM, and indexed in dune-data.fnal.gov under

"detector data"

Script importdata311dpsam.sh is a keepup script – looks at a list of all input data files, sees which
nes have been done already, and starts working on the remainder. Assumes it's the only one
running. We'll have to reorganize this when processing is more CPU intensive. Script checked in to

dunetpc.

516 GB total
data tier = raw (maybe this is not a great choice)
New dataset definitions

wa105_3x1x1dp_daq file_name=wa105%.dat wa105_3x1x1dp_larsoftimport file_name wa105%.root and data_tier raw and lbne_data.name wa105_testdata_2017

Something else I noticed: only raw::RawDigits are present; no photon detector data products

8/8/17

T. Junk and A. Norman | News

7

SLIDE 8

ProtoDUNE Overlay Discussion

New WIB frame spec in DocDB 1701
Overlay proposal in DocDB 4877 – new version after iterating with JJ
Milo Vermeulen added as a developer on dune-raw-data. Ruth says they have

initial code but needed permission to check it in

Discussion with Brett on channel numbering and making sure we have access

to fiber and crate numbers

John Freeman has written a developer's guide for dune-raw-data

https://cdcvs.fnal.gov/redmine/projects/dune-raw-data/wiki/The_dune-raw- data_developer's_guide

JJ is making progress, but has to run the overlay code live in three places:
nline/offline (dune-raw-data)

Vivado HLS – in order to verify the firmware via simulation In the ARM chip – used to verify the firmware in a live environment

8/8/17

T. Junk and A. Norman | News

8

SLIDE 9

ProtoDUNE Overlay Discussion

We want the data access methods first so we can get on the air with the cold-

box test.

Debugging methods – just pass-through of the WIB data
Event-unpacking methods – compressed and uncompressed data
Inverse methods for testing – put MC digits into raw format (compressed and

uncompressed): interesting but of lower priority

I'd like to encourage online monitoring to use raw::RawDigits
same format as we propose to get out of the overlays
can test online monitoring modules with MC
no penalty except we have to keep the entire unpacked event in memory at the same

time as the packed event. Maybe put each APA in a separate branch to save RAM? Can drop input data after use.

8/8/17

T. Junk and A. Norman | News

9

SLIDE 10

Photon Detector Overlays

I talked briefly with Zelimir
Only small changes wrt 35-ton planned for photon detector readout

for ProtoDUNE, at least as far as data format is concerned

Re-use 35-ton overlay classes, though need to evaluate how they fit

in with the events – make sure events are consistent.

Martin Haigh (Warwick) had written them for 35-ton
Need to check with J. Freeman on boardreader/fragment generator

8/8/17

T. Junk and A. Norman | News

10

SLIDE 11

dunetpc librarian duties

We would like to help Tingjun out – he tells me he does these things for

dunetpc

Updates the larsoft version number in product_deps
Builds new release locally and runs unit tests (in case something in the new larsoft

release breaks something of ours)

Merges branches.
LArSoft-provided ones
User branches on request – resolve conflicts
Fields questions about data product differences found by the CI system – okay to

generate new reference files or start debugging?

n.b. Some DUNE repositories that depend on LArSoft aren't on Lynn's umbrella to

maintain when there are breaking changes in LArSoft (header files move around for example), such as the nucleon decay, ND, and BSM repositories.

8/8/17

T. Junk and A. Norman | News

11

SLIDE 12

dunetpc release manager duties

Increase the dunetpc version number
Build release on /grid/fermiapp
Install release in CVMFS
Clean up space in /grid/fermiapp
Test that the release works
Announce to dune-reco

8/8/17

T. Junk and A. Norman | News

12

SLIDE 13

Extras

8/8/17

T. Junk and A. Norman | News

13

SLIDE 14

Received Electronics Test Info from BNL

Warm ASIC test results
/dune/data/users/bkirby/20170706_elecprod_quadFeAsic.tar
JSON file documentation coming at:

https://github.com/DUNE/femb_python/tree/master/femb_python /test_measurements/feAsicTest

Files have a formatting bug; easy enough to fix.
Contain mean, RMS, and Gain. Mean may be the pedestal,

need further clarification. Also failure flags.

I asked Brian if these ASICs are the ones to be used in the

upcoming test of APA 1 at CERN.

8/8/17

T. Junk and A. Norman | News

14

SLIDE 15

Electronics Test Results from BNL

These are warm measurements – need cold linearity scans.

But we can practice on these.

Files are in JSON format – some parsing to put them in rootfiles
one-time warm measurements – probably a database is not the

right place to put this data. But the failure flags may be.

Need channel mapping to figure out how to use this data offline.

8/8/17

T. Junk and A. Norman | News

15

SLIDE 16

Deliverables for the DUNE Computing Consortium

Discussion items sent to Mark:
Each of the subgroups
Data Management
Software Management
Production Databases
Collaborative Tools

needs a group leader and a deputy. Two-year rotations are

deliverables. Effort levels scale with need -- ProtoDUNE
perations and analysis will put more strain on the groups as we

move forwards.

8/8/17

T. Junk and A. Norman | News

16

SLIDE 17

Deliverables for Computing Consortium

Specific deliverables, though some are more ongoing services

than deliverables:

Databases for ProtoDUNE
Online conditions
Calibration
Beam Instrumentation
Hardware Database
These involve interfacing with stakeholders, defining schema,

collecting data, adding it to the database, and providing retrieval mechanisms and documentation

8/8/17

T. Junk and A. Norman | News

17

SLIDE 18

Deliverables for Computing Consortium

Data Transfer tools (FNAL SCD is doing a good job here)
Data catalog for ProtoDUNE and DUNE Metadata schema

update

Production Management Tool Upgrades and maintenance
Remote job submission support
Negotiating with OSG sites & T0 (Steve has been doing this)
Running on NERSC machines

8/8/17

T. Junk and A. Norman | News

18

SLIDE 19

Deliverables for the Computing Consortium

Lots of software development is ongoing, but it is under Physics. Computing

items:

Data Reduction tool (added to list since sending to Mark)
Prompt Processing system & payloads (added to list since sending to Mark)
Design the data production workflow -- decide what we can run on what

samples of events

Design the event stream categories for ProtoDUNE
Optimize the reco for CPU and memory MCC10, MCC11, ...
Interface with art and LArSoft teams
Weekly dunetpc code releases
updating CVMFS
Archiving and deleting old code

8/8/17

T. Junk and A. Norman | News

19

SLIDE 20

Deliverables for the Computing Consortium

Managing disk space at Fermilab: (/dune/app, /dune/data,

/dune/data2, /pnfs/dune/persistent)

VO management
Handling requests for short-term out-of-band resource needs
CI system interface, writing tests, monitoring output,

communicating with code authors Interface with infrastructure providers

Service Desk
Networking
User support for all of the above
Documentation and Wikis

8/8/17

T. Junk and A. Norman | News

20

SLIDE 21

ProtoDUNE-SP Overlay Method Discussion has started

https://indico.fnal.gov/conferenceDisplay.py?confId=14794
Data will be compressed in the RCE's or in software (FELIX)
Data compression methods – compress ADCt – ADCt-1 naturally

force data to be arranged along the time axis.

35-ton data were in nanoslices – arranged on the channel axis
Slow data access had to turn the data sideways to get it in

raw::RawDigit. Also multiple inefficient copies were made in the splitter.

No need to split events in ProtoDUNE-SP. 10000-tick events

are made by the DAQ, centered on a trigger.

8/8/17

T. Junk and A. Norman | News

21

SLIDE 22

Overlay Method Discussion

Uncompressed ProtoDUNE-SP events are ~300 MB big.
Would like to minimize copying these in an offline job, or unpacking more than
nce
JJ Russell says he will implement a table of contents in his packets so

channels can be accessed one at a time

RCE packets do not line up with the triggers – compression happens before

the trigger comes in, and doesn't happen a second time. Have some extra ticks on the ends that need trimming.

FELIX relies on downstream computing resources (fragment generator) to

compress, so no trimming needed. But buffering is required in the fragment generator to cut the events to size.

No contact yet with SSP people. Or CRT.
DP: input source written by Kevin Fusshoeller and used with 3x1x1 data

8/8/17

T. Junk and A. Norman | News

22

SLIDE 23

Software Repo Organization Proposal from John Freeman, July 3

If the code will be used in modules which will be run just for
nline monitoring (and not used in offline), it should go in dune-

artdaq.

If it will be used both in offline and for online monitoring, it

should go in dune-raw-data.

If it will be used only for offline analysis, somewhere else -

dunetpc?

8/8/17

T. Junk and A. Norman | News

23