Computing and Reconstruction in PANDA Stefano Spataro ISTITUTO - - PowerPoint PPT Presentation

Stefano Spataro

Monday, 12th May 2014

ISTITUTO ¡NAZIONALE ¡ DI ¡FISICA ¡NUCLEARE ¡ Sezione ¡di ¡Torino ¡

Computing and Reconstruction in PANDA

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

Overview

Ø Introduction to the Panda Experiment Ø The offline framework (PandaRoot) Ø Tracking and Offline Reconstruction Ø Triggerless Data Acquisition Ø Online Reconstruction

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

Facility or Antiproton and Ion Research

What is FAIR?

GSI

forest forest forest ???

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

Darmstadt (Germany)

GSI

Facility for Antiproton and Ion Research

Let’s speak about future experiments

In the upcoming future (taking data from 2018) …

High Energy Storage Ring

Storage Ring

¤ p = 1.5 – 15 GeV/c

High intensity mode

ž L = 1032 cm-2 s-1 , σp/p = 10-4 ž Electric cooling

High resolution mode

ž L = 1031 cm-2 s-1 , σp/p = 10-5 ž Stochastic cooling

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

pp, pA collisions 1.5⇒15 GeV/c (p momentum) Ø Charmonium (cc) spectroscopy Ø Open charm spectroscopy Ø Search for gluonic excitations (hybrids - glueballs) Ø Charmed hadrons in nuclei Ø Drell-Yan Ø Single and double Hypernuclei Ø Parton Dist., EM Form Factor…

2017: Detector Assembly 2018: First Data taking

More than 500 physicists from more than 54 institutions in 17 countries

Anti-proton power

Drawbacks of pp collisions

Fundamental a high performance trigger

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

Software manpower is limited and busy also with other activities

The Panda Solution Use a framework already used by other experiments Ø Less software developments for our computing group Ø More people using the same code → better debug Ø Share of the same tools by larger community

ü Data objects format ü Geometry handling ü I/O Manager ü Database connection (which DB?) ü Simulation of physics processes (G3, G4, Fluka, ?) ü Event Display ü Advanced Analysis Tools What do you need from a reconstruction software?

Panda ¡decided ¡ to ¡join-­‑> ¡ FairRoot: ¡same ¡ Base ¡package ¡ for ¡different ¡ experiments ¡

2006 ¡

The FairRoot History

MPD ¡(NICA) ¡ start ¡also ¡using ¡ FairRoot ¡ Start ¡tesOng ¡ the ¡VMC ¡ ¡ concept ¡for ¡ CBM ¡ First ¡Release ¡of ¡ CbmRoot ¡ ¡

2004 ¡

• M. Al-Turany

ASYEOS ¡joined ¡ (ASYEOSRoot) ¡ GEM-­‑TPC ¡ seperated ¡ from ¡PANDA ¡ branch ¡ (FOPIRoot) ¡ R3B ¡joined ¡ EIC ¡(Electron ¡ Ion ¡Collider ¡ BNL) ¡ EICRoot ¡

2011 ¡ 2010 ¡ 2012 ¡

SOFIA (Studies On Fission with Aladin) ¡ ENSAR-­‑ROOT ¡ CollecOon ¡of ¡ modules ¡used ¡by ¡ structural ¡nuclear ¡ phsyics ¡exp. ¡

2013

Postgresql Root files MySQL Oracle Run Manager Event Generator Magnetic Field Detector base IO Manager Tasks RTDataBase Root ¡files ¡ ¡Hits, ¡ ¡ Digits, ¡ ¡ Tracks ¡ Application Cuts, processes Event Display Track propagation TSQLServer

Virtual MC Geant3 Geant4 G4VMC G3VMC Geometry

STT MUO TOF GEM EMC MVD DIRC FTS ASCII EvtGen DPM Pythia Track finding digitizers Hit Producers Dipole Map Solenoid Map const. field

Panda Code

The PandaRoot Code Design

FairRoot

PandaRoot

CbmRoot R3BRoot MPDRoot (NICA) ASYEOSRoot EICRoot

M.Al-Turany, D.Bertini, F.Uhlig, R.Karabowicz

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

ü No executable Root macros to define the experimental setup, the tasks for reco/analysis, the configuration ü No fixed simulation model Different simulation models with the same user code (VMC) ü No fixed output structure Dynamic event structure based on Root TFolder and TTree

Data and Analysis Structure

ROOT Geometry and Event Display

TGeoManager TEve

MultiVariate Particle Identification

implementation of ROOT TMVA methods Ø EMC shower shape analysis

← Correlation Variables →

e/π separation in EMC MLP

ü K- Nearest Neighbors (KNN) ü Learning Vector Quantization (LVQ) ü Multi Layer Perceptron (MLP) ü …

Tracking on Proof on Demand

2 worker nodes 4 worker nodes 4 worker nodes PoD on external CPUs with SSH (4CPUs)+(8CPUs)+(8CPUs) a lot of work to modify the code to make it “Proof compatible”

• R. Karabowicz

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

Compiled and running

• n many Linux distributions and on MAC OS X

Using a set of self-configurating scripts (CMake) and regular checks (DashBoard) Everybody in his desktop, laptop, local farm can run the code w/o problems (hopefully)

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA Nightly → all nights Continuous → each commit Experimental → on demand

…and Rule Checker Also compatibility with “stable” data sample

F.Uhlig

PandaGrid: an Alien2-based GRID

Successfully used for: ü Central Tracker TDR ü MVD TDR ü Many Analyses Why Alien?

Ø It can run on all platforms (source distribution) Ø Several Panda institutions were hosting Alien sites ü “Reuse” of currently existing manpower ü Use of parts of already existing resources ü Strong collaboration with Alien developers

Beta-tester (now 2.20)

No further developments for Alien2 Ø Alien3? Ø PanDa? Ø Big batch farm? Still some time to design the distributed computing

Distributed T0/T1 centre embedded in Grid/Cloud

APPA CBM LQCD NuStar Panda (66k cores,12PB disks,12PB/y tape)

in 2018 300000 cores + grid 40 PB disk 40PB/y archive

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

Geometry from TGeoManager field map

target spectrometer forward spectrometer

p

target forward

MVD

p

GEM FTS STT

Global Tracking

Prefit Track Kalman Filter

(GENFIT- Munich*)

Detector Hits Track Follower

(GEANE – Pavia**)

same geometry for simulation and track following

Energy loss Not homogeneous magnetic field Different detector hits Ø 3D points – (TPC) Ø planar hits – MVD/GEM Ø tube + drift time – STT/FTS

Tracking: Global Fit

barrel forward

**A.Fontana, L.Lavezzi, A.Rotondi * C. Höppner, S.Neubert

Barrel Tracking: Pattern Recognition 3° step – Extrapolation to GEM planes 2°step – Correlation of STT tracks with MVD hits 1° step – STT (+SciTil) pattern recognition 4° step – Kalman Filter (Genfit)

MVD STT

G.Boca, R. Karabowicz , L.Lavezzi

Available information Ø Position/orientation of the tube Ø Drift radius (from drift time) Parallel tubes How STT Pattern Recognition works Skewed tubes ü XY roadmap track finding ü Association of skewed tubes ü From skewed tubes -> Z ü SciTil for track cleaning

X Y STT

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

Barrel Tracking: Performances Large Improvement from MVD/GEM detectors STT stand-alone STT+MVD+GEM

S.Costanza, L.Lavezzi

• W. Erni et al, EPJA 49 (2013) 25

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

Other Pattern Recognition Algorithms

Ø Riemann Track Finder (T. Stockmanns et al.) (see Andreas Herten talk tomorrow) Ø Barrel Track Finder (R. Karabowicz) Use at the same time of MVD + STT + GEM Ø V0 Track Finder (L. Lavezzi) For particles decaying far from the interaction point Ø Cellular Automaton @ FIAS (just started, not yet in PandaRoot)

Forward Tracking

15.00 11.91 8.90 4.06 1.50

_ p @ [GeV/c]

|B| [T]

Ø Ideal Pattern Recognition Ø Kalman Filter

muons

FTS + MVD + GEM

E.Fioravanti, I.Garzia, R.Kliemt

Realistic Forward PR (ongoing)

Based on Hough Trasformation

• M. Galuska et. al., PoS(Bormio 2013)023

Data Acquisition Challenges Ø Interaction rates of 20 MHz (50 MHz peak) Ø Event size of ∼15 kB Ø Data rates after front end preprocessing: 80GB/s ‐ 300GB/s

H.Xu, TIPP2011, Chicago

Ipeak/Iavg ≈ 2-2.5

Many benchmark channels Background & signal similar Where is our signal?

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

• Required reduction factor: ~1/1000 (all triggers in total)
• A lot of physics channel triggers → even higher reduction factor required

Events/Data acquired by DAQ (temporarily buffered)

Software Trigger Algorithms

„Trickle“ of events stored on disc

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

Physics Book criteria:

• J/psi (" base for many charmonia)

– Invariant Mass: Tracking/Momentum – Electron ID: Tracking, cluster energy, track/cluster match – Muon ID: Tracking, Muon detector information – Vertex: Tracking

• D/Ds Mesons

– Pi0s: EMC clusters – Inv. Mass: Tracking – Kaon, Pion ID: dE/dx, DIRC info (w/ track match), ToF (track match) – Vertex: Tracking

• Baryons

– Inv. Mass: Tracking – proton, pion ID: DIRC info (w/ track match) – Vertex: Tracking

• Full events: 4C fitting

Tracking & momentum → key information

Triggerless Data Acquisition

Panda DAQ

continuous sampling DAQ flash ADC self-triggered detector front-end front-end feature extraction (signal amplitude, shape, …) each signal get a time stamp (155.52 MHz) high quality clock distributed

Time Dependent Simulation

slow transition from event based to time ordered simulation

STT 0.5 ns 3 ns 20 ns 200 ns

Digi Index Time Stamp [ns] Position in Array Time Stamp [ns]

same color = same event

Ø Randomized Digi Data

MVD Digi Data Stream

Digi index

Time Based Simulation Ø Sorting Digi Data using Time Stamps + Drift Time, ToT… Ø Event Building – t0 determination

T.Stockmanns T.Stockmanns

Marius C. Mertens

Black ¡circles: ¡Early ¡isochrone ¡ Blue ¡circles: ¡Early ¡skewed ¡isochrone ¡ Green ¡circles: ¡Close ¡isochrone ¡ Red ¡circles: ¡Late ¡isochrone ¡ Black ¡dots: ¡MVD ¡hits ¡ Green ¡dots: ¡MVD ¡hits ¡r/z ¡> ¡0.3 ¡ Black+Red ¡dots: ¡Triplets/Skewlets ¡ ¡tracks: ¡Vetoed ¡ Blue ¡tracks: ¡Accepted ¡

XY-View Dual Parton Model (DPM): Standard pp background generator

15 GeV/c DPM, 50 ns mean time

Continuous Online Tracking

12th May 2014 Stefano Spataro Computing and Reconstruction In PANDA

How to deal with continuous data stream?

Ø More power to online computing Ø The (almost) whole offline reconstruction should run also online Ø Algorithms as fast as possible Ø (of course) Concurrency is the key!

No possibility to pre-filter events (lvl1)

need for more intelligence in online computing

!

algorithms run continuously

The Compute Node the actual version for Panda online computing what is intended to be used later in Panda still to be decided

Conformal ¡ transformaIon

x,y,z,r ¡ Wire position +drift distance

Legendre ¡ transformaIon Find ¡peak Fill ¡ Legendre ¡space

Simulation with ISim Pt(GeV/c)

Tracking in VHDL (FPGA)

Tracking Algorithm And tests with PC as data source and receiver see Y.Liang talk

Ø GPU threads are extremely lightweight Ø CPUs can execute 1-2 threads per core, while GPUs can maintain up to 1024 threads per multiprocessor (8-core) Ø CPUs use SIMD (single instruction is performed over multiple data) vector units, and GPUs use SIMT (single instruction, multiple threads) for scalar thread processing. SIMT does not require developers to convert data to vectors and allows arbitrary branching in threads.

CPUs and GPUs

M.Al-Turany Track Propagation with Panda Field (RK 4th order)

Speedup

Panda GPU Activities

ü GPUs for event reconstruction CHEP 2010 – Mohammad Al-Turany ü Track Finding in a High-Rate Time Projection Chamber Using GPUs CHEP 2010 – Felix Böhmer ü Track finding and fitting on GPUs, first steps toward a software trigger CHEP 2012 – Mohammad Al-Turany ü Possibility to run Cuda directly from PandaRoot (FairCUDA) ü Direct collaboration with NVIDIA ü Algorithm Developments of: Ø GPU Hough Transform Tracker Ø GPU Riemann Track Finder Ø GPU Triplet Finder

In the past… Currently

see A. Herten talk

300 ¡GB/s ¡ 20M ¡Evt/s ¡ How to distribute the processes? How to manage the data flow? How to recover processes when they crash? How to monitor the whole system? …… > 60 000 CPU-core

• r Equivalent

GPU, FPGA, …

The Online Reconstruction and analysis

Ø Highly ¡flexible: ¡different ¡data ¡paths ¡should ¡be ¡modeled. ¡ ¡ Ø AdapIve: ¡Sub-­‑systems ¡are ¡conInuously ¡under ¡development ¡and ¡improvement ¡ Ø Should ¡works ¡for ¡sim ¡and ¡real ¡data: ¡developing ¡and ¡debugging ¡ ¡the ¡algorithms ¡ Ø It ¡should ¡support ¡all ¡possible ¡hardware ¡(CPU, ¡GPU, ¡FPGA, ¡ARR?) ¡ Ø It ¡has ¡to ¡scale ¡to ¡any ¡size! ¡With ¡minimum ¡or ¡ideally ¡no ¡effort. ¡

Ø A socket library that acts as a concurrency framework Ø Messaging library, allowing to design a complex communication system without much effort

ØMQ ¡(zeromq) ¡

Experiment/detector ¡ specific ¡code ¡ Framework ¡classes ¡ that ¡can ¡be ¡used ¡ directly ¡ ¡ see M. Al-Turany talk

Summary

Computing Model Ø The MONARC model good starting point but updated by new technologies Ø Grid, Cloud, Proof, computing on FPGAs and on GPUs… Ø Multi-core CPUs and many-core GPUs → importance of scalable software Ø More democratic and flexible models ü Panda benefits from the LHC experiences and from the new IT technologies ü Taking data from 2018, still some time to take final decisions ü The trigger-less data acquisition is the real challenge Reconstruction Ø PandaRoot is our framework for simulation, reconstruction and analysis Ø Dynamic data structure, macro driven, supported on many OS Ø Advantages from a large developer community and from 3rd part packages Ø Time based simulation under realization (new concept!) Ø High importance of Online algorithms ü With LHC upgrade higher data rates and more need of software parallelization ü Many points in common with LHC experiments, mutual benefits?