a data acquisition system for the cerenkov telescope array
play

A data acquisition system for the Cerenkov Telescope Array Julien - PowerPoint PPT Presentation

Mar 07, 2023 •392 likes •801 views

A data acquisition system for the Cerenkov Telescope Array Julien HOULES, Dirk HOFFMANN CPPM/IN2P3/CNRS And the CPPM CTA group Contact : houles@cppm.in2p3.fr Camera server Global architecture Telescope N Telescope 2 Telescope 1 Camera

  1. A data acquisition system for the Cerenkov Telescope Array Julien HOULES, Dirk HOFFMANN CPPM/IN2P3/CNRS And the CPPM CTA group Contact : houles@cppm.in2p3.fr

  2. Camera server

  3. Global architecture Telescope N Telescope 2 Telescope 1 Camera front end boards Camera front end boards Camera front end boards nN... n2... n1... 5 4 3 2 1 5 4 3 2 1 5 4 3 2 1 ... Camera Camera Camera Server Server Server N 2 1 Central server CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 3

  4. Camera data flow PM event 1 event 2 event N Header Header Header PM 1 sampling data PM 1 sampling data PM 1 sampling data Front end board PM 2 sampling data PM 2 sampling data PM 2 sampling data PM 3 sampling data PM 3 sampling data PM 3 sampling data ... PM 4 sampling data PM 4 sampling data PM 4 sampling data PM 5 sampling data PM 5 sampling data PM 5 sampling data PM 6 sampling data PM 6 sampling data PM 6 sampling data PM 7 sampling data PM 7 sampling data PM 7 sampling data Whole Camera ~ 2000 PM -> 300 front end boards CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 4

  5. Camera server ● Build event ● L2 trigger on camera server (L1 on front end) : ● CPU (SSE, AVX...) ● GPU ● Compress ? ● Send data to central server (array level) CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 5

  6. Data flow hypothesis ● ~ 2000 pixels camera ● L1 trigger rate : 10 KHz ● Size of sampling data for 1 PM : 144 bytes (16 bit * 72 samples) ● No data loss (all the L1 events are sent) Max theorical bandwidth = 10000 * 2000 * 144 = 2.88 GB/s = 23 Gb/s ● 7 detectors for each front end board : 300 boards/camera Each board generates a flow of 2880/300 = 9.6 MB/s = 77 Mb/s https://portal.cta-observatory.org/WG/ACTL/SitePages/Data%20Rates.aspx CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 6

  7. Global architecture Ethernet link Camera Front end Server board CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 7

  8. Camera infrastructure 300 * 1Gb/s To central To central server Ethernet links server OR 3 * 10Gb/s Front end boards Ethernet links SFP+ 7 stacked Powerconnect 6248 7*48 ports @ 1Gb/s One or several . Dell T7500 workstations . . CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 8

  9. Dell Precision T7500 ● Two Intel Xeon X5650 (2.66GHz,6.4GT/s,12MB,6Cores) ● Memory : 24GB (6x4GB) 1333MHz ● Intel X520 DA2 10GbE Dual Port SFP+ Server Adapter, PCIe x8 ● Triple channel (maximum speed reached) ● QPI at 6.4 GT/s (maximum speed on the market) ● Memory DDR3-1333 ● 2 full speed full duplex 10 Gb/s links (PCIe x8 Gen 2) ● 1 PCIe x16 slot free (->GPU) and 1 PCIe x8 free (-> one more 10 Gbps adapter) ● SFP+ -> Copper or Optical link ~ 3500 euros CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 9

  10. Dell Powerconnect 6248 ● 48 * 1 Gb/s ports ● Backplane 184 Gb/s ● 2 * 10 Gb/s SFP+ ports included 2 more 10 Gb/s optional ports ● Up to 12 switches stackable -> 576 ports ~ 1500 euros (with 2 * 10 Gb/s) CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 10

  11. Event builder

  12. Why a prototype ? We need a prototype : ● To evaluate the maximum speed reachable ● To test several technologies ● To validate different approaches of the data processing ● To adapt our needs to what we can do CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 12

  13. Our first approach ● High modularity to make adaptation to different front end electronics easier ● Multitask approach to divide the flow processing if needed ● Use of a standard Linux distribution but take control on scheduling and memory allocation ● Constrained electronics to reach the best performances (in a first time) CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 13

  14. Event builder Trigger FE1 FE2 FE3 FE4 FEN PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H H H H H Ev1 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 Ev2 H H H H H ... ... ... ... ... PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 Ev N H H H H H Copy Copy Copy Copy Copy Ev1 H PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 ... PM1 PM2 PM3 PM4 PM5 PM6 PM7 Ev2 H PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 ... PM1 PM2 PM3 PM4 PM5 PM6 PM7 ... Ev N H PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 ... PM1 PM2 PM3 PM4 PM5 PM6 PM7 CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 14

  15. Data format : regular frame Level 2 triggering on camera server 1 frame : 1024 bytes s 4 e 2 Event n t 0 y 0 32 64 1 b header Level 1 Header flag FE Id Event number bytes 16 Time 1 Time 2 bytes 144 PM 1 sampling data bytes 144 PM 2 sampling data bytes 144 PM 3 sampling data bytes 144 PM 4 sampling data s 4 e PM 5 sampling data 4 t y 1 b s 4 e PM 6 sampling data 4 t y 1 b bytes 144 PM 7 sampling data CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 15

  16. Data format : jumbo frame Level 2 triggering on camera server 1 frame : 8192 bytes s 4 e 2 Event n t 0 y 0 32 64 1 b header Level 1 Header flag FE Id Event number bytes 16 s 4 Time 1 Time 2 e 2 Event n+1 t 0 y 1 b bytes 144 PM 1 sampling data s 4 e 2 Event n+2 t 0 y 1 b bytes 144 PM 2 sampling data s 4 e 2 Event n+3 t 0 y 1 b bytes 8192 bytes 144 PM 3 sampling data s 4 e Event n+4 2 t 0 y 1 b bytes 144 s PM 4 sampling data 4 e Event n+5 2 t 0 y 1 b s s 4 4 e PM 5 sampling data e 2 Event n+6 4 t t y 0 1 y b 1 b s s 4 4 e e 2 Event n+7 PM 6 sampling data 4 t t 0 y y 1 1 b b bytes 144 PM 7 sampling data CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 16

  17. Software overview : 1 st architecture N2 camera elements N1 camera elements Nx camera elements 1 1 1 Socket Socket Socket (blocking) (blocking) (blocking) packetReceived packetReceived packetReceived SYNC SYNC SYNC SYNC SYNC SYNC 1 1 1 taskReceiveDaq taskReceiveDaq taskReceiveDaq Packets Packets Packets ... PacketReceived PacketReceived PacketReceived Semaphore Semaphore Semaphore SYNC SYNC SYNC ... ... ... 1..n 1..n 1..n ReceiveDaqPacketBuffer ReceiveDaqPacketBuffer ReceiveDaqPacketBuffer taskBuildEventDaq taskBuildEventDaq taskBuildEventDaq C ComputeEvents N Y Semaphore S Events ... computeEventsSemaphore SYNC eventBuffer SYNC computeEventsSemaphore 1..n CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 17 taskComputeEventDaq

  18. Software overview : 2 nd architecture N2 camera elements N1 camera elements Nx camera elements . . . 1 1 1 Socket Socket Socket (blocking) (blocking) (blocking) packetReceived packetReceived packetReceived SYNC SYNC SYNC S S S Y Y Y N N N C C C 1 1 1 Task Task Task ReceiveDaq ReceiveDaq ReceiveDaq + + + BuildEvent BuildEvent BuildEvent C ComputeEvents N Y Semaphore S Events ... computeEventsSemaphore SYNC eventBuffer SYNC computeEventsSemaphore 1..n taskComputeEventDaq CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 18

  19. Stimulation configuration 10 * 2 * 1Gb/s 10 HP servers Ethernet links 2 * 10Gb/s Ethernet links SFP+ One Dell T7500 workstation CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 19

  20. Stimulation room CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 20

  21. First results : event builder 150 nodes (15 per HP server) sending data to interface 1 300 nodes simulated 150 nodes (15 per HP server) sending data to interface 2 Tests of the event building with varying packet size: 1 st Achitecture 2 nd Achitecture Jumbo frames (8192 bytes) : Jumbo frames (8192 bytes) : 19,2 Gb/s (2,4 GB/s ~ 8000 evts/s) 19,2 Gb/s (2,4 GB/s ~ 8000 evts/s) with no loss with no loss CPU usage : 300 % (3 cores/12) CPU usage : 160 % (1.6 cores/12) Regular frames (1024 bytes) : Regular frames (1024 bytes) : 6,5 Gb/s (0,82 GB/s) with no loss 8 Gb/s (1 GB/s) with no loss CPU usage : 300 % (3 cores/12) CPU usage : 170 % (1.7 cores/12) Results obtained with standard libraries/drivers (Linux). Test of a direct I/O solution to improve small frames reception in progress CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 21

  22. Integration in ACS The basic functions of the Event Builder are available from the ACS interface CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 22

  23. Stimulator

  24. Need for a real stimulator Need a stimulator to make : ● timing measurements on software ● real time validation ● algorithms validation ● trigger validation ● latency measurements on network ● front end boards and stimulator mix ● validate the complete acquisition chain CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 24

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Gamma-ray radiation from type IIb supernova remnants prospect for the Cerenkov Telescope Array

Gamma-ray radiation from type IIb supernova remnants prospect for the Cerenkov Telescope Array

Gamma-ray radiation from type IIb supernova remnants prospect for the Cerenkov Telescope Array A.marcowith (L.U.P.M.) In collaboration with M.Renaud (L.U.P.M.), V. Dwarkadas (Chicago university) & V. Tatischeff (C.S.N.S.M. Orsay)

675 views • 38 slides
Composition study of ultra-high-energy cosmic rays based on the Telescope Array surface detector

Composition study of ultra-high-energy cosmic rays based on the Telescope Array surface detector

Composition study of ultra-high-energy cosmic rays based on the Telescope Array surface detector data Yana Zhezher for the Telescope Array collaboration Institute for Nuclear Research of the Russian Academy of Sciences, Moscow, Russia July 17,

674 views • 25 slides
TA Spectrum Summary : Energy Spectrum Measurements With The Telescope Array Detectors

TA Spectrum Summary : Energy Spectrum Measurements With The Telescope Array Detectors

1 Jul 19, ICRC2017 Busan, Korea CRI125 TA Spectrum Summary : Energy Spectrum Measurements With The Telescope Array Detectors Yoshiki Tsunesada Osaka City University for The Telescope Array Collaboration Millard county, Utah, US BR LR

471 views • 19 slides
Optimization of the Data Acquisition Software (PxSuite DAQ) for the Silicon Strip Telescope at

Optimization of the Data Acquisition Software (PxSuite DAQ) for the Silicon Strip Telescope at

Optimization of the Data Acquisition Software (PxSuite DAQ) for the Silicon Strip Telescope at FTBF Clifford Denis SIST Intern, Fermilab Ramapo College of New Jersey 08/04/2014 Background What is the Silicon Strip What are the

490 views • 15 slides
Telescope Array search for EeV photons Yana Zhezher (on behalf of Mikhail Kuznetsov), Oleg

Telescope Array search for EeV photons Yana Zhezher (on behalf of Mikhail Kuznetsov), Oleg

Telescope Array search for EeV photons Yana Zhezher (on behalf of Mikhail Kuznetsov), Oleg Kalashev and Grigory Rubtsov for the Telescope Array collaboration ICRR, Univ. of Tokyo & INR RAS TAUP-2019 11 September 2019 Supported by Russian

507 views • 22 slides
Telescope Array: Status and Upgrades John Belz University of Utah Snowmass on the Mississippi

Telescope Array: Status and Upgrades John Belz University of Utah Snowmass on the Mississippi

Telescope Array: Status and Upgrades John Belz University of Utah Snowmass on the Mississippi Minneapolis, 30 July 2013 Outline Telescope Array Overview Detector Physics TA Upgrades TALE TA x 4 NICHE TA

400 views • 22 slides
TA Anisotropy Summary Eiji Kido for the Telescope Array Collaboration 1 Outline

TA Anisotropy Summary Eiji Kido for the Telescope Array Collaboration 1 Outline

TA Anisotropy Summary Eiji Kido for the Telescope Array Collaboration 1 Outline Introduction Data set for anisotropy studies Results Autocorrelation Hotspot Correlation with LSS Anisotropy in energy spectrum

557 views • 24 slides
CHERENKOV TELESCOPE ARRAY FOR INTENSITY INTERFEROMETRY Dainis Dravins Lund Observatory, Sweden

CHERENKOV TELESCOPE ARRAY FOR INTENSITY INTERFEROMETRY Dainis Dravins Lund Observatory, Sweden

Intensity Interferometry workshop Salt Lake City, January 2009 OPTIMIZING THE CHERENKOV TELESCOPE ARRAY FOR INTENSITY INTERFEROMETRY Dainis Dravins Lund Observatory, Sweden www.astro.lu.se/~dainis CTA, Cherenkov Telescope Array

705 views • 54 slides
The Cherenkov Telescope Array SST-1M camera prototype commissioning T .R.S. Njoh Ekoume*, C.

The Cherenkov Telescope Array SST-1M camera prototype commissioning T .R.S. Njoh Ekoume*, C.

Nicolaus Copernicus Astronomical Center University College Dublin Irelands Global University Taras Shevchenko National University of Centrum Bada Kosmicznych Kyiv Space Research Centre The Cherenkov Telescope Array SST-1M camera

518 views • 16 slides
Ch Cherenkov Telescope Array: k T l A The Next Generation e e t Ge e at o Gamma-ray

Ch Cherenkov Telescope Array: k T l A The Next Generation e e t Ge e at o Gamma-ray

Ch Cherenkov Telescope Array: k T l A The Next Generation e e t Ge e at o Gamma-ray Observatory ICRC 2017 (Busan, Korea), 13 July 2017 The CTA Consortium 1 , represented by Rene A. Ong 2 1 See

502 views • 38 slides
Results from the Telescope Array Experiment Charlie Jui University of Utah TeVPA 2013 Irvine,

Results from the Telescope Array Experiment Charlie Jui University of Utah TeVPA 2013 Irvine,

Results from the Telescope Array Experiment Charlie Jui University of Utah TeVPA 2013 Irvine, CA Aug. 27, 2013 http://www.physics.utah.edu/~jui/ta-tevpa2013.pdf TA, TeVPA2013 1 Telescope Array Collaboration T. Abu-Zayyad a , M. Allen a , R.

559 views • 44 slides
LOFAR DATA MANAGEMENT R. F. Pizzo ASTRON, December2 nd 2015 THE LOW FREQUENCY ARRAY KEY

LOFAR DATA MANAGEMENT R. F. Pizzo ASTRON, December2 nd 2015 THE LOW FREQUENCY ARRAY KEY

LOFAR DATA MANAGEMENT R. F. Pizzo ASTRON, December2 nd 2015 THE LOW FREQUENCY ARRAY KEY FACTS THE PROPOSALS The International LOFAR telescope (ILT) consists of an interferometric array of dipole antenna stations distributed

350 views • 13 slides
ARA detector calibration with Telescope Array Electron Light Source Romain Gaior for the Chiba

ARA detector calibration with Telescope Array Electron Light Source Romain Gaior for the Chiba

ARA detector calibration with Telescope Array Electron Light Source Romain Gaior for the Chiba group 1 ARA: Askaryan Radio Array - Detec&on of UHE cosmogenic neutrino - Expected sensi&vity

288 views • 25 slides
Atmospheric calibration of the Cherenkov Telescope Array Jan Ebr for the CTA Consortium

Atmospheric calibration of the Cherenkov Telescope Array Jan Ebr for the CTA Consortium

Atmospheric calibration of the Cherenkov Telescope Array Jan Ebr for the CTA Consortium Institute of Physics Czech Academy of Sciences, Prague www.cta-observatory.org CTA: a precision VHE gamma-ray observatory goal: systematic uncertainty

832 views • 14 slides
AGN Physics with the Cherenkov Telescope Array A. Zech (for the CTA Consortium) LUTH,

AGN Physics with the Cherenkov Telescope Array A. Zech (for the CTA Consortium) LUTH,

AGN Physics with the Cherenkov Telescope Array A. Zech (for the CTA Consortium) LUTH, Observatoire de Paris Fermi meets Jansky November 2011, St. Michaels, MD, USA The CTA Consortium 25 Countries 132 Institutes >800 Persons M. Martinez

643 views • 50 slides
The Telescope Array Low-Energy Extension (TALE) C. Jui For the TA/TALE Collaboration ISVHECRI,

The Telescope Array Low-Energy Extension (TALE) C. Jui For the TA/TALE Collaboration ISVHECRI,

The Telescope Array Low-Energy Extension (TALE) C. Jui For the TA/TALE Collaboration ISVHECRI, FNAL, July 1, 2010 Features in the UHECR spectrum: The minority straight-forward View CMBR photons interact with cosmic ray protons: Pion

520 views • 20 slides
Two weight L 2 inequality for the Hilbert transform Eric T. Sawyer reporting on joint work with

Two weight L 2 inequality for the Hilbert transform Eric T. Sawyer reporting on joint work with

Two weight L 2 inequality for the Hilbert transform Eric T. Sawyer reporting on joint work with Michael T. Lacey Chun-Yen Shen Ignacio Uriarte-Tuero Abel Symposium Oslo, Norway August 24 2012 Two weight L 2 inequality E. Sawyer (McMaster

1.67k views • 138 slides
Multi-field -attractor in fundamental theory Yusuke Yamada (Stanford Univ.) collaborators: R.

Multi-field -attractor in fundamental theory Yusuke Yamada (Stanford Univ.) collaborators: R.

Multi-field -attractor in fundamental theory Yusuke Yamada (Stanford Univ.) collaborators: R. Kallosh, A. Linde, D. Roest, A. Westphal, T. Wrase Outline <latexit

596 views • 21 slides
Control and forwarding plane separation on an open-source router Linux Kongress 2010-09-23 in N

Control and forwarding plane separation on an open-source router Linux Kongress 2010-09-23 in N

Control and forwarding plane separation on an open-source router Linux Kongress 2010-09-23 in N rnberg Robert Olsson, Uppsala University Olof Hagsand, KTH More than 10 year in production at Uppsala University Stockholm Stockholm

1.07k views • 86 slides
Building a Program from Streams Tim Williams | October 2018 Sponsored by What is a stream? A

Building a Program from Streams Tim Williams | October 2018 Sponsored by What is a stream? A

Building a Program from Streams Tim Williams | October 2018 Sponsored by What is a stream? A potentially infinite sequence of data elements, processed incre- mentally rather than as a whole. An abstraction that offers an alternative to

624 views • 30 slides
Advanced CUDA: GPU Memory Systems John E. Stone Theoretical and Computational Biophysics Group

Advanced CUDA: GPU Memory Systems John E. Stone Theoretical and Computational Biophysics Group

Advanced CUDA: GPU Memory Systems John E. Stone Theoretical and Computational Biophysics Group Beckman Institute for Advanced Science and Technology University of Illinois at Urbana-Champaign http://www.ks.uiuc.edu/Research/gpu/ GPGPU2:

851 views • 30 slides
Sunrise or Sunset: Exploring the Design Space of Big Data

Sunrise or Sunset: Exploring the Design Space of Big Data

Sunrise or Sunset: Exploring the Design Space of Big Data So7ware Stacks Panel PresentaAon at HPBDC 17 by Dhabaleswar K. (DK) Panda The Ohio

485 views • 37 slides
BSS Processes and Intermittency/Volatility Realised Quadratic Variation Turbulence Turbulence

BSS Processes and Intermittency/Volatility Realised Quadratic Variation Turbulence Turbulence

Outline BSS models Denition Key object of interest BSS Processes and Intermittency/Volatility Realised Quadratic Variation Turbulence Turbulence Stochastics background Ambit processes Intermittency BSS models Ole E. Barndor-Nielsen

894 views • 47 slides
Flavor without symmetries Alex Pomarol, UAB (Barcelona) Flavor without symmetries Alex Pomarol,

Flavor without symmetries Alex Pomarol, UAB (Barcelona) Flavor without symmetries Alex Pomarol,

Flavor without symmetries Alex Pomarol, UAB (Barcelona) Flavor without symmetries Alex Pomarol, UAB (Barcelona) Apologizes: I am not going to talk on glances at the energy frontier Flavor without symmetries Alex Pomarol, UAB (Barcelona)

585 views • 43 slides

More recommend