A data acquisition system for the Cerenkov Telescope Array Julien - - PowerPoint PPT Presentation

▶
a data acquisition system for the cerenkov telescope array
SMART_READER_LITE
LIVE PREVIEW

A data acquisition system for the Cerenkov Telescope Array Julien - - PowerPoint PPT Presentation

Mar 07, 2023 392 likes •801 views

A data acquisition system for the Cerenkov Telescope Array Julien HOULES, Dirk HOFFMANN CPPM/IN2P3/CNRS And the CPPM CTA group Contact : houles@cppm.in2p3.fr Camera server Global architecture Telescope N Telescope 2 Telescope 1 Camera

slide-1
SLIDE 1

A data acquisition system for the Cerenkov Telescope Array

Julien HOULES, Dirk HOFFMANN CPPM/IN2P3/CNRS And the CPPM CTA group Contact : houles@cppm.in2p3.fr

slide-2
SLIDE 2

Camera server

slide-3
SLIDE 3

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 3

1 2 3 4 5 n1... Camera Server 1 Telescope 1

Camera front end boards

1 2 3 4 5 n2... Camera Server 2 Telescope 2

Camera front end boards

1 2 3 4 5 nN... Camera Server N Telescope N

Camera front end boards

Central server

...

Global architecture

slide-4
SLIDE 4

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 4

Camera data flow

Header PM 1 sampling data PM 2 sampling data PM 3 sampling data PM 4 sampling data PM 5 sampling data PM 6 sampling data PM 7 sampling data Header PM 1 sampling data PM 2 sampling data PM 3 sampling data PM 4 sampling data PM 5 sampling data PM 6 sampling data PM 7 sampling data Header PM 1 sampling data PM 2 sampling data PM 3 sampling data PM 4 sampling data PM 5 sampling data PM 6 sampling data PM 7 sampling data

event 1 event N event 2 ... Whole Camera ~ 2000 PM -> 300 front end boards PM Front end board

slide-5
SLIDE 5

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 5

Camera server

  • Build event
  • L2 trigger on camera server (L1 on front end) :
  • CPU (SSE, AVX...)
  • GPU
  • Compress ?
  • Send data to central server (array level)
slide-6
SLIDE 6

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 6

  • ~ 2000 pixels camera
  • L1 trigger rate : 10 KHz
  • Size of sampling data for 1 PM : 144 bytes (16 bit * 72 samples)
  • No data loss (all the L1 events are sent)

Max theorical bandwidth = 10000 * 2000 * 144 = 2.88 GB/s = 23 Gb/s

  • 7 detectors for each front end board : 300 boards/camera

Each board generates a flow of 2880/300 = 9.6 MB/s = 77 Mb/s

Data flow hypothesis

https://portal.cta-observatory.org/WG/ACTL/SitePages/Data%20Rates.aspx

slide-7
SLIDE 7

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 7

Global architecture

Front end board Camera Server Ethernet link

slide-8
SLIDE 8

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 8

. . . Front end boards

7 stacked Powerconnect 6248 7*48 ports @ 1Gb/s One or several Dell T7500 workstations To central server

300 * 1Gb/s Ethernet links 3 * 10Gb/s Ethernet links SFP+

Camera infrastructure

To central server

OR

slide-9
SLIDE 9

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 9

  • Two Intel Xeon X5650

(2.66GHz,6.4GT/s,12MB,6Cores)

  • Memory : 24GB (6x4GB) 1333MHz
  • Intel X520 DA2 10GbE Dual Port

SFP+ Server Adapter, PCIe x8

  • Triple channel (maximum speed reached)
  • QPI at 6.4 GT/s (maximum speed on the market)
  • Memory DDR3-1333
  • 2 full speed full duplex 10 Gb/s links (PCIe x8 Gen 2)
  • 1 PCIe x16 slot free (->GPU)

and 1 PCIe x8 free (-> one more 10 Gbps adapter)

  • SFP+ -> Copper or Optical link

~ 3500 euros

Dell Precision T7500

slide-10
SLIDE 10

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 10

  • 48 * 1 Gb/s ports
  • Backplane 184 Gb/s
  • 2 * 10 Gb/s SFP+ ports included

2 more 10 Gb/s optional ports

  • Up to 12 switches stackable
  • > 576 ports

~ 1500 euros (with 2 * 10 Gb/s)

Dell Powerconnect 6248

slide-11
SLIDE 11

Event builder

slide-12
SLIDE 12

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 12

We need a prototype :

  • To evaluate the maximum speed reachable
  • To test several technologies
  • To validate different approaches of the data processing
  • To adapt our needs to what we can do

Why a prototype ?

slide-13
SLIDE 13

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 13

  • High modularity to make adaptation to different front

end electronics easier

  • Multitask approach to divide the flow processing if needed
  • Use of a standard Linux distribution but take control on

scheduling and memory allocation

  • Constrained electronics to reach the best performances

(in a first time)

Our first approach

slide-14
SLIDE 14

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 14

Event builder

H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 H PM 1 PM 2 PM 3 PM 4 PM 5 PM 6 PM 7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 ... H PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 ... H PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 PM1 PM2 PM3 PM4 PM5 PM6 PM7 ... H

Ev1 Ev2 Ev N Trigger Ev1 Ev2 Ev N

FE1 FE2 FE3 FE4 FEN Copy Copy Copy Copy Copy ... ... ... ... ... ...

slide-15
SLIDE 15

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 15

Time 1 Header flag Event number FE Id Time 2 Event n 1 2 4 b y t e s PM 1 sampling data PM 2 sampling data PM 3 sampling data PM 4 sampling data PM 5 sampling data PM 6 sampling data PM 7 sampling data 144 bytes 144 bytes 144 bytes 144 bytes 1 4 4 b y t e s 1 4 4 b y t e s 144 bytes 16 bytes 32 64

1 frame : 1024 bytes

Level 1 header

Level 2 triggering on camera server

Data format : regular frame

slide-16
SLIDE 16

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 16

Time 1 Header flag Event number FE Id Time 2 Event n Event n+1 Event n+2 Event n+3 Event n+4 Event n+5 Event n+6 1 2 4 b y t e s 1 2 4 b y t e s 1 2 4 b y t e s 1 2 4 b y t e s 1 2 4 b y t e s 1 2 4 b y t e s 1 2 4 b y t e s Event n+7 1 2 4 b y t e s PM 1 sampling data PM 2 sampling data PM 3 sampling data PM 4 sampling data PM 5 sampling data PM 6 sampling data PM 7 sampling data 144 bytes 144 bytes 144 bytes 144 bytes 1 4 4 b y t e s 1 4 4 b y t e s 144 bytes 16 bytes 32 64 8192 bytes

1 frame : 8192 bytes

Level 1 header

Level 2 triggering on camera server

Data format : jumbo frame

slide-17
SLIDE 17

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 17

... ...

Events

eventBuffer

taskComputeEventDaq

1..n

taskReceiveDaq

Socket

taskBuildEventDaq

...

SYNC SYNC

PacketReceived Semaphore

SYNC

packetReceived (blocking) ReceiveDaqPacketBuffer

1 1 1..n

Packets N2 camera elements

taskReceiveDaq

Socket

taskBuildEventDaq

...

SYNC SYNC

PacketReceived Semaphore

SYNC

packetReceived (blocking) ReceiveDaqPacketBuffer

1 1 1..n

Packets N1 camera elements

taskReceiveDaq

Socket

taskBuildEventDaq

...

SYNC SYNC

PacketReceived Semaphore

SYNC

packetReceived (blocking) ReceiveDaqPacketBuffer

1 1 1..n

Packets Nx camera elements

SYNC S Y N C SYNC

computeEventsSemaphore computeEventsSemaphore ComputeEvents Semaphore

Software overview : 1st architecture

slide-18
SLIDE 18

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 18

...

Events

eventBuffer

taskComputeEventDaq

1..n

N2 camera elements

Task ReceiveDaq + BuildEvent

Socket

SYNC S Y N C

packetReceived (blocking)

1 1

N1 camera elements Nx camera elements

SYNC S Y N C SYNC

computeEventsSemaphore computeEventsSemaphore ComputeEvents Semaphore

Software overview : 2nd architecture

Task ReceiveDaq + BuildEvent

Socket

SYNC S Y N C

packetReceived (blocking)

1 1

Task ReceiveDaq + BuildEvent

Socket

SYNC S Y N C

packetReceived (blocking)

1 1

. . .

slide-19
SLIDE 19

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 19

One Dell T7500 workstation

2 * 10Gb/s Ethernet links SFP+

Stimulation configuration

10 * 2 * 1Gb/s Ethernet links 10 HP servers

slide-20
SLIDE 20

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 20

Stimulation room

slide-21
SLIDE 21

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 21

First results : event builder

150 nodes (15 per HP server) sending data to interface 1 150 nodes (15 per HP server) sending data to interface 2 Jumbo frames (8192 bytes) : 19,2 Gb/s (2,4 GB/s ~ 8000 evts/s) with no loss CPU usage : 300 % (3 cores/12) Regular frames (1024 bytes) : 6,5 Gb/s (0,82 GB/s) with no loss CPU usage : 300 % (3 cores/12) Jumbo frames (8192 bytes) : 19,2 Gb/s (2,4 GB/s ~ 8000 evts/s) with no loss CPU usage : 160 % (1.6 cores/12) Regular frames (1024 bytes) : 8 Gb/s (1 GB/s) with no loss CPU usage : 170 % (1.7 cores/12) 1st Achitecture 2nd Achitecture 300 nodes simulated Test of a direct I/O solution to improve small frames reception in progress Results obtained with standard libraries/drivers (Linux). Tests of the event building with varying packet size:

slide-22
SLIDE 22

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 22

The basic functions of the Event Builder are available from the ACS interface

Integration in ACS

slide-23
SLIDE 23

Stimulator

slide-24
SLIDE 24

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 24

Need a stimulator to make :

  • timing measurements on software
  • real time validation
  • algorithms validation
  • trigger validation
  • latency measurements on network
  • front end boards and stimulator mix
  • validate the complete acquisition chain

Need for a real stimulator

slide-25
SLIDE 25

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 25

1Gb/s links EVOC NET-1820 Sync Signal Powerconnect 6248 stack To camera server

. . .

10 Gb/s links

Testing configuration

slide-26
SLIDE 26

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 26

Intel Atom D525 dual core processor 1.8GHz 4.0 GB RAM 6 x Intel 82574L Giga LAN (supports 9K frames and boot on LAN) 8-bit Digital I/O interface 1 x Parallel port, Serial port ~ 300 € each → ~ 15000 € for 300 ports without switches

EVOC NET-1820

Measured throughput @ CPPM: ~ 2,4 Gb/s (400 Mb/s per port) → can easily be improved.

Most promising candidate (50 € / port)

slide-27
SLIDE 27

Future

slide-28
SLIDE 28

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 28

Future work

  • Full ACS integration
  • Make the software reliable enough for production stage
  • Test of a direct I/O

solution to improve regular frames reception (in progress)

  • Perform precise

measurements on performances

  • Improve the event

builder

  • Build a full-size

stimulator

  • Work with slow

control and array server communications teams

  • Design a L2 trig

(CPU ? GPU ?)

slide-29
SLIDE 29

Interface definition

slide-30
SLIDE 30

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 30

Data format : type 1.0

Time 1 Header flag Event number Empty Time 2 Event n Event n+1 Event n+2 Event n+3 Event n+4 Event n+5 Event n+6 Event n+7 PM 1 sampling data PM 2 sampling data PM 3 sampling data PM 4 sampling data PM 5 sampling data PM 6 sampling data PM 7 sampling data Level 1 header

The front end electronics transmit all events (after a L1 triggering)

slide-31
SLIDE 31

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 31

Data format : type 1.1

Time 1 Header flag Event number Empty Time 2 Event n Event n+1 Event n+2 Event n+3 Event n+4 Event n+5 Event n+6 Event n+7 PM 1 max/tot... PM 2 max/tot... PM 3 max/tot... PM 4 max/tot... PM 5 max/tot... PM 6 max/tot... PM 7 max/tot... Level 1 header

The front end electronics transmit a single value for each PM for all events (after a L1 triggering)

slide-32
SLIDE 32

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 32

Data format : type 2.0

Time 1 Header flag Event number Empty Time 2 Event n Event n+6 Event n+24 Event n+31 Event n+34 Event n+52 Event n+67 Event n+72 PM 1 sampling data PM 2 sampling data PM 3 sampling data PM 4 sampling data PM 5 sampling data PM 6 sampling data PM 7 sampling data Level 1 header

The front end electronics only transmit L2 triggered events Be careful about latency !

slide-33
SLIDE 33

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 33

Data format : type 2.1

Time 1 Header flag Event number Empty Time 2 PM 1 max/tot... PM 2 max/tot... PM 3 max/tot... PM 4 max/tot... PM 5 max/tot... PM 6 max/tot... PM 7 max/tot... Level 1 header

The front end electronics transmit a single value for each PM in L2 triggered events

Event n Event n+6 Event n+24 Event n+31 Event n+34 Event n+52 Event n+67 Event n+72

Be careful about latency !

slide-34
SLIDE 34

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 34

Data format : type 3.0

Time 1 Header flag Event number PM IDs Time 2 Event n Event n+12 Event n+23 Event n+25 Event n+31 Event n+32 Event n+45 Event n+51 PM 3 sampling data PM 5 sampling data Level 1 header

The front end electronics only transmit triggering PM in L2 triggered events

Time 1 Header flag Event number PM IDs Time 2 PM 2 sampling data PM 5 sampling data PM 6 sampling data Level 1 header

Be careful about latency !

slide-35
SLIDE 35

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 35

Data format : type 3.1

Time 1 Header flag Event number PM IDs Time 2 Event n Event n+12 Event n+23 Event n+25 Event n+31 Event n+32 Event n+45 Event n+51 PM 3 max/tot... PM 5 max/tot... Level 1 header

The front end electronics transmit a single value for triggered PM in L2 triggered events

Time 1 Header flag Event number PM IDs Time 2 PM 2 max/tot... PM 5 max/tot... PM 6 max/tot... Level 1 header

Be careful about latency !

slide-36
SLIDE 36

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 36

sampling datas beginnings are truncated, can be applied to the types describe Example 1.0

Data format : type X.Y.1

Time 1 Header flag Event number Empty Time 2 Event n Event n+1 Event n+2 Event n+3 Event n+4 Event n+5 Event n+6 Event n+7 PM 1 sampling data PM 2 sampling data PM 3 sampling data PM 4 sampling data PM 5 sampling data PM 6 sampling data PM 7 sampling data Level 1 header Empty Time from origin Level 2 header

slide-37
SLIDE 37

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 37

Data format : discussion

The formats exposed are just a proposition and must be discussed with all the concerned teams

slide-38
SLIDE 38

Backup

slide-39
SLIDE 39

CPPM/IN2P3/CNRS CTA - ACTL France Workshop 06/04/2012 39

1st IOH 2nd IOH

PCIe2 x16 slot 1 PCIe2 x8 slot 2 PCIe2 x4 slot 3 PCIe2 x16 slot 4 PCIe2 x8 slot 5 PCIe2 x4 slot 6 QPI QPI QPI QPI

QPI @ 6.4 GT/s bandwidth < DDR3-1333 memory bandwidth

Non Uniform Memory Access