Experiment using the Unified Communication Framework Dominic - - PowerPoint PPT Presentation

Dominic Gaisbauer, Igor Konorov, Dymtro Levit, Prof. Dr. Stephan Paul Technische Universität München Institute for Hadronic Structure and Fundamental Symmetries Novosibirsk, March 3rd, 2017

Data Acquisition System for the PENeLOPE Experiment using the Unified Communication Framework

• Precision Experiment on Neutron Lifetime

Operating with Proton Extraction

• Will be located at the Forschungs-

Neutronenquelle Heinz Maier-Leibnitz (FRM II)

• Magneto-gravitational trap for ultra-cold

neutrons

• Aiming for a precision of ± 0.1 s
• Measuring protons and neutrons

2 Dominic Gaisbauer (TUM) | Novosibirsk 2017

PENeLOPE

proton detector absorber movement mechanism

• uter pressure

vessel helium vessel storage walls (electropolished) 2.5 m

• Protons are guided via magnetic and accelerated by

electrical field

• Complete detector and electronics on -30 kV
• Detector at 77 K
• Electronics at 300 K
• Active area of 0.23 m2
• Peak event rate including margin:

130.000 events/s + bg ≈ data rate: 500 Mbit/s

• Background is < 1/Ch/s

3

PENeLOPE - Proton Detector Requirements

Dominic Gaisbauer (TUM) | Novosibirsk 2017

• 6.8x14 mm2 active area
• 18x9 mm2 size
• 4.4 %/V gain sensitivity to voltage at a gain of 100
• Operational voltage of 365 V to 440 V
• No epoxy window since low energy protons would be absorbed
• Commissioning tests with samples done
• Beam test with final detector mid of 2017

4

PENeLOPE - Hamamatsu S11048 APD

Dominic Gaisbauer (TUM) | Novosibirsk 2017 30 keV proton spectrum

5

PENeLOPE - Proton Detector Readout Concept

UCF

Dominic Gaisbauer (TUM) | Novosibirsk 2017

Slow Control PC NAC Interface IPBus UDP Outside of Cryostat Inside of Cryostat Ground Potential 30 kV high Voltage Passive Optical Splitter 14x 24 APDs Slow Control DAQ PC PreAmp, Shaper, ADC SDU01 Signal Processing 24 APDs PreAmp, Shaper, ADC 4x

... ...

24 APDs PreAmp, Shaper, ADC SDU14 Signal Processing 24 APDs PreAmp, Shaper, ADC 4x

...

Bias

• 500 Mbit/s data rate
• 1496 channels

6

PENeLOPE - Proton Detector Readout Concept

SEP

Slow Control PC NAC Interface IPBus UDP Outside of Cryostat Inside of Cryostat Ground Potential 30 kV high Voltage Passive Optical Splitter 14x 24 APDs Slow Control DAQ PC PreAmp, Shaper, ADC SDU01 Signal Processing 24 APDs PreAmp, Shaper, ADC 4x ... ... 24 APDs PreAmp, Shaper, ADC SDU14 Signal Processing 24 APDs PreAmp, Shaper, ADC 4x ...

Bias

UCF

Dominic Gaisbauer (TUM) | Novosibirsk 2017

• Originates from the SODA time distribution system developed for the

PANDA experiment

• Single high-speed serial link for data, slow control, trigger, and timing

information implemented on FPGAs

• Up to 64 different communication channels (e.g. timing, slow control,

Data, JTAG, I2C, SPI, TCP, UDP…)

• Fixed latency for one channel
• Priority handling for all channels
• Self recoverable after connection losses
• Independent from physical layer

7

Unified Communication Framework (UCF)

FPGA Stage Front-end Electronics/ FPGA Stage Detector/Sensor PC/DAQ TCS

Data IPBus Data Data IPBus

Dominic Gaisbauer (TUM) | Novosibirsk 2017

8

UCF – Example Topologies

• Point-to-Point topology:
• Multiple or single 1:1 connections
• Experiments with high data rates, …
• Bidirectional on all channels

…

TCS Slow Control Data

Front-end Slave

TCS Slow Control Data

Front-end Slave

TCS Slow Control Data

Front-end Slave Master Master Master DAQ-PC TCS Slow Control PC

Data TCS IPBus

Dominic Gaisbauer (TUM) | Novosibirsk 2017

9

UCF – Example Topologies

• Star-like topology:
• Single 1:n connections
• Experiments with low data rates, time distribution systems …
• Slaves share link in time division manner
• Bidirectional on all channels

Splitter

TCS Slow Control Data

Front-end 0

TCS Slow Control Data

Front-end 1

TCS Slow Control Data

Front-end 255 Slave Slave Slave

Data TCS IPBus

Master DAQ-PC TCS Slow Control PC

Dominic Gaisbauer (TUM) | Novosibirsk 2017

10

UCF – Example Topologies

• Hybrid topology:
• Combination of point-to-point and star-like topologies
• Bidirectional on all channels

…

Slave Splitter Splitter

TCS Data TCS Data TCS Slow Control Data TCS Slow Control Data IPBus

DAQ-PC

Data

TCS

TCS

Slow Control PC

Slow Control Slow Control

Front-end 255 Front-end 0 Slave Slave Slave Front-end 255 Master Master Front-end 0

Dominic Gaisbauer (TUM) | Novosibirsk 2017

• Backbone of UCF
• Handles communication and initialization
• 8b/10b encoding scheme
• 10b K-characters for control and synchronization
• Protocol frames consist always of several character sequence:
• Start of frame
• Type of the message (either specific destination or broadcast)
• Protocol identifier
• Payload
• Remainder defining the valid bytes in the last transmission
• End of frame

11

UCF – Low Layer Protocol

SOF TYPE ID PAYLOAD PAYLOAD ... PAYLOAD REM EOF

Dominic Gaisbauer (TUM) | Novosibirsk 2017

• Fixed phase synchronization by sequence of two defined K-characters

(x”BCDC”)

• Synchronization character will be send for specific time to let the slaves

synchronize

• Attached parties are scanned by sending an initialization frame containing

different DNAs and waiting for response

• DNA can be taken as the serial number of an FPGA
• Unique ID and IP assignment for all connection parties

12

UCF – Initialization

... BCDC SOF ID DNA ... DNA EOF BCDC ...

Dominic Gaisbauer (TUM) | Novosibirsk 2017

• All 64 communication channels have different priorities
• Protocol 0 has the highest and then it cascades down to the

protocol 63 which has the lowest priority

• Frames with higher priority can always interrupt lower

priority frames

• Maintains fixed latency for the timing channel

13

UCF – Priority Handling

SOF TYPE ID x”0001” x”0203” x”0405” x”0000” REM EOF SOF TYPE ID x”0001” x”0203” x”0405” x”0600” REM EOF

Dominic Gaisbauer (TUM) | Novosibirsk 2017

• All channels are addressed via the standardized ARM AMBA AXI4 Stream interface
• Leads to easy interfacing with other IP-Cores
• Configuration of all parameters with a generic directly in the top module instantiation:
• Link speed
• Topology
• Device type (Spartan6, Virtex6, Artix7)
• ….

14

UCF – User Interface and Configuration

Dominic Gaisbauer (TUM) | Novosibirsk 2017

• Point-to-Point topology with 1 slave and 1 master
• 2.5 Gbit/s link speed
• Virtex 6 as slave and master
• Recovered clock jitter (σ) of 23 ps
• Long term test with 4 different protocols and link utilization of 99 % over two weeks

15

UCF – Tests and Measurements

Dominic Gaisbauer (TUM) | Novosibirsk 2017

• Star-like topology with 12 slaves and 1 master
• 1.25 Gbit/s link speed
• Spartan 6 FPGA as slave and Virtex 6 as master
• Switching time of 16 µs (includes character transmission and synchronization)
• Long term stability test with 99 % link utilization over two weeks
• JTAG over UCF
• IPBus over UCF

16

UCF – Tests and Measurements

Tran ansm smissi ssion

• n

Time e [µs] s] Effici cien ency cy [%] %] 25000 99,93 10000 99,84 1000 98,42 500 96,90 100 86,20

Dominic Gaisbauer (TUM) | Novosibirsk 2017

• Developed IP-Core providing unified communication of up to

64 channels via a single optical link

• Fixed latency for one channel (23 ps clock jitter)
• Standardized ARM AMBA AXI4 Stream interface for user
• Multiple 1:n and 1:1 connections possible
• Typically 98 to 99 % link utilization efficiency for star-like

topologies (16 µs switching time)

• JTAG over UCF
• IPBus over UCF

17

Conclusion

Dominic Gaisbauer (TUM) | Novosibirsk 2017

Thank you for your attention