[PPT] - PXD DAQ S. Lange (Giessen) for PXD DAQ team Hardware and firmware PowerPoint Presentation

SLIDE 1

PXD DAQ

1

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

PXD DAQ

S. Lange (Giessen)

for PXD DAQ team Hardware and firmware by IHEP Beijing, Bonn, Giessen, Kavli IPMU, KEK, TU Munich Additional software and test MC data by DESY, KIT Karlsruhe, Prague, MPI Munich

This talk is a preliminary version. Apologies. Final version will be uploaded after the CHARM'2013 conference (08/31-09/04) at Manchester.

SLIDE 2

PXD DAQ

2

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

DHHC

Belle II PXD DAQ

„ONSEN“ (Online Selector Node) because data reduction factor ¸ 10

SLIDE 3

PXD DAQ

3

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Data Reduction by Factor ≥10

ROI (Regions-of-Interest) from

1. DATCON

SVD only track finder

2. HLT

SVD + CDC track finder followed by logical OR.

SLIDE 4

PXD DAQ

4

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Status of DHH

SLIDE 5

DHH prototype

 FPGA: Virtex6 VLX75T  128 MB DDR2 memory

for a MicroBlaze softcore CPU

 SODIMM socket as interface  3x RJ45 for  ethernet,  timing

(interface to FTSW board)

 JTAG (for flash memory access

n FTSW)

 two SFP+

6 Gbps optical transceivers as interface to ONSEN system

 SFP for optical connection

to DHHC as controller interface,

 two Infiniband 4× connectors

for receiving data from DHP

SLIDE 6

PXD DAQ Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

New Design: DHH ATCA module

not a 1:1 mapping anymore, supports load balancing

SLIDE 7

DHH/DHHC final ATCA-based system

Design

AMC module, ATCA standard
Module programmed as DHH or

DHHC

Load balancing

30% peak data rate reduction

Subevent building
VIRTEX6 XC6VLX130T FPGA
2GB DDR3
DHHC data throuput : 2.5 GB/s
First modules have been

succesfully tested Production

50 modules to be producced
12 more modules are in

production and to be delivered in September ATCA Carrier board

Schematic ready
PCB layout in progress

Igor Konorov, Dima Levit

SLIDE 8

Data processing in DHH

SLIDE 9

PXD DAQ

9

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Status of ONSEN

ATCA based FPGA Compute Node

SLIDE 10

PXD DAQ

10

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

ATCA Compute Node Version #2.1, 12/2008 Version #2.1, 12/2013 ATCA Compute Node Version #1, 01/2008

ATCA Compute Node

SLIDE 11

PXD DAQ

11

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Compute Node Version #3 Carrier Board, xTCA compliant

SLIDE 12

PXD DAQ

12

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

New (2012): ATCA Compute Node Version #3

Carrier Board xTCA Daughter Board AMC (uTCA) Power Supply Board

SLIDE 13

PXD DAQ

13

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

AMC Version 1.0 AMC Version 2.0 Problem: DDR2 RAM only operable with 200 MHz + 177 MHz Problem: AMC-AMC connection some RocketIO pins were routed to LVDS pins

2 x 2 GB DDR2

SLIDE 14

PXD DAQ

14

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

DC-DC converters (12V 3.3 V) → JTAG UART-to-USB GbE (RJ45) 4 x optical 6.5 Gbps 64 MB Flash Connector for MMC (IPMI) AMC v3.2 arrived in Giessen March 12, 2013 used during DESY test May13

SLIDE 15

PXD DAQ

15

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Giessen Setup 5 ATCA Shelfs

SLIDE 16

PXD DAQ

16

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

 Virtex-4 switch FPGA on carrier board

is connected to each AMC FPGA with 8 LVDS pairs (4 × duplex)

 Some links do not work,

because low-capacitance pins (no LVDS drivers) were used

 Such pins were also used

for output to the clock fan-out chip

 The remaining links work,

if one of the data links is used for the clock makes carrier board redesign → necessary Not crucial for DESY Test → (as switch FPGA not needed)

Problem with the Carrier Board

SLIDE 17

PXD DAQ

17

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Status of DATCON

SLIDE 18

DATCON Setup for Vienna Test June13

Michael Schnell (Bonn)

SLIDE 19

PXD DAQ

19

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

2 AMC cards v2 at Bonn, 1 nTCA shelf
Successful test of communication

between FADC, FTB and DATCON with ~12 million events in Vienna

Implementation of lookup table and software for sending 'real'

(simulated) strip data from the FTB

Full preprocessing chain in DATCON done, including
Decoding of FADC protocol
Strip clustering with simple center of gravity
Coordinate translation
Internal protocol for control and inter-AMC communication (e.g.

transmission of coordinate data, update coord translation table...)

Status at Bonn

SLIDE 20

PXD DAQ

20

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Simple tracking unit (only straight tracks) for the Giessen-Bonn test

(planned for Sep 23-27, 2013)

Track extrapolation to PXD and ROI creation

TODO:

ROI transmission module to ONSEN
Bit error test with new clock input (GREFCLK)
Backplane bit error tests
DDR2 memory access for testbeam at DESY

Status at Bonn

SLIDE 21

Planned DATCON Setup for DESY Test Jan14

Michael Schnell (Bonn)

SLIDE 22

PXD DAQ

22

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Firmware

SLIDE 23

PXD DAQ

23

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Buffer management for 4 GB of DDR2 RAM (buffer for 5 seconds HLT latency) implemented in VHDL

B. Spruck (Giessen)

SLIDE 24

PXD DAQ

24

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

ATCA Firmware

1526 MB/s achieved 540 MB/s required 271.000 pointers/s achieved 3 x 30.000 pointers/s required

TCP/IP 95 MB/s

nly one receiver

UDP 60 MB/s multiple receivers

625 MB/s AURORA 8B/10B

ONSEN Firmware

SLIDE 25

PXD DAQ

25

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Reminder: ROI core receives ROIs from

DATCON
HLT

then performes logical OR. ROI selector is implemented PARALLEL Format unpacker/decoder for 3 formats: DHH (DHP subformat, DCE subformat, RAW subformat) 16-bit aligned: 1 bit of row address in encoded in column addr Frequency must be doubled because

f 16 bit

32 bit unpacking → Long-term stability standalone FPGA core run for ~10

9 random events

and for some 10

3 MC events (with random ROIs)

Future: pixel recovery

Status of ROI Firmware on ONSEN

SLIDE 26

PXD DAQ

26

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

same DHH ID different chip ID Integration of ROI Selector into Firmware

SLIDE 27

PXD DAQ

27

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

some signals had to be delayed by 1-2 clock cycles for synchronisation Integration of ROI Selector into Firmware

SLIDE 28

PXD DAQ

28

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Status of ROI core implementation for Jan14 DESY test: 41 hour test with 2xNPI(data, ROI) + ROI selector core + siTCP successful (zero errors observed) „debug mode“ switched on: ROIs are also sent out Integration into „full core“ (complete Onsen firmware)

ngoing work

→

SLIDE 29

PXD DAQ

29

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

basf2 interface

SLIDE 30

PXD DAQ

30

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

SLIDE 31

PXD DAQ

31

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

≥9 Mill. pixel hits of „cold“ pixels from DESY data reformatted (new Jan14 format) PC-PC send/receive (socket) basf2 → Landau fit to a „cold“ pixel in basf2

Bachelor Thesis Klemens Lautenbach (Giessen)

SLIDE 32

PXD DAQ

32

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Results from May13 DESY test beam

SLIDE 33

PXD DAQ

33

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Change between readout chains by re-plug infiniband cable

SLIDE 34

T est beam in May with DHH prototype

 1xDHP  1xDCD  1xFTSW  DHP interface

1.6 Gbps

 Onsen Interface

3.125 Gbps

Igor Konorov, Dima Levit

SLIDE 35

T est beam in May with DHH prototype

DHH Problem observed in the May 13 DESY test: scrambled data when DHP/DCD programmed by DHH Working scheme : programming by DHH EMULATOR and reading by DHH Reason : not correct programming sequence and of DCD by DHH Problem understood and being solved.

Igor Konorov, Dima Levit

SLIDE 36

PXD DAQ

36

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

1 uTCA Shelf (+ 1 spare)
1 AMC card (+ 1 spare)

This is the new AMC card v3.2 which arrived in Giessen in March 2013.

No xTCA carrier board
Aurora 3.125 Gbps on optical link

(not highspeed, because we didn't know beforehand if we would have the speedgrade -11 FPGAs)

No ROI selector

ONSEN Setup in May13

SLIDE 37

PXD DAQ

37

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

SLIDE 38

PXD DAQ

38

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

 optical link, L= 30 m optical cable  Buffer management was used

256 kB buffer size x 4096 buffers = 1 GB

 DATA WAS DUPLICATED - Writing data file

on ONSEN readout PC,
on BonnDAQ
on EUDAQ

(at same time) Differences between DHH+Onsen and DHH-Emulator System

SLIDE 39

PXD DAQ

39

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Onsen hardware
problems with PowerPC when 2 AMC cards in 1 uTCA shelf

configuration pins (on backplane) ?

2nd memory bank off
nly 1 bank (2 GB instead of 4 GB) operated at DESY test

→ as data size small, no problem

FPGA (Virtex-5 FX70T) almost full

issue: multi-port memory controller

siTCP
Both UDP and TCP were prepared and found o.k. in lab tests
@ DESY UDP (Onsen → BonnDAQ) stalls after some time,
rigin of problem under investigation
Unpacking routines still detect 3 problems (from chain before

ONSEN)

ghost frames: there is a trigger but no data
double 0xa0 headers
double trigger numbers

→ we used workarounds

Debugging Issues

SLIDE 40

PXD DAQ

40

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

FPGA re-boot almost not required 4-5 days start/stop/re-start w/o re-upload bitstream bitstream not changed during beam test > very stable operation Debugging Issues

SLIDE 41

PXD DAQ

41

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

SLIDE 42

PXD DAQ

42

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

SLIDE 43

PXD DAQ

43

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

 2 GeV e- beam  3mm Al target  ~1620 Hz trigger rate

triggered by scintillator with TLU

 framerate 150 kHz (6.6 us frame length)

(this is 3x faster than design (50 kHz), but possible because of smaller matrix)

 8 frames per 1 trigger (set on DHH)  zero suppressed mode  200-300 kB/s sustained rate  54.878.580 events recorded

(in 1 single run, no stop inbetween) Results (1) Long-term stable data taking

SLIDE 44

PXD DAQ

44

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Results (2) High rate test

 2 GeV e- (maximum of particles/spill)  5 mm Cu target (thickest)  ~4 kHz rate with DHH+ONSEN

stable for ~20 min

SLIDE 45

PXD DAQ

45

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

short spikes: spill structure

1 frame per 1 trigger >4 kHz 8 frame per 1 trigger >2 kHz

High rate test Screen Shot of DHH EPICS (by Dima, Michael R., Alan)

SLIDE 46

PXD DAQ

46

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

List of DHH+Onsen recorded data, ~10

8 events

SLIDE 47

PXD DAQ

47

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

5 hot pixels masked, no further processing ADC values (DHH+Onsen data), quasi-online about 30 min after data taking

SLIDE 48

PXD DAQ

48

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Preparation for Jan14 DESY test beam

SLIDE 49

PXD DAQ

49

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

What will be new compared to May 13 ?

 ROI core  May13 test was only Rx/Tx data, but Jan14

requires unpack/decode on FPGA

 Frame handling (2 halfladders

2 frames, → multiple frames per 1 trigger, ...)

 high speed optical links (6.25 Gbps)

not required in terms of data bandwidth, but for a test

 2nd AMC board

as input node for HLT ROI and DATCON ROI,

merging and matching of ROIs AMC-AMC backplane communication Onsen Preparations for Jan14 DESY Test

SLIDE 50

PXD DAQ

50

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

„Pocket-ONSEN“ System 1 uTCA shelf, 2 AMC cards

SLIDE 51

PXD DAQ

51

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Planned ONSEN Setup for DESY Jan14

Finisar FCLF-8521-3 SFP+ tp RJ45

SLIDE 52

PXD DAQ

52

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Status at Giessen

2+1 AMC cards v3 at Giessen, 3 uTCA shelfs ROI merger node was developed and tested 2 inputs from PC, 1 output to PC multiple stable overnight tests with 16 kHz done and partially tested: optical input, Aurora TODO: backplane communication to ROI selector node (backup solution: Aurora SFP SFP, → but due to decision of oscillators would only be 1 Gbps) [SFP RJ45 transceiver] + siTCP slow: bandwidth few MB/s → (siTCP support team contacted) but not critical, because bandwidth at DESY test low if PC with fast ethernet adapter

.k.

→

SLIDE 53

PXD DAQ

53

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Status at Giessen

guest from Crakow (Greg Korcyl) at Giessen for ~3 weeks debugging on UDP problem was: unstable operation during May13 test read and decode trigger numbers: done. event handling: done (multiple frames in 1 event) some error handling ready:

ROI checksum
case: HLT without DATCON
case: DATCON without HLT

debugging on interrupts ongoing for EPICS monitoring (read registers) work on IPMI ongoing (microcontroller for AMC v3) guest from IHEP (Jingzhou Zhao) in Giessen for ~4 weeks

SLIDE 54

PXD DAQ

54

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Future: Pixel Recovery

SLIDE 55

PXD DAQ

55

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

SLIDE 56

PXD DAQ

56

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

PIXEL RECOVERY If tracks (e.g. low pT pions) do not reach the CDC (or even SVD) there is no track there is no ROI data discarded → → → pixels can be „recovered“ based on e.g. dE/dx (low momentum high dE/dx Bethe-Bloch) → needs cluster data instead of hit data ( cluster finder on DHH) → „Pixel recoverer“ on ONSEN arallel with ROI selector

needs same data
does not need ROIs
needs to be synchronous with ROI selector

(maybe need buffers and wait states)

SLIDE 57

Clustering in DHH

New clustering algorithm developed for DHH FPGA

– Reconstruct any cluster shape in 250x768 pixel array – 1st step: FSMs reconstruct clusters in 64x768 array(one DHP) – 2nd step: Algorithm links clusters within 4 areas (4x64x768)

Algorithm consumes about 30% of XC6VLX130T FPGA resources

Finite State Machines Clustering algorithm

SLIDE 58

PXD DAQ

58

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Extension of ROI Core: if 1 pixel is inside ROI, then send out whole cluster

SLIDE 59

PXD DAQ

59

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

DCE3 Format Unpacker/Decoder

Cluster with a tree split simple cluster 1-pixel cluster

David Münchow (Giessen)

SLIDE 60

PXD DAQ

60

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Handling Cluster Format on ONSEN

David Münchow (Giessen)

2 data words in 32 bit

needs 2 clock cycles per 32 bit → 200 MHz →

decoder

unpacking enlarges data from 16 bit to 32 bit (but keeping same clock)

SLIDE 61

PXD DAQ

61

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Thank you.

SLIDE 62

PXD DAQ

62

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

BACKUP

SLIDE 63

PXD DAQ

63

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

· 1.3% · 0.6% Background data from A. Moll (MPI), 3rd campaign Decoded on CN Version #2 by D. Münchow (Giessen) ~ 80% of clusters are small (≤4 hits)

SLIDE 64

PXD DAQ

64

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Data Rates (for bandwidth session)

SLIDE 65

PXD DAQ

65

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Data Rates (for bandwidth discussion)

SLIDE 66

PXD DAQ

66

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Monitoring

Shared Mem Data receiver Buffer manager ROI core Data sender EPICS CLIENT PowerPC 440 Linux VHDL Registers

EPICS on ONSEN

SLIDE 67

PXD DAQ

67

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

DHH + ONSEN München Test, 16.-17.07.2012

function generator 50 kHz
for every pulse,

FTSW generates a trigger

DHH generates random frames
n each trigger
3 binary formats:
raw
DHP format (hit based)
DCE3 format (cluster based)

(using a C program running

n Microblaze)
data send with Aurora over SFP+

@ 6.25 Gbps (8B/10B)

data received and written to RAM

(1 fixed-size buffer per Aurora frame) by NPI

until 1 GB of memory is filled,

then dumped to a file via NFS

Compute Node Version #2

SLIDE 68

PXD DAQ

68

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

ONSEN Firmware München Test

Thomas Geßler, Björn Spruck (Giessen)

SLIDE 69

PXD DAQ

69

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

SLIDE 70

PXD DAQ

70

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Tests of AMC v3

SLIDE 71

PXD DAQ

71

Belle II Review, Sep 9-10, 2013 | S. Lange (Giessen)

Test of Buffer Management

Use lower 20 bit of triggerID as LUT index

(valid for t ~ 30 s)

Toy model: realistic size and HLT latency distributions

Result: mean average required memory ~ 0.6 GB/s