ECAL off detector architecture Nikitas Loukas, University of Notre - - PowerPoint PPT Presentation

▶
ecal off detector architecture
SMART_READER_LITE
LIVE PREVIEW

ECAL off detector architecture Nikitas Loukas, University of Notre - - PowerPoint PPT Presentation

Oct 02, 2023 458 likes •669 views

HL-LHC ECAL off detector architecture Nikitas Loukas, University of Notre Dame N.Loukas, Aug 30 2017 FNAL Technical Review - Barrel Calorimeter 1 HL-LHC Bio M.Sc. in Modern Electronic Technologies, University of Ioannina, Greece: 2009

slide-1
SLIDE 1

HL-LHC

ECAL off detector architecture

Nikitas Loukas, University of Notre Dame

1

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

slide-2
SLIDE 2

HL-LHC

  • M.Sc. in Modern Electronic Technologies, University of Ioannina,

Greece: 2009

  • CMS Collaboration member: Jan 2012 – present.
  • Based at CERN: Mar 2014 – present.
  • PhD at University of Ioannina (GR): 2017
  • Thesis title: CMS L1 Trigger Upgrade - The Barrel Muon Track Finder
  • Thesis main achievements: Development of the Barrel Muon Track Finder

(BMTF) firmware and installation of the system in CMS. [Conference:C15-09- 28, JINST11(2016):no.03,C03038].

  • Professional Electronics Engineer working for University of Notre

Dame, based at CERN: Jan 2017 – present.

2

Bio

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

slide-3
SLIDE 3

HL-LHC

  • Requires a significant enhancement in finer granularity in the calorimeter
  • trigger. Instead of TPs per Tower it is planned to have TPs per crystal or

small clusters.

  • The new granularity will increase bandwidth and processing power. The

build of TPs will be moved from the FE to BE.

  • The calorimeter trigger must be able to accommodate the required latency.
  • The higher rates produce higher pileup which will be mitigated by precision

timing in the calorimeter as well as spike killing.

3

ECAL upgrade on phase 2 - Modivation

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

Max Latency Pipeline (BXs) Max L1A rate Trigger Phase 2 12.5 μs 500 750MHz After Trigger Phase 1 3.2 μs 128 150 KHz Requirement: BCAL-sci-engr-002 Requirement: BCAL-sci-engr-006 Requirement: BCAL-sci-engr-005 Requirement: BCAL-sci-engr-007

slide-4
SLIDE 4

HL-LHC

4

ECAL Block Diagram of the legacy system

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

Trigger path DAQ path TTC & ctrl path

  • Separate trigger and data cards
  • Reads out only areas of interest

due to bandwidth limitation

  • 5x5 crystal trigger primitives
slide-5
SLIDE 5

HL-LHC

5

Front-End Phase 2 Upgrade

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

  • In legacy ECAL, the TPs are build in the

trigger towers

  • FENIX chip sums of 5 crystals into a strip,

analyze the digitized signal and performs Trigger and Readout functions

  • 0.8 Gb/s links are sending all data to the

VME BE.

  • The sampling rate is 40 MHz
  • In ECAL phase 2, the TPs will be build in the Back-

End

  • FE will have lpGBT chips that will concentrate row

ADC data and send them to BE via Versatile+ links

  • In addition FE will distribute the clock and will control

the VFEs via I2C

  • The sampling will be done at 160 MHz, which will

help the BE to reject signal spikes

  • Link utilization: Upstream 4 links at 10.24Gb/s with

FEC5 and downstream 1 at 2.56Gb/s

slide-6
SLIDE 6

HL-LHC

6

ECAL Phase 2 Upgrade Overview

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

Motherboard L V R V F E V F E V F E V F E V F E FE HV S 10.24 GB/s links Powerful FPGA Feature extraction Suppress isolated anomalous deposits Trigger primitive formation (signal fit, clustering, etc.) L1 Trigger DAQ TCDS + clock 2.56 GB/s links 16 GB/s links 16 GB/s links

  • Move trigger primitive generation to OFF-detector electronics

(FPGA)

  • Three main tasks:
  • Concentrate detector row data, build trigger primitives and transmit

them to L1 Trigger. (402.03.03.04)

  • Receive the LHC clock and distribute with high precision to the on

detector electronics. (402.03.03.05)

  • Form and send a event of data to the DAQ after each L1A signal

arrived though the TTC interface. (402.03.03.03)

Requirement: BCAL-sci-engr-002

slide-7
SLIDE 7

HL-LHC

  • Slide taken from:

https://indico.cern.ch/event/649204/contributions/2639554/attachments/1483850/2302504/CMS_HL- LHC_EM_Calorimeter_Upgrade_20170628.pdf

7

Clock and Control System

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

  • High Precision (HP)

clock of 30ps is required for ECAL in the phase 2 upgrade.

  • As baseline this clock

will be delivered to the Front-End (FE) by the lpGBT downstream links.

  • A fallback option is to

deliver the HP clock directly through additional fibers.

Requirement: BCAL-sci-engr-006

slide-8
SLIDE 8

HL-LHC

  • Waiting for L1A from TCDS++, build evens and send to the DAQ
  • DTH provides two different data rates 16Gb/s and 25Gb/s
  • 4x16Gb/s bidirectional links per FPGA. Requirement: BCAL-engr-039
  • 8x16Gb/s bidirectional links per BCP
  • ~120 Gb/s per BCP

8

Data Concentrator and DAQ links

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

  • η +
  • η +

B C P B C P B C P B C P B C P B C P B C P B C P B C P B C P B C P B C P D T H

8x16Gb/s 8x16Gb/s 8x16Gb/s

… …

… …

ATCA backplane

slide-9
SLIDE 9

HL-LHC

9

I/O link protocols

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

Link rate Encoding bits/ BX Effective line rate Comments

#1

800Mb/s 8b10b 16 640Mb/s ECAL shared links high rate

#2

2.4Gb/s GBT 44 1.76Gb/s HCAL control and status

#3

2.56Gb/s lpGBT 32 1.28Gb/s ECAL FE control and clock distribution – downstream

#4

3.2Gb/s 8b10b 64 2.56Gb/s TCDS++ interface and LHC clock

#5

5.0Gb/s 8b10b 96 3.84Gb/s HCAL data concentration – upstream BCAL-engr-004

#6

10.24Gb/s lpGBT 224 8.96Gb/s ECAL data Concentration – upstream BCAL-engr-004

#7

14Gb/s 64b66b 320 12.8Gb/s ECAL shared links high rate, interface to DAQ, interface to Calo Layer-1

#8

16Gb/s 64b66b 384 15.36Gb/s

  • The system will communicate with interfaces in different protocols and rates.
slide-10
SLIDE 10

HL-LHC

  • The curves show

resources of Ultrascale FPGAs (that meat the transceiver number) versus the resources of the reference XC7VX690 used in CTP7.

  • The Energy and timing

information will be extracted using DSPs with a minimum requirement of 1800 slices.

  • XCKU115 is the

baseline, capable to accommodate both ECAL and HCAL

10

FPGA baseline

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

FPGA / XC7VX690

No Family FPGA LUT (K) DSP BRAM (Mb) GTH GTY MGTs 0 Virtex-7 XC7VX690 433.200 3,600 51.68 80

  • 80

1 Kintex Us XCKU085 497.520 4,100 56.9 56

  • 56

2 Kintex Us XCKU115 663.360 5,520 75.9 64

  • 64

3 Kintex Us+ XCKU15P 1,143.000 1,968 70.6 44 32 76 4 Virtex Us XCVU160 926.400 1,560 115.2 40 36 76 5 Virtex Us XCVU190 1,074.240 1,800 132.9 40 36 76 6 Virtex Us+ XCVU5P 601.000 3,474 36

  • 76

76 7 Virtex Us+ XCVU9P 1,182.000 6,840 75.9

  • 76

76

0.000 0.500 1.000 1.500 2.000 2.500 3.000 XCKU085 XCKU115 XCKU15P XCVU160 XCVU190 XCVU5P XCVU9P LUT (K) DSP BRAM (Mb) MGTs Reference FPGA DSP threshold

Package B2104

slide-11
SLIDE 11

HL-LHC

  • ECAL Towers: 3η x 4φ
  • Region: 300 crystals
  • 15η x 20φ = 0.26 x 0.35
  • 216 UltraScale FPGAs
  • ATCA blades

11

Processing region per FPGA

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Tower

crystal

slide-12
SLIDE 12

HL-LHC

  • 12 Towers x 5 lpGBT = 60 links (48 upstream & 12 downstream)

12

Front-End to Back-End links

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72

15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Tower

20 degrees (2 Super Modules)

+ η

  • φ

Total number of EB Back-End FPGAs 216 One FPGA - 12 Towers

5 4 3 2 1 1 2 3 4 5

One Tower 5 lpGBT links 4 up 1 down

Tower

crystal Total number of FE to BE links: 12240 Number of links Per BE card is 60 Requirement: BCAL-engr-017, BCAL-engr-018

slide-13
SLIDE 13

HL-LHC

13

Shared Back-end links (in case we need)

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Top or bottom: 20 Crystals x 16bits x 40MHz = 12.8Gb/s => 2 links @16Gb/s Right or left neighbors: 30 Crystals x 16 bits x 40MHz = 19.2Gb/s => 2 links @16Gb/s Swiss cross

15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

12 Towers per BE card

Tower

Corner neighbors: 2 Crystals x 16 bits x 40MHz = 1.28Gb/s => 2 links @0.8Gb/s

8 Transceivers in total

4 Right/left 4 Top/Bottom

8 SERDES

2 on each corner

Cluster ΔηxΔφ = 3x5 xtls Requirement: BCAL-engr-023, BCAL-engr-026, BCAL-engr-027

slide-14
SLIDE 14

HL-LHC

  • From the total 384 bits per BX per link, 320 bits are used for crystal TPs

and 32 bits for protocol header, CRC, eventual metadata, etc

  • Requirements: BCAL-engr-023, BCAL-engr-026, BCAL-engr-027

14

Back-End to Layer1 Trigger Calo links

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72

15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

+ η

  • crystal

BE FPGA 12 Towers

One 16Gb/s link

Tower φ

20 degrees (2 Super Modules)

Total number of EB Back-End FPGAs 216

Tower

Total number of BE to L1T links: 3060

One 16Gb/s link One 16Gb/s link One 16Gb/s link One 16Gb/s link One 16Gb/s link

Number of links per BE FPGA is 15

One 16Gb/s link

  • Baseline: Single crystals TPs using 16Gb/s links.
  • Each optical link includes TPs from 20 crystals (4φ x 5η)

15 links x 16Gb/s Requirement: BCAL-sci-engr-004, BCAL-engr-028, BCAL-engr-029, BCAL-engr-031

slide-15
SLIDE 15

HL-LHC

15

BE to DAQ - Event size

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

  • FE links
  • lpGBT (FEC5) upstream payload: 224 bits
  • 4 lpGBTs (FEC5): 4 links x 224 bits = 896 bits
  • 12 towers: 12 towers x 896 bits = 10752 bits
  • Input readout size assuming 5 BXs: 5 BX x 10752 bits = 53760 bits
  • L1T links
  • Number of crystals: 12 towers x 25 crystals = 300 xtls
  • Crystal TPs: 16 bits x 300 xtls = 4800 bits
  • Output readout size assuming 5 BXs: 24000 bits
  • Readout Window (BXs): 53760 bits + 24000 bits = 77760 bits
  • Hence event size: ~9 KB
  • DAQ payload needed at 750 KHz Trigger rate:
  • 77760 bits x 750 KHz = ~ 59 Gb/s
  • 4 x 16Gb/s (one QUAD) to send an event to DAQ

Total number of DAQ links: 864 Requirement: BCAL-sci-engr-005

slide-16
SLIDE 16

HL-LHC

16

Summarize Back-End FPGA links

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

  • At least 56 transceivers available
  • Each transceiver consists one Transmitter (TX) and one Receiver (RX)
  • TX and RX will be used separately in different data rates and protocols
  • 56 inputs:
  • 48 @ 10.24Gb/s (lpGBT): upstream links from FEs
  • 8 @ 16Gb/s: Neighbor processor links
  • 35 outputs:
  • 12 @ lpGBT (2.56Gb/s) : downstream links to FEs
  • 8 @ 16Gb/s: Neighbor processor links
  • 15 TP links @ 16Gb/s: Back-End to Calo Trigger Layer-1
  • DAQ and TCDS++:
  • Inputs 2 links: 1 x TCDS++ stat + 1 x DAQ flow status
  • Outputs 6 links: 1 x TCDS++ ctrl + 1 x DAQ flow control + 4 @ 16Gb/s
  • 8 Low rate optical links through standard pins
slide-17
SLIDE 17

HL-LHC

  • ATCA standard

provides all the power two place two FPGAs

  • n the same blade.
  • The FPGAs would be

independent

  • System will use 108

BCP boards.

17

Processor board optical I/O requirements

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017

plGBT = 96 plGBT = 24 Trigger = 30 TCDS = 2 FPGA1 FPGA0 DAQ = 8 BCP 1 Tb/s ½ Tb/s Requirement: BCAL-engr-006, BCAL-engr-021, BCAL-engr-022, BCAL-engr-032, BCAL-engr-033, BCAL-engr-035, BCAL-engr-036 BCP = 2 BCP = 2 BCP = 2 BCP = 2 24 Towers Calo Layer-1 L1T DTH

slide-18
SLIDE 18

HL-LHC

18

ECAL demonstrator chain 2017 at CERN

FNAL Technical Review - Barrel Calorimeter

  • Adapter boards are controlled through Ethernet and configure VFEs via I2C interface. (MGPAs,

triggering test pulses, ADC mode selection)

  • In the Front-End (FE) the master GBTx receive the downstream link, recover the “LHC” clock

and distribute it to all electronics of the tower

  • Slave GBTx use the master’s clock to operate and transmits 2 GBT links
  • 5 GBTx chips (1 master and 4 slaves) receive data from 5 VFE cards using elinks. Then they

encode the data to upstream GBT links

  • The GBT links are locked in the CTP7 (current ECAL/HCAL demo card) and the captured data

are buffering until the L1A arrived.

  • The AMC13 (current DTH) synchronize the system with the “LHC” clock and provides a trigger

signal (emulating the L1A). The trigger is delivered to the AMC13’s front panel by a LEMO cable

  • When the Trigger signal arrives the CTP7 the data are send to the Host PC which plots them

plotted.

Motherboard L V R

V F E V F E V F E V F E V F E

FE HV S 5 GBT links (VFE data) 1 GBT link (clock) Host PC

bc0, trigger LHC clock

C T P 7

A M C 1 3 M C H

μTCA Crate Spy data on VFE Configure VFE Adapter Boards switch trigger

N.Loukas, Aug 30 2017

slide-19
SLIDE 19

HL-LHC

  • The architecture of ECAL is defined
  • Each ECAL ATCA blade will concentrate raw ADC data from 24 Towers
  • Total bandwidth per blade: ~1 Tb/s
  • ECAL blade will process ADC data, generate and transmit Trigger

Primitives to the L1 Trigger.

  • Total bandwidth per blade: ~0.5 Tb/s
  • The interfaces of BCP with DAQ, TCDS and L1T are known
  • The FPGA package to be used on the demonstrator board is also defined
  • We are going to use B2104
  • Baseline choice: XKCU115-2FLVB2104E
  • We finalize the system specs by getting feedback from ECAL

demonstration chain.

  • Test beams in October

19

Conclusions

FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017