ECAL off detector architecture Nikitas Loukas, University of Notre - PowerPoint PPT Presentation

HL-LHC ECAL off detector architecture Nikitas Loukas, University of Notre Dame N.Loukas, Aug 30 2017 FNAL Technical Review - Barrel Calorimeter 1

HL-LHC Bio  M.Sc. in Modern Electronic Technologies, University of Ioannina, Greece: 2009  CMS Collaboration member: Jan 2012 – present.  Based at CERN: Mar 2014 – present.  PhD at University of Ioannina (GR): 2017  Thesis title: CMS L1 Trigger Upgrade - The Barrel Muon Track Finder  Thesis main achievements: Development of the Barrel Muon Track Finder (BMTF) firmware and installation of the system in CMS. [Conference:C15-09- 28, JINST11(2016):no.03,C03038].  Professional Electronics Engineer working for University of Notre Dame, based at CERN: Jan 2017 – present. FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 2

HL-LHC ECAL upgrade on phase 2 - Modivation Max Latency Pipeline (BXs) Max L1A rate Trigger Phase 2 12.5 μ s 500 750MHz After Trigger Phase 1 3.2 μ s 128 150 KHz  Requires a significant enhancement in finer granularity in the calorimeter trigger. Instead of TPs per Tower it is planned to have TPs per crystal or small clusters. Requirement: BCAL-sci-engr-002  The new granularity will increase bandwidth and processing power . The Requirement: BCAL-sci-engr-007 build of TPs will be moved from the FE to BE .  The calorimeter trigger must be able to accommodate the required latency . Requirement: BCAL-sci-engr-005  The higher rates produce higher pileup which will be mitigated by precision timing in the calorimeter as well as spike killing . Requirement: BCAL-sci-engr-006 FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 3

HL-LHC ECAL Block Diagram of the legacy system Trigger path DAQ path TTC & ctrl path  Separate trigger and data cards  Reads out only areas of interest due to bandwidth limitation  5x5 crystal trigger primitives FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 4

HL-LHC Front-End Phase 2 Upgrade  In legacy ECAL, the TPs are build in the trigger towers  FENIX chip sums of 5 crystals into a strip, analyze the digitized signal and performs Trigger and Readout functions  0.8 Gb/s links are sending all data to the VME BE.  The sampling rate is 40 MHz  In ECAL phase 2, the TPs will be build in the Back- End  FE will have lpGBT chips that will concentrate row ADC data and send them to BE via Versatile+ links  In addition FE will distribute the clock and will control the VFEs via I2C  The sampling will be done at 160 MHz, which will help the BE to reject signal spikes  Link utilization: Upstream 4 links at 10.24Gb/s with FEC5 and downstream 1 at 2.56Gb/s FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 5

HL-LHC ECAL Phase 2 Upgrade Overview L1 Trigger 16 GB/s FE links 10.24 GB/s Powerful FPGA links DAQ V V V L V V Feature extraction 16 GB/s links V F F F F F Suppress isolated E E E R E E anomalous deposits Trigger primitive 2.56 GB/s formation (signal fit, links TCDS + clock clustering, etc.) Motherboard  HV Move trigger primitive generation to OFF-detector electronics S Requirement: BCAL-sci-engr-002 (FPGA)  Three main tasks:  Concentrate detector row data, build trigger primitives and transmit them to L1 Trigger. (402.03.03.04)  Receive the LHC clock and distribute with high precision to the on detector electronics. (402.03.03.05)  Form and send a event of data to the DAQ after each L1A signal arrived though the TTC interface. (402.03.03.03) FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 6

HL-LHC Clock and Control System Requirement: BCAL-sci-engr-006  Slide taken from: https://indico.cern.ch/event/649204/contributions/2639554/attachments/1483850/2302504/CMS_HL- LHC_EM_Calorimeter_Upgrade_20170628.pdf  High Precision (HP) clock of 30ps is required for ECAL in the phase 2 upgrade.  As baseline this clock will be delivered to the Front-End (FE) by the lpGBT downstream links.  A fallback option is to deliver the HP clock directly through additional fibers. FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 7

HL-LHC Data Concentrator and DAQ links  Waiting for L1A from TCDS++, build evens and send to the DAQ  DTH provides two different data rates 16Gb/s and 25Gb/s  4x16Gb/s bidirectional links per FPGA. Requirement: BCAL-engr-039  8x16Gb/s bidirectional links per BCP  ~120 Gb/s per BCP 8x16Gb/s 8x16Gb/s 8x16Gb/s … … B B B B B B B B B B B B D C C C C C C C C C C C T C - η + - η + P P P P P P P P P P P P H … … ATCA backplane FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 8

HL-LHC I/O link protocols  The system will communicate with interfaces in different protocols and rates. Link rate Encoding bits/ Effective Comments BX line rate #1 800Mb/s 8b10b 16 640Mb/s ECAL shared links high rate #2 2.4Gb/s GBT 44 1.76Gb/s HCAL control and status #3 2.56Gb/s lpGBT 32 1.28Gb/s ECAL FE control and clock distribution – downstream #4 3.2Gb/s 8b10b 64 2.56Gb/s TCDS++ interface and LHC clock #5 5.0Gb/s 8b10b 96 3.84Gb/s HCAL data concentration – upstream BCAL-engr-004 #6 10.24Gb/s lpGBT 224 8.96Gb/s ECAL data Concentration – upstream BCAL-engr-004 #7 14Gb/s 64b66b 320 12.8Gb/s ECAL shared links high rate, interface to DAQ, interface to Calo Layer-1 #8 16Gb/s 64b66b 384 15.36Gb/s FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 9

HL-LHC FPGA baseline  The curves show No Family FPGA LUT (K) DSP BRAM (Mb) GTH GTY MGTs resources of Ultrascale 0 Virtex-7 XC7VX690 433.200 3,600 51.68 80 - 80 FPGAs (that meat the 1 Kintex Us XCKU085 497.520 4,100 56.9 56 - 56 transceiver number) 2 Kintex Us XCKU115 663.360 5,520 75.9 64 - 64 versus the resources of 3 Kintex Us+ XCKU15P 1,143.000 1,968 70.6 44 32 76 the reference 4 Virtex Us XCVU160 926.400 1,560 115.2 40 36 76 XC7VX690 used in CTP7. 5 Virtex Us XCVU190 1,074.240 1,800 132.9 40 36 76 6 Virtex Us+ XCVU5P 601.000 3,474 36 - 76 76  The Energy and timing 7 Virtex Us+ XCVU9P 1,182.000 6,840 75.9 - 76 76 information will be 3.000 extracted using DSPs with a minimum 2.500 requirement of 1800 LUT (K) 2.000 slices. DSP BRAM (Mb) 1.500  XCKU115 is the MGTs 1.000 baseline, capable to Reference FPGA accommodate both DSP threshold 0.500 ECAL and HCAL FPGA / 0.000 XC7VX690 XCKU085 XCKU115 XCKU15P XCVU160 XCVU190 XCVU5P XCVU9P Package B2104 FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 10

HL-LHC Processing region per FPGA 15  ECAL Towers: 3 η x 4 φ 14 13  Region: 300 crystals 12  15 η x 2 0φ = 0.26 x 0.35 11 Tower crystal 10  216 UltraScale FPGAs 9 8 7  ATCA blades 6 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 11

HL-LHC Front-End to Back-End links 34 Total number of 33 15 32 31 FE to BE links: 12240 14 30 29 5 28 13 27 One Tower 26 4 12 25 Number of links 5 lpGBT links 24 11 23 Per BE card is 60 + 3 22 1 down 10 21 4 up 20 2 9 19 η 18 17 8 1 16 15 7 One FPGA - 12 Towers 14 - 13 6 1 2 3 4 5 12 11 5 10 9 4 crystal 8 7 Total number of Tower 3 6 5 EB Back-End FPGAs 216 4 2 3 2 1 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 φ Tower 20 degrees (2 Super Modules)  12 Towers x 5 lpGBT = 60 links (48 upstream & 12 downstream) Requirement: BCAL-engr-017, BCAL-engr-018 FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 12

HL-LHC Shared Back-end links (in case we need) 3 3 Right or left 2 2 Swiss cross 1 1 neighbors: 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 15 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 30 Crystals x 16 14 14 13 13 bits x 40MHz = 12 12 11 11 19.2Gb/s => 10 10 9 9 2 links @16Gb/s Tower 8 8 7 7 6 6 5 5 12 Towers per 4 4 3 3 BE card 2 2 1 Cluster Δη x Δφ = 3 x5 xtls 1 Corner neighbors: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 2 Crystals x 16 bits x 40MHz = 1.28Gb/s => Top or bottom: 20 2 links @0.8Gb/s Crystals x 16bits x 40MHz = 12.8Gb/s 8 Transceivers in total => 2 links @16Gb/s 4 Right/left 4 Top/Bottom 8 SERDES 2 on each corner Requirement: BCAL-engr-023, BCAL-engr-026, BCAL-engr-027 FNAL Technical Review - Barrel Calorimeter N.Loukas, Aug 30 2017 13