BO5-2 Calorimeter Trigger 402.06.03 W. H. Smith, U. - - PowerPoint PPT Presentation
BO5-2 Calorimeter Trigger 402.06.03 W. H. Smith, U. Wisconsin-Madison L3 Manager, HL-LHC Calorimeter Trigger Upgrade, 402.06.03 Directors Review of US-CMS HL-LHC Upgrades Fermilab, February 2, 2016 2-Feb-2016 W. H. Smith BO5-2: HL-LHC
BO5-2 Calorimeter Trigger 402.06.03
L3 Manager, HL-LHC Calorimeter Trigger Upgrade, 402.06.03 Director’s Review of US-CMS HL-LHC Upgrades Fermilab, February 2, 2016
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 1
!! WBS definition !! Basis of Estimate !! Schedule !! Cost and Labor Profiles !! Risk and Contingency !! R&D status and plans !! ES&H and QA !! Summary
2
Outline
BO5-2: HL-LHC Calorimeter Trigger Upgrade 2-Feb-2016 W. H. Smith
3
402.06 Organization Chart to L3
BO5-2: HL-LHC Calorimeter Trigger Upgrade 2-Feb-2016 W. H. Smith
402.06 Trigger Jeff Berryhill (FNAL) 402.06.03 Calorimeter Trigger Wesley Smith (UW) 402.06.04 Muon Trigger Darin Acosta (UF) 402.06.05 Track Correlator Rick Cavanaugh (UIC) Institutions involved: U. Wisconsin – Madison and Fermilab
!! W.S.*† (U. Wisconsin) – US CMS HL-LHC L3 Calorimeter Trigger Project Manager
!! CMS Trigger Project Manager 1994-2007, !! Trigger Coordinator 2007 – 2012 !! Trigger Performance and Strategy Working Group 2012 - 2015 !! US CMS L2 Trigger Project Manager (construction and operations) 1998 – present !! US CMS Phase 1 Upgrade L2 Trigger Project Manager 2013 – present
!! Sridhara Dasu*† (U. Wisconsin)
!! US CMS L3 Manager for Calorimeter Trigger (construction and operations) 1998 – present !! US CMS L3 Manager for Phase 1 Calorimeter Trigger Upgrade 2013 – present !! Author of original and upgrade cal. trig. Algorithms 1994 – present
!! Pam Klabbers*† (U. Wisconsin) – Cal. Trig. On-site Manager
!! CMS Regional Calorimeter Trigger Operations Manager (more than a decade on RCT project) !! CMS Deputy Trigger Technical Coordinator
!! Jeff Berryhill†
!! CMS & US CMS Phase-1 Stage-1 Calorimeter Trigger Project Manager 2013-present
!! Tom Gorski*† (U. Wisconsin) – Cal. Trig. Electrical Engineer – Lead Engineer
!! Over a decade of engineering on the CMS Calorimeter Trigger !! Delivered final phase of original Regional CMS Calorimeter Trigger !! Delivered Phase 1 Layer-1 Calorimeter Trigger Upgrade Electronics
!! Ales Svetek† (U. Wisconsin) – Cal. Trig. Firmware Engineer
!! 3 years on Phase 1 Calorimeter Trigger Upgrade Firmware !! (4 years ATLAS Beam Conditions Monitor Firmware, DAQ, Commissioning, Detector Operations)
!! Marcelo Vicente† (U. Wisconsin) – Cal. Trig. Firmware Engineer
!! 3 years on Phase 1 Calorimeter Trigger Upgrade Firmware + HCAL Firmware !! 2 Years on ECAL Phase 1 Upgrade Trigger Primitive Generation Electronics (oSLB, oRM)
!! Jes Tikalski† (U. Wisconsin) – Cal. Trig. Software Engineer
!! 3 years on Phase 1 Calorimeter Trigger Upgrade Software and embedded systems
!! Robert Fobes*† (U. Wisconsin) – Cal. Trig. Technician
!! 2 decades of work on CMS Calorimeter Trigger; ordering, production, testing, repairs
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 4
Team Members involved in delivering
Phase 1 upgrade† trigger systems on schedule and on budget
WBS Definition
5
BO5-2: HL-LHC Calorimeter Trigger Upgrade 2-Feb-2016 W. H. Smith
Level-1 Trigger Architecture
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 6
402.06.03 U.S. covers ~ 55% fraction
Sorting/Merging Layer Muon Track-Finder MPC CSC DT LB RPC Global Correlations (Matching, PT, Isolation, vertexing, etc.) Splitters fan-out fan-out ECAL EB HCAL HB HCAL HF
single xtalRegional Calo Trigger Layer Global Calo Trigger Layer Tracker
Track-Finding
GEM + iRPC Global Trigger Tracker Stubs HGCAL
HGCAL
Calorimeter Trigger Muon Trigger Track Trigger
!! EB/EE/HB/HE: Process individual readout granularity cells from Calorimeter Back-end (Trigger Primitive Generation – TPG) electronics to be optimally matched with track trigger information
!! Produce Tau, Jet, e/! clusters….
!! New Endcap calorimeter TPG electronic produces clusters with Tau, Jet, e/! topologies which are then processed for optimal matching with track trigger info. !! Data processed by input Regional Layer and then final Global Layer providing the output. Similar to current calorimeter trigger, essentially scaled to higher number
!! Tasks: Isolation, duplicate removal, boundaries, global energy sums !! Produces/refines candidate objects/clusters to send to the different track correlator processors
!! Also provides stand-alone calorimeter trigger
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 7
Calorimeter Trig. Design Requirements
L1 Calorimeter Trigger Upgrade
!! Example of Stand- alone Calorimeter Trigger:
!! Process individual crystal energies instead of present 5x5 towers !! Higher resolution matching to tracks: "R < 0.006 !! Improvement in stand- alone electron trigger efficiency + rate#
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 8
Crate B Crate C Crate A Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor Processor
Track Correlator
Regional Processing: Global Processing:
Model for L1 Cal. Trigger Hardware
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 9
Base processors on existing CMS Virtex7 trigger processor cards
cluster ECAL using fine granularity information for e/! candidates for track matching/veto + track isolation, and use wider H clusters behind for veto, etc.
Global Trigger
Stand-alone calo. trig.
HCAL ECAL HGCAL HF HCAL ECAL HGCAL HF HCAL ECAL HGCAL HF
!! WBS includes all Engineering and Technical activities as well as M&S to produce the calorimeter L1 trigger upgrade electronics. !! The system takes as its input the Trigger Primitive data on optical fibers from the HCAL and ECAL Back End Electronics and provides processed trigger data for the CMS Correlator and Global triggers. !! WBS includes managing production of the cards, engineering in support of the production of the cards, procurement of the optical components, FPGAs and all other components on the calorimeter L1 trigger upgrade electronics. !! WBS also includes fabrication of the PCBs and assembly of the finished calorimeter L1 trigger upgrade electronics. !! Costs associated with production of boards are assumed to be consistent with the costs experienced with the Phase 1 Trigger Upgrade with appropriate economies of scale applied when quotes justify them.
!! Less expensive optical parts and FPGAs lack bandwidth to process trigger data in planned number of cards and crates, resulting in a multiplication of the system in size by factors of 2-4, which costs significantly more in both M&S and labor.
!! Technology extrapolations may potentially reduce costs but not included.
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 10
WBS Overview
!! 402.06.03.01 Calorimeter Trigger Management
!! 402.06.03.01.01 Calorimeter Trigger Milestones, Interfaces !! 402.06.03.01.02 Calorimeter Trigger Travel
!! 402.06.03.02 Regional Calorimeter Trigger
!! 402.06.03.02.01 Regional Calorimeter Trigger M&S (Detail Next Slide) !! 402.06.03.02.02 Regional Calorimeter Trigger Engineering !! 402.06.03.02.03 Regional Calorimeter Trigger Technical Work !! 402.06.03.02.04 Regional Calorimeter Trigger FW !! 402.06.03.02.05 Regional Calorimeter Trigger SW
!! 402.06.03.03 Global Calorimeter Trigger
!! 402.06.03.03.01 Global Calorimeter Trigger M&S !! 402.06.03.03.02 Global Calorimeter Trigger Engineering !! 402.06.03.03.03 Global Calorimeter Trigger Technical Work !! 402.06.03.03.04 Global Calorimeter Trigger FW !! 402.06.03.03.05 Global Calorimeter Trigger SW
!! 402.06.03.04 Calorimeter Trigger Infrastructure
!! 402.06.03.04.01 Crates and Power Supplies M&S !! 402.06.03.04.02 Cables, Fibers and Patch Panel M&S !! 402.06.03.04.03 Test Facilities M&S !! 402.06.03.04.04 Infrastructure Engineering !! 402.06.03.04.05 Infrastructure Technical Work
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 11
402.06.03 WBS: Calorimeter Trigger I
!! 402.06.03.02.0X, (X=1,2) (Regional, Global) Calorimeter Trigger M&S
!! 402.06.03.02.0X.1 Cal. Trig. Preproduction Optics !! 402.06.03.02.0X.2 Cal. Trig. Preproduction FPGAs !! 402.06.03.02.0X.3 Cal. Trig. Preproduction Misc. Comp. !! 402.06.03.02.0X.4 Cal. Trig. Preproduction PCB Fabrication !! 402.06.03.02.0X.5 Cal. Trig. Preproduction Assembly !! 402.06.03.02.0X.6 Cal. Trig. Optics !! 402.06.03.02.0X.7 Cal. Trig. FPGAs !! 402.06.03.02.0X.8 Cal. Trig. Misc. Comp. !! 402.06.03.02.0X.9 Cal. Trig. PCB Fabrication !! 402.06.03.02.0X.10 Cal. Trig. Assembly
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 12
Calorimeter Trigger M&S Detail
!! M&S costs are based on escalated prices of similar components used for the Phase 1 upgrade of the L1
!! Labor costs are estimated from engineers currently
calculated as per the Phase 1 Trigger Upgrade
!! International travel is estimated at $3K per trip, and domestic travel is estimated at $1K per trip.
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 13
Basis of Estimate Overview
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 14
Cost Estimate (from CMS HL-LHC TP)
EB Channels (via Back-End Elect) 61200 # of xtals EE Channels (via Back-End Elect) – New Endcap 61000 Use # of Shashlik channels temporarily until #’s from HGCAL available HB/HE Chan. (via Back-End Elect) 13824 From HCAL Phase 1 HF Channels (via Back-End Elect) 1728 From HCAL Phase 1 but combine 2 measurements/PMT Information per channel (bits) 12 assume 10 bits energy and 2 bits quality Total Bits 1653024 Bandwidth (bits/sec) 6.61E+13 Data transmitted at 40 MHz Card BW (bits/sec) 4.92E+11 Assume present cards with 80x10 Gbps links running 192 bits at 40 MHz with 80% packing efficiency
135 Cards This number of cards assumes the present Phase 1 Upgrade Boards Add 33% more cards for Layer 2 179 Cards Multiply by 15 k$/card + 15% Spares + 17 k$/12 Cards Infra- structure (Power, Crate, Fibers) 3.5 M$ (US pays for 2.5 M$) NB: Estimate done with Phase 1 CTP7 (Virtex7) card capabilities and costs. For HL-LHC Expect Ultrascale+ FPGA costs higher, but fewer boards used (next slide).
Cost Driver: FPGAs
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 15
Virtex 7 Ultrascale Ultrascale+ Virtex 7 used in CMS Phase 1: 80 x 10 Gbps transceivers = 800 Gbps Ultrascale+: 120 x 30 Gbps transceivers = 3600 Gbps (e.g. 4.5 x V7) (Virtex Ultrascale+ samples available before summer) Virtex 7: retail cost now: $7.5K ea. Ultrascale+: unofficial estimated retail cost: $15K ea. (e.g. twice V7 cost) (~ 50% of Phase 1
Courtesy Xilinx, Inc.
!! R&D M&S
!! 2 prototype cards at $20k apiece in FY17,18,19 each !! $13K ($15K) in FY17 (18,19) for crates and other testing infrastructure.
!! Production M&S costs
!! (135 system, 2 test stand, 20 spare) 157 regional cards at $15.4k each, !! (45 system, 1 test stand, 7 spare) 52 global cards at $15.4k each. !! Design one card for both, different FW
!! Production Card M&S breakdown
!! Based on costs of the Phase 1 cards !! FPGA costs (Virtex plus ZYNQ) of $6.6k. !! Optical Components $3.6K, !! Miscellaneous Components $1.4k, !! Printed Circuit Board $2k !! Assembly $1.8k.
!! Production Infrastructure M&S Breakdown
!! Based on costs of the Phase 1 Infrastructure !! Power Supplies, Fibers, Patch Panels summed to $17k/crate of 12 cards
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 16
M&S for Production and R&D
NB: Assumed Phase 1 µTCA Form Factor smaller than other FF’s considered (e.g. ATCA) means costs are higher since overall card infrastructure is amortized
conservative cost model
!! Electronics Engineering and Technical work to design, produce, and test the calorimeter trigger electronics !! Software Engineering to produce the software to program, test, operate, diagnose, configure, validate and read out the calorimeter L1 trigger upgrade electronics.
!! This includes software to interface with the Trigger Online System
!! Firmware Engineering to implement the full functionality of the calorimeter L1 trigger upgrade electronics
!! Including implementing the trigger algorithms, diagnostics, data acquisition and readout.
!! SW and FW produced in releases for testing cards, commissioning cards and operating them under initial, low, medium and high luminosity HL-LHC conditions. !! All labor costs based on actual costs for corresponding tasks for the Phase 1 Trigger Upgrade
!! In most cases by the same people
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 17
Calorimeter Trigger Labor
!! The calorimeter trigger labor for the R&D period FY17-19 includes:
!! 1004 FTE hours/year divided equally between University- based Electronic, Firmware and Software Engineers !! 502 FTE hours/year of Fermilab-based Electronic Engineering for FY18,19.
!! The calorimeter trigger labor for the production phase FY20-23 includes:
!! 3012 FTE hours/year divided equally between University- based Electronic, Firmware and Software Engineers !! 251 FTE hours/year of Fermilab-based Electronic Engineering.
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 18
Calorimeter Trigger Labor FTE
19
Construction Schedule
BO5-2: HL-LHC Calorimeter Trigger Upgrade 2-Feb-2016 W. H. Smith
!"#$% !"#&% !"#'% !"##% !"#(% !"#)% !"(*% !"(+% !"(,%
CD4 CD1 CD2 CD3 CD0
Specification and Technology R&D Trigger TDR TDR
Pre- production
Installation
LS 2 LS 3 Physics Physics
LHC Schedule
TDR
CDR PDR CD3A FDR
Prototyping and Demonstrators Production Readiness Review Production and Test Test & Commission
Reality Check:
1st Proto: Q4/CY13 6 Pre. Prod. Proto: Q2-3/CY14 1st Production Board: Q1/CY15 50 Board Production Complete Q2CY15 Installation Complete: Q2/CY15 Commissioned, Operations start: Q3/CY15 21 Months: 1st prototype to pp operations
20
Cost in k$
Cost = AY $M (No Contingency)
L3 Area M&S* Labor Total R&D Calorimeter Trigger 2537 1044 3581 583
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade
*Includes $30K Travel NB: escalation at 3%/year starting in FY17
)% #))% &))%
+))% ()))% (#))% !"(-% !"(,% !"(+% !"(*% !"#)% !"#(% !"##% !"#'% !"#&% !"#$%
AY K$
./0% 12345678923%
21
Cost Profile
BO5-2: HL-LHC Calorimeter Trigger Upgrade 2-Feb-2016 W. H. Smith
)% ):#% ):&% ):-% ):+% (% (:#%
!"(-% !"(,% !"(+% !"(*% !"#)% !"#(% !"##% !"#'% !"#&% !"#$%
!;<=%>3?@3AA6% !;<=%BA8C% D3@EA64@5F%>3?@3AA6% D3@EA64@5F%BA8C%
22
Labor FTE Profile
BO5-2: HL-LHC Calorimeter Trigger Upgrade 2-Feb-2016 W. H. Smith
!! M&S: 50%
!! Based on conceptual design documented in CMS HL-LHC Technical Proposal !! Items based on existing Ph. 1 Trigger Upg. with documented costs
!! Travel: 20%
!! Based on LOE required as determined from Ph. 1 Trigger Upgrade
!! Labor: 50%
!! Based on expert judgment using documented experience of similar work required for the Phase 1 Trigger Upgrade !! Development of activities defined at a conceptual level informed by the experience of the Phase 1 Trigger Upgrade !! Technical requirements are a moderately challenging, but straightforward extrapolation from the Phase 1 Trigger Upgrade
!! Note: Phase 1 Upgrade production/installation finished in
!! Using recent data for labor, M&S in the cost, contingency, risk estimates
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 23
Contingency Calculation
!! C&S understood since based on Phase 1 Trigger Upgrade Systems experience !! Cards are extrapolations of existing Phase 1 Trigger Upgrade Cards
!! Using most conservative costing model (construct with Phase 1 hardware) !! R&D and technology advances offer opportunities to reduce cost.
!! C&S based on experience of the same team that built and wrote software and firmware for Phase 1 Trigger Upgrade
!! Exploiting new commercial tools offer opportunities to reduce amount of custom written FW and SW !! Opportunities already realized in Phase 1 Cal. Trig. Upgrade
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 24
Cost and Schedule Risks
DR Question 2
!! Senior Engineer becomes unavailable (Low Risk)
!! Hire new engineer, subcontract to consulting firm, use FNAL engineer
!! Software or Firmware does not meet requirements (Low Risk)
!! Hire extra expert effort to recover schedule and help personnel
!! Boards are delayed (design, manufacture or testing) (Low Risk)
!! Hire extra effort to speed up testing schedule
!! Vendor non-performance (Low Risk)
!! Acquire spending authority to use alternative vendors (while
!! Input or output electronics (non-trigger) delayed (Low Risk)
!! Built in capabilities of trigger electronics provide signals for their
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 25
Managed Trigger Risks & Mitigation
!! None !! Since calorimeter trigger has full self-test capabilities, satisfying performance requirements validated w/o requiring connected inputs or outputs. !! Trigger electronics can store up sequences of test patterns and inject them into the front end of the trigger electronics at speed !! These patterns are based on actual LHC pp data taken before LS3 !! Trigger electronics can receive its output, process and record this at speed for subsequent readout by DAQ.
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 26
External Dependencies
!! Safety: follows procedures in CMS-doc-11587, FESHM
!! L3 Manager (W.S.) responsible for applying ISM to trigger upgrade.
!! Modules similar to others built before, of small size and no high voltage
!! Quality Assurance: follows procedures in CMS-doc-11584
!! Regularly evaluate achievement relative to performance requirements and appropriately validate or update performance requirements and expectations to ensure quality. !! QA: Equipment inspections and verifications; Software code inspections, verifications, and validations; Design reviews; Baseline change reviews; Work planning; and Self-assessments. !! All modules have hardware identifiers which are tracked in a database logging QA data through all phases of construction, installation, operation and repair.
!! Graded Approach:
!! Apply appropriate level of analysis, controls, and documentation commensurate with the potential to have an environmental, safety, health, radiological, or quality impact. !! Four ESH&Q Risk levels are defined and documented in CMS-doc-11584.
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 27
Trigger ESH&Q
!! After full testing at institute, shipped to CERN
!! All tests recorded (of all types) for individual boards in database !! Tests use and validate software and firmware test release
!! Acceptance Testing in Electronics Integration Center (EIC) at CERN
!! Individual labs for CSC and Calorimeter Trigger !! Boards retested to validate institute test results !! Tests use software and firmware test release
!! Integration Testing in EIC
!! Row of racks with DAQ, Trigger, Central Clock, Crates of other subsystem electronics !! Operation of a vertical slice with electronics to be tested installed. !! Tests use and validate software and firmware commissioning release
!! Integration Testing at P5: Global Runs
!! Test with all CMS with cosmics when beam not running/with beam when running !! Electronics installed in final locations with final cables !! Full-scale tests with full CMS DAQ/Trigger/Clocking !! Tests use software and firmware commissioning release
!! Handover to Operations at P5: Global Runs/Parallel Operation
!! After testing completes, continue with Global Runs/Parallel Operation !! Validate software and firmware initial operational release
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 28
QA/QC: Testing and Validation
HL-LHC Trigger Algorithm R&D
!! Goal:
!! Allow development of calorimeter, correlation trigger electronics – specify:
!! Plan (with CMS HL-LHC Technical Proposal Milestones):
!! Initial definition of trigger algorithms, primitive objects and inter- layer objects (TP.L1.1) – 2QCY16 !! Baseline definition of trigger algorithms, primitive objects and interchange requirements with subdetectors. (TP.L1.3) – 2QCY17 !! Detailed Software emulator demonstrates implementation of core HL-LHC trigger menu with baseline objects (TP.L1.4) – 4QCY17
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 29
HL-LHC Trigger Hardware R&D - I
!! 2 R&D activities:
!! Calorimeter Trigger Processor !! Track Correlator Processor
!! Hardware R&D Milestones - I
!! Initial demonstration of key implementation technologies (TP.L1.2) – 4QCY16
feasibility of trigger design, leads to:
!! Definition of hardware technology implementation baseline (TP.L1.5) – 1QCY18
for #
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 30
HL-LHC Trigger Hardware R&D - II
!! Hardware R&D Milestones – II
!! Full-function prototypes produced which allow local comparison with emulator (TP.L1.6) – 4QCY18
bandwidth optical links to meet the requirements of the final boards
however, and only local comparisons will be possible between hardware behavior and the emulator.
!! Demonstrator system shows integration and scaling, global/full- chain comparison with emulator (TP.L1.7) – 4QCY19
multiple full-capability prototype boards and the prototype full-capability infrastructure
testing environment capable of being connected to its front end detector for test-beam validation to follow.
!! Final Milestone:
!! HL-LHC Trigger TDR (TP.L1.8) – 1Q2020
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 31
!! Phase 1 upgrade: two generations (V5, V6) before production boards—similar path reasonable for HL-LHC !! HL-LHC upgrade working terminology: APD
!! “APDx”—Advanced Processor Demonstrator, APD1 for gen-1, APD2 for gen-2, etc.
advances in FPGA, SoC, PCB, embedded OS and optical technologies
!! APM—Advanced Processor Module—HL-LHC production platform
!! Today: CTP7 a very capable “Gen 0” demonstrator
!! Supporting HL-LHC Tracking Trigger and Calorimeter Trigger R&D !! Comparatively “young” platform (< 2 years old) w/ new technology !! Receiving interest from other groups as an upgrade and/or HL-LHC R&D platform
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 32
HL-LHC Cal. Trig. Demonstrators
!!12 MGT MicroTCA backplane links !!67 Rx and 48 Tx 10G optical links
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 33
Gen-0 Demonstrator: U. Wisconsin CTP7 Card for Phase 1 Cal. Trig.
Virtex-7 690T FPGA (Data Processor) ZYNQ `045 System-on-Chip (SoC) Device (embedded Linux control platform)
!! Production:
!! 50 Boards – 100% yield
!! Phase 1 L1 Trigger Deployment:
!! Stage 1 and Stage 2 Layer-1 Calorimeter Trigger !! 22 CTP7s !! Stage-1 was main calorimeter trigger for 2015 !! Stage-2 operating in parallel since September – main cal. Trig. for 2016
!! HL-LHC R&D: Cornell Track Trigger prototypes
!! 4 CTP7s @ CERN !! Got running over a weekend! !! 2nd setup at Cornell: 2 CTP7s
!! HL-LHC Cal, Correlator Trigger prototypes: platforms for FW development and testing !! HL-LHC EMU Readout prototype: FW development and testing
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 34
CTP7 Deployment Phase 1 & HL-LHC
!! Embedded Linux
!! Functional Linux system (network, file system, shell) !! Low latency access point tightly integrated with workhorse FPGA !! Basic card level infrastructure with very little new code—Ethernet, I2C, USART, GPIO drivers, ssh, file system, etc.—all standard !! Paid for itself in time saved in the first project cycle
!! AXI Architecture
!! Industry standard on-chip interconnection scheme for FPGAs !! Straightforward to implement AXI interfaces for registers and memory !! AXI infrastructure bridged into the Virtex-7 (“Chip2Chip” core), a single integrated address space for both devices !! 95% of CTP7 generic infrastructure from ZYNQ hard cores and Library IP catalog, no custom HDL needed—it’s in the tools !! Improved status and access fo advances applications
!! XVC – Embedded Linux Xilinx Virtual Cable (e.g. JTAG)
!! Debug Card at P5 via TCP/IP just as if on the bench in the lab
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 35
CTP7 introduces HL-LHC technologies
!! MicroTCA.0—the MicroTCA for Phase 1 !! MicroTCA.4—a standard with a rear transition module (RTM) about the same size as a double-width AMC
!! MicroTCA.4 shares payload power between AMC and RTM !! New Vadatech chassis (VT815) supports 12 full size AMC+RTM combinations with 120W per slot
!! ATCA—older standard, physically larger
!! Shape of the RTM in ATCA limits its utility, but overall ATCA provides about 2X the board and frontpanel area as MicroTCA.4 !! IPMI: CMS-common IPMI solutions (MMC, System Manager) supplied by Wisconsin can easily migrate to ATCA
!! Board costs and FPGAs
!! Xilinx UltraScale+ at about same cost/gate as Virtex-7, but gate/MGT ratios are higher in UltraScale+ —MGTs drive part selection in CMS !! FPGA costs are going to dominate over form factor costs in the high performance applications
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 36
R&D on Upgrade Form Factors
Processi ng FPGA(s)
!! Next-gen FPGA and ZYNQ SoC devices !! General upgrade to embedded Linux platform over CTP7 !! Direct optical interfaces for the ZYNQ PL section !! DDR4 SDRAM on main FPGA for higher density and bandwidth !! Optical module mix for compatibility with current and next-gen
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 37
APD Architecture Example
Processing
FPGA(s) ZYNQ SoC
High BW DDR4 SDRAM
System Memory
GbE Control Path
Optical Interfaces
Front Side Optical Interface
Flash File System
Backplane/RTM MGT Links
Summary
38
BO5-2: HL-LHC Calorimeter Trigger Upgrade 2-Feb-2016 W. H. Smith
!! R&D Program will result in designs for the HL-LHC Calorimeter Trigger Upgrade that will meet technical performance requirements !! Scope and Specifications of this trigger upgrade are sufficiently well-defined to support the C&S estimates !! Trigger upgrade based upon common hardware platforms and components !! ES&H, QA plans, C&S based on experience with
!! Management and Engineering teams are experienced with sufficient design skills, having designed and built
2-Feb-2016 W. H. Smith BO5-2: HL-LHC Calorimeter Trigger Upgrade 39
Conclusions: HL-LHC Cal. Trigger