DAQ for Sensor R&D at FNAL Ryan A. Rivera 2014 Detector R&D - - PowerPoint PPT Presentation

DAQ for Sensor R&D at FNAL

Ryan A. Rivera 2014 Detector R&D DOE Review 29 October, 2014

Comprehensive Approach to Tracking Detector R&D

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 2

Sensor Front End Data Transmission L1 Trigger

Radiation hardness in new sensors (3D columnar + Diamond) New ASIC developments to give integrated track segment information Multi-wavelength optical DAQ Track trigger

Track Fitting

FPGA, GPU’s and Associative Memory based

• n xTCA

Creating a tracking and trigger system that can withstand the projected HL-LHC luminosities is perhaps the most important detector challenge in the field of High Energy Physics. There must be a comprehensive approach (emphasis on FNAL/SCD contributions):

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 3

• Motivation
• simple DAQ

 It is 6” x 6” and, for many systems, the only external

connections needed are a 3.5V power supply and a standard Ethernet cable.

• flexible DAQ

 The user can stack compatible boards in different

combinations to give unique functionality.

• scalable DAQ

 In addition to the vertical stacking, the stacks can be

repeated arbitrarily and connected with one or many PCs in an Ethernet network. Compact And Programmable daTa Acquisition Node (CAPTAN )

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 4

CAPTAN User Template

OPTICAL BUS COOLING CHANNEL VERTICAL BUS LATERAL BUS MOUNTING HOLE ELECTRONICS

• The CAPTAN

architecture consists of a few core boards but is intended to be augmented by custom boards designed and built by users.

CAPTAN: Applications

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 5

• Developed in 2008/2009, the CAPTAN system was

designed to handle common data acquisition, control, and processing challenges within high energy physics.

• Examples of such applications are tracker readout

systems, R&D test stands, and parallel data processing.

• As the CAPTAN system is a modular system it can be

used for a wide range of applications, from very small to very large.

• Quite a number of groups at Fermilab and other

institutes in the US, China, and Europe have acquired the system for their test-stands. We work with them to provide hardware and software support.

• We are currently working on the next generation

CAPTAN with a new FPGA.

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 6

QIE10 Single Event Upset Testing

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 7

Tevatron: T980 Crystal Collimation Telescope

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 8

CAPTAN Stacks Power Supply DUT HV Supply Pixel Telescope Frame Ethernet Router Scintillator

The CAPTAN pixel telescope is 8 silicon pixel planes leftover from CMS, with space for 2-4 DUTs in the middle. Pixel size is 100 µm x 150 µm. Data acquisition with the CAPTAN system.

Fermilab Test Beam Facility

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 9

Fermilab Test Beam continued:

• Old pixel telescope DAQ is based on CAPTAN
• Triggered, 2.5cm2 coverage, and 8µm track resolution
• New strip telescope is based on CAPTAN too.
• Dead-timeless, 16cm2 coverage, and 5µm track resolution
• For the last 6 years CAPTAN supported all versions of the CMS pixel chip
• Recently tested the VIPIC Read Out Chip from FNAL/PPD using CAPTAN

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 10

• Telescope is part of the FTBF facility and has been used by many

experiments as a high resolution tracking tool to characterize different Devices Under Test (DUTs)

• List of Fermi Test Beam Facility Experiments using the Telescope
• T992 - Radiation-Hard Sensors for the HL-LHC (ongoing)
• T995 - Scintillator Muon/Tail Catcher R&D with SiPM Readout
• T979 - Fast Timing Counters – PSEC Collaboration
• T1004 - Total Absorption Dual Readout Calorimetry R&D
• T1006 - Response and Uniformity Studies of Directly Coupled Tiles
• T1017 - CIRTE (COUPP Iodine Recoil Threshold Experiment)
• T958 – FP420 Fast Timing Group
• T1038 – PHENIX Muon Piston Calorimeter
• We will continue to support test beam experiments that want to use the

pixel telescope

FTBF Telescope User Community

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 11

FNAL/PPD Collaboration

• 3D ASIC test stand

and test beam efforts, another run in November scheduled.

• SCREAM (Single CCD Readout

Module): compact low-cost, CCD readout system. Using CAPTAN firmware/software

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 12

• New CMS pixel digital ROC Test Stand
• Distributed to US collaborators in Colorado, Purdue and to

Italian collaborators in Milan, Lecce and Torino. Collaboration with CMS

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 13

Leveraging Event Building and xTCA for CMS

• Data Processing in an FPGA
• Receives all CMS Calorimeter data
• 276 Gbps in to a single FPGA (Xilinx Virtex 7)
• Must aggregate/summarize information and

pass to next stage in < 400ns

• 20 Gbps out from FPGA

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 14

• xTCA (Telecommunications Computing

Architecture)

• Spec put forth by PICMG (PCI Industrial Computer

Manufacturers Group: a consortium of over 250 companies).

• xTCA encompasses MicroTCA and ATCA.
• Large experiments are already using xTCA or

planning to:

– CMS – ATLAS – LHCb – LBNE

xTCA

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 15

• Completed a test beam project at Fermilab
• Real-time event assembly conducted in MicroTCA form

factor.

CAPTAN and xTCA

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 16

• MicroTCA.4 Standard
• Specification finalized in 2012 for the physics community.
• We are collaborating with CERN to use MicroTCA cards

developed in Europe: GLIB, MP7, FC7. 8U 12-slot MTCA.4 shelf

MicroTCA effort

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 17

ATCA effort

12U 14-slot ATCA shelf

• ATCA
• Advanced Telecommunications Computing Architecture
• More space for I/O
• Possible collaboration with SLAC on RCE

development for LBNE

• Work led by Ted Liu

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 18

Other Data Processing Platforms

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 19

• A set of applications running extensible software

components to be customized by experimenters to create a DAQ system.

• Lariat, DarkSide-50, LBNE and Mu2e experiments;

partial use in Nova. Chosen for “Off-the-Shelf” DAQ.

• Recompilation is not needed in to change parameters –

done through configuration scripts that load plug-ins

• artdaq-demo allows users to get a toy artdaq-based

DAQ system up-and-running from scratch in about 10 minutes What is artdaq?

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 20

• For CAPTAN:
• Update FPGA
• Further develop software on LINUX -

web based graphical user interface using HTML5 and JavaScript.

• Build out artdaq support
• Exploit parallel processing power and

mTCA integration

• Provide user support for their

applications

• Work with possible users on new

applications

Next Steps for DAQ for Sensor R&D

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 21

• For xTCA:
• Continue to explore big data

applications

• There are possibilities for event

aggregating and tracking based trigger systems.

• For parallel processing:
• Compare xTCA with GPU and co-

processor fronts

 Intel PHI  CUDA  OpenCL

Next Steps cont.

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 22

• For DAQ systems:
• Proceed with “Off-the-Shelf” DAQ concept
• This proposal is intended to demonstrate the

feasibility of a low-cost, high-bandwidth, commercial approach to data acquisition based on standard networking technology.

• We can no longer afford the costs associated with

developing new back-end systems from scratch for each new experiment. Experiments are asking for an “off-the-shelf”, commodity solution. The Computing Sector is all about “service” and perhaps it’s time FNAL consider a “DAQ as a service” approach.

• Effort will leverage CAPTAN, artdaq, and test beam

experience.

Next Steps cont.

Conclusion

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 23

Sensor Front End Data Transmission L1 Trigger

Radiation hardness in new sensors (3D columnar + Diamond) New ASIC developments to give integrated track segment information Multi-wavelength optical DAQ Track trigger

Track Fitting

FPGA, GPU’s and Associative Memory based

• n xTCA

The efforts related to DAQ for Sensor R&D are intimately tied into every piece of the puzzle for a tracking and trigger system that can withstand the projected HL-LHC luminosities .

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 24

Thank you.

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 25

Backup slides…

CAPTAN Core Boards

“Green Board” NPCB – Node Processing and Control Board “Blue Board” DCB – Data Conversion Board “Red Board” PDB – Power Distribution Board

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 27

Design Philosophy

• Commercial Slow Control hardware and software is used for

development

– needed early, can’t wait for custom HW/SW – hopefully, utility extends into production and running – LabVIEW or EPICS

• Embedded Processors

– use commercial modules if possible to allow early software development – embedded Ethernet and HTTP with eye towards debugging ease

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 28

artdaq

Best Future DAQ R&D Avenues

• Ethernet/InfiniBand: InfiniBand prevalent as the interconnect between nodes
• f the highest performing super computers
• Off-the-shelf components
• xTCA
• GPUs: Fastest computer in the world, Tianhe-1A in China uses 7,168

NVidia Fermi GPUs and 14,336 Intel Xeon CPUs. Would require 50,000 CPUs for same performance with CPUs alone.

• Optics: Commonly 100 Gbps with distances over 100 m. Becoming

interconnect of choice in high speed data centers and HPC clusters. There are 40 Gbps and 100 Gbps optical Ethernet standards.

• FPGAs
• EPICS: The tool is designed to help develop systems which often feature

large numbers of networked computers providing control and feedback.

Data Acquisition Systems - Ryan Rivera 29 Detector R&D Retreat

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 30

DAQ Program Resource Issues

• Limited manpower

– Has hindered xTCA effort

• xTCA development accessibility

– Community MMC design effort

• FPGA power/expense/expertise

31

Conclusions

• For medium-small project and test stands we think that the

CAPTAN should be supported as any other commercial option.

• We believe that especially for small DAQ most of the overhead

comes from designing of the software and firmware. Having an integrated software/firmware with an infrastructure and a user configurable part must be the goal.

• The advantages that we see with the CAPTAN is the fact that the

board is really simple with a direct Ethernet connection and a lot

• f IO.
• For any DAQ system we have done in the past we always had to

create a custom board that was the interface between the detector and the FPGA.

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 32

10/29/2014 Ryan A. Rivera | DAQ for Sensor R&D at FNAL 33