Front-end Electronics and Data Acquisition in Particle Physics Igor - - PowerPoint PPT Presentation

Igor Konorov Institute for Hadronic Structure and Fundamental Symmetries (E18) Technical University of Munich Department of Physics Advanced Workshop on Modern FPGA-Based Technology for Scientific Computing ICTP Trieste

Front-end Electronics and Data Acquisition in Particle Physics

Motivation DAQ and Frond-end electronics Data Processing in FPGA Continuous DAQ Intelligent FPGA-based Data Acquisition System

Plan

CERN Accelerator and Experiments

CERN Accelerators

LHC Experiments

• CMS
• ATLAS
• LHCb
• ALICE
• TOTEM, LHCf, MeEDAL

Fixed Target Experiments

• COMPASS
• NA61/SHINE
• NA62
• DIRAC
• LOUD
• …

LHC CMS Experiment

CERN Accelerators

LHC CMS Experiment

CMS Experiment

3 other LHC experiments

• ATLAS
• LHCb
• ALICE

COMPASS Experiment

1. Detector Front-End Electronics (FEE) 2. Data Acquisition and (Sub-)Event Building

• 3. Trigger Logic

FPGAs in High-Energy Physics

TDC ADC

Data Reduction Steps

• Zero Suppression
• Pedestal Calculation
• Time stamp assignment
• Signal detection
• Feature Extraction
• Signal Amplitude
• Signal Time

 The process of sampling detector signals  Conversion to digital form  Data processing  Transmission to PC for further processing, visualization, and storage FPGA tasks

• Glue logic
• Control and synchronization
• Data processing
• Computer interface

Data Acquisition System

Transducer

Volta ltage Curr rrent

Digitizer

Digital Logic

Amplifier Front-End Electronics DAQ

Experiment : Electron Energy Measurement

Curr rrent

Digitizer FPGA Scintillation Counter 1

e‾

Scintillation Counter 2 Electromagnetic Calorimeter TRIGGER

TRIGGER – define time when detector signal to be measured Why read only when trigger and not continuously ?

• Not feasible to measure continuous data flow
• Not feasible to transmit such amount of data
• Not feasible to sore such amount of data

DAQ Architecture in Particle Physics

Thr

Delay ay cable bles

Trigger igger Logic gic

PCIe Ie Etherne rnet

Inhibit

ADC ADC ADC

Trigger Readout

Front-End Acquisition Slow Control

Twait

Tdead = Tread Twait + Tread

Tread

DAQ Architecture in Particle Physics

Thr

Delay ay cable bles

Trigger igger Logic gic

PCIe Ie Etherne rnet

Inhibit

ADC ADC ADC

Trigger Readout

Front-End Acquisition Slow Control

Twait

Tdead = Tread Twait + Tread

Tread

Efficiency of data taking

RO Sequence : Trigger –> Busy – Read Out -> Release Busy (Ready for next event) = T busy Probability of events described by Poisson distribution J – number of triggers and r – trigger rate A rule of thumb:

Dead Time =

𝑈𝑐𝑣𝑡𝑧 λ

Tbusy – DAQ busy time λ – average time between triggers Tav >> Tbusy

Example: 1kHz => λ = 1ms Tbusy = 50 useconds DeadTime = 0.05/1 = 0.05 or 5% Tbusy 𝑟𝑘 𝑢 = 𝑠(𝑠𝑢)𝑘−1𝑓−𝑠𝑢

𝑘−1 !

, if j = 1 𝑟1 𝑢 = 𝑠𝑓−𝑠𝑢

Pipe Line Front Ends

FIFO

FIFO for N events

FIFO

ADC

100MHz CLK CLK

FIFO Logic

Trigger

FIFO

FIFO for N events

FIFO

ADC

100MHz CLK CLK

FIFO Logic

Trigger

Multi Event buffer: de-randomization buffer Input : Poisson distribution Output : more like a Gaussian centered around average value

DAQ efficiency vs FIFO Depth

DAQ Efficiency Trigger rate/DAQ rate capability

Multi Channel Data Flow

Front-End

Trigger Logic

Event Builder PC Data Concentrator PC

Time Reference

Classical method:

– TRIGGER is a reference – SIGNAL time is measured respectively to TRIGGER

Alternative method for big experiments:

– Distribute CLOCK , why clock?

• Easier to distribute with very low jitter

– Measure absolute time respectively to CLOCK phase

Tsig = Ns Tclk+ tsig Ttrg = Nt Tclk+ ttrg Clock and Data are encoded and transmitted from single source to multiple destinations

Detector Signals Trigger Signal

∆T ∆T

Trigger Signal

Common CLOCK

tsig tsig ttrg

Detector Signals

Encoding Data

Ns T0

Multiple Trigger Levels

Front-End

Trigger Logic

Event Builder

PC

Data Concentrator

PC

Front-End

Trigger Logic

Event Builder

PC

Data Concentrator

PC

Trigger Logic

Front-End

Trigger Logic

Event Builder

PC

Data Concentrator

PC

Trigger Logic Trigger Logic Storage/CDR Storage/CDR Storage/CDR

 Front-end electronics, detector specific

• Conversion of detector analog signal to digital form
• Derandomization
• Data processing: signal detection, extraction of signals’ parameters Time and/or Amp…

 Trigger Logic

• reduce amount of stored data
• define time when of interesting event

 Trigger Distribution system => Time Distribution System  Slow Control System

• Control and monitoring of PS, Gas system, Temperature, Humidity,…
• Programming of Front-ends

 Acquisition System => Event builder

• Data acquisition – moving data from FE to PCs
• Data flow control
• Real time Software
• Run control

DAQ Elements

FPGA

FRONT-END ELETRONICS DATA PROCESSING

1. Detector Front-End Electronics (FEE) 2. Data Acquisition and (Sub-)Event Building  Pipeline Architecture  Reliability

• 3. Trigger Logic

 High-speed  Advanced event identification abilities

 Doublet/triplet finder  Track fitter

FPGAs in High-Energy Physics

TDC ADC

Data Reduction Steps

• Zero Suppression
• Pedestal Calculation
• Time stamp assignment
• Signal detection
• Feature Extraction
• Signal Amplitude
• Signal Time

ADC readout – Avalanche Photodiodes (APDs)

• Semiconductor detector: Reverse-

biased p-n-junction

• Primary photoelectrons accelerated in

electric field → avalanche

• Created charge proportional to

deposited energy

• suitable for detection of every ionizing

radiation

ADC readout – Preamplifier and Shaper

• Charge-sensitive (CS)

preamplifier

• Shaping: CR differentiator

and RC integrator

30 keV proton 20 keV proton Created charge proportional to energy deposit  Amplitude measurement  Signal Time defines EVENT TIME

Sampling ADC FPGA CLK

> 10 MHz 12-16 bit

Noise on Signals

Data reduction steps

Challenges in presence of noise

• Pedestal Calculation
• Averaging over large number of samples
• Signal detection
• Discrimination between signals and noise

(signal-to-noise ratio)

• Feature Extraction
• Precise determination of amplitude or signal

shape

Important quantity: 𝑡/𝑜 = 𝜈 𝜏noise 𝜈 𝜏

• Implementing a low pass filter means averaging over number of samples N:

ҧ 𝑡 =

σ𝑗=0

𝑂−1 𝑡𝑗

𝑂

It is convenient to choose 𝑂 = 2𝑛 so that division can be implemented as a bit shift. (e.g. a/8 = a >> 3)

• Measurement of pedestal when there is no signal
• Very precise method
• Allows to determine noise distribution (RMS): 𝜏2 = 𝑦2 − ҧ

𝑦2 =

σ𝑗=0

𝑂−1 𝑡𝑗 2

𝑂

−

σ𝑗=0

𝑂−1 𝑡𝑗

𝑂 2

• Measurement in the presence of signals: accuracy of measured value depends on

signal rate

Pedestal Calculation – Low Pass Filter

• Continuous calculation of pedestal during data taking with signals and exclude signal

samples for calculation: Σped = Σped + 𝑡𝑗 − ҧ 𝑡 ҧ 𝑡 = Τ Σped 𝑂

• Allows for continuous calculation of RMS of noise:

Σdiff = Σdiff + 𝑡𝑗 − ҧ 𝑡 2 − 𝜏2 𝜏2 = Τ Σdiff 𝑂

Pedestal Calculation – Baseline Follower

Evolution of pedestal and noise RMS at the startup

Conditions 1. Amplitude over threshold: 𝑡𝑗 > thr 2. Window over threshold: σ 𝑡𝑗

𝑂 > thr

3. Difference between consecutive samples over threshold: σ 𝑡𝑗 − 𝑡𝑗−1 > thr 4. Window over baseline: 5. For baseline follower and continuous RMS calculation: 𝑡𝑗 − ҧ 𝑡 > 𝑦 ∙ 𝜏 ∧ 𝑡𝑗+1 − ҧ 𝑡 > 𝑦 ∙ 𝜏 ∧ 𝑡𝑗+2 − ҧ 𝑡 > 𝑦 ∙ 𝜏 ∧ ⋯ ∧ 𝑡𝑗+𝑜 − ҧ 𝑡 > 𝑦 ∙ 𝜏

Signal Detection

Comparison Between Algorithms

Algorithm 3 Algorithm 4

Areas where trigger condition is fulfilled are highlighted. S/N = 6 Parameter for algorithm 3: N = 15 Parameter for algorithm 4: length of averaging wiondows = 16, distance = 30

• Time reference – sampling clock
• Signal time = coarse time & fine time
• Coarse time - sample number
• Fine time
• fraction of clock period
• Algorithms:
• Leading edge discriminator = time of the first sample above threshold
• Constant fraction discriminator = time of 1st sample above fraction of max amplitude

Feature Extraction – Signal Time Extraction

Leading edge algorithm introducing walk Constant fraction algorithm

• Difference of two signals 𝐶𝑗
• Scaled samples 𝑔 ∙ 𝑡𝑗
• Delayed samples 𝑡𝑗−𝑢
• Zero crossing defines signal time
• Coarse time = i if Bi < 0 and Bi+1 > 0
• Fine time 𝑒𝑢 =

𝐶𝑗 𝐶𝑗−𝐶𝑗+1 ∙ 𝑢𝑑𝑚𝑙

Signal Time Extraction – Constant Fraction Discriminator

Digital Constant Fraction Discriminator

Additional signal compared to original:

• Delay of 2 samples
• scaled

Time measurement:

• Subtraction of delayed signal from original
• Interpolation of zero crossing

Signal Time Extraction – Center of Gravity Method

Simulated pulse with noise

Center of Gravity Definition: Results in fine time of 1/4 to 1/8: Implementation:

• Goal: Reduce effect of noise ∝ 𝑂−2
• Trade-off between precise determination and resource utilization
• Different algorithms:
• Sample with maximum amplitude
• Integral over N samples
• Finite Impulse Response filters
• FFT for noise reduction and increase of S/N

Feature Extraction – Amplitude Determination

CONTINUOUS DAQ

Triggered DAQ

Curr rrent

Digitizer FPGA Scintillation Counter 1

e‾

Scintillation Counter 2 Electromagnetic Calorimeter TRIGGER

TRIGGER – define time when detector signal to be measured Why read only when triggered data and not continuously ?

• Not feasible to measure continuous data flow
• Not feasible to transmit such amount of data
• Not feasible to store such amount of data

FPGA technology : 45 nm => 28 nm => 20 nm => 16 nm

• Higher density => more complex circuits
• Lower operational voltages and faster
• Lower power consumption

ASIC – being developed by Particle Physics Research Labs

Low power fast Sampling ADC Digitization and first data processing implemented in ASICs High speed serial interfaces integrated in ASICs Trigger less architecture

Technology give possibility to process detector information without external triggers => Trigger less DAQ

Technology Progress

Trigger-less DAQ

Take everything A time when something interesting happen is not easy to define Taking video => allows to achieve higher precision measurement Why to read triggered data :

• Not feasible to digitize continuously
• Not feasible to transmit such amount of data
• Not feasible to store such amount of data

Evolution of DAQ Architecture

Front-End Storage/CDR

PC

Event Builder

PC

HLT Storage Trigger Logic Continuous Front-End HLT HLT

PC

HLT

intelligent FPGA-based DAQ

Novel iFDAQ Architecture

Challenges of Event Building

Event Building Challenges I

Event Building Challenges II

Traffic Pattern Causes Congestion Problem

Progress of FPGA technology

 High bandwidth of serial links: 3000 Gbps per single FPGA  High bandwidth of external memory interfaces 10 GB/s per single FPGA

Motivation for iFDAQ

Chip Manufacturer Technology Transistor count

Duo-core + GPU Iris Core i7 Broadwell-U Intel 14 nm 1 900 000 000 22-core Xeon Broadwell-E5 Intel 14 nm 7 200 000 000 Virtex 7 Xilinx 28 nm 6 800 000 000 Virtex Ultra Scale Xilinx 20 nm 20 000 000 000

FPGA technology advantages

Emerging technology, rapidly extends application fields Highly parallel architecture Enormous IO bandwidth Low cost Long development time => Software tools for a moment behind complexity of HW technology

FPGA is ideal technology for development reliable, high performant, low cost DAQ system

iFDAQ (intelligent FPGA DAQ) Reliability achieved by smart recovery algorithms included in FPGA

DAQ Architectures

Continuous Front-End

Trigger Logic Storage

Event Builder

FPGA WORLD PC WORLD LHC IFDAQ Future LHC PC

Concept of iFDAQ

• Minimize amount of real-time software

processes and implement Event Builder in FPGA Advantages

• Increased compactness
• Increased reliability
• Reduced cost

iFDAQ Frame work

• Implementing congestion free Event

Builder in FPGA

• Intelligent data handling
• Unified Interfaces
• Unified IP Cores
• Integrated Digital Trigger (to be impl.)

Data Concentrators

Event Building CPU vs FPGA

CPU FPGA

Advantages:

• Uses mass-produced

components

• Easy integration of

redundancy elements

• Availability of libraries and

templates

Disadvantages:

• Throughput limited by EB

network

• Performance of sequential

execution strongly depends on algorithm complexity

• Recovery of crashed

processes takes significant time

Advantages:

• Only FPGA allows to build

real real-time system

• High scalability
• High reliability
• Low cost

Disadvantages:

• Long firmware development

progress: high level simulation tools like System Verilog and OSVVM

 Motivation

• Minimize real time SW

processes

• Development of highly

autamotized and reliable DAQ

iFDAQ Architecture

iFDAQ Architecture

Data Concentrator

Event builder in FPGA

• Events are processed simultaneously and buffered in DDR, no congestion
• Events distributed between outgoing links in round robin manner
• 2.5 GB/s throughput => 10 GB/s

TCS TCS

Compact : Before : 30 online PCs Now : one VME 6U crate + 1 rack (8 computers)

iFDAQ

Performance : Up Time in 2017

BACKUP SLIDES

Multiplexer

UCF

UCF UCF

Triggered DAQ

At certain time

– when something interesting happened

Take time measurement=> TRIGGERED DAQ Photo shooting => TRIGGERED DAQ

Electron Energy Measurement

Curr rrent

Digitizer FPGA Scintillation Counter 1

e‾

Scintillation Counter 2 Electromagnetic Calorimeter TRIGGER

How much data the system should handle ? Poisson distribution , probability mass function :

𝜇 – average number of events k - number of events occur f(x) =

𝜇𝑙𝑓−𝜇 𝑙!

Electron Energy Measurement

Curr rrent

Digitizer FPGA Scintillation Counter 1

e‾

Scintillation Counter 2 Electromagnetic Calorimeter TRIGGER

TRIGGER – defines time to store SIGNAL VALUE Data path delay should be equal to trigger delay with correction for time of flight ! Why Triggered and not continuous ?

• Feasibility to handle continuous stream
• No need to collect all data