Single-phase TPS Warm Interface Some thoughts E. Hazen, M. Johnson, - - PowerPoint PPT Presentation

2016-02-08

• E. Hazen - DUNE Warm Interface

1/18

Single-phase TPS Warm Interface Some thoughts

• E. Hazen, M. Johnson, R. Van Berg

2016-02-08

• E. Hazen - DUNE Warm Interface

2/18

Focus on...

• 1. Slow Controls
• 2. Clock and

synchronous controls

• 3. DAQ

2016-02-08

• E. Hazen - DUNE Warm Interface

3/18

Slow Controls

• Requirements (actually very few!)

– Must provide read/write access to registers – Must be reliable

• Must tolerate cold+warm ops, 25m cable, DC offsets etc

– Should be simple to implement cold+warm ends – Should be easy to integrate to custom/COTS system

• Proposed Implementation (COLDATA interface):

– LVDS electrical standard – Three LVDS pairs: CLK in, Data in, Data out – “I2C-Like” protocol, – Details t.b.d. (FNAL will provide a proposal)

• By far the easiest way to accommodate this is with an FPGA or

microcontroller on the warm interface board

– Downstream interface would be a COTS standard specified by the slow

controls group (RS-232, Modbus, CAN bus, Ethernet, whatever)

2016-02-08

• E. Hazen - DUNE Warm Interface

4/18

Clock and Controls

• Requirements (to support measurements of):

– TPC times to ¼ of ADC conversion time (125ns)

(to be better than sqrt(12) ADC sampling time resolution)

– Provide “fast” control path to TPC front-end

(ADC convert and a few other anticipated signals)

– Link to Universal Time to tag beam arrival (sub-µs)

• COLDATA interface (proposed)

– One pair with 50 MHz clock – One pair with encoded controls – Details t.b.d. (FNAL will provide a proposal)

• Upstream (distribution system, only a proposal)

– Central clock module near top of cryostat – Above ground GPS receiver with ~ 3 fibers down to cryostat – Distribution system for clock, controls cables and to photon detectors

2016-02-08

• E. Hazen - DUNE Warm Interface

5/18

Timing Considerations

• Ideally we know the time of each ADC sample

in the system to < 1 conversion period

• COLDATA requirements:

– Timestamp each data packet with 16-bit count,

incremented every (50 MHz) clock cycle

– Reset/check 16-bit count on receipt of RESET

command, if count ≠ 0xNNNN on RESET, flag error

– RESETs are counted elsewhere in clock system

and on warm interface FPGA to provide many more bits of timestamp.

2016-02-08

• E. Hazen - DUNE Warm Interface

6/18

DAQ – Cold HW Review

ADC

12 bits

16 ch ADC

12 bits

16 ch ADC

12 bits

16 ch ADC

12 bits

16 ch

COLDDATA ASIC

Two output streams 200Mb/s per ADC Two 1.2 Gb/s

• utput streams

(8b10b code) 2 MSPS

This is ½ of a Front-End Board (64 channels)

Raw sampled data volume: 16 ch * 12 bits * 4 ADC * 2 MSPS = 1536 Mbits/s ADC adds 4 bits per stream per sample; this is an additional: 8 streams * 4 bits * 2 MSPS = 64 Mbits/s 1536 + 64 = 1600 Mbits/s

2016-02-08

• E. Hazen - DUNE Warm Interface

7/18

DAQ Warm Interface

(from Cold perspective)

• Receive digitized data:

– ADC output (each 500ns):

• 16 Channels, 12 bits on two output streams

– 96 bits plus 4 header bits = 100 bits per stream

• COLDATA inputs 8 such streams, so 800 bits every 500ns

(1600 Mb/s)

– COLDATA outputs (there are two):

• 1.2 Gb/s w/ 8b10b so 1.0 Gb/s payload (500 bits per 500ns)

(Two links gives 2000 Mb/s)

• This provides 20% spare bandwidth (2000 / 1600)
• Transmit to DAQ:

– Few “requirements” here, but some thoughts:

• Avoid constraining the DAQ choice
• Allow for DAQ on the surface if that seems best

2016-02-08

• E. Hazen - DUNE Warm Interface

8/18

DAQ Interface Considerations

• 25m cables seriously

degrade the 1.2Gb/s

– At or below threshold for

LVDS / CML inputs

• Ideally one would use a low-cost FPGA to receive

and aggregate these links to e.g. 10Gb or 40Gb fibers

– Receiving with normal I/O on FPGA is cheapest

(but may not work, needs a bit of R&D)

– Receiving with high-speed SERDES (as on SBND board)

would work, but more expensive

– In either case an active equalizer circuit is likely required,

at least on the long cables

2016-02-08

• E. Hazen - DUNE Warm Interface

9/18

One Proposal

COLDDATA COLDDATA

FPGA- Based Data Merge

1.2 Gb Cu 10 GbE Fiber

DWDM Multplex

32x 10 GbE X 32 x8 Total 12k links Total 1500 links Total 50 links 10 GbE to DAQ Up shaft to surface DWDM Demux

2016-02-08

• E. Hazen - DUNE Warm Interface

10/18

FPGA Multiplex

(conceptual design)

FPGA

Kintex-7 Class

• r Altera

equivalent Optional active equalizer (long cables only)

SDRAM

Generic IO SERDES

• r high-speed SERDES

10Gb capable SERDES 10Gb optical transceiver

(DWDM channel chosen) (Entire block could be repeated using larger FPGA to minimise

• verall cost)

8-10x 1.2Gb LVDS Inputs Firmware combines 8-10 streams into a single 10Gb stream (could be std. TCP/IP – firmware already exists for Muon g-2)

(RAM required for TCP/IP buffering)

2016-02-08

• E. Hazen - DUNE Warm Interface

11/18

FPGA Multiplex

• Quite similar to SBND board and other proposals
• Benefits:

– Decouples DAQ from COLDATA protocol – Standard TCP/IP an easy option – Reduces link count by 8-10x – Can use DWDM to reduce fiber count such that entire DAQ can

be on surface

– Can include clock and slow controls in same box/board

• Possible Optimizations:

– Use 40 or 100Gb output – Two-stage using inexpensive FPGA (e.g. Arria-V)

to receive 1.2Gb, second stage to handle fast outputs

– Implement cable equalizer on plug-in board

2016-02-08

• E. Hazen - DUNE Warm Interface

12/18

DWDM

(Dense Wavelength Division Multiplexing)

• Up to 40 x 10Gb

streams on one single- mode fiber

• SFP fiber transceivers

available on 40 different

• ptical channels
• DWDM mux/demux is a

passive, unpowered unit

2016-02-08

• E. Hazen - DUNE Warm Interface

13/18

DWDM

• Benefits:

– Could transport all data for one TPC to surface on

~50 duplex fibers (simplex if not TCP/IP)

– In principle putting the DAQ on the surface would

be less expensive

• Costs:

– Must purchase (more) expensive fiber transceivers – Must purchase DWDM mux/demux units

2016-02-08

• E. Hazen - DUNE Warm Interface

14/18

COLDATA Warm I/O Summary

Three pairs CLK, Tx + Rx “I2C-Like” protocol Two Pairs CLK + encoded control Two pairs 1.2 Gb/s 8b10b out One packet per ADC conversion

2016-02-08

• E. Hazen - DUNE Warm Interface

15/18

Summary

• It seems reasonable to implement a warm

interface with one or more FPGAs near the flange with the following functions:

– Slow controls interface: “I2C-like” to std protocol – Local distribution of clock and encoded controls

with local decoding for time-stamping of data

– Receive 1.2Gb LVDS links – Aggregate N x 1.2Gb links and retransmit on fiber

at industry standard speed (10/40/100 Gb/s)

2016-02-08

• E. Hazen - DUNE Warm Interface

16/18

Backup Slides

2016-02-08

• E. Hazen - DUNE Warm Interface

17/18

Additional Concerns

(outside scope of this meeting)

• We are concerned that reliability be addressed

– For example, a COLDATA failure takes out a

significant number of U, V and Y channels

– Should we consider a different architecture with

more links? COLDATA per ADC?

• We should ensure that the COLDATA control

logic work correctly under fault conditions:

– Shorted or spuriously driven inputs

• Should we add a provision for an analog

monitor output for selected channels?

2016-02-08

• E. Hazen - DUNE Warm Interface

18/18

Clock / Fast Controls

• Requirements on clock/controls system

– Must distribute 50MHz master clock to all FEBs – Must distribute synchronous commands (precision = 1 clock tick)

• CONVERT every 500 ns
• RESET periodically
• TEST pulse
• Others..

– Must provide a monitoring scheme for distributed clock/controls

• Proposed Implementation (general features)

– Two LVDS pairs carries clock, controls – Overall DC balanced transmission – Easy / unambiguous algorithm to decode – Minimum effect on clock jitter

• Details t.b.d. but FNAL will provide a detailed proposal