Optical Links Ba Babak Abi DAQ FD D "D "Des esig - - PowerPoint PPT Presentation

Networking 7.2.6 Optical Links

Ba Babak Abi

DAQ FD D "D "Des esig ign" " Wor

• rkshop 2

26 Ja Jan 2018

1

Outline:

• Data link Components and suggestion for current standard
• Costs per link and Gb/s
• Rough Estimation of number fibres needed

1 2 3 4 Timing interface? Trade of between fiber cost and Transceiver cost?

• In case of 10G links RCE-FELIX need ~3000 TR module!
• 1500 SNAP to CE

Data Links Options

• Main questions :

1. What are the data links current and future options?

• Link speed, form factor power consumption

2. Long-haul and short-haul links from the WIB to the DAQ

1. Technologies with coper for 30m till 40Gb 2. Long-haul 200m > should be optical links

3. Cost and trade-offs of each

• BUT Networking and high-speed solution and components have a very broad phase space since

we are on the verge of a technological change in the landscape of data links.

• The IEEE is proposing more Ethernet standards from 2015 to 2020 (five years) than its released

in the last 40. There is even realistic talk of 10 TbE towards the end of the next decade.

• Should we focus our designs on current or future standard ?
• We need to review future standard since the hardware system design level would be effected if want to use future

standard

• We present the suggestion base on current standard

Ethernet Alliance Technology Roadmap 2014 paper

3

Optical MUX

• Optical MUX
• Simplex MUX, needs two fibre, cheaper
• Duplex MUX, needs ONE fibre, expensive
• CWDM or DWDM?
• Dense wavelength division multiplexing (DWDM) up to 80

(theoretically) channels can be multiplexed into on a single optical fiber.

• CWDM (Coarse Wavelength Division Multiplexing) Up to 18 channel

spacing 20 nm, 1270 nm to 1610 nm, Cost-effective

• If we assume we fix design on SFP+ (10G)
• DWDM 40 links per fiber (400Gb) and CWDM 18 links per fiber

(180Gb)

• There are hacks for increase it like using qsfp+ in 1310nm line
• Limitation on fibres we have only 96

DWDM 38Tb/s and CWDM 19Tb/s

4

Simplex MUX Duplex MUX

CWDM and DWDM module

• FMU 1U Rack Mount, LC/UPC, with Monitor Port, 3.0 to 6dB Typical IL
• Could add a 22dBm Output Booster DWDM EDFA C-band 24dB Gain for long-haul
• Price:
• Retail price $1700 for 40 channels DWDM (4 APA if we use 10x10GbE )
• Retail price $710 for 18 channels CWDM ( almost 2 APA if we use 10x10GbE )
• BUT in mass order or FOB can be much cheaper
• Mux muddles are fairly cheap but SFP module
• CWDM SFP+ starts from $80
• DWDM SFP+ starts from $270
• But prices are falling down , need extrapolation for 2024?
• Current cost for MUX end to end included TRX and MUX module but fibre :
• DWDM ~$55 + 10 per Gb
• CWDM ~$15 + 10 per Gb
• DWDM 400Gb per fibre and CWDM 180Gb per fibre!
• Fibre selection: Zero-water-peak fibres might be needed (backup slide)
• Further options& considerations ?
• BiDi SFP+ not widely used + we do not need downstream fibre??
• Enterprise Ethernet network switch !

(32*100GbE) 100GbE Spine/Core Layer for $8000

5

40km 10km

Estimation of number fibres needed

• Number of fibres needed from RCEs for

three data stream: 1) lossless data 2) lossy data (removed for now) 3) trigger primitives

• stream. I am also assuming:

1- WDM is only supported on 10Gb/s FSP+ for

• now. No WDM for 40/100Gb/s.

2- One-sided traffic ratio less than 1:10 3- One COB per APA, No Compression, No link sharing between COBS. 4- in PD since the links are 1Gbs we have to use the 10Gbs switch to reduce numbers of fibres and I used a factor 3 overhead for congestion risk!

6

1-TPC Lossless 4-Only TPC Total DATA stream from RCE to surf/FELIX Total single fiber RCE output links APA 10Kt totalDUNE RCE output links APA 10Kt totalDUNE 10Gbs links 8 1200 4800 10Gbs links 11 1650 6600 10Gbs links CWDM 0.44444 66.66667 266.666667 10Gbs links CWDM 0.611111 91.66667 366.66667 10Gbs links DWDM 0.2 30 120 10Gbs links DWDM 0.275 41.25 165 40Gbs links 2 300 1200 40Gbs links 3 450 1800 100Gbs links 1 150 600 100Gbs links 2 300 1200 2-DATA stream from surf/FELIX to RCE RCE output links APA 10Kt totalDUNE 10Gbs links 1 150 600 6- TPC + PD total 10Gbs links CWDM 0.05556 8.333333 33.3333333 Total single fiber 10Gbs links DWDM 0.025 3.75 15 RCE output links APA 10Kt totalDUNE 40Gbs links 1 150 600 10Gbs links 19 2850 11400 100Gbs links 1 150 600 10Gbs links CWDM 0.744444 111.6667 446.66667 10Gbs links DWDM 0.335 50.25 201 3- Trigger Primitives ;bidirectional 40Gbs links 3.075 461.25 1845 RCE output links APA 10Kt totalDUNE 100Gbs links 2.03 304.5 1218 10Gbs links 2 300 1200 10Gbs links CWDM 0.11111 16.66667 66.6666667 10Gbs links DWDM 0.05 7.5 30 40Gbs links ? 0 ? 100Gbs links ? 0 ?

General conclusion

• How much cost of different schemes (price Jan/2018)
• DWDM $65 per Gb
• CWDM $25 per Gb
• DWDM 400Gb per fibre and CWDM 180Gb per fibre!
• 100Gb $ 10-15 per Gb
• QSFP+ , CFP/CFP2/CFP4/CXP
• What is the recommended rate :
• 10G, links if fiber is more expensive then Transceiver modules
• 100Gb if We can get the signal integrities right!
• We need at least 200/450 fibres DWDM/CWDM for TPC+PD
• This is the worst case scenario, RCE in service cavern and FELIX is on Surface
• If FELIX is in service room then triggered DATA needs even less fibres
• So those number seems an upper limit of estimation
• How much fibre cost?

7

Backup slides

8

water-peak fi fibres

9

Custom-made MTP(MPO) Module

• Parallel optical link module to reduce the footprint of components to

CE’s interface.

• It is possible to make them in house, The knowledge of How-to exist

already in wide range of experiments. (in case if they way expensive

• r difficult to find in future)

10 Axel Roggenbuck ATLAS-Pixeldetektor Diplomarbeit, Universit¨at Siegen,

Link roadmap; speed

• TbE (Terabit Ethernet) vs. GbE (Gigabit Ethernet)
• GbE link speeds to 100Gb and TbE links speeds from 100Gb
• If we want to use TbE we need to design hardware now
• CWDM and DWDM dominant transceivers are SFP and SFP+

EthernetAlliance_Roadmap_whitepaper_FINAL-032015-21 11