The White Rabbit project an Ethernet-based solution for sub-ns - - PowerPoint PPT Presentation

▶
the white rabbit project
SMART_READER_LITE
LIVE PREVIEW

The White Rabbit project an Ethernet-based solution for sub-ns - - PowerPoint PPT Presentation

Jul 20, 2023 464 likes •1.23k views

Introduction WR Network Time Distribution Data Distribution Applications Summary The White Rabbit project an Ethernet-based solution for sub-ns synchronization and deterministic delivery Grzegorz Daniluk CERN BE-CO Hardware and Timing

slide-1
SLIDE 1

Introduction WR Network Time Distribution Data Distribution Applications Summary

The White Rabbit project

an Ethernet-based solution for sub-ns synchronization and deterministic delivery

Grzegorz Daniluk

CERN BE-CO Hardware and Timing section

30 January 2014

  • G. Daniluk

White Rabbit 1/36

slide-2
SLIDE 2

Introduction WR Network Time Distribution Data Distribution Applications Summary

Outline

1

Introduction

2

White Rabbit Network

3

Time Distribution

4

Data Distribution

5

Applications

6

Summary

  • G. Daniluk

White Rabbit 2/36

slide-3
SLIDE 3

Introduction WR Network Time Distribution Data Distribution Applications Summary

Outline

1

Introduction

2

White Rabbit Network

3

Time Distribution

4

Data Distribution

5

Applications

6

Summary

  • G. Daniluk

White Rabbit 3/36

slide-4
SLIDE 4

Introduction WR Network Time Distribution Data Distribution Applications Summary

What’s in a name ?

Oh dear! Oh dear! I shall be too late!

The White Rabbit in charge of real time

  • G. Daniluk

White Rabbit 4/36

slide-5
SLIDE 5

Introduction WR Network Time Distribution Data Distribution Applications Summary

What is White Rabbit?

Renovation of accelerator’s control and timing Based on well-known technologies Open Hardware and Open Software with commercial support International collaboration Many users: CERN, GSI, KM3NET, cosmic ray detectors, metrology labs...

  • G. Daniluk

White Rabbit 5/36

slide-6
SLIDE 6

Introduction WR Network Time Distribution Data Distribution Applications Summary

Why we use Open Hardware ?

Get a design just the way we want it Peer review Healthier relationship with companies

  • G. Daniluk

White Rabbit 6/36

slide-7
SLIDE 7

Introduction WR Network Time Distribution Data Distribution Applications Summary

White Rabbit features

Ethernet-based

thousands-nodes system tens-km span

Synchronism

sub-ns accuracy tens-ps precision

Determinism

upper-bound low-latency high reliability

  • G. Daniluk

White Rabbit 7/36

slide-8
SLIDE 8

Introduction WR Network Time Distribution Data Distribution Applications Summary

Outline

1

Introduction

2

White Rabbit Network

3

Time Distribution

4

Data Distribution

5

Applications

6

Summary

  • G. Daniluk

White Rabbit 8/36

slide-9
SLIDE 9

Introduction WR Network Time Distribution Data Distribution Applications Summary

White Rabbit Network

Standard Ethernet network Ethernet features (VLAN) & protocols (SNMP) High accuracy synchronization Reliable and low-latency Control Data

  • G. Daniluk

White Rabbit 9/36

slide-10
SLIDE 10

Introduction WR Network Time Distribution Data Distribution Applications Summary

White Rabbit Switch

Central element of WR network Designed from scratch 18 ports 1000BASE-BX10 SFPs: up to 10 km, single-mode fiber Open design (H/W and S/W)

  • G. Daniluk

White Rabbit 10/36

slide-11
SLIDE 11

Introduction WR Network Time Distribution Data Distribution Applications Summary

White Rabbit Node

Modular hardware kit: set of Mezzanine boards: ADC, DAC, TDC, Fine delay... set of carriers for various needs: PCIe, VME64x, PXIe... all carriers equipped with a White Rabbit port

  • G. Daniluk

White Rabbit 11/36

slide-12
SLIDE 12

Introduction WR Network Time Distribution Data Distribution Applications Summary

White Rabbit Node - example

  • G. Daniluk

White Rabbit 12/36

slide-13
SLIDE 13

Introduction WR Network Time Distribution Data Distribution Applications Summary

White Rabbit Node - example

  • G. Daniluk

White Rabbit 12/36

slide-14
SLIDE 14

Introduction WR Network Time Distribution Data Distribution Applications Summary

White Rabbit Node - example

  • G. Daniluk

White Rabbit 12/36

slide-15
SLIDE 15

Introduction WR Network Time Distribution Data Distribution Applications Summary

White Rabbit Node - example

  • G. Daniluk

White Rabbit 12/36

slide-16
SLIDE 16

Introduction WR Network Time Distribution Data Distribution Applications Summary

White Rabbit Node - example

  • G. Daniluk

White Rabbit 12/36

slide-17
SLIDE 17

Introduction WR Network Time Distribution Data Distribution Applications Summary

White Rabbit PTP Core

Fancy Ethernet MAC with White Rabbit support Open IP Core Easily integrated into custom FPGA-based designs

  • G. Daniluk

White Rabbit 13/36

slide-18
SLIDE 18

Introduction WR Network Time Distribution Data Distribution Applications Summary

Open Hardware Repository (OHWR)

All schematics, HDL designs and software sources available in OHWR Over 100 projects currently hosted 11 scientific institutes and 16 companies involved http://www.ohwr.org

  • G. Daniluk

White Rabbit 14/36

slide-19
SLIDE 19

Introduction WR Network Time Distribution Data Distribution Applications Summary

Outline

1

Introduction

2

White Rabbit Network

3

Time Distribution

4

Data Distribution

5

Applications

6

Summary

  • G. Daniluk

White Rabbit 15/36

slide-20
SLIDE 20

Introduction WR Network Time Distribution Data Distribution Applications Summary

Time Distribution in White Rabbit Network

Synchronization with sub-ns accuracy tens-ps precision Combination of

Precision Time Protocol (IEEE1588) synchronization Layer 1 syntonization Phase measurements

  • G. Daniluk

White Rabbit 16/36

slide-21
SLIDE 21

Introduction WR Network Time Distribution Data Distribution Applications Summary

Precision Time Protocol (IEEE1588)

Simple calculations:

link delayms: δms = (t4−t1)−(t3−t2)

2

clock offsetms = t2 − t1 + δms

  • G. Daniluk

White Rabbit 17/36

slide-22
SLIDE 22

Introduction WR Network Time Distribution Data Distribution Applications Summary

Precision Time Protocol (IEEE1588)

Simple calculations:

link delayms: δms = (t4−t1)−(t3−t2)

2

clock offsetms = t2 − t1 + δms

Disadvantages

assumes symmetry of medium all nodes have free-running oscillators frequency drift compensation vs. message exchange traffic

  • G. Daniluk

White Rabbit 17/36

slide-23
SLIDE 23

Introduction WR Network Time Distribution Data Distribution Applications Summary

Layer 1 Syntonization

  • G. Daniluk

White Rabbit 18/36

slide-24
SLIDE 24

Introduction WR Network Time Distribution Data Distribution Applications Summary

Phase measurements

Monitor phase of bounced-back clock Enhance PTP timestamps with phase measurement Phase-locked loop in the slave follows the phase changes

  • G. Daniluk

White Rabbit 19/36

slide-25
SLIDE 25

Introduction WR Network Time Distribution Data Distribution Applications Summary

WR synchronization performance

  • G. Daniluk

White Rabbit 20/36

slide-26
SLIDE 26

Introduction WR Network Time Distribution Data Distribution Applications Summary

WR synchronization performance

  • G. Daniluk

White Rabbit 20/36

slide-27
SLIDE 27

Introduction WR Network Time Distribution Data Distribution Applications Summary

WR synchronization performance

ISPCS Plug Fest WR: most accurate PTP implementation in the world!

  • G. Daniluk

White Rabbit 20/36

slide-28
SLIDE 28

Introduction WR Network Time Distribution Data Distribution Applications Summary

WR Standardization under IEEE1588

We want to standardize!

  • G. Daniluk

White Rabbit 21/36

slide-29
SLIDE 29

Introduction WR Network Time Distribution Data Distribution Applications Summary

WR Standardization under IEEE1588

We want to standardize! Intention by 1588 Standardization Group expressed in Project Authorization Request

  • G. Daniluk

White Rabbit 21/36

slide-30
SLIDE 30

Introduction WR Network Time Distribution Data Distribution Applications Summary

WR Standardization under IEEE1588

We want to standardize! Intention by 1588 Standardization Group expressed in Project Authorization Request Enhanced Accuracy Options / Profile

  • G. Daniluk

White Rabbit 21/36

slide-31
SLIDE 31

Introduction WR Network Time Distribution Data Distribution Applications Summary

Outline

1

Introduction

2

White Rabbit Network

3

Time Distribution

4

Data Distribution

5

Applications

6

Summary

  • G. Daniluk

White Rabbit 22/36

slide-32
SLIDE 32

Introduction WR Network Time Distribution Data Distribution Applications Summary

Data Distribution in a White Rabbit Network

  • G. Daniluk

White Rabbit 23/36

slide-33
SLIDE 33

Introduction WR Network Time Distribution Data Distribution Applications Summary

Deterministic data delivery

Types of data distinguished by 802.1Q tag:

Control Data (strict priority) Standard Data (Best Effort)

Control Data characteristics:

Sent by Data Master(s) Broadcast (one-to-many) Deterministic and low-latency Reliable delivery

Low-latency WR Switch by design ( < 10us )

  • G. Daniluk

White Rabbit 24/36

slide-34
SLIDE 34

Introduction WR Network Time Distribution Data Distribution Applications Summary

Data Redundancy (Node)

Forward Error Correction (FEC) – transparent layer:

One message encoded into 4 Ethernet frames Recovery of message from any 2 frames

  • G. Daniluk

White Rabbit 25/36

slide-35
SLIDE 35

Introduction WR Network Time Distribution Data Distribution Applications Summary

Data Redundancy (Node)

Forward Error Correction (FEC) – transparent layer:

One message encoded into 4 Ethernet frames Recovery of message from any 2 frames

FEC can prevent data loss due to:

  • G. Daniluk

White Rabbit 25/36

slide-36
SLIDE 36

Introduction WR Network Time Distribution Data Distribution Applications Summary

Data Redundancy (Node)

Forward Error Correction (FEC) – transparent layer:

One message encoded into 4 Ethernet frames Recovery of message from any 2 frames

FEC can prevent data loss due to:

bit errors

  • G. Daniluk

White Rabbit 25/36

slide-37
SLIDE 37

Introduction WR Network Time Distribution Data Distribution Applications Summary

Data Redundancy (Node)

Forward Error Correction (FEC) – transparent layer:

One message encoded into 4 Ethernet frames Recovery of message from any 2 frames

FEC can prevent data loss due to:

bit errors network reconfiguration

  • G. Daniluk

White Rabbit 25/36

slide-38
SLIDE 38

Introduction WR Network Time Distribution Data Distribution Applications Summary

Topology Redundancy (Switch)

Ideas:

Using VLANs H/W switch-over to the backup link WR Rapid Spanning Tree Protocol WR Shortest Path Bridging

Seamless redundancy requires Forward Error Correction

  • G. Daniluk

White Rabbit 26/36

slide-39
SLIDE 39

Introduction WR Network Time Distribution Data Distribution Applications Summary

Topology reconfiguration performance

  • G. Daniluk

White Rabbit 27/36

slide-40
SLIDE 40

Introduction WR Network Time Distribution Data Distribution Applications Summary

Outline

1

Introduction

2

White Rabbit Network

3

Time Distribution

4

Data Distribution

5

Applications

6

Summary

  • G. Daniluk

White Rabbit 28/36

slide-41
SLIDE 41

Introduction WR Network Time Distribution Data Distribution Applications Summary

Distributed oscilloscope

Common clock in the entire network: no skew between ADCs. Ability to sample with different clocks Internal time triggers or external asynchronous triggers time tagged with a TDC

  • G. Daniluk

White Rabbit 29/36

slide-42
SLIDE 42

Introduction WR Network Time Distribution Data Distribution Applications Summary

CERN Neutrinos to Gran Sasso project

Investigation of neutrino oscillation Time of Flight measurement

  • G. Daniluk

White Rabbit 30/36

slide-43
SLIDE 43

Introduction WR Network Time Distribution Data Distribution Applications Summary

CERN Neutrinos to Gran Sasso project

  • G. Daniluk

White Rabbit 31/36

slide-44
SLIDE 44

Introduction WR Network Time Distribution Data Distribution Applications Summary

CERN Neutrinos to Gran Sasso project

WR transferring UTC from GPS receiver to the measurement point 8km of fiber between WR Switches WR Switch in the cavern serves various experiments Performance monitoring Results from ∼31 days:

Accuracy: 0.517 ns Precision: 0.119ns (std. dev)

  • G. Daniluk

White Rabbit 31/36

slide-45
SLIDE 45

Introduction WR Network Time Distribution Data Distribution Applications Summary

Other WR Applications

CERN and GSI

  • G. Daniluk

White Rabbit 32/36

slide-46
SLIDE 46

Introduction WR Network Time Distribution Data Distribution Applications Summary

Other WR Applications

CERN and GSI HiSCORE: Gamma&Cosmic-Ray experiment

  • G. Daniluk

White Rabbit 32/36

slide-47
SLIDE 47

Introduction WR Network Time Distribution Data Distribution Applications Summary

Other WR Applications

CERN and GSI HiSCORE: Gamma&Cosmic-Ray experiment The Large High Altitude Air Shower Observatory

  • G. Daniluk

White Rabbit 32/36

slide-48
SLIDE 48

Introduction WR Network Time Distribution Data Distribution Applications Summary

Other WR Applications

CERN and GSI HiSCORE: Gamma&Cosmic-Ray experiment The Large High Altitude Air Shower Observatory MIKES: Centre for metrology and accreditation

  • G. Daniluk

White Rabbit 32/36

slide-49
SLIDE 49

Introduction WR Network Time Distribution Data Distribution Applications Summary

Other WR Applications

CERN and GSI HiSCORE: Gamma&Cosmic-Ray experiment The Large High Altitude Air Shower Observatory MIKES: Centre for metrology and accreditation KM3NET: European deep-sea research infrastructure

Full list of WR users: http://www.ohwr.org/projects/white-rabbit/wiki/WRUsers

  • G. Daniluk

White Rabbit 32/36

slide-50
SLIDE 50

Introduction WR Network Time Distribution Data Distribution Applications Summary

Outline

1

Introduction

2

White Rabbit Network

3

Time Distribution

4

Data Distribution

5

Applications

6

Summary

  • G. Daniluk

White Rabbit 33/36

slide-51
SLIDE 51

Introduction WR Network Time Distribution Data Distribution Applications Summary

White Rabbit Family

Successful international collaboration of institutes, universities and companies WR Users: http://www.ohwr.org/projects/white-rabbit/wiki/WRUsers

  • G. Daniluk

White Rabbit 34/36

slide-52
SLIDE 52

Introduction WR Network Time Distribution Data Distribution Applications Summary

White Rabbit Family

Successful international collaboration of institutes, universities and companies WR Users: http://www.ohwr.org/projects/white-rabbit/wiki/WRUsers

  • G. Daniluk

White Rabbit 34/36

slide-53
SLIDE 53

Introduction WR Network Time Distribution Data Distribution Applications Summary

Pushing frontiers

Scientific, open (H/W & S/W), with commercial support

  • G. Daniluk

White Rabbit 35/36

slide-54
SLIDE 54

Introduction WR Network Time Distribution Data Distribution Applications Summary

Pushing frontiers

Scientific, open (H/W & S/W), with commercial support More applications than ever expected

  • G. Daniluk

White Rabbit 35/36

slide-55
SLIDE 55

Introduction WR Network Time Distribution Data Distribution Applications Summary

Pushing frontiers

Scientific, open (H/W & S/W), with commercial support More applications than ever expected A versatile solution for general control and data acquisition

  • G. Daniluk

White Rabbit 35/36

slide-56
SLIDE 56

Introduction WR Network Time Distribution Data Distribution Applications Summary

Pushing frontiers

Scientific, open (H/W & S/W), with commercial support More applications than ever expected A versatile solution for general control and data acquisition Fulfilling all our needs in synchronization and determinism

  • G. Daniluk

White Rabbit 35/36

slide-57
SLIDE 57

Introduction WR Network Time Distribution Data Distribution Applications Summary

Pushing frontiers

Scientific, open (H/W & S/W), with commercial support More applications than ever expected A versatile solution for general control and data acquisition Fulfilling all our needs in synchronization and determinism Standard-compatible and standard-extending

  • G. Daniluk

White Rabbit 35/36

slide-58
SLIDE 58

Introduction WR Network Time Distribution Data Distribution Applications Summary

Pushing frontiers

Scientific, open (H/W & S/W), with commercial support More applications than ever expected A versatile solution for general control and data acquisition Fulfilling all our needs in synchronization and determinism Standard-compatible and standard-extending Active participation in IEEE1588 revision process

  • G. Daniluk

White Rabbit 35/36

slide-59
SLIDE 59

Introduction WR Network Time Distribution Data Distribution Applications Summary

Pushing frontiers

Scientific, open (H/W & S/W), with commercial support More applications than ever expected A versatile solution for general control and data acquisition Fulfilling all our needs in synchronization and determinism Standard-compatible and standard-extending Active participation in IEEE1588 revision process

  • G. Daniluk

White Rabbit 35/36

slide-60
SLIDE 60

Introduction WR Network Time Distribution Data Distribution Applications Summary

Thank you

More information: http://www.ohwr.org/projects/white-rabbit/wiki

  • G. Daniluk

White Rabbit 36/36

slide-61
SLIDE 61

Even more stuff...

Link Delay Model

  • G. Daniluk

White Rabbit 37/36

slide-62
SLIDE 62

Even more stuff...

Link Delay Model

  • G. Daniluk

White Rabbit 37/36

slide-63
SLIDE 63

Even more stuff...

Link Delay Model

delayms = ∆txm + δms + ∆rxs delaysm = ∆txs + δsm + ∆rxm

master fixed delays ∆txm ∆rxm δsm δms variable delays fixed delays ∆rxs ∆txs slave t1 t4 t3 t2

  • G. Daniluk

White Rabbit 38/36

slide-64
SLIDE 64

Even more stuff...

Link Delay Model

delayms = ∆txm + δms + ∆rxs delaysm = ∆txs + δsm + ∆rxm

master fixed delays ∆txm ∆rxm δsm δms variable delays fixed delays ∆rxs ∆txs slave t1 t4 t3 t2

Relative Delay Coefficient (α) for 1000base-X over a Single-mode Optical Fibre δms = (1 + α) δsm

  • G. Daniluk

White Rabbit 38/36

slide-65
SLIDE 65

Even more stuff...

Link Delay Model

delayms = ∆txm + δms + ∆rxs delaysm = ∆txs + δsm + ∆rxm

master fixed delays ∆txm ∆rxm δsm δms variable delays fixed delays ∆rxs ∆txs slave t1 t4 t3 t2

Measuring fixed delays is hard but we use mathematical tricks for that - WR Calibration procedure (http://www.ohwr.org/documents/213)

  • G. Daniluk

White Rabbit 38/36

slide-66
SLIDE 66

Even more stuff...

White Rabbit extension to PTP

White Rabbit requires:

WR-specific states Exchange of WR-specific information asymmetry estimation based on Link Delay Model

WR PTP

PTP extensions mechanisms Enhanced precision t1, t2, t3, t4 Correction for asymmetry Interoperability with PTP gear

  • G. Daniluk

White Rabbit 39/36

slide-67
SLIDE 67

Even more stuff...

White Rabbit Network

White Rabbit Switch White Rabbit Node (White Rabbit PTP Core)

  • G. Daniluk

White Rabbit 40/36

slide-68
SLIDE 68

Even more stuff...

White Rabbit Switch

Functionality of a professional Gigabit Ethernet Switch with White Rabbit extensions 3 layers of design:

  • G. Daniluk

White Rabbit 41/36

slide-69
SLIDE 69

Even more stuff...

WR Switch: hardware

Xilinx Virtex 6, Atmel AT91SAM9G45 18 cages for Gigabit SFPs, 10/100 Ethernet management port 5 SMC connectors (1-PPS in/out, CLK in/out) designed and produced by Seven Solutions in cooperation with CERN

  • G. Daniluk

White Rabbit 42/36

slide-70
SLIDE 70

Even more stuff...

WR Switch: gateware

Implemented in Xilinx Virtex6 FPGA:

  • G. Daniluk

White Rabbit 43/36

slide-71
SLIDE 71

Even more stuff...

WR Switch: software

Running on ARM processor: Embedded Linux kernel 2.6.39 with patches and modules for HDL components Hardware Abstraction Layer RTU daemon PTP daemon with WR extension CLI and SNMP support coming

  • G. Daniluk

White Rabbit 44/36

slide-72
SLIDE 72

Even more stuff...

White Rabbit Node - WR PTP Core

HDL IP-Core developed on Xilinx Spartan 6 but not tied to Xilinx it is a fancy Ethernet MAC interfaces user-defined module sending/receiving Ethernet frames with PHY layer provides precise timing by implementing WR protocol ready to be integrated in user’s devices requires only two tunable oscillators and EEPROM to store the configuration

  • G. Daniluk

White Rabbit 45/36

slide-73
SLIDE 73

Even more stuff...

WR PTP Core: interfaces

clocks and reset frame interface (WR Fabric) timecode and 1-PPS output PHY interface (GTP/GTX tested and supported) I2C, 1-Wire, UART, GPIO

  • G. Daniluk

White Rabbit 46/36

slide-74
SLIDE 74

Even more stuff...

WR PTP Core: HDL design

  • G. Daniluk

White Rabbit 47/36

slide-75
SLIDE 75

Even more stuff...

WR PTP Core: HDL design

  • G. Daniluk

White Rabbit 48/36