via Dynamic Multi-Path Routing in a Full Mesh Network European - - PowerPoint PPT Presentation

Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP2375650 and EP2634981 1

Based on the patents: EP2375650: Apparatus and method provisioning synchronous locked cell transfers in a network without central control EP2634981: Cell based data transfer with dynamic multi-path routing in a full mesh network without central control

István Vadász

Dipl.-Informatiker (Univ.) Independent Embedded Computing Expert

Curd-Juergens-Str. 4 81739 Muenchen, Germany  +49-89-6990446  istvan@vdsz.com  http://www.vdsz.com

Presentation at the VSO meeting in Portsmouth, NH 23/09/2015

Ultimate System Bandwidth via Dynamic Multi-Path Routing in a Full Mesh Network without Central Control

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 2

 Background

 Personal  Technical  The idea was born  Patents

 The substance of the inventions

 Decentral self-synchronizing deterministic networking  Dynamic multi-path routing over a full mesh backplane

 Utilization

 Topologies  Options

 Perspectives

 Demonstration, prototyping  FPGA, Board design, Backplane design  Application areas per topology  Cooperation, Licensing, etc.  Disclaimers

Thanks for the Invitation - Welcome to the Presentation

21 September 2015

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 3

 István Vadász

 Hungarian, living in Germany  Dipl.-Informatiker (TU Muenchen)  Worked 27 years at the same site

 Force Computers  Force Computers – Solectron Subsidiary  Motorola Embedded Communications Computing  Emerson Network Power – Embedded Computing

 Positions: Hardware design, ASIC/FPGA design, Chief Architect  Several years representing the company at

 VITA – VSO: VME64 Extentions, Live Insertion, 2eSST, Processor PMC, etc.  IEEE – BASC: IEEE 1101.10, IEEE 1101.11, Futurebus+  PICMG: CompactPCI Hot Swap, PICMG 2.16, AdvancedTCA, AdvancedMC, µTCA

 Independent Embedded Computing Expert since 2010

 System/Board review of schematics and layout  Industrial computing design advisor  Inventor of synchronous networking without central control

 Patents granted in 2013 and 2015 by the European Patent Office

The Presenter

21 September 2015

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 4

 Bus – Address/Data cycles over 8 / 16 / 32 / 64 parallel connections – optimally

controlled by source synchronous dual edge strobes

 Switched – Networks with Star or Dual Star topology over a Backplane

 Differential pair routing, up to 4 Lanes per Channel, separate Paths per direction  One or two Slots reserved for central control (Switching, clocking)  Subsidiary specifications support packet protocols like Ethernet, InfiniBand, Fibre Channel, SRIO  Support for PCI Express

 Full Mesh topology – Backplane routing specified

 Routing is a superset of the Dual Star routing  No subsidiary specifications developed to exploit the excessive bandwidth of the Full Mesh Links  Specific implementations use Ethernet switches on all inserted Boards and execute direct transmission Board-to-Board without needing a central switch resource  Backplanes are expensive without providing the bandwidth benefit lacking a standard

 Protocols

 All considered architectures are Packet based

 Any exceptions?

Technical Background System Architectures (VITA/PICMG)

21 September 2015

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 5

 I participated in the AdvancedTCA specification development, incl. the introduction of the Full Mesh routing topology  I understood the advantages and disadvantages encountered by redundant clocking and switching in the Dual Star topology  I was involved in a cell based customer project – certainly centrally controlled...  I started to systhematically look for a way to exploit the bandwidth potential of the Full Mesh network without central clocking and switching

 I came to the principle that the Full Mesh Backplane connection of a Board has to be considered as

• ne conduit, i.e. the additional solution shall be invisible for upper layer protocols and also avoid

complexity by „intelligent“ additions. Boards are now Nodes of this network.  Inherently predictable conditions shall enable a deterministic solution for the data routing  A Cell level substructure is the right choice for predictable conditions as opposed to packets  Cells shall be aligned over the network, without dropping data (locked)  It is mandatory to achieve the Cell Alignment without central control  The Nodes shall be able to dynamically assign Paths per Cell Period – without central control  The Nodes shall be able to forward each other‘s data  Dissemination of request and capability information to all other Nodes  A replicated arbitration produces the Path assignments for the next Cell Period in each Node

 I found viable technical solutions for each of the elementary ideas

The Idea Was Born

21 September 2015

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 6

 Physical: high speed differential (SERDES) interface  Topology: Full Mesh Backplane

 A typical application uses 16 boards e.g. in an ATCA or VITA-46 VPX system

 Determinism without central control: Cell level synchronicity

 Achieved by insertion and deletion of timing compensation Symbols as computed from measured and communicated Cell offset data (according to patent)

 Routing: Dynamic Multi-Path Routing controlled by replicated arbitration

 Nodes are built to support payload forwarding, i.e. each Node is able to provide payload forwarding supporting the data traffic of the other Nodes (switching)  Nodes disseminate their payload transportation capability and payload transportation request information to all other Nodes in each Cell period (in predefined Symbol positions of the Cell)  Nodes execute identical replicated arbitration for each Cell period

 Paths are assigned as well as forwarding buffers in the Nodes  Arbitration results are extracted from the local arbitration within each Node independently

 A simultaneous data transmission of payload segments takes place over multiple Paths between sources and destinations. A convention is applied for payload data segmentation and serial reassembly.

 Fault tolerant:

 Can be enhanced to tolerate single point failures

High Performance Backplane Communication Principles

22 September 2015

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 7

 The invention was understood as a single item, therefore I formulated a single

patent application. The patent offices enforced a separation into two applications,

• ne for the synchronization and another one for the dynamic multi-path routing

 European patents:

 EP2375650: Apparatus and method provisioning synchronous locked cell transfer in a network without central control (Germany, UK, France, Sweden)  EP2634981: Cell based data transfer with dynamic multi-path routing in a full mesh network without central control

 Japan:

 JP5466788: (Synchronous locked cell transfer) - Patent  2013-255681: (Dynamic multi-path routing) – Grant decision received

 Taiwan:

 I478535: A single document covers all - Patent

 US:

 US 13/639,911: Application (Synchronous locked cell transfer) – “Allowed for issuance as a patent”

 Canada:

 CA2790945: Application (Synchronous locked cell transfer)  CA2856660: Application (Dynamic multi-path routing)

Patents

21 September 2015

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 8 21 September 2015

Full Duplex Links - Paths

APP_A APP_B APP_C GA = 3 GA = 2 GA = 1

BA CB AB BC AC CA

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 9 21 September 2015

Links and Paths Links comprise two directed Paths

This is a logical aspect Physical appearance can be

 The same cable or cable pair used for both directions, e.g.

• Telephone connections
• DSL, ADSL
• 1000BASE-T Ethernet

 Separate directed cables, cable pairs, or fibers e.g.

• RS-232
• 10BASE-T and 100BASE-TX Ethernet
• 1000BASE-KX Ethernet
• InfiniBand
• PCI-Express
• SRIO
• SATA

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 10 21 September 2015

Cell vs. Packet - Transfer of Symbols

Cell transmission scheme:

N N + 1 N + 2

Packet transmission scheme:

Header Trailer Payload

• Synch. only or empty

Fixed length Cell start In-band or out-band structuring

Atomic components: Symbols

Variable length

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 11 21 September 2015

Elements of a Cell

Cell start 1

S1

2 3 4 5 6 7 n-1 n n-2

S2 S3 S4 S5 S6 S7

A Symbol First Symbol Last Symbol Symbol position Cell length: n Symbols Cell Period Symbol Period

Sn Sn-1 Sn-2

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 12 21 September 2015

Remarks to Timing Control of Transmissions

Source synchronous clocking

 Source synchronous timing (clocking) of data streams increases the maximum bandwidth for bus architectures  Embedded clocking of e.g. 8b/10b encoded data streams over Node-To- Node Links further increases the transfer rate

Independency of data transmission timing per direction

 There is usually no timing coordination between the transmissions on the directed Paths of a Link – the data streams are independent  There are several reasons for keeping the clocking of individual Nodes independent

 Data transmissions are done by each Node at a rate derived from a respective local clock

 An exception to this is represented by the telco transmission systems which rely on highly precise clock rates (not locked though), but even in this case the clocks for the two directed paths of a link may be different.  Another type of exception: In the case of a Physical Layer which uses the same physical cables for both transfer directions simultaneously, echo cancellation electronics needs to be implemented. The idea of echo cancellation requires that the physical signaling in both directions is done synchronous at the elementary signaling level. Therefore 1000BASE-T PHYs operate as a clocking Master/Slave pair at both ends of a Link, - the Slave PHY uses the recovered clock of the received signal stream for the transmission toward the Master.

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 13 21 September 2015

Deterministic Conditions

 Centrally clocked architectures are known for deterministic behavior  Some centrally clocked architectures are Cell based – and deterministic  Packet based networking has enabled communication beyond the system

• boundaries. This is per definition non-deterministic.

 Packet based networking found its way into system internal communication – despite being non-deterministic.  Architectures without central control used to be per definition non-deterministic

 With significant software effort it is possible to enforce determinism at a higher layer

 Deterministic conditions for serial linked interconnects are introduced here as a native feature of the network

 The two directed Paths of a Link are brought into a timing relation via the Alignment of the Cells transmitted in the two directions of the Link

 It is not binding to assign all Symbol positions of the Cells for the deterministic

• communication. It is absolutely possible to assign a subset of the Symbols for IP

communication – the transmissions over these Signal positions automatically lose the deterministic feature.

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 14 21 September 2015

Signaling Delay Conventions

 For the proper control of the Cell timing signaling delay conventions are settled as follows

 Signaling delays for the two Paths of a Link are nominally equal  Signaling delays for multiple Lanes of a Path are nominally equal  A maximum value for the signaling delay of a Link is specified  Signaling delays are nominally constant (might vary at a low rate)  Signaling delays of different Links may differ between 0 and a specified maximum  The signaling delay of the Links are not defined explicitly toward the connected Nodes

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 15 22 September 2015

Cell Alignment Requirements

 The following conditions shall be met for Cell Alignment:

 A Node uses identical Cell transmission timing to all Links  All Cells received by a Node arrive starting within a predefined number of Symbol periods before or after the transmission of the Node’s own Cell Start Symbol

 The offset between Symbols received in identical positions in the Cell over different Links must be within a predefined limit

 The offset of a received Symbol in a Cell to the same Symbol position in the transmitted Cell must not exceed a predefined number of Symbol periods

Cell Start chronology

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 16 21 September 2015

Cell Alignment Enables Deterministic Evaluation  The Cell Alignment grants:

Coherent Cell Periods across the Network Status of resources can be identified per Cell Period Requests for resources can be issued per Cell Period Resources can be granted per Cell Period Received Symbols can be retransmitted at higher numbered Symbol positions during the Cell Period of the

• riginal transmission

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 17 21 September 2015

A Method for Cell Synchronization in the Network

 A Node uses identical Cell transmission timing to all Links  The offset of a received Cell in relation to a transmitted Cell is measured

 The trivial unit for this measurement is the Symbol period

 The measured offset values include the signaling delays which are not explicitly known

 With high bit rates (e.g. 10Gb/s) the signaling delay may amount to several Symbol periods

 The key idea for enabling Cell Start to Cell Start OFFSET determination is the dissemination of offset measurement data

 The inaccuracy encountered due to the different local clocks is negligible  It is sufficient to transmit the measured offset for a Link only to the Node connected via the respective Link itself  The offset measurement data is transmitted via a predefined Symbol position in the Cell

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 18 21 September 2015

Cell Start Timing OFFSET Calculation

Symbol period

Tx

APP_A

Rx AB BA Rx

APP_B

Tx DAB DBA

MAB MBA

XT

XR

OFFSET : MBA MAB 2

C O M C O M

APP_B APP_B  APP_A APP_A  APP_B 

C O M C O M

 APP_B APP_A APP_A

 

G R E N E G R E N E B L E U B L E U

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 19 21 September 2015

How to Control the OFFSET ?  Control the Symbol rate ?

Principally possible but very accurate PLL circuitry needed

 Add Symbols for timing adjustments

A minimum and maximum number to be specified

• Normally the range [0..2] is adequate

The midpoint is the default number for the Adjustment Symbols A Node applies the same number of Adjustment Symbols to all connected Links Simple algorithm needed for the calculation of the number

• Certainly without central control

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 20 21 September 2015

Initial Synchronization - Coarse Alignment

 A certain level of synchronicity can be achieved when measuring the Cell start timing of the connected Nodes  Initially the received Cell starts may be distributed through the complete Cell Period

 A prioritization based machinery may be used

 E.g. the geographical address may be used for a priority structure. For this option it is required that each Node disseminates its geographical address

 Coarse Alignment is performed until the misalignment has decreased below a specified threshold

 Coarse Alignment must ensure that all possible live-lock situations are securely resolved  Coarse Alignment must ensure that the procedure does not extend beyond a reasonable time period even if the number of the Nodes is at the specified maximum  The Coarse Alignment procedure must not rely on the natural diversity of the Symbol periods  It is a good idea to generate Adjustment Symbols within the otherwise specified range

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 21 21 September 2015

Cell Synchronization - Fine Alignment

 When the misalignment is below a certain threshold offset measurement information can be exchanged between the connected Nodes

 The Fine Alignment procedure can start  During Fine Alignment the measured offset value is transmitted via each Link by both of the connected Nodes using a specified Symbol position and defined encoding  The respective OFFSET of the Cell starts is calculated as the difference of the measured and the received offset values

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 22 21 September 2015

Cell Transmission-Reception Timing Adjustments

Symbol period

Tx

APP_A

Rx AB BA Rx

APP_B

Tx DAB DBA

MAB MBA

XT

XR

OFFSET : MBA MAB 2

C O M S K P S K P S K P S K P C O M

APP_B APP_B  APP_A APP_A 

S K P

APP_B 

C O M S K P S K P S K P C O M

 APP_B APP_A APP_A

S K P S K P

 

G R E N E G R E N E B L E U B L E U

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 23 21 September 2015

Calculations for Each Link

 The OFFSET is to be determined for each Link  The list of the calculated OFFSETs allows to build a chronology

• f the Cell Starts for each Node

 The Node itself adds its own OFFSET value to the list as zero

 The minimum and the maximum OFFSET value are determined  The midpoint between the maximum and the minimum OFFSET value is the target for the timing of the next Cell Start

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 24 21 September 2015

Chronology Analysis (example)

1 2 3 4 5

• 5
• 4
• 3
• 2
• 1

The Node itself has per definition 0 OFFSET first last Correction target

1 7 2 4 5 9 3 8 10 9 6

Correction target : first + last 2

Different Nodes may have slightly different views of this chronology

Nodes identified by GA:

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 25 21 September 2015

Gravitate Toward the Midpoint of the Range

 The number of the applied Adjustment Symbols (SKP) has to gravitate toward the midpoint of the specified range (of the Adjustment

Symbols)

 This grants room for correction of the Cell Start in both directions  This is absolutely necessary to avoid a growth beyond the limit

 The number of Adjustment Symbols on each Link is analyzed

 The maximum and the minimum numbers are determined  The midpoint is determined  If the midpoint deviates from the midpoint of the specified range for the Adjustment Symbols then a correction is applied to the target point

• This equates a collective correction of the Cell Start timing by all Nodes
• Note: this calculation provides an identical result for each Node

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 26 21 September 2015

Practical Aspects

 How many Adjustment Symbols are required?

 Resonators with 50 ppm tolerance are readily available at no price premium  The worst case tolerance calculates as a deviation of 1 Symbol period per 10 000 Symbols – which will be normally less than 1 Symbol period of deviation per Cell Period  Therefore: corrections are needed regularly but not always  Preferred range for the number of Adjustment Symbols:

Minimum: 0 Default: 1 Maximum: 2

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 27 21 September 2015

Independent Clocking of the Nodes Clock source

Local clock sources

Timing base for predefined nominal Symbol rate Specified tolerance e.g. 50 ppm Each Node generates an independently calculated number of Adjustment Symbols between the Cells Through cooperative compensation activity of the Nodes achieve and sustain the Cell Alignment

High precision external source may be used

 It is allowed that exactly one Node does not cooperate but uses a “reference clock” and always inserts the predefined default number of Adjustment Symbols

E.g. Telco signal is used as a clock source

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 28 21 September 2015

Status Sequence

 A cooperative behavior of all Nodes is beneficial through all status steps to establish the Cell Alignment status for the Network  Insertion and extraction as well as the power up timing of a Node are unpredictable for the other Nodes  A status sequence is proposed to limit the multitude of possibilities

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 29 21 September 2015

Status Dissemination

 Nodes are disseminating their status for each of their Links via predefined Symbol positions of the transmitted Cells  When the Cell Alignment is better than a specified threshold, the Fine Alignment status is entered and the Nodes are transmitting the measured offset information for each Fine Aligned Link appropriately  When the Cell Alignment is better than a specified threshold, the Link enters the Locked Status

 Under healthy conditions the Locked Status is sustained forever

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 30 21 September 2015

Status Sequence Diagram

S0 S0 No activity S1 S1 All transmitters start signaling of presence S2 Wait period

>1

S5 S5 All other Links locked ?

YES

S6 S6 Start Cell transmission S8 S8 Fine Alignment S9 S9 Locked Status S3 S3 Check number of detected transmitters S7 S7 Coarse Alignment

=0

S4 S4 Check number of inputs receiving Cells

=1 <2 >1 NO

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 31 21 September 2015

Multiple Lanes per Path

 A Cell Locked Network may include Paths which consist of several Lanes

 The Cell Start Symbol must be used on each Lane simultaneously  Adjustment Symbols must be used on each Lane identically

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 32 21 September 2015

Options (subset only)

 Bit rate

 Around 30 Gb/s is the top bit rate available today using FPGA with LVDS channels  1 Gb/s is an option for implementations using 1000BASE-T PHYs

 Physical interface

 LVDS, 1000BASE-T are readily available  Fiber optical interfaces are also possible

 Cell length

 The range of 1000 to 4000 Symbols looks like the optimum area

 Interconnect length

 A limitation arises from the necessity to maintain exchanges within a Cell Period  In the case of a Chain/Ring the sum of all wire delays and internal forwarding delays of all Nodes shall be less than the Cell Period

 Lanes per Link

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 33 21 September 2015

Predictable Communication in the Full Mesh

 Cell Alignment over the network

 System level locked Cell Period  Deterministic conditions

 Cell Period is the basis for the Dynamic Multi-Path Routing

 Dissemination of requests and capabilities  Arbitration  Assignment of Path usage  Assignment of forwarding resources in the Nodes  Transmission of Payload

 Cells used as the unit of granularity for Payload

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 34 21 September 2015

How to Exploit the Potential of Full Mesh Use direct Path to the target Node

 Local switch distributes transmissions to Paths  Better than the transmission via a central switch  Most Paths remain idle most of the time

• Most applications do not rely on intensive transmissions from

each Node to each other Node

• Excess bandwidth available

Use other Nodes as switches

 Not worse than when a central switch is used  The appropriate assistant Node (switch) has

• Idle input Path from the source Node
• Idle output Path to the target Node

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 35 21 September 2015

Concepts for Traffic Control in the Full Mesh Network

 Idea of a cable connection from the viewpoint of the connected devices No acknowledge No retry No checking No history No flow control Addition and removal of devices in a live network Structure transparent to software Available for whichever higher level protocol

• Exception: A transmission is carried out only if a route is available

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 36 21 September 2015

Transmission Requests

 A Node asserts pending transmission requests to several target Nodes simultaneously  The requests have to specify

 The target Nodes  The amount – the number of Cell Payloads for each of the target Nodes

 The list of the transmission requests is disseminated to all Nodes identically

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 37 21 September 2015

Receiver Capabilities

 Each Node declares the availability of its Link inputs for the subsequent Cell Period

 The availability may be the number of the Lanes in the case of multiple Lanes per Link

 The list of the available Links is disseminated to all Nodes

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 38 21 September 2015

Replicated Arbitration

 Each Node performs the complete arbitration for the data transmission to be accomplished during the subsequent Cell Period

 This is possible since the complete set of the request and capability information is available in each Node  The results are not communicated since the arbiter in each Node produces all results

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 39 21 September 2015

Payload Forwarding Alternatives

 Within the same Cell Period (preferred by the inventor)

 No long-term resource handling  A portion of the Cell cannot be used for Payload  No Payload storage required  Disadvantage with short Cells

 In the subsequent Cell Period

 More complexity for the arbitration  A Cell worth of Payload need to be stored for forwarding  Possible collision of interest for the assignment of the direct connection  Maximal usage of Cell contents for the Payload  Easier implementation for short Cells

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 40 21 September 2015

F1: Forwarding in the Same Cell Period

 Two Cell Formats required

CF1:

Transmission from initiator Node to target Node via the direct Path Transmission from initiator Node to forwarding Node

CF2:

Transmission from forwarding Node to target Node

Note: The applicable Cell Formats are identified based on the arbiter results

 Cell segments

D: Payload data segment W: Wasted segment (usable for something else) Pipeline: Request5 Request6 Request7 Request8 Perform4 Perform5 Perform6 Perform7 Cell Period 5: Cell Period 6: Cell Period 7: Cell Period 8: Stage 1 Stage 2

CF1: D W CF2: D W

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 41 21 September 2015

F1: A Simple Example for Payload Forwarding

APP_A APP_B APP_C

GA = 3 GA = 2 GA = 1

BA CB AB BC AC CA

AB2 AB2 AB2 AB: AB1 W AB1 W W W W AB2 W AC: APP_A: AB: CB: APP_B: AC: CB: APP_C:

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 42 21 September 2015

F1: Second Example for Payload Forwarding

AB1 AB3

AB

AC4 AC1 AC2 AC3 BA1 BA2 BA4 CA3 BC1 BA3 BA5 AC4 CA1 BA3 BA5 CA2 AB2 CB

A→C B→A B→C C→A C→B A→B

CA3

CB AB AC BA BC CA CB AB1 AB2 AB3

A→B A→C B→A B→C C→A C→B

AB3 AC1 AC2 AC1 AC2 AC3 AC2 AC3 AC4 AC3 AC4 BA1 BA2 BA3 BA2 BA3 BA4 BA5 BA4 BA5 BC1 CA1 CA2 CA3

CP_2 CP_3 CP_4 CP_5

AB1 AB2 AB3 BA1 BA2 BC1 CA1 CA2 CA3

CP_1

APP_A APP_B APP_C

GA = 3 GA = 2 GA = 1

BA CB AB BC AC CA

AB2 AB2 AB2 Transmitting AB2 Payload in Cell Format CF1 from the source APP_A to a forwarding Node Transmitting AB2 Payload in Cell Format CF2 from the forwarding Node to the target APP_B

Path Assignments APP_C egress queues APP_B egress queues APP_A egress queues Cell Periods

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 43 21 September 2015

F2: Forwarding in the Subsequent Cell Period  A longer pipeline structure is used

Stage 1:

Transmission requests issued

Stage 2:

 Source to target transmissions executed except if blocked  Source to forwarding Node transmissions

Stage 3:

 Forwarding Node to target transmissions

 Remark:

 It may occur that a transmission cannot be assigned to the direct connection Path because the Path is occupied by the prior forwarding Node to target transmission

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 44 21 September 2015

F2: Second Example for Payload Forwarding

APP_A APP_B APP_C

GA = 3 GA = 2 GA = 1

BA CB AB BC AC CA

AB1 AB3 AC3

AB

AB2 AC1 AC2 AC4 BA1 BA2 BA4

BA

BC1 BA3 BA5 AC3 CA1 BA3 BA5 AB2

CB

A→C B→A B→C C→A C→B A→B

CA2

CB AB AC BA BC CA CB AB1 AB2 AB3

A→B A→C B→A B→C C→A C→B

AB3 AC1 AC2 AC1 AC2 AC3 AC2 AC3 AC4 AC4 BA1 BA2 BA3 BA2 BA3 BA4 BA5 BA4 BA5 BC1 CA1 CA2 CA3

CP_2 CP_3 CP_4 CP_5

AB1 AB2 AB3 BA1 BA2 BC1 CA1 CA2 CA3

CP_1

CA AB AC

CA2

BC

CA3

CB CA3

CP_6

AC3 Transmitting AC3 Payload from source APP_A Storing received Payload into buffer storage AC3 Transmitting AC3 Payload from buffer storage

Path Assignments

APP_C APP_B APP_A

Cell Periods

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 45 21 September 2015

Annotations Used in the Prior Two Figures

Egress queues:

APP_A, APP_B, APP_C: Protocol agnostic egress queues of the identified

• Nodes. Queues are provided toward each possible target Node respectively.

AB1 AB2 AB3

Bold face: Transmission carried out in the current Cell Period using the direct source to

target Path

Color face: Transmission carried out in the current Cell Period using a forwarding Node Light face: Transmission requested

AB

No transmission assigned for the Path in the current Cell Period

AB1

Transmission of the given Payload segment assigned for the Path in the current Cell Period

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 46 21 September 2015

Network Interface – NWIF – Example

PAIQ x15 PAEQ x15 FIFO x15 MUX 2:1 x15 ARB CSE CSI CLC DAT CLN x15 CLN x15 MUX 2:1 x15i MUX 15:15 MUX 15:15 MUX 15:1 REPL 1:15 MUX 15:15 MUX 2:1 x15 DAT

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 47 21 September 2015

Network Interface – NWIF – Explanations

 ARB: Arbiter for Link assignments

The building block includes the control of multiplexers during the Cell Period

 PAIQ: Protocol agnostic ingress queue  PAEQ: Protocol agnostic egress queue  CLC: Cell locking control  CSE: Control Symbol extractor  CSI: Control Symbol injector  MUX 15:15: Multiplexer of 15 inputs to any of 15 outputs  MUX 2:1 x15: 15 channels of 2:1 multiplexers (common control)  MUX 2:1 x15i: 15 channels of 2:1 multiplexers (individual control)  MUX 15:1 REPL 1:15: 15:1 multiplexer followed by 1:15 replicator  FIFOx15: 15 channels of input FIFO building block

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 48 21 September 2015

The Arbiter

 The preferred arbitration method relies on a cyclic priority based assignment of the forwarding Node function

 This is described in all detail in the patent text

 It may be advantageous to apply a basic arbitration algorithm several times, every time for a limited number of the pending transmissions per source-target pair - this is helpful to achieve fair results under heavy load

 This is surely simpler to implement than one single high sophisticated procedure

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 49 21 September 2015

Parameters for a First Competitive Product

 Link bit rate: 25 Gb/s (per direction)  Lanes/Link: 1  Boards (Nodes) per Backplane Network: 16

 Assuming that the direct connection and 10 forwarding Nodes are available:

25 Gb/s * 11 = 275 Gb/s Node-to-Node (maximum) per direction  200 Gb/s Payload bandwidth available Board-to-Board

This is the double of the 100 Gb/s offered with switched networks

Several transmission relations at this rate may coexist

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 50 22 September 2015

Perspective of 1 Terabit/s Board-to-Board

 Link bit rate: 30 Gb/s (per direction)  Lanes/Link: 4

120 Gb/s per Link in each direction

 Boards (Nodes) per Network: 16

 Assuming that the direct connection and 10 forwarding Nodes are available:

120 Gb/s * 11 = 1320 Gb/s Node-to-Node (maximum) per direction 1 Tb/s Payload bandwidth available Board-to-Board Optical Backplane is needed for the central switch model Several transmission relations at this rate may coexist

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 51 21 September 2015

Demonstration for the Full Mesh Topology

 VITA or PICMG Full Mesh Backplanes could be used for basic demonstration, so a Backplane design is not needed  Some Board products do exist which provide FPGA with direct connections of SERDES to the backplane Fabric Channels, so a Board design is not needed either  An FPGA design is needed, using 4, 6, or 8 Links

 The design will have to use the FPGA (XILINX or ALTERA) according to the Board used  Single Lane routing is fully sufficient for a demonstration setup  The presenter does not have himself the necessary experience with FPGA design tools to execute the project. For this task a small but experienced design group is required.  In a second step the project should be expanded to show something useful, e.g. work as a high performance Ethernet switch

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 52 21 September 2015

Implementations for Dynamic Multi-Path Routing

 Existing VITA 46 and AdvancedTCA Systems and Backplanes are able to carry the here proposed architecture  Since the key investment of a System is in the high performance Backplane and no backwards compatibility is needed, it is rather inviting to make more thoughts about how a dedicated System architecture could look like

 In the following slides a Backplane routing comparison is shown. A dedicated routing for the here proposed architecture has the potential to significantly decrease the cost for ultra high speed Backplanes and trigger new developments to accomplish this.  Remark: In this context there is no relevant idea with a fall back to Star or Dual Star. It is much more so that if some Slots remain unused then they should be filled in with Boards which carry only an FPGA supporting a default Node function providing only the switching (forwarding) function for maximal bandwidth performance of the System for the other Boards.

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 53 21 September 2015

Full Mesh Topology per VITA/PICMG

1

      

2

      

3

      

4

      

5

      

6

      

7

      

8

      

Primary Hub Slot Secondary Hub Slot Node Slots Mesh Enabled Slots

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 54 21 September 2015

Full Mesh Topology Optimization for Layout

1

      

2

      

3

      

4

      

5

      

6

      

7

      

8

      

Note: 4 layers are required at one position In-line routes are available for higher Slot counts as well

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 55 21 September 2015

Applications for the Dynamic Multi-Path Routing

 High performance computing  Telecommunication switching, etc.  High energy physics  Astronomy  High performance simulation  Highest rate switching

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 56 21 September 2015

Alternative Topologies

 The originally targeted topology was the Full Mesh Backplane

 Adjoining Secondary Cell Locked Networks are described in the patent text

 Chain and Ring

 A Chain or Ring built with Nodes connected by Cell Locked Links represents a deterministic Field Bus

 Both the Coarse and the Fine Alignment processes needed additional non-trivial ideas  The presenter built a demonstration setup of a 4 Nodes Ring

• The Cell substructure was designed for up to 64 Nodes
• The demonstration was shown at the Embedded World 2015 in Nürnberg, Germany
• A detailed description of the demo is available via http://www.vdsz.com

 A preferred usage: Uniform segmentation of the Cell with each Node disseminating its sourced data in the first segment toward both Links and in the further segments forwarding received segments from the respective opposite Link. This makes all data available at all Nodes in the Ring – with the property that this information is available per Cell Period exactly IDENTICALLY at all Nodes, so a distributed decision-making is allowed, granting identical results.  A subset of the Symbol positions may be exempted from the deterministic handling and be used e.g. for IP communication.

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 57 21 September 2015

Demonstration and Prototyping for the Ring Topology

 Ring with LVDS

 Existing demo: 4 Nodes built of FPGA design kits interconnected at 2.5 Gb/s LVDS

This is to be understood as a required prerequisite for all other topologies since the basic Cell Alignment architecture was implemented the first time ever

 Demo for up to 64 Nodes requires a small prototype module hardware needed with e.g. a Lattice ECP5 FPGA device like the LFE5UM-25 with 2 SERDES channels.

 The logic block requirements of the Cell Locked Network and other here proposed additions are at a level which allows to estimate that the above named smallest FPGA is used to less than 50% if no sophisticated additions are included.

 To make the prototype module - I need partners. The logic design is still within my reach.  A prototype module with a CPU interface would enable a useful construction.

 Ring with 1000BASE-T electric layer

 For industrial applications more robustness and longer cables can be supported with the transformer coupled electrical layer  I need the right to acquire NDA documents for Marvell’s PHY (88E1119R) in order to make the necessary coding for the FPGA. The design kit I am using does provide the hardware I need.

 Marvell persistently denies the support of NDA documents to a freelance outside the US.

 To make this design happen I need some sort of support, at least a 3rd party which would convince Marvell to provide me design documents / data sheet of the PHY.

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 58 21 September 2015

Photo of the Demonstration Setup:

Photo:  by Peter Schuller

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 59 21 September 2015

Applications for the Ring Topology

 Using a bidirectional forwarding mechanism: an identical set of information is available at all Nodes, redundancy when one Link is broken

 Interconnect for hot replaceable fully redundant computing resources  The high performance alternative for any Field Bus application for industrial control (possibly sometimes with 1000BASE-T physical links)

 Smart robots  Machine control + camera services

 Vehicle entertainment and control systems (using vehicle approved physical links)  Digital audio networking – conferencing  Digital audio networking – studio and stage  Outdoors measurements with instruments control in geology, earthquake research, nature observation  Airport and railway control systems

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 60 21 September 2015

Cooperation, Licensing, etc.

 I do not intend to monopolize on this technology  I am searching for cooperation partners

 Companies large and small  Institutions of reasearch and education  Standardization groups like VITA-VSO but also for Field Bus, etc.

 Development partners are supported with FPGA source code  I am searching for licensors / partners

 Preferably flat rate for application areas or whatever else  RAND licensing possible as long as I have the patent rights  Sublicensors for industry areas or technology subsets  Patent monetization institutions / companies  Selling of the patent rights is one possibility

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 61 21 September 2015

Disclaimers

 It cannot be guaranteed that the patented inventions or any here submitted design proposals are reasonable or viable for a specific implementation.  This presentation does not include the transfer of any property or rights whatsoever.  No warranty can be given whether an implementation of the presented architectures would violate any 3rd party rights.  No warranty can be given for direct or indirect damages which might be brought into connection with the referenced patents or this presentation.  The purpose of this presentation is to provide information to the audience at the 09/23/2015 VSO meeting and to other member companies of the VITA organization.  This presentation shall in no way constrain the rights of the presenter in cooperating with other entities.

Ultimate bandwidth via dynamic multi-path routing in a full mesh network without central control  István Vadász Dipl.-Informatiker (Univ.) Presentation for VITA-VSO, Portsmouth, NH, 09/23/2015 European Patents EP10003791 and EP2634981 62 21 September 2015

Thanks

Questions ? Thanks !

Feel free to contact me with questions, proposals, etc.  istvan@vdsz.com The invention is an obligation – the innovative step of technology development is yours advancing to the next sphere of backplane performance