The Correlator for the Tianlai Experiment Jie Hao National ASIC - - PowerPoint PPT Presentation

▶
the correlator for the tianlai experiment
SMART_READER_LITE
LIVE PREVIEW

The Correlator for the Tianlai Experiment Jie Hao National ASIC - - PowerPoint PPT Presentation

Aug 26, 2022 322 likes •682 views

The Correlator for the Tianlai Experiment Jie Hao National ASIC Design Engineering Center, Institute of Automation, Chinese Academy of Sciences Email: jie.hao@ia.ac.cn Outline q System Design q Hardware Design q Algorithm Design q Control

slide-1
SLIDE 1

The Correlator for the Tianlai Experiment

Jie Hao

National ASIC Design Engineering Center, Institute of Automation, Chinese Academy of Sciences

Email: jie.hao@ia.ac.cn

slide-2
SLIDE 2

Outline

q System Design q Hardware Design

q Algorithm Design

q Control Software Design q Questions and Solutions

slide-3
SLIDE 3

AD AD SRIO Switch Correlator (DSP Array) Correlator (DSP Array) 8bits PFB& FFT PFB& FFT 250MSPS,14bits SATA Array 32bits FPGA AD AD PFB& FFT PFB& FFT

192 inputs

10G Ethernet FPGA 504Ports

System Design

clock trigger GPS 10M 10M to 250M Control PC AD/FFT data

slide-4
SLIDE 4

Data Flow

AD 2K points FFT Correlator Float point(IEEE 754 std.)/ Fixed point 14bits 8bits 32bits

FPGA DSP

slide-5
SLIDE 5

Tianlai Digital Backend

slide-6
SLIDE 6

q Data Sampling(ADC+FFT)

  • ADC based on FMC(FPGA Mezzanine Card) carrier board
  • FFT based on FPGA(Virtex-6) board
  • Rear board

q Data processing (Correlator)

  • based on DSP(TMS320C6678)

q Data switch

  • RapidIO switch

q Data storage

  • 10G Ethernet
  • SATA Array

Hardware Design

slide-7
SLIDE 7

FMC(HPC) FMC(HPC)

GTX(X4)/20Gbps GTX(X4)/20Gbps

QSFP

GTX(X4)/20Gbps

FPGA2 FPGA1

GTX(X4)/20Gbps GTX(X4)(550T)

DDR3 FPGA0 Virtex5 (50T)

POWER

POWER GROUP XP1 XP2 XP3 XP4 XP5 XP6 XP7 XP8 XP9

GTX(X4)/20Gbps LVDS GTX(X16)/80Gbps PCIex1

  • nly for

550T/475T GTX(X16)80Gbps

QDR DDR3 QDR

LVDS 10/100/1000M Ethernet GTX(X8)(only for 550T/475T) clock trigger

USB

GD2FPGA Data Flow

slide-8
SLIDE 8

Data Sampling(ADC+FFT)

GD2FPGA board FPGA(Virtex6) FPGA(Virtex6) FPGA (Virtex5)

slide-9
SLIDE 9

Reconfigurable mother board

slide-10
SLIDE 10

GD2FPGA Board

Features

Data processing board+ FMC carrier board

  • 6U standard size
  • 2xVirtex-6 FPGA (XC6VLX240T-2FFG1759/XC6VLX315T-2FFG1759

XC6VLX550T-2FFG1759/XC6VLX475T-2FFG1759)

  • 16G DDR3-SDRAM
  • 288M QDR
  • 2xFMC (HPC) Expansion Slots
  • 1xUSB2.0
  • 2xPCIe
  • 10/100/1000M Ethernet
  • 2xGTX—40Gbps(RapidIO) through frontplane(QSFP connector)
  • 8x GTX(X4) through backplane(ZD connector)—

160Gbps(XC6VLX240T-2FFG1759/XC6VLX315T-2FFG1759)

  • or 12xGTX(X4) through backplane(ZD connector)—

240Gbps(XC6VLX550T-2FFG1759/XC6VLX475T-2FFG1759)

  • Soft IP(high speed interface)
  • Rapid IO
  • 10G Ethernet
slide-11
SLIDE 11

Roach2

Roach2 GD2FPGA 4XSFP+ FPGA Power PC

slide-12
SLIDE 12

Rear board

l Connecting boards through cable(Molex) l Signal testing 10Gb Ethernet rear board l 4xBCM8747 l 12xSfp+

slide-13
SLIDE 13
  • 8 channels
  • 4xADS62P49
  • External trigger&clock
  • 250 MSPS, 14-bits A/D
  • Supports multiple clock
  • Versatile and industry-

standard VITA 57.1 FMC

FMC card (ADC)

slide-14
SLIDE 14

P O W E R

DSP7 DSP6 DSP8 DSP5 DSP1 DSP4 DSP2 DSP3 Swtich

SRIO(X4)/20Gbps

XP1

FPGA S6

XP2 XP3 XP4 XP5 XP6 XP7 XP8 XP9

PCIe(x1)

RJ45x2 88E1111 88E1111

SGMII MDI

POWER GROUP

Data processing (correlator)

slide-15
SLIDE 15

GD8DSP (based on DSP)

Data processing board

slide-16
SLIDE 16

GD8DSP Board

Features

Data processing board

  • 6U standard size
  • 8xTMS320C6678
  • 16G DDR3-SDRAM
  • 4xRapidIO(X4) through backplane(ZD connector)—80Gbps
  • 8xPCIe(X2) through backplane(ZD connector)—80Gbps
  • 2560GMAC(Multiply and Accumulate)
slide-17
SLIDE 17

Data switch board

GDSRIOSW board

Front plane Rear plane

slide-18
SLIDE 18
  • 6U standard size
  • 4xIDT CPS1848(5~6.25Gbps/lane) serial RapidIO switches
  • Supports Rapid IO 1.0,2.0,2.1
  • 8xQSFP—160Gbps
  • 28xRapidIO(X4) through backplane(ZD connector)— 560Gbps
  • Supports backplane star topology

Features

GDSRIOSW board

slide-19
SLIDE 19

RapidIO vs. 10G Ethernet

FPGA 10GE PHY

2.5Gbps 2.5Gbps 2.5Gbps 2.5Gbps GTX GTX GTX GTX

FPGA

5Gbps 5Gbps 5Gbps 5Gbps GTX GTX GTX GTX

RapidIO 10 GE Virtex6’s GTX transceivers: up to 6.6 Gb/s

slide-20
SLIDE 20

The rear board of GDSRIOSW board

28xRapidIO(X4) GDSRIOSW board & rear board

slide-21
SLIDE 21

Pod1 pod8

...

128 servers

Switching System(fat tree)

slide-22
SLIDE 22 A31 A11 A41 A21 B11 B31 B21 B41

A11 A13 A12 B11 B12 B13 A14 B14 B16 A17 A16 B17 C11 B18 A18 C12 D11 C14 C13 D12 D13 C16 D14 D16 D17 C17 D18 C18 A21 A23 A22 B21 B22 B23 A24 B24 B26 A27 A26 B27 C21 B28 A28 C22 D21 C24 C23 D22 D23 C26 D24 D26 D27 C27 D28 C28 A31 A33 A32 B31 B32 B33 A34 B34 B36 A37 A36 B37 C31 B38 A38 C32 D31 C34 C33 D32 D33 C36 D34 D36 D37 C37 D38 C38 A41 A43 A42 B41 B42 B43 A44 B44 B46 A47 A46 B47 C41 B48 A48 C42 D41 C44 C43 D42 D43 C46 D44 D46 D47 C47 D48 C48

C31 C11 C41 C21 D11 D31 D21 D41 A32 A12 A42 A22 B12 B32 B22 B42 C32 C12 C42 C22 D12 D32 D22 D42 A33 A13 A43 A23 B13 B33 B23 B43 C33 C13 C43 C23 D13 D33 D23 D43 A34 A14 A44 A24 B14 B34 B24 B44 C34 C14 C44 C24 D14 D34 D24 D44

C25 D25 A25 B25 C45 D45 A45 B45 C35 D35 A35 B35 C15 D15 A15 B15

A35 A15 A45 A25 B15 B35 B25 B45 C35 C15 C45 C25 D15 D35 D25 D45 A36 A16 A46 A26 B16 B36 B26 B46 C36 C16 C46 C26 D16 D36 D26 D46 A37 A17 A47 A27 B17 B37 B27 B47 C37 C17 C47 C27 D17 D37 D27 D47 A38 A18 A48 A28 B18 B38 B28 B48 C38 C18 C48 C28 D18 D38 D28 D48

Pod 3 Pod 1 Pod 2 Pod 8 Pod 7 Pod 6 Pod 5 Pod 4

Switching System

slide-23
SLIDE 23

3.125Gbps 5Gbps 1.25Gbps 2.5Gbps

High Speed Interface Testing

slide-24
SLIDE 24

q SATA Array+10G Ethernet card q parallel processing q >3.2Gbps

Data storage

slide-25
SLIDE 25

Algorithm Design

q Algorithm on FPGA q Algorithm on DSP q Interface program q Switch schedule

slide-26
SLIDE 26

Based on Simulink Development Environment

ADC FFT Correlator PFB

slide-27
SLIDE 27

Correlation

Plan A:Full Correlation Plan B:1D DFT correlation The Fourier domain cross-correlation is：

slide-28
SLIDE 28

q System bootload

ü Flash ü Ethernet

q Control instruction

ü System/Algorithm parameter ü Control word format

q Information Display(Eclipse)

ü Original AD data display ü Board information collection ü High speed interface information display ü Board temperature information display ü Power voltage and Current information display

q Remote Control

System Control software

slide-29
SLIDE 29

Control and Display Interface

System bootload control High speed Interface Information

slide-30
SLIDE 30

Control and Display Interface

Original AD data display

slide-31
SLIDE 31

Control and Display Interface

Board temperature information Power voltage and Current information

slide-32
SLIDE 32

Questions and Solutions

“Noise”

slide-33
SLIDE 33

How does it work?

Block Block 1 Block N-1 Block Block 1 Block N-1 MSMCSRAM One piece of correlation results (INT) DDR Accumulations (FLOAT) Block (INT) Block (FLOAT) Inverse int to float and accumulate it to the float loop0 loop1 loopN-1 loop0 loop1 loopN-1 EDMA EDMA loop0 loop1 loopN-1 L1DRAM

accumulation

slide-34
SLIDE 34

How did that happen Solution

Since we use the EDMA for the data transfer, it's not controlled by the

  • cores. If the data in loop n flushed in before the results moved out in

loop n-1, the final results in loop n-1 could be covered by the data in loop n, resulting in the “Noises”. In order to avoid this, we control the time gaps between the loops next to each other by inserting a proper time delay.

slide-35
SLIDE 35

Thank You!