The Washington University Smart Port Card John DeHart Washington - - PowerPoint PPT Presentation

▶
the washington university smart port card
SMART_READER_LITE
LIVE PREVIEW

The Washington University Smart Port Card John DeHart Washington - - PowerPoint PPT Presentation

Nov 06, 2022 45 likes •405 views

The Washington University Smart Port Card John DeHart Washington University jdd@arl.wustl.edu http://www.arl.wustl.edu/~jdd Washington Kits Workshop January 10,11 2000 1 WASHINGTON UNIVERSITY IN ST LOUIS SPC Personnel Dave Richard -

slide-1
SLIDE 1

January 10,11 2000

Kits Workshop

1

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

The Washington University Smart Port Card

John DeHart Washington University jdd@arl.wustl.edu http://www.arl.wustl.edu/~jdd

slide-2
SLIDE 2

January 10,11 2000

Kits Workshop

2

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

SPC Personnel

  • Dave Richard - Overall Hardware Design
  • Dave Taylor - System FPGA
  • Mike Richards - Board Layout
  • Ed Spitznagel - Embedded NetBSD kernel
  • John DeHart - Integration and Testing
  • Anshul Kantawala - Embedded NetBSD kernel
  • Zubin Dittia - APIC Driver
  • Berkley Shands - APIC Driver
  • Will Eatherton - Original SPC design
  • Toshiya Aramaki - Original SPC design
slide-3
SLIDE 3

January 10,11 2000

Kits Workshop

3

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Outline

  • Hardware:

– SPC as a PC – SPC Hardware Components

  • Software

– Kernel Building and Loading – Network Configuration

  • Preliminary Performance Numbers
slide-4
SLIDE 4

January 10,11 2000

Kits Workshop

4

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Typical Pentium PC

CPU North- Bridge Cache DRAM SouthBridge (PIIX3) (PIC, PIT, …) PCI Bus ISA Bus PCI Devices ISA Devices BIOS Super-IO BIOS

RTC Uarts Kbd/Mse Floppy Parallel ...

Addr/Data Ctrl Ctrl Addr/Data/Ctrl Intr NMI INIT

slide-5
SLIDE 5

January 10,11 2000

Kits Workshop

5

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

What SPC Needs

CPU North- Bridge Cache DRAM SouthBridge (PIC, PIT, …) PCI Bus APIC BIOS BIOS

Addr/Data Ctrl Ctrl Addr/Data/Ctrl Intr NMI INIT

RTC Uarts

slide-6
SLIDE 6

January 10,11 2000

Kits Workshop

6

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

System FPGA Intel Embedded Module

SPC Architecture

CPU North- Bridge Cache DRAM PCI Bus APIC

Addr/Data Ctrl Ctrl Addr/Data/Ctrl Intr NMI INIT

PIT PIC RTC’ BIOS ROM UART1 Interface UART2 Interface UART1 UART2

Link Interface Switch Interface

slide-7
SLIDE 7

January 10,11 2000

Kits Workshop

7

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

SPC Photo Tour

Switch Interface Link Interface Serial Ports APIC CPU Module PCI Bus System FPGA DRAM

slide-8
SLIDE 8

January 10,11 2000

Kits Workshop

8

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

SPC Components

  • APIC
  • PCI Bus Master
  • Pentium Embedded Module

– 166 MHz MMX Pentium Processor

  • L1 Cache: 16KB Data, 16KB Code

– L2 cache: 512 KB – NorthBridge - 33 MHz, 32 bit PCI Bus

  • PCI Bus Master
  • System FPGA
  • PCI Bus Slave

– Xilinx XC4020XL-1 FPGA – 20K Equivalent Gates – ~ 75% used

slide-9
SLIDE 9

January 10,11 2000

Kits Workshop

9

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

SPC Components (continued)

  • Memory

– EDO DRAM – 64MB (Max for current design) – SO DIMM

  • Switch Interface - 1 Gb Utopia
  • Link Interface - 1 Gb Utopia
  • UART

– Two Serial Ports

  • NetBSD system console
  • TTY port
slide-10
SLIDE 10

January 10,11 2000

Kits Workshop

10

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

System FPGA

  • Coded in VHDL
  • PCI slave device
  • Replaces some of the PIIX3 (south bridge)
  • Replaces some of the BIOS
  • Replaces some of the Super IO Chip
slide-11
SLIDE 11

January 10,11 2000

Kits Workshop

11

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

System FPGA: PIIX3 Functionality

  • Programmable Interrupt Controller (PIC)

– Four Interrupts supported and statically assigned:

  • PIT (IRQ 0)
  • APIC (IRQ 5)
  • COM1 (IRQ 4)
  • COM2 (IRQ 3)

– Static fully-nested interrupt priority structure. – Specific End of Interrupt is the only EOI mode supported

  • Programmable Interval Timer (PIT)

– generates a clock interrupt for NetBSD every ~10ms

slide-12
SLIDE 12

January 10,11 2000

Kits Workshop

12

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

System FPGA: BIOS Functionality

  • Interrupt functionality replaced by static values
  • Simple 16 word by 32-bit “ROM”

– implements loop waiting for location 0xFFE00 to change value – then jumps to boot loader code

  • Does NOT perform configuration of Northbridge

– This will be done by the boot loader

  • Does NOT perform PCI configuration of APIC

– This will be done by the APIC Driver

slide-13
SLIDE 13

January 10,11 2000

Kits Workshop

13

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

System FPGA: Super IO Chip Functionality

  • UART Interface

– Two Serial lines supported – Fixed IRQs

  • Real Time Clock

– only the register accesses of the RTC are supported – no interrupts supported – i.e. supported only so NetBSD didn’t need to change – i.e. no alarms will be generated

slide-14
SLIDE 14

January 10,11 2000

Kits Workshop

14

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Software Overview

  • Building a Kernel

– config – compile

  • Getting Kernel onto device that is to execute it

– download via APIC

  • Initiating Kernel execution

– boot loader

  • Configuring Devices

– APIC

slide-15
SLIDE 15

January 10,11 2000

Kits Workshop

15

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Steps to an Executing Kernel on an SPC

  • Configure Kernel

– Start with a typical NetBSD kernel config file – add memory disk (filesystem will reside in kernel!) – serial console port – remove “extra” devices (ethernet, mouse, …)

  • Filesystem

– build filesystem

  • vnconfig(8) - configure vnode pseudo disk as a regular file
  • disklabel(8) - label it
  • newfs(8) - construct a new file system

– mknod(8) - make device files – populate regular files from an existing system

slide-16
SLIDE 16

January 10,11 2000

Kits Workshop

16

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

… Steps to an Executing Kernel on an SPC

  • Compile Kernel
  • Make a symbol only version of kernel for /netbsd

– ps, netstat, … use symbols from /netbsd – real kernel with file system CANNOT reside in itself!

  • Add File System to Kernel

– mdsetimage(8)

  • Create an image version of Kernel

– sets up bss section and symbol table

  • Kernel bss file is ready to download!
slide-17
SLIDE 17

January 10,11 2000

Kits Workshop

17

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

… Steps to an Executing Kernel on an SPC

  • Reset SPC
  • Download Boot Loader

– via APIC control cells – loaded at physical memory location: 0x0

  • Flip bit at location 0xFFE00 to start boot loader

– configure NorthBridge

  • cache, memory size, memory type, timing, etc

– wait for kernel download to complete

  • looks for location 0x3FC to change from 0 to 1

– set up parameters on stack and jump to kernel

  • kernel located at 0x100000
  • Download kernel bss file
  • Flip bit at location 0x3FC - kernel starts booting...
slide-18
SLIDE 18

January 10,11 2000

Kits Workshop

18

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

APIC Driver at SPC Boot Time

  • PCI Configuration Space:

– Base memory address:

  • APIC Physical memory in SPC starts at 0xF0000000

– Memory mapping options:

  • Memory Enable
  • Master Enable
  • Write Invalidate Enable

– IRQ

  • Driver normally would read this from PCI config space
  • APIC statically assigned IRQ 5 by System FPGA
  • For consistency, driver writes 5 to Interrupt Line Register
slide-19
SLIDE 19

January 10,11 2000

Kits Workshop

19

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

PCI Configuration Space

Configuration Space Header

Device ID Vendor ID Status Command Class Code

Revision ID

BIST Header Type Cardbus CIS Pointer Subsystem ID

Subsystem Vendor ID

Expansion ROM Base Address Reserved Reserved Max_Lat Min_Gnt Latency Timer

Cache Line Size

Base Address Registers Interrupt Pin Interrupt Line

slide-20
SLIDE 20

January 10,11 2000

Kits Workshop

20

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Defining a Route on APIC Network Interface

  • Configure the APIC

> ifconfig apic0 inet 192.168.10.1 netmask 0xffffffff0

  • network interface: apic0
  • our IP address: 192.168.10.1
  • Configure a VCI

> atm_ifconfig apic0 0xc5 open -aal5 -llc -besteffort

  • maxqueuebufs 100 -inport 0 -outports 1
  • network interface: apic0
  • vci: 0xc5
  • besteffort (vs. paced or lowdelay)
  • Tx buffer queue limit for this VCI: ‘maxqueuebufs 100’
  • Configure the route

> route add -iface 192.168.10.5 -link apic0:0.0.0.c5

  • destination: 192.168.10.5
  • network interface: apic0
  • vci: 0xc5
slide-21
SLIDE 21

January 10,11 2000

Kits Workshop

21

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

atm_ifconfig

> atm_ifconfig Usage: atm_ifconfig <interface> <vpivci> [open|close|modify [options...]] Options are:

  • aal0 | -aal5
  • llc | -nollc <protocol>: use LLC/SNAP?

protocol is inet | ratm | value of ethernet type

  • loopback
  • inport <input_port_number>
  • outports <bitvector_of_output_ports>

bitvector: 1 => port 0, 2 => port 1, 3 => both

  • lowdelay | -paced <rate> | -besteffort
  • maxqueuebufs <max_descriptors_in_tx_queue>

<rate> is in Kbits/sec

slide-22
SLIDE 22

January 10,11 2000

Kits Workshop

22

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

IPP OPP SPC TI

Switch Fabric

IPP OPP SPC TI IPP OPP

TI

Control Processor

APIC Configuration

6 5 6 5 6 5

IPP OPP SPC TI IPP OPP SPC TI

. . .

Sys. FPGA 32-64 MB Pentium Cache North Bridge

APIC

3 3 3 6 6 6

SPC

slide-23
SLIDE 23

January 10,11 2000

Kits Workshop

23

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

IPP OPP TI

Sys. FPGA 32-64 MB Pentium Cache North Bridge

APIC

Inport 1

Fiber

Outport 2 Outport 1 Inport 0

Defining APIC Ports for Routing

atm_ifconfig apic0 0xca open -aal5 -llc -besteffort -inport $INPORT -outports $OUTPORTs

Switch INPORT OUTPORTs 0 1 0 2 1 1 1 2 0 3 1 3

slide-24
SLIDE 24

January 10,11 2000

Kits Workshop

24

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Using the SPC in our Active Network Demo

Video Source Cross Traffic Source 1 Video Display 1 Video Display 2 Video Display 3 Cross Traffic Sink Cross Traffic Source 2

SPC1 SPC2 SPC2

Switch Fiber

slide-25
SLIDE 25

January 10,11 2000

Kits Workshop

25

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

SPC1: APIC Configuration

#!/bin/sh ifconfig apic0 inet 192.168.10.1 netmask 0xffffffff0 #loopback route add 192.168.10.1 127.0.0.1 # to SPC2 atm_ifconfig apic0 0x0000ca open -aal5 -llc -besteffort

  • maxqueuebufs 100 -inport 0 -outports 1

route add -iface 192.168.10.10 -link apic0:0.0.0.ca

slide-26
SLIDE 26

January 10,11 2000

Kits Workshop

26

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

SPC1: APIC Configuration

# to nmvc0 - Video Source atm_ifconfig apic0 0x0000c2 open -aal5 -llc -besteffort

  • maxqueuebufs 100 -inport 1 -outports 2

route add -iface 192.168.10.2 -link apic0:0.0.0.c2 # to nmvc1 - Video Display 1 atm_ifconfig apic0 0x0000c3 open -aal5 -llc -besteffort

  • maxqueuebufs 100 -inport 1 -outports 2

route add -iface 192.168.10.3 -link apic0:0.0.0.c3 # to nmvc2 - Video Display 2 atm_ifconfig apic0 0x0000c4 open -aal5 -llc -besteffort

  • maxqueuebufs 100 -inport 0 -outports 1

route add -iface 192.168.10.4 -link apic0:0.0.0.c4

slide-27
SLIDE 27

January 10,11 2000

Kits Workshop

27

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

SPC1: APIC Configuration

# to nmvc3 atm_ifconfig apic0 0x0000c5 open -aal5 -llc -besteffort -maxqueuebufs 100 -inport 0 -outports 1 route add -iface 192.168.10.5 -link apic0:0.0.0.c5 # to wooster atm_ifconfig apic0 0x0000c6 open -aal5 -llc -besteffort -maxqueuebufs 100 -inport 1 -outports 2 route add -iface 192.168.10.6 -link apic0:0.0.0.c6 spc.cfg: unmodified: line 1 to gussie # to gussie through gantry route add 192.168.10.7 192.168.10.10 # open multipoint VC for video atm_ifconfig apic0 0x0000d1 open -aal5 -llc -besteffort -maxqueuebufs 100 -inport 1 -outports 2 # open multipoint VC for CT atm_ifconfig apic0 0x0000d6 open -aal5 -llc -besteffort -maxqueuebufs 100 -inport 1 -outports 2 atm_ifconfig apic0 0x0000c7 open -aal5 -llc -besteffort -maxqueuebufs 100 -inport 0 -outports 1 route add -iface 192.168.111.112 -link apic0:0.0.0.c7 # multipoint out atm_ifconfig apic0 0x0000e1 open -aal5 -llc -besteffort -maxqueuebufs 100 -inport 0 -outports 1 route add -iface 192.168.111.111 -link apic0:0.0.0.e1

slide-28
SLIDE 28

January 10,11 2000

Kits Workshop

28

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Linux on an SPC - What would we need?

  • Interrupts

– End of interrupt usage

  • RTC?
  • Boot procedure

– stack parameters – symbol table

  • Embedded file system

– kernel config, utilities for building downloadable kernel

  • APIC driver

– Currently have:

  • skeleton
  • Berkley’s user space code
slide-29
SLIDE 29

January 10,11 2000

Kits Workshop

29

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Preliminary SPC Performance (ANN Kernel)

  • Throughput and Delay
  • Processing:

– active forwarding – payload adder

  • Systems:

– SPC (166 MHz P5) – 450 MHz PII

slide-30
SLIDE 30

January 10,11 2000

Kits Workshop

30

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Test Setup

Switch

SPC APIC

CPU MEM

Ping through SPC Delay Measurement Ping Baseline Delay Measurement Throughput Measurement

slide-31
SLIDE 31

January 10,11 2000

Kits Workshop

31

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Delay (Forward)

100 200 300 400 500 600 700 800 900 1000 5 6 3 4 4 6 3 2 9 2 1 2 8 1 4 9 6 1 7 8 4 2 7 2 2 3 6 2 6 4 8 2 9 3 6 3 2 2 4 3 5 1 2 3 8 4 8 8 4 3 7 6 4 6 6 4 4 9 5 2 5 2 4 5 5 2 8 5 8 1 6 6 1 4 6 3 9 2 6 6 8 6 9 6 8 7 2 5 6 7 5 4 4 7 8 3 2 8 1 2 packet size microseconds SPC 450 MHz

slide-32
SLIDE 32

January 10,11 2000

Kits Workshop

32

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Delay (Adder)

500 1000 1500 2000 2500 3000 3500 5 6 3 4 4 6 3 2 9 2 1 2 8 1 4 9 6 1 7 8 4 2 7 2 2 3 6 2 6 4 8 2 9 3 6 3 2 2 4 3 5 1 2 3 8 4 8 8 4 3 7 6 4 6 6 4 4 9 5 2 5 2 4 5 5 2 8 5 8 1 6 6 1 4 6 3 9 2 6 6 8 6 9 6 8 7 2 5 6 7 5 4 4 7 8 3 2 8 1 2 packet size microseconds SPC 450 MHz

slide-33
SLIDE 33

January 10,11 2000

Kits Workshop

33

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Throughput (Forward)

5000 10000 15000 20000 25000 30000 35000 4 4 3 3 2 6 2 9 8 1 1 9 6 1 4 8 4 1 7 7 2 2 6 2 3 4 8 2 6 3 6 2 9 2 4 3 2 1 2 3 5 3 7 8 8 4 7 6 4 3 6 4 4 6 5 2 4 9 4 5 2 2 8 5 5 1 6 5 8 4 6 9 2 6 3 8 6 6 6 8 6 9 5 6 7 2 4 4 7 5 3 2 7 8 2 8 1 8 packet size packets per second SPC 450 MHz PII

slide-34
SLIDE 34

January 10,11 2000

Kits Workshop

34

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Throughput (Adder)

5000 10000 15000 20000 25000 30000 35000 4 4 3 3 2 6 2 9 8 1 1 9 6 1 4 8 4 1 7 7 2 2 6 2 3 4 8 2 6 3 6 2 9 2 4 3 2 1 2 3 5 3 7 8 8 4 7 6 4 3 6 4 4 6 5 2 4 9 4 5 2 2 8 5 5 1 6 5 8 4 6 9 2 6 3 8 6 6 6 8 6 9 5 6 7 2 4 4 7 5 3 2 7 8 2 8 1 8 packet size packets per second SPC 450 MHz PII

slide-35
SLIDE 35

January 10,11 2000

Kits Workshop

35

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Throughput - Mb/s (Forward)

50 100 150 200 250 300 350 400 450 4 4 3 3 2 6 2 9 8 1 1 9 6 1 4 8 4 1 7 7 2 2 6 2 3 4 8 2 6 3 6 2 9 2 4 3 2 1 2 3 5 3 7 8 8 4 7 6 4 3 6 4 4 6 5 2 4 9 4 5 2 2 8 5 5 1 6 5 8 4 6 9 2 6 3 8 6 6 6 8 6 9 5 6 7 2 4 4 7 5 3 2 7 8 2 8 1 8 packet size Mbit/s SPC 450 MHz PII

slide-36
SLIDE 36

January 10,11 2000

Kits Workshop

36

Washington

WASHINGTON UNIVERSITY IN ST LOUIS

Throughput - Mb/s (Adder)

50 100 150 200 250 4 4 3 3 2 6 2 9 8 1 1 9 6 1 4 8 4 1 7 7 2 2 6 2 3 4 8 2 6 3 6 2 9 2 4 3 2 1 2 3 5 3 7 8 8 4 7 6 4 3 6 4 4 6 5 2 4 9 4 5 2 2 8 5 5 1 6 5 8 4 6 9 2 6 3 8 6 6 6 8 6 9 5 6 7 2 4 4 7 5 3 2 7 8 2 8 1 8 packet size Mbit/s SPC 450 MHz PII