Networking in Biscuit Sebastian Voit, Paul Emmerich January 18, - - PowerPoint PPT Presentation

▶
networking in biscuit
SMART_READER_LITE
LIVE PREVIEW

Networking in Biscuit Sebastian Voit, Paul Emmerich January 18, - - PowerPoint PPT Presentation

Feb 26, 2023 230 likes •588 views

Chair of Network Architectures and Services Department of Informatics Technical University of Munich Networking in Biscuit Sebastian Voit, Paul Emmerich January 18, 2019 Chair of Network Architectures and Services Department of Informatics

slide-1
SLIDE 1

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Networking in Biscuit

Sebastian Voit, Paul Emmerich

January 18, 2019 Chair of Network Architectures and Services Department of Informatics Technical University of Munich

slide-2
SLIDE 2

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Biscuit

  • Biscuit is an operating system kernel written in Go
  • Evaluate the impact of high-level languages on low-level systems
  • We took a look at Biscuit’s network driver (ixgbe)

Sebastian Voit, Paul Emmerich — Networking in Biscuit 2

slide-3
SLIDE 3

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Overview

  • 1. Golang summary
  • 2. Kernel network drivers
  • 3. Introduction to PCIe device programming
  • 4. Biscuit’s ixgbe driver

Sebastian Voit, Paul Emmerich — Networking in Biscuit 3

slide-4
SLIDE 4

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Golang summary Yet another programming language: Go(lang)

  • Go is a compiled (statically linked binaries) programming language
  • Runtime with garbage collection, memory safe, concurrency
  • Chosen for biscuit for asm support, compilation to machine code, ...

→look at preventable C bugs that cannot occur in memory safe languages

Sebastian Voit, Paul Emmerich — Networking in Biscuit 4

slide-5
SLIDE 5

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Golang summary Yet another programming language: Go(lang)

  • Go is a compiled (statically linked binaries) programming language
  • Runtime with garbage collection, memory safe, concurrency
  • Chosen for biscuit for asm support, compilation to machine code, ...

→look at preventable C bugs that cannot occur in memory safe languages

  • Syntactically similar to C:

package main import "fmt" import "time" func main() { for i := 0; i < 10; i++ { go func (num int){ fmt.Printf("counting up, currently at %v\n", num) }(i) } time.Sleep(1 * time.Second) }

Sebastian Voit, Paul Emmerich — Networking in Biscuit 4

slide-6
SLIDE 6

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Kernel network drivers

  • Enable communication between NIC and the rest of the system:
  • Hand received packets to network stack and
  • Expose sending functionality

Sebastian Voit, Paul Emmerich — Networking in Biscuit 5

slide-7
SLIDE 7

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Kernel network drivers

  • Enable communication between NIC and the rest of the system:
  • Hand received packets to network stack and
  • Expose sending functionality
  • NICs are PCIe devices

Sebastian Voit, Paul Emmerich — Networking in Biscuit 5

slide-8
SLIDE 8

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Kernel network drivers

  • Enable communication between NIC and the rest of the system:
  • Hand received packets to network stack and
  • Expose sending functionality
  • NICs are PCIe devices
  • How do we communicate with PCIe devices?

Sebastian Voit, Paul Emmerich — Networking in Biscuit 5

slide-9
SLIDE 9

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Introduction to PCIe device programming

  • 1. Allocate shared memory

Sebastian Voit, Paul Emmerich — Networking in Biscuit 6

slide-10
SLIDE 10

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Introduction to PCIe device programming

  • 1. Allocate shared memory
  • 2. Write driver

Sebastian Voit, Paul Emmerich — Networking in Biscuit 6

slide-11
SLIDE 11

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Introduction to PCIe device programming

  • 1. Allocate shared memory
  • 2. Write driver
  • 3. ???

Sebastian Voit, Paul Emmerich — Networking in Biscuit 6

slide-12
SLIDE 12

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Introduction to PCIe device programming

  • 1. Allocate shared memory
  • 2. Write driver
  • 3. ???
  • 4. Profit

Sebastian Voit, Paul Emmerich — Networking in Biscuit 6

slide-13
SLIDE 13

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Introduction to PCIe device programming How it’s actually done:

Sebastian Voit, Paul Emmerich — Networking in Biscuit 7

slide-14
SLIDE 14

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Introduction to PCIe device programming How it’s actually done:

  • Communicate with PCIe devices via MMIO (Memory Mapped I/O):
  • Allocate DMA memory for BAR (Base Address Register)

→ Reading from/writing to memory area reads from/writes to the device

  • Datasheet describes registers, offsets, values, etc.

Sebastian Voit, Paul Emmerich — Networking in Biscuit 7

slide-15
SLIDE 15

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Introduction to PCIe device programming How it’s actually done:

  • Communicate with PCIe devices via MMIO (Memory Mapped I/O):
  • Allocate DMA memory for BAR (Base Address Register)

→ Reading from/writing to memory area reads from/writes to the device

  • Datasheet describes registers, offsets, values, etc.
  • In user space we would use mmap()
  • In kernel space the memory is controlled directly
  • Not the driver’s responsibility though

Sebastian Voit, Paul Emmerich — Networking in Biscuit 7

slide-16
SLIDE 16

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Introduction to PCIe device programming How to communicate with the NIC:

  • Use the shared memory
  • Datasheet tells us how:
  • 1. Figure out what to do
  • 2. Look up register offset with control values up in the datesheet
  • 3. Read+set registers accordingly
  • Example: Blinking LED

Sebastian Voit, Paul Emmerich — Networking in Biscuit 8

slide-17
SLIDE 17
slide-18
SLIDE 18
slide-19
SLIDE 19

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Example: LEDs const ( LEDCTL = 0x00200 LED0_BLINK_OFFS = 7 ) leds := atomic.LoadUint32(&bar0[LEDCTL]) //compiler barrier atomic.StoreUint32(&bar0[LEDCTL], leds | (1 << LED0_BLINK_OFFS))

Sebastian Voit, Paul Emmerich — Networking in Biscuit 11

slide-20
SLIDE 20

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Example: LEDs const ( LEDCTL = 0x00200 LED0_BLINK_OFFS = 7 ) leds := atomic.LoadUint32(&bar0[LEDCTL]) //compiler barrier atomic.StoreUint32(&bar0[LEDCTL], leds | (1 << LED0_BLINK_OFFS))

  • Define constants according to datasheet
  • Register write and store: compiler barrier! (volatile in C, atomic.Load/Store in go)
  • Change value in the correct register to turn LEDs on/off

Sebastian Voit, Paul Emmerich — Networking in Biscuit 11

slide-21
SLIDE 21

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

More info In case you want more info on PCIe programming/NIC drivers/biscuit/etc.

  • Watch our talk at 35c3: Safe and Secure Drivers in High-Level Languages1
  • Visit our ixy-languages repository (many more implementations then just go)2
  • Read the seminar paper
  • Look at biscuit’s ixgbe driver3

1https://media.ccc.de/v/35c3-9670-safe_and_secure_drivers_in_high-level_languages 2https://github.com/ixy-languages/ixy-languages 3https://github.com/mit-pdos/biscuit/blob/master/biscuit/src/ixgbe/ixgbe.go Sebastian Voit, Paul Emmerich — Networking in Biscuit 12

slide-22
SLIDE 22

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Biscuit’s ixgbe driver First of all, what is an ”ixgbe”?

Sebastian Voit, Paul Emmerich — Networking in Biscuit 13

slide-23
SLIDE 23

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Biscuit’s ixgbe driver First of all, what is an ”ixgbe”?

  • Intel ixgbe family: 82599ES (aka X520), X540, X550, Xeon D embedded NIC
  • Very common with good datasheet (as seen before)
  • Contrary to other NICs:
  • No message exchange, direct access
  • Hardware features directly exposed instead of black-box NIC

Sebastian Voit, Paul Emmerich — Networking in Biscuit 13

slide-24
SLIDE 24

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Biscuit’s ixgbe driver Now how does biscuit’s driver work?

  • 1. Initialize and set up the device

Sebastian Voit, Paul Emmerich — Networking in Biscuit 14

slide-25
SLIDE 25

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Biscuit’s ixgbe driver Now how does biscuit’s driver work?

  • 1. Initialize and set up the device
  • 2. Forever:
  • Hand received packets over to network stack (interrupt on incoming packets)
  • Instruct NIC to send packets (exposed sending methods)

Sebastian Voit, Paul Emmerich — Networking in Biscuit 14

slide-26
SLIDE 26

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Biscuit’s ixgbe driver: Introduction to tx&rx queues Queues are the basis of the driver!

  • Receive (rx) and transmit (tx) queues are a ring buffer of descriptors
  • Each descriptor is associated with a packet buffer
  • Descriptor contain information about the packets
  • Two formats: read & write-back
  • Read format: write to descriptor to send information to the NIC
  • Write-back format: NIC provides information about the packet’s status
  • Ring structure: Hardware controls the head pointer, software the tail pointer

Sebastian Voit, Paul Emmerich — Networking in Biscuit 15

slide-27
SLIDE 27
slide-28
SLIDE 28

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Biscuit’s ixgbe driver: Initialization Initialization? How?

  • 1. Initialization sequence described in datasheet (section 4.6.3):
  • Global device reset
  • Disable flow control, wait for DMA initialization and enable interrupts

Sebastian Voit, Paul Emmerich — Networking in Biscuit 17

slide-29
SLIDE 29

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Biscuit’s ixgbe driver: Initialization Initialization? How?

  • 1. Initialization sequence described in datasheet (section 4.6.3):
  • Global device reset
  • Disable flow control, wait for DMA initialization and enable interrupts
  • 2. Configure and enable receive (rx) queues:
  • Configure receive features (e.g. VLAN, IP checksumming, ...)
  • Allocate DMA memory for descriptors
  • Allocate packet buffer for each descriptor (also DMA memory)
  • Initialize descriptors (write packet address to NIC)
  • Initialize receive head&tail pointers and enable queue

Sebastian Voit, Paul Emmerich — Networking in Biscuit 17

slide-30
SLIDE 30

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Biscuit’s ixgbe driver: Initialization Initialization? How?

  • 1. Initialization sequence described in datasheet (section 4.6.3):
  • Global device reset
  • Disable flow control, wait for DMA initialization and enable interrupts
  • 2. Configure and enable receive (rx) queues:
  • Configure receive features (e.g. VLAN, IP checksumming, ...)
  • Allocate DMA memory for descriptors
  • Allocate packet buffer for each descriptor (also DMA memory)
  • Initialize descriptors (write packet address to NIC)
  • Initialize receive head&tail pointers and enable queue
  • 3. Configure and enable transmit (tx) queues:
  • Very similar to rx queues, but allocate multiple queues
  • Configure transmission features (e.g. CRC offloading + stripping, ...)
  • Allocate descriptor & packet memory and configure queues
  • Initialize transmit head & tail pointers and enable queues

Sebastian Voit, Paul Emmerich — Networking in Biscuit 17

slide-31
SLIDE 31

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Biscuit’s ixgbe driver: Initialization Initialization? How?

  • 1. Initialization sequence described in datasheet (section 4.6.3):
  • Global device reset
  • Disable flow control, wait for DMA initialization and enable interrupts
  • 2. Configure and enable receive (rx) queues:
  • Configure receive features (e.g. VLAN, IP checksumming, ...)
  • Allocate DMA memory for descriptors
  • Allocate packet buffer for each descriptor (also DMA memory)
  • Initialize descriptors (write packet address to NIC)
  • Initialize receive head&tail pointers and enable queue
  • 3. Configure and enable transmit (tx) queues:
  • Very similar to rx queues, but allocate multiple queues
  • Configure transmission features (e.g. CRC offloading + stripping, ...)
  • Allocate descriptor & packet memory and configure queues
  • Initialize transmit head & tail pointers and enable queues
  • 4. Configure and enable interrupts:
  • Enable interrupts and clear previous interrupts
  • Start interrupt handler as a goroutine (runs parallel)
  • Interrupts: tx&rx queue and link change interrupts

Sebastian Voit, Paul Emmerich — Networking in Biscuit 17

slide-32
SLIDE 32

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Biscuit’s ixgbe driver: Receiving Got a packet! What to do with it!?

  • Receiving is handled via interrupts
  • Will receive an rx interrupt when the NIC receives a packet
  • Figure out how many packets are in the queue and hand all of them to the network stack
  • Reset descriptors and update tail pointer

Sebastian Voit, Paul Emmerich — Networking in Biscuit 18

slide-33
SLIDE 33

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Biscuit’s ixgbe driver: Sending System: I have this packet I want to send. Driver: I got this!

  • Expose sending functions: Tx [raw|ipv4|tcp|tcp tso]
  • Sending is asynchronous

→keep track of pending buffers

  • Enqueueing packets: check for DD and eop flags and figure out how many packets can be

sent

  • Hand control over to NIC by updating tail pointer

Sebastian Voit, Paul Emmerich — Networking in Biscuit 19

slide-34
SLIDE 34

Chair of Network Architectures and Services Department of Informatics Technical University of Munich

Summary

Drivers are not magic!

Just work through the code diligently and have the datasheet at hand.

Sebastian Voit, Paul Emmerich — Networking in Biscuit 20