Hardware and Device Drivers Device virtualization Device drivers - - PowerPoint PPT Presentation

▶
hardware and device drivers
SMART_READER_LITE
LIVE PREVIEW

Hardware and Device Drivers Device virtualization Device drivers - - PowerPoint PPT Presentation

Oct 02, 2022 481 likes •620 views

Outline Faculty of Computer Science Institute for System Architecture, Operating Systems Group What's so different about device drivers? Hardware access Writing device drivers on L4 Reuse of legacy drivers Hardware and Device

slide-1
SLIDE 1

Faculty of Computer Science Institute for System Architecture, Operating Systems Group

Hardware and Device Drivers

Dresden, 2007-11-13 Björn Döbel

TU Dresden, 2007-11-13 Device Drivers Slide 2 von 45

Outline

  • What's so different about device drivers?
  • Hardware access
  • Writing device drivers on L4

– Reuse of legacy drivers

  • Device virtualization
  • Device drivers and security

TU Dresden, 2007-11-13 Device Drivers Slide 3 von 45

Why are Drivers so hard?

  • Typically need kernel privileges because of

– Interrupt handling – Hardware access

  • data transfer
  • config registers
  • firmware

– Special memory management – DMA

  • physical adresses
  • DMA vs. paging

– Synchronization & Timing mechanisms

  • complex state machines
  • Unsexy solution: keep drivers in the kernel

TU Dresden, 2007-11-13 Device Drivers Slide 4 von 45

Idea: User-level Drivers

  • Isolate components

– device drivers (disk, network, graphic, USB cruise missiles, ...) – stacks (TCP/IP, file systems, ...)

  • Separate address spaces for each
  • More robust components

– Crashing network device does not hurt the file system

  • HW multiplexing
slide-2
SLIDE 2

TU Dresden, 2007-11-13 Device Drivers Slide 5 von 45

System Layout

  • Devices connected by busses (USB, PCI,

PCIx)

  • Host chipset (DMA logic, IRQ controller)

connects busses and CPU

CPU Chipset Memory System Bus Memory Bus PCI Controller Network Card Sound Card PCI Bridge USB Host Controller PCI Bus USB Bus USB Coffee Maker PCI Wifi PCI Bus PCI Bus

TU Dresden, 2007-11-13 Device Drivers Slide 6 von 45

Real World Example

Intel 925x Chipset (source: http://www.intel.com)

TU Dresden, 2007-11-13 Device Drivers Slide 7 von 45

Buses and Devices

  • A long long time ago:

– device architecture hard-coded

  • Problem: more and more devices

– need means of dynamic device discovery

  • Probing: try out every driver to see if it works
  • Plug&Play: first try of dynamic system

description – device manufacturers provide unique IDs

  • PCI: dedicated config space
  • ACPI: system description without relying on

underlying bus/chipset

TU Dresden, 2007-11-13 Device Drivers Slide 8 von 45

Buses: PCI

  • Peripheral Component Interconnect
  • Hierarchy of busses, devices and functions
  • Configuration via I/O ports
  • Address + data register (0xcf8-0xcff)

Chipset

Device 1 Device 2 PCI-to-PCI Bridge Device 3 Device 4 Func 1 Func 2

slide-3
SLIDE 3

TU Dresden, 2007-11-13 Device Drivers Slide 9 von 45

Buses: PCI (2)

  • PCI config space
  • 64 byte header

– Busmaster DMA – Interrupt line – I/O port regions – I/O memory regions – + 192 byte additional space

  • must be provided by every device function
  • must be managed to separate device drivers

TU Dresden, 2007-11-13 Device Drivers Slide 10 von 45

Buses: USB

  • Intel, 1996
  • One bus to rule them all?

– Firewire has always been faster

  • Tree of devices

– root = Host Controller (UHCI, OHCI, EHCI) – Device drivers use HC to communicate with their device via USB Request Blocks (URBs) – USB is a serial bus

  • HC serializes URBs
  • Wide range of device classes (input, storage,

peripherals, ...)

  • Device classes allow generic drivers

TU Dresden, 2007-11-13 Device Drivers Slide 11 von 45

Interrupts

  • Signal device state change
  • Programmable Interrupt Controller (PIC, APIC)

– map HW IRQs to CPU's IRQ lines – prioritize interrupts

Chipset CPU

System Bus Memory Bus PCI Bus

IDE NET USB

INT PIC

INT A INT B INT C

Memory

TU Dresden, 2007-11-13 Device Drivers Slide 12 von 45

Interrupts (2)

  • Handling interrupts involves

– examine / manipulate device – program PIC

  • acknowledge/mask/unmask interrupts

Chipset CPU

System Bus Memory Bus PCI Bus

IDE NET USB

INT PIC

INT A INT B INT C

Memory

slide-4
SLIDE 4

TU Dresden, 2007-11-13 Device Drivers Slide 13 von 45

L4: Interrupt handling

  • IRQs mapped to IPC
  • threads call ipc_recv() at a dedicated interrupt

kernel thread

  • exactly one thread can wait for exactly one

interrupt – no IRQ sharing support in the kernel

l4_ipc_receive(irq_id,…)

CPU Chipset Device Hardware Software

Kernel

Fiasco Microkernel

Device Driver

TU Dresden, 2007-11-13 Device Drivers Slide 14 von 45

L4: Problems with interrupts

  • IRQ lines are limited – IRQ sharing is needed
  • Multiple user-level drivers need to access PIC

– locking – we don't want to trust our drivers'

  • IRQ priorities vs. thread priorities -> priority

inversion

  • IRQs are a resource and must be managed!
  • Some drivers need to disable interrupts.

TU Dresden, 2007-11-13 Device Drivers Slide 15 von 45

IRQ management: Omega0

  • Omega0

– Hides all interrupt logic from device drivers

  • one interface for all architectures

– Provides sharing of interrupts – Provides interrupt hand-over to another thread – Prevents priority inversion – Implementation may require a user-level server, depending on the kernel APIs

TU Dresden, 2007-11-13 Device Drivers Slide 16 von 45

Omega0 layout

  • x86: user-level management server
  • ack IRQs immediately at PIC -> prevent

priority inversion

Chipset CPU

PIC Device 1 Device 2

INT INT A INT B

Hardware Software

Kernel

Fiasco Microkernel

Omega0

IRQ A IRQ B IRQ A IRQ B

Device Driver 1 Device Driver 2

slide-5
SLIDE 5

TU Dresden, 2007-11-13 Device Drivers Slide 17 von 45

Omega0: Multiple IRQs per Thread

  • e.g.: one IDE driver for both IDE ports

Device Driver Chipset CPU

PIC Device 1 Device 2

INT INT A INT B

Hardware Software

Kernel

Fiasco Microkernel

Omega0

IRQ A IRQ B IRQ A IRQ B

TU Dresden, 2007-11-13 Device Drivers Slide 18 von 45

Omega0: IRQ sharing

  • e.g., USB storage and USB missile launcher
  • n the same IRQ line

Chipset CPU

PIC Device 1 Device 2

INT INT A

Hardware Software

Kernel

Fiasco Microkernel

Omega0

IRQ A IRQ A

Device Driver 1 Device Driver 2

TU Dresden, 2007-11-13 Device Drivers Slide 19 von 45

Disabling interrupts

  • CLI – only with IO Privilege Level (IOPL) 3
  • Should not be allowed for every user-level

driver

– unstrusted drivers – security risk

  • Observation: drivers often don't need to

disable IRQs globally, but only access to their

  • wn IRQ

– in this case dropping pending IRQs at omega0 is enough, no need to mingle with others

TU Dresden, 2007-11-13 Device Drivers Slide 20 von 45

Hardware access

  • Apart from IRQs, drivers need to access

device hardware

– data transfer – configuration

  • Approaches

– memory-mapped I/O – I/O ports (x86-specific)

slide-6
SLIDE 6

TU Dresden, 2007-11-13 Device Drivers Slide 21 von 45

I/O ports

  • x86-specific feature
  • I/O ports define own I/O address space

– Each device uses its own area within this address space

  • Special instruction to access I/O ports

– in / out: I/O read / write – Example: read byte from serial port mov $0x3f8, %edx in (%dx), %al

  • Everyone who can access I/O ports of a device can

manipulate the device

  • Need to restrict I/O port access

– Allow device drivers access to I/O ports used by its device only

TU Dresden, 2007-11-13 Device Drivers Slide 22 von 45

I/O Bitmap

  • Tasks have I/O privilige level

(IOPL)

  • If IOPL > current PL, all

accesses are allowed (kernel mode)

  • Else: I/O bitmap is checked
  • 1 bit per I/O port

– 65536 ports -> 8kB

  • Controls port access

(0 == ok, 1 == GPF)

  • L4: per-task I/O bitmap

– Switched during task switch – Allows per-task grant/deny of I/O port access

I/O Map Base Address

. . .

#0x0000 #0xffe0 #0xfff0 TSS 1111111111111111 1111111111011011 1111111111111111

TU Dresden, 2007-11-13 Device Drivers Slide 23 von 45

I/O Flexpages

  • Reuse kernel's map/grant mechanism for

mapping I/O port rights -> I/O flexpages

  • Kernel detects type of flexpage and acts

accordingly

  • Task with all I/O ports mapped is raised to

IOPL 3 1111

Base Port log2#Ports

0000

Map/Grant L4.Fiasco I/O flexpage format

TU Dresden, 2007-11-13 Device Drivers Slide 24 von 45

I/O Memory

  • Devices often contain on-chip memory (NICs, graphcis

cards, ...)

  • Instead of accessing through I/O ports, drivers can

map this memory into their address space just like normal RAM – no need for special instructions – increased flexibility by using underlying virtual memory management

CPU Chipset Memoy

Device

Hardware Software

Kernel

Fiasco Microkernel Memory

Driver

Memory

slide-7
SLIDE 7

TU Dresden, 2007-11-13 Device Drivers Slide 25 von 45

I/O memory (2)

  • Device memory looks just like phys. memory
  • Chipset needs to

– map I/O memory to exclusive address ranges – distinguish physical and I/O memory access

CPU Chipset Memory

Device

Hardware Software

Kernel

Fiasco Microkernel Memory

Driver

TU Dresden, 2007-11-13 Device Drivers Slide 26 von 45

I/O memory in L4

  • Like all memory, I/O memory is owned by sigma0
  • Sigma0 implements protocol to request I/O memory

pages

  • Abstraction: Dataspaces containing I/O memory

CPU Chipset Memory Hardware Software

Kernel

Fiasco Microkernel

Driver 1

Device 1 Device 2

Driver 2 Sigma0

TU Dresden, 2007-11-13 Device Drivers Slide 27 von 45

Direct Memory Access (DMA)

  • Bypass CPU by directly transferring data from

device to RAM

– improved bandwidth – relieved CPU

  • DMA controller either programmed by driver
  • r by device's DMA engine (Busmaster DMA)

CPU Chipset Memory Device Controller

DMA Engine

TU Dresden, 2007-11-13 Device Drivers Slide 28 von 45

Problems with DMA

  • Most architectures use physical addresses for

DMA

  • Drivers need to know these addresses in
  • rder to correctly use buffers etc.
  • Buffers must not be paged out (pinned pages)

during DMA

– close interaction with memory management

  • DMA with phys. addresses bypasses VM

management

– Drivers can overwrite any phys. address

  • DMA is a security risk.
slide-8
SLIDE 8

TU Dresden, 2007-11-13 Device Drivers Slide 29 von 45

Idea: I/O MMU

  • Like traditional MMU maps virtual to physical

addresses

– implemented in PCI bridge – manages a pagetable – I/O-TLB

  • Drivers access buffers through virtual

addresses

– I/O MMU translates accesses – restrict access to phys. memory by only mapping certain DMA addresses into driver's address space

TU Dresden, 2007-11-13 Device Drivers Slide 30 von 45

I/O MMUs

  • Per-Bus IOMMU already in 64bit architectures
  • Per-device IOMMU not available yet

Chipset

Device 1 Device 2 I/O-MMU Device 3

Memory Chipset Memory

Device 3 I/O-MMU Device 2 I/O-MMU Device 1 I/O-MMU

TU Dresden, 2007-11-13 Device Drivers Slide 31 von 45

I/O MMU Architecture

  • I/O MMU manages by yet another resource

manager

  • Before accessing I/O memory, drivers use

manager to establish a virt->phys mapping

CPU Chipset Memory Hardware Software I/O-MMU Device Device Manager Dataspace Manager I/O-MMU Manager Driver Client Application

TU Dresden, 2007-11-13 Device Drivers Slide 32 von 45

Summary: Driver support in L4

  • Interrupts -> IPC
  • I/O ports and memory -> flexpage mappings
  • User-level resource manager -> L4IO

CPU Chipset Memory Devices Hardware Software

Kernel

Fiasco Microkernel

L4IO

  • Omega0
  • I/O Ports / Memory
  • PCI

Dataspace Manager

  • Phys. Addresses
  • Pinned Memory

Driver

lib_l4io lib_dm

Driver

lib_l4io lib_dm

Driver

lib_l4io lib_dm

slide-9
SLIDE 9

TU Dresden, 2007-11-13 Device Drivers Slide 33 von 45

Untrusted Device Drivers

  • Problem: How to enforce device access policies on

untrusted drivers?

  • Solution:

– I/O manager needs to manage device resources – external driver loader configures policy

  • General concept: Split policy configuration and

enforcement

CPU Chipset Devices Hardware Software

Kernel

Fiasco Microkernel

I/O manager

  • knows all devices
  • manages I/O

resources

Driver loader

  • decide which drivers

to load and which re- sources to assign

configure policy

TU Dresden, 2007-11-13 Device Drivers Slide 34 von 45

Implementing Device Drivers

  • Just like in any other OS:

– Specify an interface (IDL in our case) – Implement interface, use the access methods provided by the L4 Environment

  • Highly optimized code possible
  • Hard to maintain, implementation time-consuming
  • Many vendors won't give you a spec.
  • Why reimplement drivers if there are already

versions available? – Linux, BSD – Open Source – Windows – Binary drivers

TU Dresden, 2007-11-13 Device Drivers Slide 35 von 45

Reusing legacy device drivers

  • Exploit virtualization: Device Driver OS

Source: LeVasseur et. al.: "Unmodified Device Driver Reuse and Improved System Dependability via Virtual Machines”, OSDI 2004

TU Dresden, 2007-11-13 Device Drivers Slide 36 von 45

Reusing Legacy Device Drivers

  • More work-intensive idea: wrap legacy source

code with a dedicated environment

– implement Linux interface in a library (which maps this interface to L4) – Device Driver Environment (DDE)

  • Reasonable less overhead than using a

complete VM for each driver

  • Maintainable code
  • Reasonable effort

– DDE can be reused

Environment Interface Code Original Driver Code Glue Code

slide-10
SLIDE 10

TU Dresden, 2007-11-13 Device Drivers Slide 37 von 45

Emulating Linux: dde_linux

  • Multiple L4 threads provide a Linux

environment

– Workqueues – SoftIRQs / Bottom Halves – Timers

  • Emulate SMP-like system (each thread

assumed to be one processor)

  • Special synchronization mechanism:

– irq_lock instead of CLI/STI (tamed Linux)

TU Dresden, 2007-11-13 Device Drivers Slide 38 von 45

DDE Structure

CPU Chipset Memory Devices Hardware Software

Kernel

Fiasco Microkernel

L4IO Dataspace Manager Emulation Library (dde_linux) Linux Driver Source Code

L4 Server Code

Client Application

Client Library lib_l4io lib_dm Work Queues SoftIRQs IRQs Timer

TU Dresden, 2007-11-13 Device Drivers Slide 39 von 45

DDEKit – another abstraction layer

  • Next task: I want to reuse FreeBSD drivers,

too.

  • Don't implement the same stuff from scratch.
  • Extract common mechanisms into another

layer – DDEKit – and build DDEs upon it:

– IRQ handling – I/O port management – PCI config space access – Locks – Threads – Timers – ...

TU Dresden, 2007-11-13 Device Drivers Slide 40 von 45

DDEKit (2)

  • Implementation overhead for single DDEs

gets much smaller

  • Performance overhead still reasonable

– e.g., no visible increase of network latency in user-level ethernet driver

  • L4-specific parts (sloccount):

– standalone DDE Linux 2.4: ~ 3.000 LoC – DDEKit ~ 2.000 LoC – DDEKit-based DDE Linux 2.6: ~ 1.000 LoC – Standalone Linux VM: > 500.000 LoC

  • Highly customizable: implement DDE base

library and support libs (net, disk, sound, ...)

slide-11
SLIDE 11

TU Dresden, 2007-11-13 Device Drivers Slide 41 von 45

DDEKit (3)

CPU Chipset Memory Devices Hardware Software

Kernel

Fiasco Microkernel

L4IO Dataspace Manager Emulation Library (ddekit) dde_linux Linux Driver Source Code

L4 Server Code

Client Application

Client Library Linux-specifics (SoftIRQs, KThreads)

TU Dresden, 2007-11-13 Device Drivers Slide 42 von 45

DDE(Kit): disk driver

driver_write_block() { /* prog device */ /* wait4completion */ } write_block() add_wait_queue() schedule() driver_init() { /* find device */ /* init ISR */ } server_init() pci_find_device() request_irq() lib_l4io driver_irq() driver_bh() { /* do work */ /* notify */ } irq_thread() bh_thread() wake_up()

IRQ

lib_ddekit

TU Dresden, 2007-11-13 Device Drivers Slide 43 von 45

DDE(Kit): Use Cases

  • DDELinux2.4

– IDE Disk Driver – Virtual Ethernet Interface – USB Webcam – TCP/IP Network Stack – OSS sound server

  • DDELinux 2.6

– Virtual Ethernet Interface – ATA disk driver – ALSA sound server

  • DDEFreeBSD

– ATA disk driver

TU Dresden, 2007-11-13 Device Drivers Slide 44 von 45

Literature

  • Omega0

– http://os.inf.tu-dresden.de/~jork/papers/omega0.pdf

  • Safe interrupt handling

– http://os.inf.tu-dresden.de/papers_ps/part98.pdf

  • DMA and I/O MMU

– http://os.inf.tu-dresden.de/papers_ps/tr-ioarch-2003.pdf – http://os.inf.tu-dresden.de/papers_ps/mehnert-phd.pdf

  • DDE and DDE-based drivers

– http://os.inf.tu-dresden.de/paper_ps/helmuth-diplom.pdf – http://os.inf.tu-dresden.de/papers_ps/griessbach-diplom.pdf – http://os.inf.tu-dresden.de/papers_ps/menzer-diplom.pdf – http://os.inf.tu-dresden.de/papers_ps/friebel-diplom.pdf

  • Device Driver OS

– http://i30www.ira.uka.de/research/documents/l4ka/2004/ .. .. LeVasseur04UnmodifiedDriverReuse.pdf

slide-12
SLIDE 12

TU Dresden, 2007-11-13 Device Drivers Slide 45 von 45

Coming soon

  • Tomorrow: Exercise on “The nucleus of a

multiprogramming system”

  • Next week: Resource management