From L3 to seL4 Background What Have We Learnt in 20 Years of L4 - - PowerPoint PPT Presentation

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

From L3 to seL4 What Have We Learnt in 20 Years of L4 Microkernels?

Presented by Andrew Shugarts

• K. Elphinstone
• G. Heiser

October 23, 2013

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Outline

Background From L3 to L4 L3 L4 L4 Development The Retained The Abandoned The Replaced and Added The Design of seL4 Security Memory Management Object Independence Preemption & Notifications Open Problems Conclusions

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Design Goal

◮ Create an operating system kernel with bare minimum

functionality.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Design Goal

◮ Create an operating system kernel with bare minimum

functionality.

◮ We only need support for virtual address spaces, thread

management, and inter-process communication, so

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Design Goal

◮ Create an operating system kernel with bare minimum

functionality.

◮ We only need support for virtual address spaces, thread

management, and inter-process communication, so

◮ Move features like device drivers, network stacks, and

file systems into userland (called servers).

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

The Problem

◮ Traditional system services are implemented as

programs in userspace

◮ So programs that require a service need to use

inter-process communication (IPC)

◮ However, this typically costs 100 µs to complete a

• ne-way message pass

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Why?

Let’s say you need to read 1000 bytes from a file. In a monolithic kernel,

• 1. The process calls read which invokes the kernel.
• 2. The kernel retrieves the data and copies it into a buffer

supplied by the process.

• 3. The kernel returns and the processer switches back to

user mode.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Why? (Cont.)

In comparison, in a micro-kernel,

• 1. The process uses IPC to send a message to the file

server

• 2. This invokes the microkernel who copies the message

into the file server’s address space

• 3. The microkernel calls the scheduler and we

context-switch into the file server.

• 4. The file server receives the message, retrieves the data

and then sends it to the process.

• 5. The microkernel is invoked, copies the data into the

process and so on...

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Why? The Picture

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

The L3 Microkernel

◮ In the early 80s, Jochen Liedtke developed the L3

microkernel.

◮ L3 was similar to other microkernels, including poor IPC

performance.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

The L3 Microkernel

◮ In the early 80s, Jochen Liedtke developed the L3

microkernel.

◮ L3 was similar to other microkernels, including poor IPC

performance.

◮ Liedtke recognizes a key improvement for IPC that

becomes the basis for L4.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

The L4 Microkernel

◮ Originally implemented in assembler for i486 by Liedtke

and then ported to Pentium.

◮ Various others reimplemented L4 in different

combinations of languages and platforms.

◮ Most recently, seL4 was designed and implemented to

be formally verified.

◮ Liedtke’s original L4 and others derivations achieve sub

microsecond IPC performance.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Overview

For the last two decades of L4 development, we’ll discuss some of the major features that were

◮ retained ◮ abandoned ◮ and replaced or added

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Retained Concepts

In 1995, Liedtke outlines a minimality principle for the design

• f L4 which was retained for seL4:

A concept is tolerated inside the microkernel only if moving it outside the kernel, i.e. permitting competing implementations, would prevent the implementation of the systems required functionality. User-level drivers and interrupts implemented as IPC were retained.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Abandoned Concepts

Features that were abandoned include:

◮ Recursive address spaces ◮ “Long” IPC messages and IPC timeouts ◮ Hierarchical task management and communication

control – “Clans and chiefs”

◮ Process kernel and virtual TCB addressing ◮ Non-standard calling conventions and assembler code

for performance

◮ Non-portable implementations

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Recursive Address Spaces

◮ L4 provided address spaces through “donation”. ◮ Spaces start empty and are given pages (via mappings)

from another address space.

◮ Allowed for the implementation of traditional virtual

paged memory (and other devices) outside the kernel

◮ Bookkeeping typically consumed 25-50% of kernel

memory

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

IPC Features

“Long” Messages

◮ Implemented by temporarily mapping the source page

into the destination page to only copy the message once

◮ Required handling nested exceptions and added

significant kernel complexity, but rarely used practically

Timeouts

◮ Intended to protect against denial of service ◮ Added complexity to manage floating point values used

and in managing wakeup lists

◮ Rarely used as is – reduced down to zero and infinite

timeouts

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

“Clans and Chiefs”

◮ Process hierarchy whereby IPC can only be sent to a

process’ parent (chief) or siblings (clan)

◮ Forced messages sent outside the clan to be routed

through both chiefs

◮ Generated lots of overhead and generally not

implemented in other versions

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Stack Approach and Addressing

◮ Originally a seperate kernel stack for each thread ◮ Use much of the per-thread memory overhead and

increase kernel’s cache footprint

◮ Warton [2005] showed that the process kernel did not

significantly outperform the event in one benchmark, but in another the event kernel performed 20% better.

◮ Nourai [2005] found that there is no performance

benefit to virtual addressing

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Performance Optimizations

◮ Originally, kernel was implemented entirely in

assembler1

◮ When later kernels were written predominantly in

C/C++, cost of calling convention mismatch became pronounced.

◮ Led to implementation of hand-crafted assembly

fast-paths where necessary

◮ Abandoned in part due to maintenance costs ◮ Implementation of seL4 found that careful construction

• f the fast-path in C with correct hints to the compiler

produced competitive performance

1Which is clearly non-portable

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Replaced/Added Concepts

Features that were replaced include:

◮ Traditional Thread IDs with “ports” for IPC ◮ Lazy scheduling with Benno scheduling ◮ Direct process switch subject to priorities ◮ Physical with virtual message registers

Asynchronous notifications for IPC messages were also added starting with embedded versions.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Thread IDs

◮ Originally messages were sent to threads via their

thread ID – poor information hiding

◮ IPC endpoints, implemented as separate port-like kernel

• bjects, instead.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Scheduling

◮ Frequent queue manipulations by the scheduler decrease

IPC performance

◮ Originally minimized through lazy scheduling

◮ Leaves the thread in the runnable queue until next

invocation of the scheduler

◮ In worst case, the scheduler is only bounded by the

number of threads

◮ Replaced by Benno scheduling

◮ Threads that block during IPC are removed from the

runnable queue

◮ A thread that unblocks during IPC is not inserted into

the runnable queue.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Direct Process Switching

Direct Process Switch

When a thread blocks on IPC, the kernel switches to a runnable thread and uses the previous thread’s time slice to execute, often ignoring priorities in the process. Instead, we only allow direct-process switching if it makes sense given the thread priorities

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Message Registers

◮ Originally, as much of the message as possible was

transferred using physical registers

◮ Instead, use virtual message registers by pinning part of

the address space

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Asynchronous Notifications

◮ Asynchronous notifications added to keep

multi-threading out of otherwise simple systems.

◮ Sender delivers the payload in the receiver’s address

space and returns

◮ Receiver can choose to block on notification or poll for

it.

◮ Interrupts can then be modeled as asynchronous

notifications

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Capabilities

seL4 uses capabilities to implement security

Capability

A capability is a set of permissions directly associated with a kernel object and which can be conferred to other programs. Example: A Unix-like file descriptor This adheres to the following design goals

• 1. All authority is explicitly conferred (via capabilities).
• 2. Data access and authority can be confined.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Design

In seL4, all in-kernel objects

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Design

In seL4, all in-kernel objects

◮ are allocated as first-class objects in the ABI

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Design

In seL4, all in-kernel objects

◮ are allocated as first-class objects in the ABI ◮ and are ensured not to change size.

This is done to link kernel objects with capabilities for security.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Allocation

Requirements:

◮ The model must ensure that no two objects overlap. ◮ User-mode code cannot uncontrollably modify kernel

• bjects.

We allocate a kernel object from an Untyped Memory object which has full capability to a region of memory then retype that space into the desired object. To ensure the second property, the only kernel objects mapped into a program’s virtual address space are Frame

• bjects.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Re-Use

To reclaim memory we impose the next requirement:

Requirement:

No reference that would allow access to an object must remain after reclamation. As a consequence of capabilities, when access to an object is revoked, the capability derivation tree is updated.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Re-Use

To reclaim memory we impose the next requirement:

Requirement:

No reference that would allow access to an object must remain after reclamation. As a consequence of capabilities, when access to an object is revoked, the capability derivation tree is updated. Only once the last capability referring to a location in memory (a leaf node in the tree) has been revoked will the

• bject be deleted.

Reclamation is bounded only be memory, so it will need to be preemptible (discussed later).

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Kernel Objects

Object Description TCB Thread control block Cnode Capability Storage Synchronous Endpoint Port-like rendezvous object for IPC Asynchronous Endpoint Port-like object for notifications PageDirectory Top-level page table for virtual memory PageTable Leaf page table Frame 4 KiB, 64 KiB, 1 MiB, and 16 MiB objects that can be mapped by page tables to form virtual memory Untyped Memory Power-of-2 region of physical memory from which other kernel objects can be allocated

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Requirements

In order to allow kernel objects to be reclaimed we require the following invariants:

◮ Every object must be reclaimable in isolation. ◮ Any capabilities used to couple objects must be allowed

to be decoupled when authority is revoked.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Possibilities for Dependence

◮ Objects may refer to each other with internal pointers.

◮ When an object is removed, we have to remove the

reference and make the object consistent.

◮ Objects contain capabilities to other objects.

◮ The capability is implemented as a copy, so we can

decouple them by revoking the capability.

◮ The capability contains book-keeping data.

◮ This is used predominantly for sharing Frames among

processes and updating corresponding Page Table Entry

• bjects.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Preemption

seL4 runs with interrupts disabled in kernel code to keep concurrency out of the kernel. To compromise, potentially long running operations are broken into pieces which can be run piecewise with threads allowed to preempt between pieces. As well, some systems calls are made restartable by storing progress in zombie capabilities.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Notifications

◮ Asynchronous endpoints allow a thread to be notified of

events even while blocked on a synchronous endpoint.

◮ Achieved by binding asynchronous endpoints to a

particular thread.

◮ Allows for event-based service implementations.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Open Issues

Scheduling policies have still not been ported into userspace, though there is an incentive to do so. seL4 was designed for a uniprocessor for formal verification, though a clustered multikernel has been advanced as an

• pportunity to keep most of the formal verification.

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Thoughts

◮ An interesting design and appealing for system safety ◮ Could be non-intuitive for an application programmer ◮ Reminded me of a paper for a formally verified kernel

for the browser Quark (which actually references the paper for the verification of seL4!)

From L3 to seL4 What Have We Learnt in 20 Years

• f L4

Microkernels?

• K. Elphinstone,
• G. Heiser

Background From L3 to L4

L3 L4

L4 Development

The Retained The Abandoned The Replaced and Added

The Design of seL4

Security Memory Management Object Independence Preemption & Notifications Open Problems

Conclusions

Questions?