We Crashed, Now What? Lorenzo Cavallaro Cristiano Giuffrida Andrew - - PowerPoint PPT Presentation

▶
we crashed now what
SMART_READER_LITE
LIVE PREVIEW

We Crashed, Now What? Lorenzo Cavallaro Cristiano Giuffrida Andrew - - PowerPoint PPT Presentation

Apr 02, 2024 167 likes •773 views

We Crashed, Now What? Lorenzo Cavallaro Cristiano Giuffrida Andrew S. Tanenbaum Vrije Universiteit Amsterdam 6 th Usenix Workshop on Hot Topics in System Dependability October 3, 2010, Vancouver, BC, Canada We Crashed, Now What? Cristiano

slide-1
SLIDE 1

We Crashed, Now What?

Cristiano Giuffrida Lorenzo Cavallaro Andrew S. Tanenbaum

Vrije Universiteit Amsterdam

6th Usenix Workshop on Hot Topics in System Dependability October 3, 2010, Vancouver, BC, Canada

1 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-2
SLIDE 2
slide-3
SLIDE 3

OS Dependability Threats

3 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-4
SLIDE 4

OS Dependability Threats

3 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-5
SLIDE 5

OS Dependability Threats

3 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-6
SLIDE 6

Are Core Components Safe?

4 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-7
SLIDE 7

Are Core Components Safe?

”We’re getting bloated and huge.

4 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-8
SLIDE 8

Are Core Components Safe?

”We’re getting bloated and huge. Yes, it’s a problem.

4 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-9
SLIDE 9

Are Core Components Safe?

”We’re getting bloated and huge. Yes, it’s a problem. [. . .] I’d like to say we have a plan.”

4 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-10
SLIDE 10

Are Core Components Safe?

”We’re getting bloated and huge. Yes, it’s a problem. [. . .] I’d like to say we have a plan.” Linus Torvalds on the Linux kernel, 2009

4 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-11
SLIDE 11

High-coverage Crash Recovery

Rapid evolution and huge size cause more bugs Crash recovery solution with smaller TCB needed Whole-OS crash recovery

5 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-12
SLIDE 12

High-coverage Crash Recovery

Rapid evolution and huge size cause more bugs Crash recovery solution with smaller TCB needed Whole-OS crash recovery

How?

5 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-13
SLIDE 13

High-coverage Crash Recovery

Rapid evolution and huge size cause more bugs Crash recovery solution with smaller TCB needed Whole-OS crash recovery

How?

  • 1. Extend existing work on isolated subsystems to the entire OS

5 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-14
SLIDE 14

High-coverage Crash Recovery

Rapid evolution and huge size cause more bugs Crash recovery solution with smaller TCB needed Whole-OS crash recovery

How?

  • 1. Extend existing work on isolated subsystems to the entire OS
  • 2. Design a new high-coverage crash recovery infrastructure

5 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-15
SLIDE 15

Isolated Subsystems

? Entire OS

Work on extensions and drivers

e.g., Safedrive, Nooks, Minix 3

Filesystems

e.g., Membrane

Assume isolated untrusted parties with well-defined interfaces Several recoverer-recoveree pairs to scale to the entire OS

Complex and hard-to-maintain recovery infrastructure

High exposure of the recovery code to the programmer

6 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-16
SLIDE 16

Isolated Subsystems

? Entire OS

Work on extensions and drivers

e.g., Safedrive, Nooks, Minix 3

Filesystems

e.g., Membrane

Assume isolated untrusted parties with well-defined interfaces Several recoverer-recoveree pairs to scale to the entire OS

Complex and hard-to-maintain recovery infrastructure

High exposure of the recovery code to the programmer

. . . it is like a dog chasing its tail!

6 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-17
SLIDE 17

Emerging High-coverage Solutions

Shadow kernel vs Pure instrumentation e.g., Otherworld e.g., Recovery Domains

7 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-18
SLIDE 18

Emerging High-coverage Solutions

Shadow kernel vs Pure instrumentation e.g., Otherworld e.g., Recovery Domains Best-effort (weak failure model)

7 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-19
SLIDE 19

Emerging High-coverage Solutions

Shadow kernel vs Pure instrumentation e.g., Otherworld e.g., Recovery Domains Best-effort (weak failure model) Heavyweight (high complexity) (poor performance) (poor scalability)

7 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-20
SLIDE 20

WWW: What We Want

8 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-21
SLIDE 21

WWW: What We Want High coverage

8 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-22
SLIDE 22

WWW: What We Want Low complexity

8 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-23
SLIDE 23

WWW: What We Want Reasonable performance and scalability

8 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-24
SLIDE 24

WWW: What We Want Good maintainability

8 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-25
SLIDE 25

WWW: What We Want Address the many challenges of the crash recovery problem

8 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-26
SLIDE 26

The Crash Recovery Problem — I

Crash detection Detect crashes proactively or reactively Isolate crashes so they do not disrupt the recovery process

9 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-27
SLIDE 27

The Crash Recovery Problem — I

Crash detection Detect crashes proactively or reactively Isolate crashes so they do not disrupt the recovery process State transfer Create a new execution context to restart execution Transfer the state from the old execution context

9 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-28
SLIDE 28

The Crash Recovery Problem — I

Crash detection Detect crashes proactively or reactively Isolate crashes so they do not disrupt the recovery process State transfer Create a new execution context to restart execution Transfer the state from the old execution context State consistency Restore a stable and consistent state in the new context Allow for deterministic execution upon restart

9 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-29
SLIDE 29

The Crash Recovery Problem — II

State dependency tracking Preserve state dependencies among different contexts Allow for a globally coherent state upon restart

10 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-30
SLIDE 30

The Crash Recovery Problem — II

State dependency tracking Preserve state dependencies among different contexts Allow for a globally coherent state upon restart State corruption Detect arbitrary data corruption Attempt to recover from arbitrary data corruption

10 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-31
SLIDE 31

The Crash Recovery Problem — II

State dependency tracking Preserve state dependencies among different contexts Allow for a globally coherent state upon restart State corruption Detect arbitrary data corruption Attempt to recover from arbitrary data corruption Restart Determine a safe execution point to resume operation Attempt to avoid further crashes

10 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-32
SLIDE 32

Our Approach

Combine OS design and lightweight instumentation

11 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-33
SLIDE 33

Our Approach

Combine OS design and lightweight instumentation

OS Design Reduce complexity at recovery time Good performance and scalability

11 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-34
SLIDE 34

Our Approach

Combine OS design and lightweight instumentation

OS Design Reduce complexity at recovery time Good performance and scalability Lightweight Compiler-based Instrumentation High coverage and component-agnostic recovery Good maintainability and evolvability

11 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-35
SLIDE 35

OS Architecture

App App App App . . . App NET SCH VFS VM . . . PM NDD HDD PRN SND . . . RS Microkernel R3 R0

We break down the OS into several userspace components Multiserver microkernel architecture based on message-passing

12 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-36
SLIDE 36

The Programming Model

O.S. Component

We rely on an event-driven model

13 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-37
SLIDE 37

The Programming Model

O.S. Component

Events trigger execution of the task loop

13 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-38
SLIDE 38

The Programming Model

O.S. Component

Idempotent messages possible within the task loop

13 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-39
SLIDE 39

The Programming Model

O.S. Component

Idempotent messages possible within the task loop

13 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-40
SLIDE 40

The Programming Model

O.S. Component

Idempotent messages possible within the task loop

13 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-41
SLIDE 41

The Programming Model

O.S. Component

Push non-idempotent messages to the end

13 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-42
SLIDE 42

The Programming Model

O.S. Component

Back to the top of the loop!

13 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-43
SLIDE 43

The Programming Model

O.S. Component

Pending interactions are remembered in the state

13 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-44
SLIDE 44

State Management

Identify state of data and state of execution Both well-defined and consistent at the top of the task loop The top of the loop is a local stable state point Global state consistency by design

14 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-45
SLIDE 45

Instrumentation-based Recovery

The task loop is the recovery window Lightweight instrumentation to track local state changes Used by the recovery code to revert to the last stable state Different strategies possible

15 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-46
SLIDE 46

Our Implemented Instrumentation

Maintain shadow state regions Track dynamic memory allocations Track changes on state objects Use alias analysis to detect changes at the object granularity Automatically commit changes at the end of the task loop (i.e., it synchronizes shadow and main state regions)

16 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-47
SLIDE 47

The Crash Recovery Process

App App App App . . . App NET SCH VFS VM . . . PM PMR NDD HDD PRN SND . . . RS Microkernel R3 R0

17 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-48
SLIDE 48

The Crash Recovery Process

App App App App . . . App NET SCH VFS VM . . . PM PMR

#PF

NDD HDD PRN SND . . . RS Microkernel R3 R0 1 1

An OS component crashes: the system manager detects the crash and initiates recovery (the microkernel actually signals the system manager)

17 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-49
SLIDE 49

The Crash Recovery Process

App App App App . . . App NET SCH VFS VM . . . PMR NDD HDD PRN SND . . . RS Microkernel R3 R0 PM 2 2

The system manager selects a new replica and tells the microkernel (virtual ids make transparent recovery possible!)

17 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-50
SLIDE 50

The Crash Recovery Process

App App App App . . . App NET SCH VFS VM . . . PMR NDD HDD PRN SND . . . RS Microkernel R3 R0 PM 3 PM

The system manager yields control to the new replica for state transfer. . . (libary-based recovery code starts executing. . . )

17 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-51
SLIDE 51

The Crash Recovery Process

App App App App . . . App NET SCH VFS VM . . . PMR NDD HDD PRN SND . . . RS Microkernel R3 R0 PM 3 PM

. . . the component is brought back to the last stable state and resumes operation (shadow and main state regions are synced!)

17 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-52
SLIDE 52

The Crash Recovery Process

App App App App . . . App NET SCH VFS VM . . . NDD HDD PRN SND . . . RS Microkernel R3 R0 PM PM 4

The system manager cleans up the dead replica (the new replica may even be involved in the process!)

17 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-53
SLIDE 53

The Crash Recovery Process

App App App App . . . App NET SCH VFS VM . . . NDD HDD PRN SND . . . RS Microkernel R3 R0 PMR PM 5

The system manager spawns a new replica (if needed) (per-component recovery policies apply)

17 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-54
SLIDE 54

The Crash Recovery Process

App App App App . . . App NET SCH VFS VM . . . NDD HDD PRN SND . . . RS Microkernel R3 R0 PMR PM

The system manager spawns a new replica (if needed) (per-component recovery policies apply)

The system keeps running as nothing bad ever happened!

17 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-55
SLIDE 55

Prototype

Implemented on top of Minix 3 Restructured OS processes to fit our event-driven model Instrumentation implemented as a series of LLVM passes Successfully recovered even the most critical components Early experiments confirmed key properties of our design

18 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-56
SLIDE 56

Scalability Properties

0.5 0.6 0.7 0.8 0.9 1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Normalized Relative Overhead N POSIX Suite Postmark

19 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-57
SLIDE 57

Summary

A new high-coverage approach to OS crash recovery Combines OS design and compiler-based instrumentation Low complexity, good performance, scalability, maintainability No heavy burden for the OS programmer Addresses many of the crash recovery challenges efficiently

20 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-58
SLIDE 58

Future Work

Finer-grained instrumentation to track the state Realistic fault injection scenarios Experiment and evaluate restart strategies Recover from state corruption Per-component recovery policies

21 We Crashed, Now What? Cristiano Giuffrida, Lorenzo Cavallaro, Andrew S. Tanenbaum

slide-59
SLIDE 59

We Crashed, Now What? Thank you! Any questions?

Cristiano Giuffrida, Lorenzo Cavallaro, Andy Tanenbaum {giuffrida,sullivan,ast}@cs.vu.nl

Vrije Universiteit Amsterdam