Goals for Today Learning Objective: Learn how to achieve - - PowerPoint PPT Presentation

goals for today
SMART_READER_LITE
LIVE PREVIEW

Goals for Today Learning Objective: Learn how to achieve - - PowerPoint PPT Presentation

Goals for Today Learning Objective: Learn how to achieve reliability + availability in storage!! Announcements, etc: MP3 is out! Due April 18th . MP3 Walkthrough on Monday, slides up later today Reminder : Please put away devices


slide-1
SLIDE 1

CS 423: Operating Systems Design 1

Goals for Today

Reminder: Please put away devices at the start of class

  • Learning Objective:
  • Learn how to achieve reliability + availability in storage!!
  • Announcements, etc:
  • MP3 is out! Due April 18th.
  • MP3 Walkthrough on Monday, slides up later today
slide-2
SLIDE 2

CS 423: Operating Systems Design

Professor Adam Bates Spring 2018

CS 423
 Operating System Design: Reliable Storage

slide-3
SLIDE 3

CS 423: Operating Systems Design

Storage is hard ; - (

3

“In each cluster's first year, it's typical that 1,000 individual machine failures will occur; thousands of hard drive failures will occur; one power distribution unit will fail, bringing down 500 to 1,000 machines for about 6 hours; 20 racks will fail, each time causing 40 to 80 machines to vanish from the network; 5 racks will "go wonky," with half their network packets missing in action; and the cluster will have to be rewired once, affecting 5 percent of the machines at any given moment over a 2-day span, Dean said. And there's about a 50 percent chance that the cluster will overheat, taking down most of the servers in less than 5 minutes and taking 1 to 2 days to recover.”

  • Jeff Dean, Google Fellow (2008)
slide-4
SLIDE 4

CS 423: Operating Systems Design 4

Storage reliability: data fetched is what you stored

■ Problem when machines randomly fail! ■

Storage availability: data is there when you want it

■ Problem when disks randomly fail! ■ More disks => higher probability of some disk failing ■ Data available ~ Prob(disk working)^k ■ If failures are independent and data is spread across k

disks

■ For large k, probability system works -> 0

Storage Goals

slide-5
SLIDE 5

CS 423: Operating Systems Design

File System Reliability

5

What can happen if disk loses power or machine software crashes?

■ Some operations in progress may complete ■ Some operations in progress may be lost ■ Overwrite of a block may only partially complete ■

File systems need durability (as a minimum!)

■ Data previously stored can be retrieved (maybe after some

recovery step), regardless of failure

slide-6
SLIDE 6

CS 423: Operating Systems Design

Storage Reliability Problem

6

Single logical file operation can involve updates to multiple physical disk blocks

■ inode, indirect block, data block, bitmap, … ■

At a physical level, operations complete one at a time

■ Want concurrent operations for performance ■

How do we guarantee consistency regardless of when crash

  • ccurs?
slide-7
SLIDE 7

CS 423: Operating Systems Design

Transaction Concept

7

A transaction is a grouping of low-level operations that are related to a single logical operation

Transactions are atomic — operations appear to happen as a group, or not at all (at logical level)

At physical level of course, only a single disk/flash write is atomic

Transactions are durable — operations that complete stay completed

Future failures do not corrupt previously stored data

(In-Progress) Transactions are isolated — other transactions cannot see the results of earlier transactions until they are committed

Transactions exhibit consistency — sequential memory model

slide-8
SLIDE 8

CS 423: Operating Systems Design

Reliability Attempt #1: Careful Ordering

8

Sequence operations in a specific order

Careful design to allow sequence to be interrupted safely

Post-crash recovery

Read data structures to see if there were any operations in progress

Clean up/finish as needed

Approach taken in FAT, FFS (fsck), and many app-level recovery schemes (e.g., Word)

slide-9
SLIDE 9

CS 423: Operating Systems Design

Reliability Attempt #1: Careful Ordering

9

FAT: Append Data to File

Add data block

Add pointer to data block

Update file tail to point to new MFT entry

Update access time at head of file

fjle 9 block 3 fjle 9 block 0 fjle 9 block 1 fjle 9 block 2 fjle 12 block 0 fjle 12 block 1 fjle 9 block 4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

MFT Data Blocks

slide-10
SLIDE 10

CS 423: Operating Systems Design

Reliability Attempt #1: Careful Ordering

10

FAT: Append Data to File

Add data block

Add pointer to data block

Update file tail to point to new MFT entry

Update access time at head of file Recovery

Scan MFT

If entry is unlinked, delete data block

If access time is incorrect, update

fjle 9 block 3 fjle 9 block 0 fjle 9 block 1 fjle 9 block 2 fjle 12 block 0 fjle 12 block 1 fjle 9 block 4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

MFT Data Blocks

slide-11
SLIDE 11

CS 423: Operating Systems Design

Reliability Attempt #1: Careful Ordering

11

FAT: Create New File

Allocate data block

Update MFT entry to point to data block

Update directory with file name -> file number

What if directory spans multiple disk blocks?

Update modify time for directory fjle 9 block 3 fjle 9 block 0 fjle 9 block 1 fjle 9 block 2 fjle 12 block 0 fjle 12 block 1 fjle 9 block 4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

MFT Data Blocks

slide-12
SLIDE 12

CS 423: Operating Systems Design

Reliability Attempt #1: Careful Ordering

12

FAT: Create New File

Allocate data block

Update MFT entry to point to data block

Update directory with file name -> file number

What if directory spans multiple disk blocks?

Update modify time for directory Recovery

Scan MFT

If any unlinked files (not in any directory), delete

Scan directories for missing update times fjle 9 block 3 fjle 9 block 0 fjle 9 block 1 fjle 9 block 2 fjle 12 block 0 fjle 12 block 1 fjle 9 block 4 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

MFT Data Blocks

slide-13
SLIDE 13

CS 423: Operating Systems Design

Reliability Attempt #1: Careful Ordering

13

FFS: Create New File

■ Allocate data block ■ Write data block ■ Allocate inode ■ Write inode block ■ Update bitmap of free blocks ■ Update directory with

file name -> file number

■ Update modify time for directory

Inode Array

File Metadata Indirect Pointer

  • Dbl. Indirect Ptr.
  • Tripl. Indirect Ptr.

Inode Indirect Blocks Double Indirect Blocks Triple Indirect Blocks

DP Direct Pointer DP DP DP DP DP DP DP DP DP Direct Pointer

slide-14
SLIDE 14

CS 423: Operating Systems Design

Reliability Attempt #1: Careful Ordering

14

FFS: Create New File

■ Allocate data block ■ Write data block ■ Allocate inode ■ Write inode block ■ Update bitmap of free blocks ■ Update directory with

file name -> file number

■ Update modify time for directory

Recovery

■ Scan inode table ■ If any unlinked files (not in any

directory), delete

■ Compare free block bitmap against inode

trees

■ Scan directories for missing update/

access times

Recovery time is proportional to size of disk!

Inode Array

File Metadata Indirect Pointer

  • Dbl. Indirect Ptr.
  • Tripl. Indirect Ptr.

Inode Indirect Blocks Double Indirect Blocks Triple Indirect Blocks

DP Direct Pointer DP DP DP DP DP DP DP DP DP Direct Pointer

slide-15
SLIDE 15

CS 423: Operating Systems Design

Reliability Attempt #1: Careful Ordering

15

FFS: Move a File

■ Remove filename from old

directory

■ Add filename to new directory

Inode Array

File Metadata Indirect Pointer

  • Dbl. Indirect Ptr.
  • Tripl. Indirect Ptr.

Inode Indirect Blocks Double Indirect Blocks Triple Indirect Blocks

DP Direct Pointer DP DP DP DP DP DP DP DP DP Direct Pointer

slide-16
SLIDE 16

CS 423: Operating Systems Design

Reliability Attempt #1: Careful Ordering

16

FFS: Move a File

■ Remove filename from old

directory

■ Add filename to new directory

Recovery

■ Scan all directories to

determine set of live files

■ Consider files with valid

inodes and not in any directory

New file being created?

File move?

File deletion?

Inode Array

File Metadata Indirect Pointer

  • Dbl. Indirect Ptr.
  • Tripl. Indirect Ptr.

Inode Indirect Blocks Double Indirect Blocks Triple Indirect Blocks

DP Direct Pointer DP DP DP DP DP DP DP DP DP Direct Pointer

slide-17
SLIDE 17

CS 423: Operating Systems Design

Reliability Attempt #1: Careful Ordering

17

Application Level

■ Write name of each open file to

app folder

■ Write changes to backup file ■ Rename backup file to be file

(atomic operation provided by file system)

■ Delete list in app folder on clean

shutdown Recovery

■ On startup, see if any files were

left open

■ If so, look for backup file ■ If so, ask user to compare

versions

slide-18
SLIDE 18

CS 423: Operating Systems Design

Reliability Attempt #1: Careful Ordering

18

FFS: Move and Grep

■ Observation — careful ordering is not a panacea… ■ Will Process B always see the contents of the file?

Process A moves file from x to y

mv x/file y/

Process B greps across x and y

grep x/* y/*

slide-19
SLIDE 19

CS 423: Operating Systems Design

Reliability Attempt #1: Careful Ordering

19

Pros

■ Works with minimal support from the disk drive ■ Works for most multi-step operations

Cons

■ Can require time-consuming recovery after a failure ■ Difficult to reduce every operation to a safely-interruptible

sequence of writes

■ Difficult to achieve consistency when multiple operations

  • ccur concurrently (e.g., FFS grep)
slide-20
SLIDE 20

CS 423: Operating Systems Design

Reliability Attempt #2: Copy-on-Write

20

To update file system, write a new version of the file system containing the update

■ Never update in place ■ Reuse existing unchanged disk blocks ■

Seems expensive! But…

■ Updates can be batched ■ Almost all disk writes can occur in parallel ■

Approach taken in network file server appliances (WAFL, ZFS)

slide-21
SLIDE 21

CS 423: Operating Systems Design

Reliability Attempt #2: Copy-on-Write

21

Indirect Blocks Data Blocks Inode Array (in Inode File) Fixed Location Anywhere Root Inode Slots Inode File’s Indirect Blocks Indirect Blocks Data Blocks Inode Array (in Inode File) Root Inode Slots Inode File’s Indirect Blocks Update Last Block of File

Copy on Write (Write Anywhere File Layout)

slide-22
SLIDE 22

CS 423: Operating Systems Design

Reliability Attempt #2: Copy-on-Write

22

Indirect Blocks Data Blocks Inode Array (in Inode File) Fixed Location Anywhere Root Inode Slots Inode File’s Indirect Blocks Indirect Blocks Data Blocks Inode Array (in Inode File) Root Inode Slots Inode File’s Indirect Blocks Update Last Block of File

Copy on Write (Write Anywhere File Layout)

slide-23
SLIDE 23

CS 423: Operating Systems Design

Reliability Attempt #2: Copy-on-Write

23

Batch Updates

Root Inode Root Inode’s Indirect Blocks Inode File File’s Indirect Blocks File’s Data Blocks New Data Blocks New Data Block of Inode File New Indirect Nodes New Indirect Nodes of Inode File New Root Inode

slide-24
SLIDE 24

CS 423: Operating Systems Design

Reliability Attempt #2: Copy-on-Write

24

Batch Update

Root Inode Root Inode’s Indirect Blocks Inode File File’s Indirect Blocks File’s Data Blocks New Data Blocks New Data Block of Inode File New Indirect Nodes New Indirect Nodes of Inode File New Root Inode

slide-25
SLIDE 25

CS 423: Operating Systems Design

Reliability Attempt #2: Copy-on-Write

25

FFS Updates (updates are in-place)

Update Bitmap Update Inode Update Indirect Block New Data Block

slide-26
SLIDE 26

CS 423: Operating Systems Design

Old Bitmap Old Inode Old Indirect Block Update Inode Update Indirect Block Update Bitmap New Data Block

Reliability Attempt #2: Copy-on-Write

26

Write Anywhere File Layout (WAFL) Updates (Uses Copy-on-Write)

slide-27
SLIDE 27

CS 423: Operating Systems Design

Reliability Attempt #2: Copy-on-Write

27

Garbage Collection

For write efficiency, want contiguous sequences of free blocks

■ Spread across all block groups ■ Updates leave dead blocks scattered ■

For read efficiency, want data read together to be in the same block group

■ Write anywhere leaves related data scattered ■

Solution? Background coalescing of live/dead blocks

slide-28
SLIDE 28

CS 423: Operating Systems Design 28

Pros

■ Correct behavior regardless of failures ■ Fast recovery (root block array) ■ High throughput (best if updates are batched)

Cons

■ Potential for high latency ■ Small changes require many writes ■ Garbage collection essential for performance

Reliability Attempt #2: Copy-on-Write

slide-29
SLIDE 29

CS 423: Operating Systems Design

Logging File Systems

29

Instead of modifying data structures on disk directly, write changes to a journal/log

■ Intention list: set of changes we intend to make ■ Log/Journal is append-only ■

Once changes are on log, safe to apply changes to data structures on disk

■ Recovery can read log to see what changes were intended ■

Once changes are copied, safe to remove log

slide-30
SLIDE 30

CS 423: Operating Systems Design

Redo Logging

30

■ Prepare

Write all changes (in transaction) to log

■ Commit

Single disk write to make transaction durable

■ Redo

Copy changes to disk

■ Garbage collection

Reclaim space in log

Recovery

Read log

Redo any operations for committed transactions

Garbage collect log

slide-31
SLIDE 31

CS 423: Operating Systems Design 31

Before transaction start

Log:

Storage

Mike = $100 Tom = $200 Mike = $100 Tom = $200

Cache Nonvolatile

Redo Logging

slide-32
SLIDE 32

CS 423: Operating Systems Design 32 Tom = $100 Mike = $200

Storage

Mike = $100 Tom = $200 Mike = $200 Tom = $100

Cache

Log:

Nonvolatile

Redo Logging

After Updates are Logged

slide-33
SLIDE 33

CS 423: Operating Systems Design

After commit logged

33

Redo Logging

Tom = $100 Mike = $200 COMMIT

Storage

Mike = $200 Tom = $100 Mike = $200 Tom = $100

Cache

Log:

Nonvolatile

slide-34
SLIDE 34

CS 423: Operating Systems Design

After copy back

34

Redo Logging

Log:

Storage

Mike = $200 Tom = $100 Mike = $200 Tom = $100

Cache Nonvolatile

slide-35
SLIDE 35

CS 423: Operating Systems Design

After garbage collection

35

Redo Logging

Log:

Storage

Mike = $200 Tom = $100 Mike = $200 Tom = $100

Cache Nonvolatile

slide-36
SLIDE 36

CS 423: Operating Systems Design 36

Questions

What happens if machine crashes?

■ Before transaction start ■ After transaction start, before operations are logged ■ After operations are logged, before commit ■ After commit, before write back ■ After write back before garbage collection ■

What happens if machine crashes during recovery?

Redo Logging

slide-37
SLIDE 37

CS 423: Operating Systems Design 37

Performance

Log written sequentially

■ Often kept in flash storage ■

Asynchronous write back

■ Any order as long as all changes are logged before commit,

and all write backs occur after commit

Can process multiple transactions

■ Transaction ID in each log entry ■ Transaction completed iff its commit record is in log

Redo Logging

slide-38
SLIDE 38

CS 423: Operating Systems Design

Transaction Isolation

38

Process A moves file from x to y

mv x/file y/

Process B greps across x and y

grep x/* y/*

■ What if grep starts after changes are logged but before they

are commited?

slide-39
SLIDE 39

CS 423: Operating Systems Design

■ What if grep starts after changes are logged but before they

are commited?

■ Two Phase Locking: Release locks only AFTER transaction

commit.

Prevents a process from seeing results of a transaction that might not commit!

Transaction Isolation

39

Process A moves file from x to y

mv x/file y/

Process B greps across x and y

grep x/* y/*

Process A moves file from x to y

Lock x, y mv x/file y/ Commit & Release x, y

Process B greps across x and y

Lock x, y grep x/* y/* Release x, y

slide-40
SLIDE 40

CS 423: Operating Systems Design 40

With two phase locking and redo logging, transactions appear to occur in a sequential order (serializability)

■ Either: grep then move or move then grep ■

Other implementations can also provide serializability

■ e.g., Optimistic concurrency control: abort any transaction

that would conflict with serializability

Serializability

slide-41
SLIDE 41

CS 423: Operating Systems Design 41

Storage reliability: data fetched is what you stored

■ Transactions, redo logging, etc. ■

Storage availability: data is there when you want it

■ More disks => higher probability of some disk failing ■ Data available ~ Prob(disk working)^k ■ If failures are independent and data is spread across k

disks

■ For large k, probability system works -> 0

Storage Availability

slide-42
SLIDE 42

CS 423: Operating Systems Design

RAID

42

“Redundant Array of Inexpensive Disks”

Multiple disk drives provide reliability via redundancy.

Speeds up access times even beyond sequential.

Increases the mean time to failure

slide-43
SLIDE 43

CS 423: Operating Systems Design

RAID

43

RAID

■ multiple disks work cooperatively ■ Improve reliability by storing redundant data ■ Striping (RAID 0) improves performance with disk

striping (use a group of disks as one storage unit)

■ Mirroring (RAID 1) keeps duplicate of each disk ■ Striped mirrors (RAID 1+0) or mirrored stripes (RAID

0+1) provides high performance and high reliability

■ Block interleaved parity (RAID 4, 5, 6) uses much less

redundancy

slide-44
SLIDE 44

CS 423: Operating Systems Design

RAID Level 0

44

Level 0 is nonredundant disk array

Files are striped across disks, no redundant info

High read throughput

Best write throughput (no redundant info to write)

Any disk failure results in data loss

slide-45
SLIDE 45

CS 423: Operating Systems Design

RAID Level 1

45

Mirrored Disks

Data is written to two places

On failure, just use surviving disk (easy to rebuild)

On read, choose fastest to read

Write performance is same as single drive, read performance is 2x better

Expensive (high space

  • verhead)
slide-46
SLIDE 46

CS 423: Operating Systems Design

RAID Level 0+1

46

Stripe on a set of disks

Then mirror of data blocks is striped on the second set.

slide-47
SLIDE 47

CS 423: Operating Systems Design

RAID Level 1+0

47

Pair mirrors first.

Then stripe on a set of paired mirrors