Optimistic Crash Consistency Vijay Chidambaram Thanumalayan - PowerPoint PPT Presentation

Optimistic Crash Consistency Vijay Chidambaram Thanumalayan Sankaranarayana Pillai Andrea Arpaci-Dusseau Remzi Arpaci-Dusseau

Crash Consistency Problem Single file-system operation updates multiple on-disk data structures System may crash in middle of updates File-system is partially (incorrectly) updated SOSP 13 2

Performance OR Consistency Crash-consistency solutions degrade performance Users forced to choose between high performance and strong consistency - Performance differs by 10x for some workloads Many users choose performance - ext3 default configuration did not guarantee crash consistency for many years - Mac OSX fsync() does not ensure data is safe “ The Fast drives out the Slow even if the Fast is wrong ” - Kahan SOSP 13 3

Ordering and Durability Crash consistency is built upon ordered writes File systems conflate ordering and durability - Ideal: {A, B} -> {C} (made durable later) - Current scenario • {A, B} durable • {C} durable Inefficient when only ordering is required SOSP 13 4

Can a file system provide both high performance and strong consistency? Is there a middle ground between: high performance but no consistency strong consistency but low performance? SOSP 13 5

Our solution Optimistic File System (OptFS) Journaling file system that provides performance and consistency by decoupling ordering and durability Such decoupling allows OptFS to trade freshness for performance while maintaining crash consistency SOSP 13 6

Results Techniques: checksums, delayed writes, etc. OptFS provides strong consistency - Equivalent to ext4 data journaling OptFS improves performance significantly - 10x better than ext4 on some workloads New primitive osync() provides ordering among writes at high performance SOSP 13 7

Outline Introduction Ordering and Durability in Journaling Optimistic File System Results Conclusion SOSP 13 8

Outline Introduction Ordering and Durability in Journaling - Journaling Overview - Realizing Ordering on Disks - Journaling without Ordering Optimistic File System Results Conclusion SOSP 13 9

Journaling Overview Before updating file system, write note describing update Make sure note is safely on disk Once note is safe, update file system - If interrupted, read note and redo updates SOSP 13 10

Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) DATA METADATA APPLICATION FILE SYSTEM DISK SOSP 13 11 Journal

Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) DATA METADATA APPLICATION FILE SYSTEM D DISK SOSP 13 11 Journal

Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) DATA METADATA APPLICATION FILE SYSTEM DISK D SOSP 13 11 Journal

Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (J M ) DATA METADATA APPLICATION FILE SYSTEM J M DISK D SOSP 13 11 Journal

Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (J M ) DATA METADATA APPLICATION FILE SYSTEM DISK D J M SOSP 13 11 Journal

Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (J M ) - Logging Commit (J C ) DATA METADATA APPLICATION FILE SYSTEM J C DISK D J M SOSP 13 11 Journal

Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (J M ) - Logging Commit (J C ) DATA METADATA APPLICATION FILE SYSTEM DISK D J M J C SOSP 13 11 Journal

Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (J M ) - Logging Commit (J C ) - Checkpointing (M) DATA METADATA APPLICATION FILE SYSTEM M DISK D J M J C SOSP 13 11 Journal

Journaling Overview Workload: Creating and writing to a file Journaling protocol (ordered journaling) - Data write (D) - Logging Metadata (J M ) - Logging Commit (J C ) - Checkpointing (M) DATA METADATA APPLICATION FILE SYSTEM DISK D M J M J C SOSP 13 11 Journal

How Writes are Ordered Original Disks with Disks Write Buffers A B A B A Flush B Disk B A B Cache Disk Disk A A B Platter SOSP 13 13

Journaling with Flushes Journaling protocol - Data write (D) DATA METADATA APPLICATION FILE SYSTEM DISK CACHE DISK PLATTER Journal SOSP 13 14

Journaling with Flushes Journaling protocol - Data write (D) DATA METADATA APPLICATION FILE SYSTEM D DISK CACHE DISK PLATTER Journal SOSP 13 14

Journaling with Flushes Journaling protocol - Data write (D) DATA METADATA APPLICATION FILE SYSTEM DISK CACHE D DISK PLATTER Journal SOSP 13 14

Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (J M ) DATA METADATA APPLICATION FILE SYSTEM J M DISK CACHE D DISK PLATTER Journal SOSP 13 14

Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (J M ) DATA METADATA APPLICATION FILE SYSTEM DISK CACHE D J M DISK PLATTER Journal SOSP 13 14

Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (J M ) DATA METADATA APPLICATION FILE SYSTEM DISK CACHE D J M FLUSH DISK PLATTER Journal SOSP 13 14

Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (J M ) DATA METADATA APPLICATION FILE SYSTEM DISK CACHE FLUSH DISK PLATTER D J M Journal SOSP 13 14

Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (J M ) - Logging Commit (J C ) DATA METADATA APPLICATION FILE SYSTEM J C DISK CACHE FLUSH DISK PLATTER D J M Journal SOSP 13 14

Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (J M ) - Logging Commit (J C ) DATA METADATA APPLICATION FILE SYSTEM DISK CACHE J C FLUSH DISK PLATTER D J M Journal SOSP 13 14

Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (J M ) - Logging Commit (J C ) DATA METADATA APPLICATION FILE SYSTEM DISK CACHE J C FLUSH FLUSH DISK PLATTER D J M Journal SOSP 13 14

Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (J M ) - Logging Commit (J C ) DATA METADATA APPLICATION FILE SYSTEM DISK CACHE FLUSH FLUSH DISK PLATTER D J M J C Journal SOSP 13 14

Journaling with Flushes Journaling protocol - Data write (D) - Logging Metadata (J M ) - Logging Commit (J C ) - Checkpointing (M) DATA METADATA APPLICATION FILE SYSTEM M DISK CACHE FLUSH FLUSH DISK PLATTER D J M J C Journal SOSP 13 14

Journaling without Ordering Practitioners turn off flushes due to performance degradation - Ex: ext3 by default did not enable flushes for many years Observe crashes do not cause inconsistency for some workloads We term this probabilistic crash consistency - Studied in detail SOSP 13 16

Journaling without Ordering DATA METADATA APPLICATION FILE SYSTEM D M J M J C DISK CACHE FLUSH FLUSH DISK PLATTER Journal SOSP 13 17

Journaling without Ordering DATA METADATA APPLICATION FILE SYSTEM D M J M J C DISK CACHE DISK PLATTER Journal SOSP 13 17

Journaling without Ordering Without flushes, blocks may be reordered DATA METADATA APPLICATION FILE SYSTEM DISK CACHE D M J M J C DISK PLATTER Journal SOSP 13 17

Journaling without Ordering Without flushes, blocks may be reordered - Ex: J C and J M written first as disk head near journal DATA METADATA APPLICATION FILE SYSTEM DISK CACHE D M DISK PLATTER J M J C Journal SOSP 13 17

Journaling without Ordering Without flushes, blocks may be reordered - Ex: J C and J M written first as disk head near journal DATA METADATA APPLICATION FILE SYSTEM DISK CACHE DISK PLATTER D M J M J C Journal SOSP 13 17

Probabilistic Crash Consistency D M J M J C MEMORY Time DISK SOSP 13 18

Probabilistic Crash Consistency D M J M J C MEMORY Time J C DISK SOSP 13 18

Probabilistic Crash Consistency D M J M J C MEMORY Time D M J C J M DISK SOSP 13 18

Probabilistic Crash Consistency Re-ordering leads to windows of vulnerability D M J M J C MEMORY Time D M J C J M DISK Window Total I/O Time P-inconsistency = Time in window(s) / Total I/O Time SOSP 13 18

Optimistic Crash Consistency Vijay Chidambaram Thanumalayan - PowerPoint PPT Presentation

Optimistic Crash Consistency Vijay Chidambaram Thanumalayan Sankaranarayana Pillai Andrea Arpaci-Dusseau Remzi Arpaci-Dusseau Crash Consistency Problem Single file-system operation updates multiple on-disk data structures System may crash

The problem: crash consistency Single operaBon updates mulBple blocks System might crash in

Application Crash Consistency and Performance with CCFS Thanumalayan Sankaranarayana Pillai,

TxFS: Leveraging File-System Crash Consistency to Provide ACID Transactions Yige Hu, Zhiting

to Systematically Test File-System Crash Consistency Ashlie Martinez Vijay Chidambaram

From Crash Consistency to Transactions Yige Hu Youngjin Kwon Vijay Chidambaram Emmett Witchel

Specifying and Checking File System Crash-Consistency Models Steven Lang September 4, 2016

Web Cache Consistency Web Cache Consistency Web Cache Consistency Web Cache Consistency

Finding Crash-Consistency Bugs with Bounded Black-Box Testing Jayashree Mohan , Ashlie Martinez,

Understanding and Finding Crash-Consistency Bugs in Parallel File Systems Jinghan Sun , Chen Wang,

Specifying and Checking File System Crash-Consistency Models James Bornholt Antoine Kaufmann

Consistency - Chapter 5 Introduce several notions of Local Consistency: arc consistency,

Constraint Programming - An overview Node-consistency Arc-consistency Path-consistency

Exa- to Yotta-scale Data An Optimistic View Rob Farber PNNL Optimistic about Storage Bandwidth

Consistent Storage or Scalable Storage Why Not Both? CONSISTENCY Strong Consistency

Just-Right As available as possible, consistent when necessary Consistency The CAP theorem

Arizona Crash Report Presentation by Glen Robison State Custodian of Crash Records Prepared

Disks and I/O Scheduling Don Porter Portions courtesy Emmett Witchel 1 COMP 530: Operating

Tinea Capitis: Current Status R. J. Hay Received: 26 June 2016 / Accepted: 24 August 2016 /

GINKGOMED COMPANY TEACHING SLIDES ORAL PATHOLOGY OP0001 OP0019 Oral

Chapter 2 State Space Models General format : (valid for any nonlinear causal system) CT : x

Fault-Tolerance, Fast and Slow: Exploiting Failure Asynchrony in Distributed Systems Ramnatthan

A Modeling Approach for Storage Workloads Christina Delimitrou 1 , Sriram Sankar 2 , Kushagra Vaid

Disks and RAID CS 4410 Opera5ng Systems Spring 2017 Cornell University Lorenzo Alvisi Anne

CSE 232A Graduate Database Systems Arun Kumar Topic 1: Data Storage Chapters 8 and 9 of Cow