Scalable File Storage Jeff Chase Duke University Why a - - PowerPoint PPT Presentation
D D u k e S y s t t e m s Scalable File Storage Jeff Chase Duke University Why a shared network file service? Data sharing across people and their apps common name space (/usr/project/stuff )
D D u k e S y s t t e m s
syscall layer
VFS VFS
syscall layer
user programs
Virtual File System (VFS) enables pluggable file system implementations as OS kernel modules (“drivers”).
A master server stores metadata (names, file maps) and acts as lock server. Clients call master to open file, acquire locks, and obtain metadata. Then they read/write directly to a scalable array of data servers for the actual data. File data may be spread across many data servers: the maps say where it is.
Handles failures automatically, e.g., restarts tasks if a node fails; Runs multiple copies of a task so a slow node does not limit the job.
The client asks the master for a list of replicas, and which replica holds the lease to act as primary. If no one has a lease, the master grants a lease to a replica it
when the master wants to disable mutations on a file that is being renamed).
A master server stores metadata (names, file maps) and acts as lock server. Clients call master to open file, acquire locks, and obtain metadata. Then they read/write directly to a scalable array of data servers for the actual data. File data may be spread across many data servers: the maps say where it is.
We use leases to maintain a consistent mutation order across replicas. The master grants a chunk lease to one of the replicas, which we call the primary. The primary picks a serial order for all mutations to the chunk. All replicas follow this order when applying mutations. Thus, the global mutation order is defined first by the lease grant order chosen by the master, and within a lease by the serial numbers assigned by the primary. The lease mechanism is designed to minimize management overhead at the
chunk is being mutated, the primary can request and typically receive exten- sions from the master indefinitely. These extension requests and grants are piggybacked
safely grant a new lease to another replica after the old lease expires.
– lock service may recall or evict – holder may release or relinquish
acquire grant acquire
x=x+1
grant release x=x+1
– i.e., that its expiration time is in the past – Even if they can’t communicate!
– synchronized clocks
– |T(Ci) – T(Cj)| < ε – Build in slack time > ε into the lease protocols as a safety margin.
Cr ashed route r A network partition is any event that blocks all message traffic between subsets of nodes.
acquire grant acquire
x=x+1
grant release x=x+1
– rename – snapshots
...The master may sometimes try to revoke a lease before it expires (e.g., when the master wants to disable mutations on a file that is being renamed). Even if the master loses communication with a primary, it can safely grant a new lease to another replica after the old lease expires…. ...When the master receives a snapshot request, it first revokes any
ensures that any subsequent writes to these chunks will require an interaction with the master to find the lease holder. This will give the master an opportunity to create a new copy of the chunk first.....
– Master gives lease to a new primary. – The client’s answer may be cached and may be stale.
Since clients cache chunk locations, they may read from a stale replica before that information is refreshed. This window is limited by the cache entry’s timeout and the next open of the file, which purges from the cache all chunk information for that file.
– Master increments it when it issues a lease. – Client and replicas get it from the master.
– If replica fails/disconnects, its lease number lags. – Easy to detect by comparing lease numbers.
– If a replica misses updates to a chunk, its version falls behind – Client checks for stale chunks on reads. – Replicas report chunk versions to master: master reclaims any stale chunks, creates new replicas.
Whenever the master grants a new lease on a chunk, it increases the chunk version number and informs the up-to-date replicas…before any client is notified and therefore before it can start writing to the chunk. If another replica is currently unavailable, its chunk version number will not be advanced. The master will detect that this chunkserver has a stale replica when the chunkserver restarts and reports its set of chunks...
Primary chooses an arbitrary total
that cross chunk boundaries are not atomic: the two chunk primaries may choose different orders. Records appended atomically at least once. …but file may contain duplicates and/or padding. Anything can happen if writes fail: a failed write may succeed at some replicas but not others. Reads from different replicas may return different results.
– Summer internship at Microsoft Research – Now at Columbia
– GFS – BlueFS (MSR scalable storage behind Hotmail etc.)
35
Primary Client write ACK write read read value/error write write write ACK ACK/error Master:
ACK
36
n Counter-example 1 – Non-atomic writes
¨ In GFS, files are split into chunks, replicated by separate groups ¨ So, writes to file can be split and executed non-atomically
n Counter-example 2 – Stale reads
¨ In GFS, no strong membership checks are done for reads ¨ So, a read can go to a stale replica
n Counter-example 3 – Read uncommitted
¨ In GFS, a read can go to any replica ¨ So, a read can return the value of an in-progress write
Chunks write
– Primary pads to chunk boundary; client retries (on next chunk).
– May cause duplicates on subset of replicas where earlier write succeeded..
The primary checks to see if appending the record to the current chunk would cause the chunk to exceed the maximum size (64 MB). If so, it pads the chunk to the maximum size, tells secondaries to do the same, and replies to the client indicating that the operation should be retried on the next chunk. … If a record append fails at any replica, the client retries the operation. As a result, replicas of the same chunk may contain different data possibly including duplicates of the same record in whole or in part. GFS does not guarantee that all replicas are bytewise identical. It only guarantees that the data is written at least once as an atomic unit. …
– “Regardless of consistency and concurrency issues, this approach has served us well.”
Appending is far more efficient and more resilient to application failures than random writes. In one typical use, a writer generates a file from beginning to end. It … periodically checkpoints how much has been successfully written. Checkpoints may include application-level checksums. Readers verify and process only the file region up to the last checkpoint, which is known to be in the defined state. …Checkpointing allows writers to restart incrementally... …Moreover, as most of our files are append-only, a stale replica usually returns a premature end of chunk rather than outdated data. When a reader retries and contacts the master, it will immediately get current chunk locations.
the system doesn’t have to solve them.
GFS applications can accommodate the relaxed consistency model with a few simple techniques already needed for other purposes: relying on appends rather than overwrites, checkpointing, and writing self-validating, self-identifying records.... Readers deal with the occasional padding and duplicates as follows. Each record prepared by the writer contains extra information like checksums so that its validity can be verified. A reader can identify and discard extra padding and record fragments using the checksums. If it cannot tolerate the
which are often needed anyway…
– operation log on the master for namespace – chunk inventories are soft state on master
log snapshot
volatile memory log snapshot Your program (or file system
executes transactions that read or change the data structures in memory. Your data structures in memory Push a checkpoint or snapshot of the entire structure to disk periodically Log transaction events as they occur After a failure, replay the log into the last snapshot to recover the data structure.
recoverable storage (e.g., disk).
record progress of T.
complete detectably in order.
– Append/write entry to log – Truncate older entries up to time t
pushed to disk) before any effects of T become visible. – Called write-ahead logging
LSN 11 XID 18 LSN 13 XID 19 LSN 12 XID 18 LSN 14 XID 18 commit
Log Sequence Number (LSN) Transaction ID (XID) commit record
All metadata is kept in the master’s memory. The first two types (names- paces and file-to-chunk mapping) are also kept persistent by logging mutations to an operation log…. The operation log contains a historical [and persistent] record of critical metadata changes… it also serves as a logical time line that defines the order of concurrent operations…the master’s
The log records contain the sequence of commands/requests executed, rather than the updates made as they executed: this is called operational logging. The master recovers its file system state by replaying the operation log… loading the latest checkpoint from local disk and replaying only the limited number of log records after that. “Replay” the log by re-executing the operations listed in the log, in order. They are deterministic: the result is the same as their execution before the crash. First it rolls back to the state of the most recent checkpoint. Then it re-executes
the results from all preceding operations.
We must store [the operation log] reliably and not make changes visible to clients until metadata changes are made persistent. …Therefore, we replicate it on multiple remote machines and respond to a client operation only after flushing the corresponding log record to disk both locally and remotely. The master batches several log records together before flushing thereby reducing the impact of flushing and replication on overall system throughput. The log is made durable by writing copies to multiple disks/machines. The log writes must complete before the master responds to the operation request. This is a form of write-ahead logging also called output commit. This is a common optimization called group commit. It increases disk write bandwidth for the log at the price of increasing commit latency.
[Renu Tewari, IBM]
Asymmetric
Symmetric
pNFS Clients Block (FC) / Object (OSD) / File (NFS) Storage NFSv4+ Server
data [David Black, SNIA]
Modifications to standard NFS protocol (v4.1, 2005-2010) to offload bulk data storage to a scalable cluster of block servers or OSDs. Based on an asymmetric structure similar to GFS and Ceph.
specified elsewhere, e.g.
– SCSI Block Commands (SBC) over Fibre Channel (FC) – SCSI Object-based Storage Device (OSD) over iSCSI – Network File System (NFS)
pNFS Clients Block (FC) / Object (OSD) / File (NFS) Storage NFSv4+ Server
data [David Black, SNIA]
Clients Storage NFSv4+ Server
layout [David Black, SNIA]
– DNS, Web
– It may be OK for clients to read data that is “a little bit stale”. – In some cases, the clients themselves don’t change the data.
Applications
Applications
Applications
Applications
Applications
Applications
Applications Read(server=xx.xx…, inode=i27412, blockID=27, …)
Applications
Applications
Applications Write(server=xx.xx…, inode=i27412, blockID=27, …)
Applications
Applications
Applications What if either of the other clients reads that block? Will it get the right data? What is the “right” data? Will it get the “last” version of the block written? How to coordinate reads/writes and caching on multiple clients? How to keep the copies “in sync”?
– Must ensure that another client can see all updates before it is able to acquire a lease allowing it to read or write.
– Callback RPC from server to lock holder: “please release now.” – Writers get a grace period to push cached writes and release.
This approach is used in NFS and various other networked data services.
Atomic Consistent Independent Durable
– Each transaction either commits or aborts: it either executes entirely or not at all. – Transactions don’t interfere with one another (I).
Consistent state to another.
– Committed effects survive failure.
Transaction bodies must be coded correctly!
“commit or abort?” “here’s my vote” “commit/abort!” TM/C RM/P precommit
vote decide notify RMs validate Tx and prepare by logging their local updates and decisions TM logs commit/ abort (commit point) If unanimous to commit decide to commit else decide to abort
Invariant: ¡append ¡new ¡data ¡to ¡log ¡before ¡applying ¡to ¡DB ¡ Called ¡“write-‑ahead ¡logging” ¡
What ¡if ¡we ¡crash ¡here ¡(between ¡3,4)? ¡ On ¡reboot, ¡reapply ¡commiPed ¡updates ¡in ¡log ¡order. ¡
What ¡if ¡we ¡crash ¡here? ¡ On ¡reboot, ¡discard ¡uncommiPed ¡updates. ¡
– Redo logging: the log records contain sufficient info to “redo” T after a crash.
– “Redo” by reissuing writes or reinvoking the operations/methods. – Redo in order (old to new) – Skip the records of uncommitted Ts – Skip records of any Ts that committed before the atomic checkpoint was taken.
LSN 11 XID 18 LSN 13 XID 19 LSN 12 XID 18 LSN 14 XID 18 commit
Log Sequence Number (LSN) Transaction ID (XID) commit record
– No longer need the entries to recover.
– Checkpoints are expensive, BUT – Long logs take up space. – Long logs increase recovery time.
– Is it safe to redo/replay records whose effect is already in the checkpoint? – Checkpoint “between” transactions, so checkpoint records a consistent state. – Lots of approaches
LSN 11 XID 18 LSN 13 XID 19 LSN 12 XID 18 LSN 14 XID 18 commit
Log Sequence Number (LSN) Transaction ID (XID) commit record