The Google File System Sanjay Ghemawat, Howard Gobioff, and Shun- - - PowerPoint PPT Presentation

The Google File System

Sanjay Ghemawat, Howard Gobioff, and Shun- Tak Leung SOSP 2003 Presented by Wenhao Xu University of British Columbia

Outline

• Background
• Assumption and Google Workloads
• Architecture
• System Interaction
• Master
• Relaxed Consistency Model
• Experiment
• Conclusion
• Discussion

Background

• In 2003,

– 50+ GFS clusters – Each with thousands of storage nodes – Managing petabytes of data – GFS is under lots of applications and other systems in Google, such as bigtable, etc.

• Built on top of commodity machine and

disks

Tailored for Google workloads

Assumption & Google Workloads

• Failures occur everywhere

– Disks, Network, Machines

• Store a modest number of large files

– A few million files; Each typically 100MB or larger in size

• Read: Large streaming reads and small

random reads

– Typically read hundreds of KBs of an individual

• peration, more commonly 1MB or more

– Small reads typically read a few KBs at arbitrary

• ffsets, which are usually batched and sorted to

avoid seeking back and forth

• Mutation: Mostly appending new data
• Concurrent clients
• Design flexibility: Co-designing the

application and the file system API

High availability: Replicas, Heartbeat, Fast recovery, checksums Large data chunk & No Client data cache Make appending fast Relaxed consistency model & Atomicity Record Append Customizable API & Don’t need to conform to POSIX semantic! Throughput is more important

Architecture

A Chunk

• Size: 64MB
• 64 bit handler
• Stored as a Linux file,

lazy space allocation

• 3+ replicas

Single Master

• Single point of Failures?  Shadow

masters, operation log

• Performance bottlenecks?  Less

involvment

• Simplicity: concurrency control,

metadata management, etc.

• Global knowledge: make sophisticated

placement and replication decisions.

System interaction – Mutation Data flow

• Leases – Primary replica
• Master is only involved in

1 & 2

• Separated data flow and

control flow

– Employs network topology to transfer data – Easily to control the

• peration order

System Interaction - Heartbeat

• What is a heartbeat message like?

– “hey, I am still healthy. And I have chunks a, b, c …. Their versions are v1, v3, v9, …”

• What is heartbeat for?

– Chunkserver status – Chunk placement – Garbage collection – Stale Replica deletion – Lease extension

Master - Metadata Management

• Three major types

– File and chunk namespaces – (File, chunk) mapping – Location of each chunk’s replicas

• All metadata is kept in the memory

– Possibility: (<64B/chunk, 226 (64M) * 64 = 4G, a few millions

• f files)

– Fast access & scanning

• Operation log

– Store First two types of metadata – Any changes to the first two types of metadata are first written to the operation log – “It is replicated on multiple remote machines and respond to a client operation only after flushing the corresponding log record to disk both locally and remotely” – Checkpoints are made to avoid larger log and long starting up time for masters.

Master – Namespace Management & locking

• Namespace is organized into a lookup

table

– (file name, metadata) mapping – Does not support aliases for the same file

• r directory
• Mutation lock

– Read lock of all parents directory – Write lock of mutation directory – E.g: /d1/d2/…/dn/leaf creation

• Read lock: /d1, /d1/d2, …, /d1/d2/…/dn
• Write lock: /d1/d2/…/dn/leaf

Master - Replica Creation and Placement

• Consideration

– Balance disk space utilization – Performance: Limit the number of “recent” creations (hot pot) on each chunkserver – Availability: different racks; as far as possible – Maximize network utilization: as less switches as possible; exploit the aggregate bandwidth of multiple racks; – Multiple racks:

• Tradeoff: read bandwidth & write traffic
• Re-replication & rebalancing

Relaxed Consistency model

• Two state definitions:

– Consistent: All the clients will always see the same data, regardless of which replicas they read from – Defined: Consistent & Clients will see what the mutation writes in its entirety

• The state after mutations depends on

– whether the ops succeeds or fails – Whether there are concurrent mutations

• Move work to applications/clients

– Error code  check consistency – Application checksum  check if it is defined – Identifier for record  figure out duplicates – Retry

• File namespace mutation are atomic

– Handled exclusively by the master

Atomic Record Appends

• Used heavily by Google applications
• Clients specify data

– Not specify offset in files

• GFS appends it to the files

– At least once atomically – Successfully written regions are defined – Does not guarantee that all replicas are identical

• Intervening regions are inconsistent

Experiments

Conclusion

• Gfs is designed for Google

applications

• It is simple, but good enough for

Google applications

• It is important for Google to store

data for other applications.

Discussion (1/5)

• It is designed specifically to serve

search needs. For what other applications this filesystem is suitable

• for. And for what applications, it is not
• ptimal?
• Reply

– Depends on:

• Major file size (small or large)
• Major file reads (streaming or not)
• Major writing modes (appending or not)

Discussion (2/5)

• When the Master decides on which replicas should a

particular chunk goes in, shouldn't it consider the potential applications that is going to access that chunk ?

– Reply: Not really. More replicas if it is hot spot.

• We don't want to see unbalanced disk usages. In the paper

it mentioned that it "rebalances" for load balancing but a way of doing it is not mentioned.

– Reply : Master knows the disk usages of all chunkservers

• In case of a shadow master is operating, it does not

provide any mutation operations. But the processes that got lease from the primary before primary got crashed will continue on writing to the disks right ? So how can we keep track of those writes that is taken place while the shadow master is in operation ?

– Reply : It can run until finished.

Discussion (3/5)

• What happens in GFS if the primary master is

down? Will one of the “shadow” masters become a new primary master? If so, how to pick one from several “shadow” ones?

– Reply : Maybe.

• Will the centralized master-server architecture

degrade the GFS's scalability?

– Reply : Seems not under Google’s workload.

• GFS uses the "closest" client for replication.

However, the term "closest" is never explained - is it based on network latency, or does it factor in network load on the closest chunkserver, does it factor in the system latency? How does one chunkserver figure out the closest server on the fly?

– Reply : Deduced from IP addresss

Discussion (4/5)

• The concept of 'records' appears frequently in the
• paper. GFS can guarantee that a record is appended

atomically, and can associate an application defined checksum with each record. This seems to imply that GFS keeps metadata for each record. How is this done?

– Reply : The checksum is in the record data, not in the metadata.

• Clients cache chunk locations, so if one chunk server

cannot contact the master it may unwittingly serve stale data to clients. Comments?

– Reply : The master includes the chunk version number when it informs clients.

Discussion (5/5)

• What do you think about the performance gain of

“Concurrent appends to the same file operation” and the “relaxed” consistency model under normal workload? (I believe this is only suits well to google). Since appending is key to GFS write performance in a multi-writer environment, it might be that GFS would give up much of its performance advantage even in large serial

• writes. What do you think about this?
• Reply:

– It is a tradeoff between complexity between GFS and its clients. – I think it is good for performance. – It is tailored for google’s workloads (appending).

Thanks for your attention!