Haystack FACEBOOKS PHOTO STORAGE AKIB ZAMAN MOTIVATION Facebook - - PowerPoint PPT Presentation
Finding a needle in Haystack FACEBOOKS PHOTO STORAGE AKIB ZAMAN MOTIVATION Facebook stores an enormous amount of data: 260 billion images 20 petabytes of data Traditional filesystems perform poorly under their workload
FACEBOOK’S PHOTO STORAGE AKIB ZAMAN
Facebook stores an enormous amount of data:
260 billion images 20 petabytes of data Traditional filesystems perform poorly under their workload Several disk operations were necessary to read a single photo Address long tail issue Design pre-requisites:
Data is written once, read often, never modified and rarely deleted
“In our experience, we find that the disadvantages of a traditional
POSIX based filesystem are directories and per file metadata.” Explain how this disadvantage becomes the limiting factor for the read throughput.
High throughput and low latency Fault-tolerant Cost-effective Simple
“We accomplish this by keeping all metadata in main memory,…”.
Why did keeping metadata in memory become a challenge in Facebook’s system? Is it possible just to keep metadata of the most popular files in memory and to achieve the objective (“at most one disk operation per read”) by exploiting access locality?
“That simplicity lets us build and deploy a working system in a few
months instead of a few years.” Comment on this statement (why can Haystack be considered as simple adaptation of UNIX file systems?)
“Haystack takes a straight-forward approach: it stores multiple
photos in a single file and therefore maintains very large files.” Is there such a need to apply the technique in conventional file systems? If applied, what are its potential issues (give two examples)?
Compare serving a photo in Haystack with GFS architecture.
“.. we explored whether it would be useful to build a system similar
to GFS.” Comment on the statement. Why does “Serving photo requests in the long tail represents a problem” on GFS?
Haystack Directory Haystack Cache Haystack Store
Photo Read Photo Write Photo Delete
The Cache “… caches a photo only if two conditions are met: (a)
the request comes directly from a user and not the CDN and (b) the photo is fetched from a write-enabled Store machine.” Please explain this design choice.
“To retrieve needles quickly, each Store machine maintains an in-
memory data structure for each of its volumes.” What is this data structure about?
“As Haystack disallows overwriting needles, photos can only be
modified by adding an updated needle with the same key and alternate key. “ Could you think of reason(s) why Haystack disallows
“Store machines maintain an index file for each of their volumes.”
What is this index and why is it needed? Does maintaining the index significantly increase disk load?
“Store machines maintain an index file for each of their volumes.”
What is this index and why is it needed? How is space for deleted photos reclaimed?
Store machine uses XFS XFS has two main advantages:
The block maps for several large files can be small enough to be stored
in main memory
XFS provides efficient file pre-allocation and avoids fragmentation. XFS helps to eliminate disk operation for metadata for reading a photo.
Haystack needs to tolerate a variety of failures- faulty hard drives,
misbehaving RAID controllers, bad motherboards.
They use the following techniques to tolerate failures:
Pitch-Fork: Background task that periodically checks the health of the
machine.
Bulk Sync: Reset the data of a Store machine using the volume files supplied
by a replica.
Haystack provides a fault-tolerant and simple solution to store pictures. Done at dramatically less cost and higher throughput than a
traditional approach using NAS appliances.
Haystack is incrementally scalable