Cloud-Based File Synchronization Services Exploding in popularity - PowerPoint PPT Presentation

*-Box (star-box) Towards Reliability and Consistency in Dropbox-like File Synchronization Services Yupu Zhang , Chris Dragga, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau University of Wisconsin - Madison 6/27/2013 1

Cloud-Based File Synchronization Services • Exploding in popularity – Numerous providers: Dropbox, Google Drive, SkyDrive … – Large user base: Dropbox has more than 100 million users • Key benefit – Automatic synchronization across clients/devices – Reliable data storage on the server through replication “your stuff is safe in Dropbox and will never be lost ” - Dropbox tour, page 1 6/27/2013 2

Is Your Data Really Safe? • Data corruption – Uploaded from local machine to cloud – Propagated to other devices/clients 6/27/2013 3

Is Your Data Really Safe? • Crash consistency – Inconsistent data ends up everywhere – “Out -of- sync” synchronization 1 2 1 1 1 1 after reboot sync client thinks everything is in sync 6/27/2013 4

Your Data is NOT Really Safe • False sense of safety – Many copies do NOT always make your data safe • Why? – Semantic gap between local file system and cloud – Separately designed and loosely linked 6/27/2013 5

Project *-Box (star-box) • Goal – Close the gap between local file system and cloud – Provide * without too much infrastructure changes • * represents desired properties – e.g ., reliable, consistent, fast, private … • Currently focus on two properties – Reliable: Data corruption – Consistent: Crash consistency 6/27/2013 6

Outline • Introduction • Data Corruption • Crash Consistency • Current Status • Conclusion 6/27/2013 7

Corruption Problem • Data corruption is not uncommon – Comes from disk media, firmware, controllers [Bairavasundaram07, Anderson03] – Remains local w/o synchronization • With synchronization – Corruption may propagate and pollute other copies • Synchronization is a double-edged sword – Make sure synchronized data is good 6/27/2013 8

foo [v1] (4MB data chunks) foo [v0] (4MB data chunks) Dropbox Background Server C1 C2 C3 C1’ Changed Multiple Times D2 ’ Local Database Dedup Rsync Dropbox Name Attributes Name Attributes NOT a duplicate duplicate Client foo I’ foo I NO offline C1’ C2 C3 changes read foo overwrite D2 (chunk by chunk) foo was modified File I’ D2 ’ Inotify System I1 I D1 D1 D2 … … Disk foo: inode 4KB data blocks 6/27/2013 9

Inject foo [v0] (4MB data chunks) Dropbox Server C1 C2 C3 Corruption File Inotify System Corrupt D1 M1 I D1 D1 D1 D1 D2 … … Disk foo: inode 4KB data blocks 6/27/2013 10

foo [v0] (4MB data chunks) Dropbox Start Client Server C1 C2 C3 Local Database Dedup Rsync Dropbox Name Attributes Client foo I NO offline changes File Inotify System I I D1 D1 D2 … … Disk foo: inode 4KB data blocks 6/27/2013 11

foo [v1] (4MB data chunks) foo [v0] (4MB data chunks) Dropbox Data Write Server C1 C2 C3 C1’ D1 D2 ’ Local Database Dedup Rsync Dropbox Name Attributes Name Attributes NOT a duplicate Client foo I’ foo I C1’ read foo overwrite D2 (chunk by chunk) foo was modified File I’ I D2 ’ Inotify System I D1 D1 D2 … … Disk foo: inode 4KB data blocks 6/27/2013 12

Metadata foo [v1] (4MB data chunks) foo [v0] (4MB data chunks) Dropbox Server C1 C2 C3 Change C1’ D1 D2 ’ Local Database Dedup Rsync Dropbox Name Attributes Name Attributes NOT a duplicate Client foo foo I’ I C1’ read foo touch -m (chunk by chunk) foo’s metadata was changed File I I’ Inotify System I D1 D1 D2 D2 … … Disk foo: inode 4KB data blocks 6/27/2013 13

More Results L: Local corruption G: Global corruption Metadata Changes FS Service Data Writes mtime ctime atime Dropbox L G L G L G L ext4 ownCloud L G L G L L (Linux) FileRock L G L G L L Dropbox L G L G L L ownCloud L G L G L L HFS+ GoogleDrive L G L G L L (Mac OS X) SugarSync L G L L L Syncplicity L G L G L L • Corruption is propagated when there is a change to file data • Even if there is no data change, corruption may still be uploaded 6/27/2013 14

Summary • Bad bits are promoted to resilient bad bits – ALL copies polluted – Cloud copies protected by checksum • Fundamental problem, not implementation bugs – FS monitoring services only provide file-level notification – Sync clients cannot tell legitimate changes from corruption • Redundant data on the cloud is not fully utilized – If corruption can be detected, local FS can recover from corruption using cloud copies 6/27/2013 15

Outline • Introduction • Data Corruption • Crash Consistency • Current Status • Conclusion 6/27/2013 16

Crash Recovery Techniques • Copy-on-write (e.g., ZFS, btrfs) – Always roll back to a consistent version • Journaling (e.g., ext4) – Data journaling mode • Both data and metadata are logged • Provide data consistency – Ordered journaling mode • Only journal metadata • Data blocks are written before metadata is logged • Cannot guarantee data consistency 6/27/2013 17

Ordered Mode 1. Write dirty data blocks to home locations 2. Write metadata blocks to journal 3. Write journal commit block to the journal 4. Checkpoint journaled metadata blocks to home locations overwrite D1 &D2 File I’ I’ I’ I D1 ’ D2 ’ D2 ’ D1 ’ System 2 3 4 1 1 I D1 D1 D2 … Disk C Journal / Log foo: inode 4KB data blocks 6/27/2013 18

Crash in Ordered Mode • Crash during step 1 – Write dirty data blocks to home locations overwrite D1 &D2 File I’ I’ I D1 ’ D2 ’ D2 ’ D1 ’ System Inconsistent Data Consistent Data 1 1 I D1 D1 D2 … Disk foo: inode 4KB data blocks 6/27/2013 19

Ext4 Ordered Mode + Dropbox • Case 1 – Inconsistent data is propagated • Case 2 – Consistent data is NOT synchronized 6/27/2013 20

Case 1 foo [v0] (4MB data chunks) Dropbox Server C1 C2 C3 Crash NOT fully updated Local Database Dedup Rsync Dropbox Name Attributes Name Attributes Client foo I’ foo I overwrite D1 &D2 foo was modified File I’ I D1 ’ D1 ’ D2 ’ Inotify System crash AFTER database is changed I D1 D2 … … Disk inconsistent data foo: inode 4KB data blocks on disk 6/27/2013 21

Case 1 foo [v1] (4MB data chunks) foo [v0] (4MB data chunks) Dropbox Server C1 C2 C3 Reboot C1’ NOT fully updated inconsistent data D1’ D2 on cloud Local Database Dedup Rsync Dropbox Name Attributes Client foo I’ Sync! File Inotify System I I D 1’ D1 D2 … … Disk inconsistent data foo: inode 4KB data blocks on disk 6/27/2013 22

Case 2 foo [v0] (4MB data chunks) Dropbox Server C1 C2 C3 Crash Local Database Dedup Rsync Dropbox Name Attributes Client foo I overwrite D1 &D2 (O_SYNC) File I’ I D1 ’ D1 ’ D2 ’ D2 ’ Inotify System crash BEFORE database is changed I D1 D2 … … Disk consistent data foo: inode 4KB data blocks on disk 6/27/2013 23

Case 2 foo [v0] (4MB data chunks) Dropbox Server C1 C2 C3 Reboot Server and other devices still have v0 Local Database Dedup Rsync Dropbox Name Attributes Client foo I NO offline NO sync! changes File Inotify System This machine has v1 I I D 1’ D1 D 2’ … … Disk consistent data foo: inode 4KB data blocks on disk 6/27/2013 24

Recover using Data on Cloud? • Data on server does not always reflect a consistent state on disk – Dropbox uploads data asynchronously – Dropbox reorders file uploading – Actively modified files may get delayed • When crash occurs, files on server could be inconsistent with respect to disk 6/27/2013 25

Summary • Inconsistent content gets propagated • “Out -of- sync” files may exist – Different client/devices see different versions of the same file • Need in-depth communication between local FS and cloud – Cloud has very weak sense of actual FS state 6/27/2013 26

Outline • Introduction • Data Corruption • Crash Consistency • Current Status • Conclusion 6/27/2013 27

Current Status of *-Box • Finished – Data checksumming in ext4 – Dropbox-aided corruption recovery in ext4 – Fine-grained inotify in Linux • Add a ranged file-update notification • Adapt ownCloud (an open-source sync service) to use it • In-progress – Use in-memory snapshot to facilitate crash recovery in ext4 ordered mode 6/27/2013 28

Outline • Introduction • Data Corruption • Crash Consistency • Possible Solutions • Conclusion 6/27/2013 29

Conclusion • Many copies do NOT always make your data safe – Propagation of corrupt data and inconsistent state – Synchronized files are out-of-sync • Propose *-Box project – Solve problems by reducing the semantic gap between existing local FS and cloud storage • Ultimately may need a cohesive system that provides capabilities unachievable in isolation 6/27/2013 30

Thank you! Questions? Advanced Systems Lab (ADSL) University of Wisconsin-Madison http://www.cs.wisc.edu/adsl Wisconsin Institute on Software-defined Datacenters in Madison http://wisdom.cs.wisc.edu/ 6/27/2013 31