[PPT] - Decentralized Deduplication in SAN Cluster File Systems Austin T. PowerPoint Presentation

SLIDE 1

Decentralized Deduplication in SAN Cluster File Systems

Austin T. Clements∗ Irfan Ahmad Murali Vilayannur Jinyuan Li

VMware, Inc.

∗MIT CSAIL Decentralized Deduplication in SAN Cluster File Systems

SLIDE 2

Storage Area Networks

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 3

Storage Area Networks

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 4

Storage Area Networks

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 5

Storage Area Networks

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 6

Storage Area Networks

1.3 TB

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 7

Storage Area Networks

1.3 TB 237 GB

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 8

Storage Area Networks

1.3 TB 237 GB 5X reduction

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 9

Deduplication

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 10

Deduplication

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 11

Deduplication

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 12

Deduplication

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 13

Deduplication

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 14

Deduplication

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 15

Deduplication

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 16

Deduplication

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 17

The Problem

Slow — Lots of IO in the write path. Can’t cache the index.

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 18

The Problem

Slow — Lots of IO in the write path. Can’t cache the index. Very Slow — Writes require allocation and thus coordination. No hope of disk locality.

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 19

The Problem

Slow — Lots of IO in the write path. Can’t cache the index. Very Slow — Writes require allocation and thus coordination. No hope of disk locality. Hopelessly Slow — Multi-host lock contention on shared index.

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 20

Three-Stage Deduplication

Decentralized Deduplication

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 21

Three-Stage Deduplication

DeDe

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 22

Three-Stage Deduplication

DeDe

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 23

Three-Stage Deduplication

DeDe

Out-of-band deduplication of live, primary storage Process duplicates efficiently, in large batches Minimize contention on the index Resilient to stale index information Unique blocks remain mutable and sequential = ⇒ No overhead for blocks that don’t benefit from deduplication

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 24

Three-Stage Deduplication

DeDe

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 25

Three-Stage Deduplication

DeDe

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 26

Three-Stage Deduplication

DeDe

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 27

Three-Stage Deduplication

DeDe

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 28

Three-Stage Deduplication

DeDe

f4bea9.. f2a4d2.. 0e7a26.. 15ba2b.. d5e341.. bc6887.. ab7373.. 4ee207.. 9b1f28.. 9b575b.. 94c9c9.. 288bc7..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 29

Three-Stage Deduplication

DeDe

f4bea9.. f2a4d2.. 0e7a26.. 15ba2b.. d5e341.. bc6887.. ab7373.. 4ee207.. 9b1f28.. 9b575b.. 94c9c9.. 288bc7..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 30

Three-Stage Deduplication

DeDe

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. f4bea9.. f2a4d2.. 0e7a26.. 15ba2b.. d5e341.. bc6887.. ab7373.. 4ee207.. 9b1f28.. 9b575b.. 94c9c9.. 288bc7..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 31

Three-Stage Deduplication

DeDe

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. f4bea9.. f2a4d2.. 0e7a26.. 15ba2b.. d5e341.. bc6887.. ab7373.. 4ee207.. 9b1f28.. 9b575b.. 94c9c9.. 288bc7..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 32

Three-Stage Deduplication

DeDe

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. ab7373.. 4ee207.. 9b1f28.. 9b575b.. 94c9c9.. 288bc7..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 33

Three-Stage Deduplication

DeDe

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. ab7373.. 4ee207.. 9b1f28.. 9b575b.. 94c9c9.. 288bc7..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 34

Three-Stage Deduplication

DeDe

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. ab7373.. 4ee207.. 9b1f28.. 9b575b.. 94c9c9.. 288bc7..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 35

Three-Stage Deduplication

DeDe

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 36

Three-Stage Deduplication

DeDe

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 37

Three-Stage Deduplication

DeDe

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

? ? ? ?

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 38

Three-Stage Deduplication

DeDe

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

? ? ? ?

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 39

Three-Stage Deduplication

DeDe

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

? ? ? ?

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 40

Implementation

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 41

Implementation

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 42

Implementation

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 43

Implementation

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 44

Implementation

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 45

VMFS

Compare-and-share

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 46

VMFS

Compare-and-share

?

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 47

VMFS

Compare-and-share

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 48

VMFS

Compare-and-share DeDe finds duplicates. VMFS eliminates them.

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 49

Write Monitor

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 50

Write Monitor

A lightweight kernel module monitors writes, computes hashes

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 51

Write Monitor

A lightweight kernel module monitors writes, computes hashes It buffers the write log in userspace before writing it to disk

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 52

Write Monitor

A lightweight kernel module monitors writes, computes hashes It buffers the write log in userspace before writing it to disk

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 53

Write Monitor

A lightweight kernel module monitors writes, computes hashes It buffers the write log in userspace before writing it to disk Safe to buffer the log because index is resilient

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 54

Write Monitor

A lightweight kernel module monitors writes, computes hashes It buffers the write log in userspace before writing it to disk Safe to buffer the log because index is resilient 150 MB of regular writes → 1 MB sequential log write

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 55

The Index

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Map from hashes to block locators, list sorted by hash

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 56

The Index

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared Map from hashes to block locators, list sorted by hash

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 57

The Index

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 58

The Index

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 59

The Index

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 60

The Index

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable A virtual arena stores COW references to all shared blocks

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 61

The Index

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable A virtual arena stores COW references to all shared blocks

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 62

The Index

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared Map from hashes to block locators, list sorted by hash Unique blocks are located in files and remain mutable A virtual arena stores COW references to all shared blocks

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 63

Indexing and Duplicate Elimination

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 64

Indexing and Duplicate Elimination

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 65

Indexing and Duplicate Elimination

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 66

Indexing and Duplicate Elimination

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared 12067c.. 6cd412.. c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 67

Indexing and Duplicate Elimination

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared 12067c.. 6cd412.. c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 68

Indexing and Duplicate Elimination

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared 12067c.. 6cd412.. c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 69

Indexing and Duplicate Elimination

0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared 12067c.. 6cd412.. c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 70

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared 6cd412.. c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 71

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared 6cd412.. c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 72

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared 6cd412.. c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 73

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 74

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 75

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 76

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared

?

c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 77

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 78

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared c277d6..

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 79

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 80

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 81

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 82

Indexing and Duplicate Elimination

12067c.. 0e7a26.. 15ba2b.. 6cd412.. ab7373.. bc6887.. c277d6.. d5e341.. f2a4d2.. f4bea9.. Unique Shared

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 83

Phew.

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 84

Evaluation

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 85

Evaluation

How much space does DeDe save?

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 86

Evaluation

How much space does DeDe save? How much overhead does DeDe introduce?

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 87

Evaluation

How much space does DeDe save? How much overhead does DeDe introduce? How fast can DeDe deduplicate?

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 88

Space Savings: VDI Cluster

Corporate Virtual Desktop Infrastructure cluster Desktop XP VM’s 6–12 months of active use Originally cloned from small number of base images ...

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 89

Space Savings: VDI Cluster

Corporate Virtual Desktop Infrastructure cluster Desktop XP VM’s 6–12 months of active use Originally cloned from small number of base images ...

                          

113 VM’s

  

1.9 TB

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 90

Space Savings: VDI Cluster

Corporate Virtual Desktop Infrastructure cluster Desktop XP VM’s 6–12 months of active use Originally cloned from small number of base images ...

                          

113 VM’s

  

1.9 TB 1.3 TB

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 91

Space Savings: VDI Cluster

1.3 TB

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 92

Space Savings: VDI Cluster

1.3 TB 237 GB

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 93

Space Savings: VDI Cluster

1.3 TB 237 GB 173 GB unique 61 GB shared

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 94

Space Savings: VDI Cluster

1.3 TB 237 GB 173 GB unique 61 GB shared 2.7 GB

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 95

Space Savings: VDI Cluster

1.3 TB 237 GB 173 GB unique 61 GB shared 2.7 GB 1.3 GB index file 194 MB

v. arena

1.1 GB FS metadata

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 96

Runtime Effects

Write monitoring Disk array caching

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 97

Runtime Effects

Write monitoring Disk array caching

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 98

Runtime Effects

Write monitoring Disk array caching EMC CLARiiON CX3-40

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 99

Runtime Overhead: Write Monitoring

Worst-case benchmark: 100% sequential write IO, No computation

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 100

Runtime Overhead: Write Monitoring

Worst-case benchmark: 100% sequential write IO, No computation Baseline Write Monitor CPU 33% 220%

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 101

Runtime Overhead: Write Monitoring

Worst-case benchmark: 100% sequential write IO, No computation Baseline Write Monitor CPU 33% 220% Bandwidth (MB/s) 233 233 Latency (ms) 8.6 8.6

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 102

Runtime Overhead: Write Monitoring

Worst-case benchmark: 100% sequential write IO, No computation Baseline Write Monitor CPU 33% 220% Bandwidth (MB/s) 233 233 Latency (ms) 8.6 8.6 (See paper for database application benchmark)

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 103

Runtime Gains: Disk Array Caching

Reduced storage footprint Better caching Less IO

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 104

Runtime Gains: Disk Array Caching

Reduced storage footprint Better caching Less IO

10 20 30 40 50 60 70 80 90 2 4 6 8 10 12 14 16 18 20

Average boot time (secs) # VMs booting concurrently VDI "boot storm"

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 105

Runtime Gains: Disk Array Caching

Reduced storage footprint Better caching Less IO

10 20 30 40 50 60 70 80 90 2 4 6 8 10 12 14 16 18 20

Average boot time (secs) # VMs booting concurrently VDI "boot storm" Full copies

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 106

Runtime Gains: Disk Array Caching

Reduced storage footprint Better caching Less IO

10 20 30 40 50 60 70 80 90 2 4 6 8 10 12 14 16 18 20

Average boot time (secs) # VMs booting concurrently VDI "boot storm" Full copies Deduplicated

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 107

Out-of-band Deduplication Rate

Index scan COW sharing

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 108

Out-of-band Deduplication Rate

Index scan 6.6 GB/sec COW sharing Virtually no cost for unique blocks

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 109

Out-of-band Deduplication Rate

Index scan 6.6 GB/sec COW sharing 2.6 MB/sec Virtually no cost for unique blocks

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 110

Out-of-band Deduplication Rate

Index scan 6.6 GB/sec COW sharing 2.6 MB/sec (It’s a prototype!) Virtually no cost for unique blocks

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 111

Out-of-band Deduplication Rate

Index scan 6.6 GB/sec COW sharing 2.6 MB/sec (It’s a prototype!) Virtually no cost for unique blocks 9 GB of new shared blocks per hour (And provisioning can be special-cased)

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 112

Related Work

Centralized archival

Venti Data Domain Foundation

Centralized primary storage

NetApp ASIS Microsoft Single Instance Store

Distributed

Farsite

SAN with Coordinator

DDE

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 113

Conclusion

Decentralized, out-of-band, live file system deduplication.

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 114

Conclusion

Decentralized, out-of-band, live file system deduplication. Deduplication is effective.

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 115

Conclusion

Decentralized, out-of-band, live file system deduplication. Deduplication is effective. Deduplication is hard.

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 116

Conclusion

Decentralized, out-of-band, live file system deduplication. Deduplication is effective. Deduplication is hard. Three-stage deduplication has only modest performance overhead.

Decentralized Deduplication in SAN Cluster File Systems

SLIDE 117

Conclusion

Decentralized, out-of-band, live file system deduplication. Deduplication is effective. Deduplication is hard. Three-stage deduplication has only modest performance overhead. Thank you.

Decentralized Deduplication in SAN Cluster File Systems