Kurma: Secure Geo-distributed Multi-cloud Storage Gateways Ming - - PowerPoint PPT Presentation

▶
kurma secure geo distributed multi cloud storage gateways
SMART_READER_LITE
LIVE PREVIEW

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways Ming - - PowerPoint PPT Presentation

Mar 06, 2024 134 likes •653 views

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways Ming Chen and Erez Zadok Stony Brook University File Systems and Storage Lab (FSL) Cloud Storage Gateways l Benefits of cloud gateways Public NAS u Combine advantages of both Cloud

slide-1
SLIDE 1

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways

Stony Brook University File Systems and Storage Lab (FSL)

Ming Chen and Erez Zadok

slide-2
SLIDE 2

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 31 SYSTOR’2019

Cloud Storage Gateways

Public Cloud NAS (NFS)

Consistency Rich Semantics Performance Security Scalability Economy Availability Accessibility

l Benefits of cloud gateways

u Combine advantages of both

clouds and traditional NAS

u High security without relying on

trusted third parties

u Allow clients to use public

clouds using network-attached storage (NAS) protocols but still share across regions

slide-3
SLIDE 3

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 32 SYSTOR’2019

Kurma Design Goals

  • 1. Strong Security

u Use clouds to store only encrypted blocks u Share metadata directly among gateways

  • 2. High availability

u Use multiple public clouds u Each gateway is highly available (ZooKeeper)

  • 3. High performance

u Extensive caching for data and metadata u Asynchronous replication of metadata

  • 4. High flexibility

u Replication, erasure coding, and secret sharing

slide-4
SLIDE 4

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 33 SYSTOR’2019

Region 1 Clients Region 2

metadata metadata metadata

Region 3

Azure S3 Google

Untrusted

Kurma Gateway

Public Clouds

Rackspace storage

Kurma Architecture

slide-5
SLIDE 5

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 34 SYSTOR’2019

Region 1 Clients Region 2

metadata metadata metadata

Region 3

Azure S3 Google

Untrusted

Kurma Gateway

Public Clouds

Rackspace storage

Kurma Architecture

Multiple clouds

slide-6
SLIDE 6

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 35 SYSTOR’2019

Region 1 Clients Region 2

metadata metadata metadata

Region 3

Azure S3 Google

Untrusted

Kurma Gateway

Public Clouds

Rackspace storage

Kurma Architecture

Multiple clouds Replicate metadata (versions)

slide-7
SLIDE 7

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 36 SYSTOR’2019

Region 1 Clients Region 2

metadata metadata metadata

Region 3

Azure S3 Google

Untrusted

Kurma Gateway

Public Clouds

Rackspace storage

Kurma Architecture

Multiple clouds Replicate metadata (versions) Distributed gateways

slide-8
SLIDE 8

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 37 SYSTOR’2019

Background

l ZooKeeper: A distributed coordination service

u Coordinate Kurma servers u Store Kurma FS metadata u Execute transactions of metadata changes

l Hedwig: A publish-subscribe system

u Provide guaranteed delivery u Replicate Kurma metadata

l Thrift: A RPC framework

u Define FS metadata format u RPC among Kurma servers

a b znode zpath:/a/b

slide-9
SLIDE 9

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 38 SYSTOR’2019

Components

1 2 3

Clients

Azure S3 Google

Public Clouds Other Gateways Kurma Gateway

Rackspace

slide-10
SLIDE 10

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 39 SYSTOR’2019

Components

1 2 3

Clients

Azure S3 Google

Public Clouds Other Gateways Kurma Gateway

Rackspace

slide-11
SLIDE 11

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 40 SYSTOR’2019

Components

1 2 3

Clients

Azure S3 Google

Public Clouds Other Gateways Kurma Gateway

Rackspace

slide-12
SLIDE 12

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 41 SYSTOR’2019

Components

1 2 3

Clients

Azure S3 Google

Public Clouds Other Gateways Kurma Gateway

Rackspace

slide-13
SLIDE 13

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 42 SYSTOR’2019

Components

1 2 3

Clients

Azure S3 Google

Public Clouds Other Gateways Kurma Gateway

Rackspace

slide-14
SLIDE 14

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 43 SYSTOR’2019

Metadata Management

l Defined using Thrift l Stored in ZooKeeper l Replicated cross-regions using Hedwig

slide-15
SLIDE 15

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 44 SYSTOR’2019

Metadata Management

l Defined using Thrift l Stored in ZooKeeper l Replicated cross-regions using Hedwig

slide-16
SLIDE 16

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 45 SYSTOR’2019

Metadata Management

l Defined using Thrift l Stored in ZooKeeper l Replicated cross-regions using Hedwig

slide-17
SLIDE 17

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 46 SYSTOR’2019

Metadata Management

l Defined using Thrift l Stored in ZooKeeper l Replicated cross-regions using Hedwig

slide-18
SLIDE 18

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 47 SYSTOR’2019

Metadata Management

l Defined using Thrift l Stored in ZooKeeper l Replicated cross-regions using Hedwig

slide-19
SLIDE 19

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 48 SYSTOR’2019

Metadata Management

l Defined using Thrift l Stored in ZooKeeper l Replicated cross-regions using Hedwig

slide-20
SLIDE 20

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 49 SYSTOR’2019

Metadata Management

l Defined using Thrift l Stored in ZooKeeper l Replicated cross-regions using Hedwig

slide-21
SLIDE 21

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 50 SYSTOR’2019

Metadata Management

l Defined using Thrift l Stored in ZooKeeper l Replicated cross-regions using Hedwig

slide-22
SLIDE 22

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 51 SYSTOR’2019

Kurma Security

l Only file data blocks are saved in clouds l Blocks are authenticated and encrypted l Per-file secret key protected by gateway master keys l Detect swap and replay attacks

slide-23
SLIDE 23

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 52 SYSTOR’2019

Multi-Cloud Redundancy

Replication Erasure Coding Secret Sharing Parameters (e.g., 4 clouds) n=4 k=3, m=1 n=4, k=3, r=2 Apply to a block n identical 1MB blocks k+m non-identical 1/k MB block n non-identical 1/k MB block Write a block n × 1MB (k+m) × 1/k MB (k+m) × 1/k MB Read a block any 1 cloud any k clouds any k clouds Tolerate failure of clouds n=f+1 m=f n-k=f Write amplifications f+1 (f+1)/k (f+1)/k Example 2 × 1MB blocks 4 × 340KB blocks 4 × 340KB blocks

slide-24
SLIDE 24

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 53 SYSTOR’2019

Hybrid Consistency Model

l FIFO consistency across gateways

u Updates made by a single gateway are seen

by other gateways in the order they occur, but updates from different gateways may be seen in any interleaved order

u FS metadata is asynchronously replicated

among all regions using Hedwig which does not order message across gateways

u Resolves inter-gateway conflicts as needed

l Region-level NFS consistency

u Same as traditional NFS u Data freshness in the same region

slide-25
SLIDE 25

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 54 SYSTOR’2019

Implementation

l NFS Servers built on top of NFS-Ganesha

u FSAL_PCACHE u FSAL_KURMA

l Gateway Servers

u File-System Module uses Thrift u Metadata Module uses Apache Curator (ZooKeeper) u Security Module uses Java 8 standard cryptographic library u Cloud Module uses cloud Java drivers u Redundancy uses Jerasure and CAONS-RS secret sharing

Components Language LoC Kurma NFS Server C/C++ 15,802 Kurma Gateway Server Java 27,976 Secret Sharing JNI C/C++ 2,480 RPC & Metadata Definition Thrift 668

slide-26
SLIDE 26

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 55 SYSTOR’2019

Optimizations

  • 1. Avoid high-latency of ZooKeeper

u Batch metadata changes using transactions u Use in-memory cache for hot znodes

  • 2. Avoid performance variations of clouds

u Sort clouds online every N seconds

  • 3. Reduce metadata size

u Compress file-system metadata u Use large block sizes

slide-27
SLIDE 27

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 56 SYSTOR’2019

Evaluation

l Experimental setup

u Two regions with a network RTT of 100ms u Each region contains VMs for

§ 3 Metadata Servers running ZooKeeper and Hedwig § 1 Gateway Server § 1 NFS Server with persistent cache on an Intel SSD § 1 NFS client § Each VM has two cores and 4GB of RAM running Fedora

25 with Linux 4.8.10 kernel

u Baseline: traditional NFS server

§ Runs NFS-Ganesha FSAL_VFS § Uses an Intel SSD formatted with Ext4

l Security tests

u Availability test u Integrity tests: swapping and replay attacks

slide-28
SLIDE 28

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 57 SYSTOR’2019

32 64 128 256 512 1024 2048 4096 8192 16KB 64KB 256KB 1MB 4MB

Latency (ms, log2) Cloud Object Size

AWS Azure Google Rackspace

Cloud Read Latency

slide-29
SLIDE 29

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 58 SYSTOR’2019

32 64 128 256 512 1024 2048 4096 8192 16KB 64KB 256KB 1MB 4MB

Latency (ms, log2) Cloud Object Size

AWS Azure Google Rackspace

Cloud Read Latency

1.1✕

slide-30
SLIDE 30

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 59 SYSTOR’2019

32 64 128 256 512 1024 2048 4096 8192 16KB 64KB 256KB 1MB 4MB

Latency (ms, log2) Cloud Object Size

AWS Azure Google Rackspace

Cloud Read Latency

3.1✕ 1.3✕

slide-31
SLIDE 31

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 60 SYSTOR’2019

32 64 128 256 512 1024 2048 4096 8192 16KB 64KB 256KB 1MB 4MB

Latency (ms, log2) Cloud Object Size

AWS Azure Google Rackspace

Cloud Read Latency

43✕

slide-32
SLIDE 32

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 61 SYSTOR’2019

32 64 128 256 512 1024 2048 4096 8192 16KB 64KB 256KB 1MB 4MB

Latency (ms, log2) Cloud Object Size

AWS Azure Google Rackspace

Cloud Read Latency

slide-33
SLIDE 33

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 62 SYSTOR’2019

Multi-Cloud Redundancy

l N-replica: Save N identical replicas in N clouds l Erasure coding: Reed-Solomon (k = 3, m = 1) l Secret sharing: CAONS-RS (n = 4, k =3, r = 2)

1 2 3 4 5 6 7

Latency (sec) Multi-Cloud Redundancy

1 replica

1.6

2 replicas

1.4

3 replicas

1.7

4 replicas

1.5

Erasure coding

2.2

Secret sharing

2.5

10 20 30 40 50 60 70

Latency (sec) Multi-Cloud Redundancy

1.8 4.9 4.6 57.4 20.8 20.8

(a) Read a 16MB file (b) Write a 16MB file

slide-34
SLIDE 34

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 63 SYSTOR’2019

Multi-Cloud Redundancy

l N-replica: Save N identical replicas in N clouds l Erasure coding: Reed-Solomon (k = 3, m = 1) l Secret sharing: CAONS-RS (n = 4, k =3, r = 2)

1 2 3 4 5 6 7

Latency (sec) Multi-Cloud Redundancy

1 replica

1.6

2 replicas

1.4

3 replicas

1.7

4 replicas

1.5

Erasure coding

2.2

Secret sharing

2.5

10 20 30 40 50 60 70

Latency (sec) Multi-Cloud Redundancy

1.8 4.9 4.6 57.4 20.8 20.8

(a) Read a 16MB file (b) Write a 16MB file

slide-35
SLIDE 35

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 64 SYSTOR’2019

Multi-Cloud Redundancy

l N-replica: Save N identical replicas in N clouds l Erasure coding: Reed-Solomon (k = 3, m = 1) l Secret sharing: CAONS-RS (n = 4, k =3, r = 2)

1 2 3 4 5 6 7

Latency (sec) Multi-Cloud Redundancy

1 replica

1.6

2 replicas

1.4

3 replicas

1.7

4 replicas

1.5

Erasure coding

2.2

Secret sharing

2.5

10 20 30 40 50 60 70

Latency (sec) Multi-Cloud Redundancy

1.8 4.9 4.6 57.4 20.8 20.8

(a) Read a 16MB file (b) Write a 16MB file

slide-36
SLIDE 36

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 65 SYSTOR’2019

Multi-Cloud Redundancy

l N-replica: Save N identical replicas in N clouds l Erasure coding: Reed-Solomon (k = 3, m = 1) l Secret sharing: CAONS-RS (n = 4, k =3, r = 2)

1 2 3 4 5 6 7

Latency (sec) Multi-Cloud Redundancy

1 replica

1.6

2 replicas

1.4

3 replicas

1.7

4 replicas

1.5

Erasure coding

2.2

Secret sharing

2.5

10 20 30 40 50 60 70

Latency (sec) Multi-Cloud Redundancy

1.8 4.9 4.6 57.4 20.8 20.8

(a) Read a 16MB file (b) Write a 16MB file

slide-37
SLIDE 37

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 66 SYSTOR’2019

Multi-Cloud Redundancy

l N-replica: Save N identical replicas in N clouds l Erasure coding: Reed-Solomon (k = 3, m = 1) l Secret sharing: CAONS-RS (n = 4, k =3, r = 2)

1 2 3 4 5 6 7

Latency (sec) Multi-Cloud Redundancy

1 replica

1.6

2 replicas

1.4

3 replicas

1.7

4 replicas

1.5

Erasure coding

2.2

Secret sharing

2.5

10 20 30 40 50 60 70

Latency (sec) Multi-Cloud Redundancy

1.8 4.9 4.6 57.4 20.8 20.8

(a) Read a 16MB file (b) Write a 16MB file

64 256 1024 4096 16384 65536 262144 1048576 16KB 64KB 256KB 1MB 4MB

Latency (ms, log2) Cloud Object Size

AWS Azure Google Rackspace

slide-38
SLIDE 38

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 67 SYSTOR’2019

Multi-Cloud Redundancy

l N-replica: Save N identical replicas in N clouds l Erasure coding: Reed-Solomon (k = 3, m = 1) l Secret sharing: CAONS-RS (n = 4, k =3, r = 2)

1 2 3 4 5 6 7

Latency (sec) Multi-Cloud Redundancy

1 replica

1.6

2 replicas

1.4

3 replicas

1.7

4 replicas

1.5

Erasure coding

2.2

Secret sharing

2.5

10 20 30 40 50 60 70

Latency (sec) Multi-Cloud Redundancy

1.8 4.9 4.6 57.4 20.8 20.8

(a) Read a 16MB file (b) Write a 16MB file

64 256 1024 4096 16384 65536 262144 1048576 16KB 64KB 256KB 1MB 4MB

Latency (ms, log2) Cloud Object Size

AWS Azure Google Rackspace

3X

slide-39
SLIDE 39

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 68 SYSTOR’2019

Cross-Gateway Replication

l Create a file in one gateway and read it

in another gateway

20 40 60 80 100 10 20 30 40 50 60

Replication Latency (sec) Cache Write−Back Wait Time (s)

64K−file 1M−file 16M−file

slide-40
SLIDE 40

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 69 SYSTOR’2019

Cross-Gateway Replication

l Create a file in one gateway and read it

in another gateway

20 40 60 80 100 10 20 30 40 50 60

Replication Latency (sec) Cache Write−Back Wait Time (s)

64K−file 1M−file 16M−file

slide-41
SLIDE 41

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 70 SYSTOR’2019

Data Operations

l Read and write files with a hot cache

0.1 0.2 0.3 0.4 64K 256K 1M 4M 16M

Latency (sec) File Size (log4)

NFS Read Kurma Read NFS Write Kurma Write

slide-42
SLIDE 42

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 71 SYSTOR’2019

Data Operations

l Read and write files with a hot cache

0.1 0.2 0.3 0.4 64K 256K 1M 4M 16M

Latency (sec) File Size (log4)

NFS Read Kurma Read NFS Write Kurma Write

slide-43
SLIDE 43

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 72 SYSTOR’2019

Metadata Operations

l Kurma metadata operations are slower than the baseline

u A metadata operation requires changes to multiple ZooKeeper

nodes (znodes). For example, create a file: 1.

create file znode

2.

create keymap znode

3.

update parent directory’s znode

u Each ZooKeeper change incurs multiple network hops.

200 400 600 800 1000 1200 create delete

Throughput (ops/sec) Operation

NFS

1059 519

Kurma

545 278

slide-44
SLIDE 44

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 73 SYSTOR’2019

Metadata Operations

l Kurma metadata operations are slower than the baseline

u A metadata operation requires changes to multiple ZooKeeper

nodes (znodes). For example, create a file: 1.

create file znode

2.

create keymap znode

3.

update parent directory’s znode

u Each ZooKeeper change incurs multiple network hops.

200 400 600 800 1000 1200 create delete

Throughput (ops/sec) Operation

NFS

1059 519

Kurma

545 278

  • 49%
slide-45
SLIDE 45

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 74 SYSTOR’2019

Metadata Operations

l Kurma metadata operations are slower than the baseline

u A metadata operation requires changes to multiple ZooKeeper

nodes (znodes). For example, create a file: 1.

create file znode

2.

create keymap znode

3.

update parent directory’s znode

u Each ZooKeeper change incurs multiple network hops.

200 400 600 800 1000 1200 create delete

Throughput (ops/sec) Operation

NFS

1059 519

Kurma

545 278

  • 49%
  • 46%
slide-46
SLIDE 46

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 75 SYSTOR’2019

Filebench Workloads

500 1000 1500 2000 2500 NFS Server File Server Mail Server

Throughput (ops/sec) Filebench Workload

NFS

610 2196 1606

Kurma

554 1145 950

slide-47
SLIDE 47

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 76 SYSTOR’2019

Filebench Workloads

500 1000 1500 2000 2500 NFS Server File Server Mail Server

Throughput (ops/sec) Filebench Workload

NFS

610 2196 1606

Kurma

554 1145 950 91%

slide-48
SLIDE 48

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 77 SYSTOR’2019

Filebench Workloads

500 1000 1500 2000 2500 NFS Server File Server Mail Server

Throughput (ops/sec) Filebench Workload

NFS

610 2196 1606

Kurma

554 1145 950 91% 52%

slide-49
SLIDE 49

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 78 SYSTOR’2019

Filebench Workloads

500 1000 1500 2000 2500 NFS Server File Server Mail Server

Throughput (ops/sec) Filebench Workload

NFS

610 2196 1606

Kurma

554 1145 950 91% 52% 59%

slide-50
SLIDE 50

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 79 SYSTOR’2019

Conclusions

l Kurma: secure distributed multi-cloud gateways

u Protect file data with authenticated encryption u Store file metadata on-premises ZooKeeper u Securely share data across regions u Multi-cloud: replication, erasure coding, secret sharing

l Implementation and evaluation

u Data operations are as fast as traditional NFS u Metadata performance: 51-54% u Filebench workloads: 52-91% u http://github.com/sbu-fsl/kurma

slide-51
SLIDE 51

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways 80 SYSTOR’2019

Kurma: Secure Geo-distributed Multi-cloud Storage Gateways

Q&A