[PPT] - Topology-Aware Cooperative Data Protection in Blockchain-Based PowerPoint Presentation

SLIDE 1

ISIT 2020

Topology-Aware Cooperative Data Protection in Blockchain-Based Decentralized Storage Networks

Siyi Yang1, Ahmed Hareedy2, Robert Calderbank2, Lara Dolecek1

1Electrical and Computer Engineering Department, UCLA 2Electrical and Computer Engineering Department, Duke University

06/2020

SLIDE 2

w Introduction

Ø Decentralized storage networks Ø Existing work

w Cooperative Data Protection

Ø Preliminaries Ø ECC hierarchy Ø Single-level cooperation

w Multi-level cooperation

Ø Cooperation graphs and compatible graphs Ø Construction over compatible graphs

w Conclusion

Outline

SLIDE 3

w Introduction

Ø Decentralized storage networks Ø Existing work

w Cooperative Data Protection

Ø Preliminaries Ø ECC hierarchy Ø Single-level cooperation

w Multi-level cooperation

Ø Cooperation graphs and compatible graphs Ø Construction over compatible graphs

w Conclusion

Outline

SLIDE 4

w Invention of Blockchain technology makes the concept of

“decentralization” a popular term

Ø Higher privacy Ø Better scalability and flexibility

w Decentralization has potential to universally revolutionize

various applications

Decentralized Storage Networks

1

Ø Blockchain-based decentralized

storage networks (DSNs)

Ø Masterless coded distributed

computation

Ø Federated learning Ø Wireless sensor networks

SLIDE 5

w Invention of Blockchain technology makes the concept of

“decentralization” a popular term

Ø Higher privacy Ø Better scalability and flexibility

w Decentralization has potential to universally revolutionize

various applications

1

Ø Blockchain-based decentralized

storage networks (DSNs)

Ø Masterless coded distributed

computation

Ø Federated learning Ø Wireless sensor networks

Decentralized Storage Networks

SLIDE 6

2

w Typical Structures

Ø Nodes form multiple clusters (clouds), each of which has a

representative (master) node

Ø Master nodes form a distributed network Ø Nodes in each cluster form a centralized network around the

representative master node

Decentralized Storage Networks

Ce Centralized Storage De Decentralized Storage Di Distributed St Storage

Messages are encoded and stored distributively among non-master nodes in each cloud

SLIDE 7

Practical Concerns in DSNs

3

w Hierarchical erasure correction

Ø Each node provides different levels of robustness for codeword

stored at it through accessing different set of nodes

Failures in reading nodes result in erased symbols
Extra information improves the reliability, at a cost of higher latency

w Topology-awareness

Ø Scheme optimized for DSNs with a specific topology can result

in bad performance in DSNs with other topologies

Different reading frequencies (hot/cold) and reliabilities among nodes
Inter-cloud transmission is much slower than intra-cloud transmission

Our goal is to construct topology-aware coding schemes that also provide hierarchical erasure correction at each node

SLIDE 8

Existing Literature

4

w Distributed storage[1]

Ø No explicit consideration of clustering nature of network nodes

w Multi-rack storage[2-8]

Ø Network topologies are typically not considered to be general Ø Capacities of the communication links are typically considered

to be homogeneous

[1] A. G. Dimakis et al., “Network coding for distributed storage systems”, IEEE Trans. Inf. Theory, vol. 56, no. 9, pp. 4539-4551, 2010 [2] Z. Kong et al., “Decentralized coding algorithms for distributed storage in wireless sensor networks”, IEEE JSAC, vol. 28, no. 2, pp. 261- 267, 2010 [3] M. Ye et al., “Cooperative repair: Constructions of optimal MDS codes for all admissible parameters”, IEEE Trans. Inf. Theory, vol. 65, no. 3, pp. 1639-1656, 2018 [4] N. Prakash et al., “The storage versus repair-bandwidth trade-off for clustered storage systems”, IEEE Trans. Inf. Theory, vol. 64, no. 8,

pp. 5783-5805, 2018

[5] J. Li et al., “Tree-structured data regeneration in distributed storage systems with regenerating codes”, IEEE INFOCOM, 2010 [6] Y. Wang et al., “Non-homogeneous two-rack model for distributed storage systems”, IEEE INFOCOM, 2014 [7] H. Hou et al., “Rack-aware regenerating codes for data centers”, IEEE Trans. Inf. Theory, 2019 [8] Z. Chen et al., “Explicit constructions of MSR codes for clustered distributed storage: the rack-aware storage model”, [Online]. Available: https://arxiv.org/abs/1901.04419, 2019

SLIDE 9

Outline

w Introduction

Ø Decentralized storage networks Ø Existing work

w Cooperative Data Protection

Ø Preliminaries Ø ECC hierarchy Ø Single-level cooperation

w Multi-level cooperation

Ø Cooperation graphs and compatible graphs Ø Construction over compatible graphs

w Conclusion

SLIDE 10

w A DSN is modeled as a graph 𝐻(𝑊, 𝐹)

Ø Sets of master nodes and edges between them: 𝑊, 𝐹 Ø Neighborhood of 𝑤(: 𝒪

( = 𝑤+,, 𝑤+-, 𝑤+.

Ø Codeword stored at 𝑤(: 𝐝( Ø Message stored at 𝑤(: 𝐧( Ø Number of nodes: 𝑞 Ø 𝐧 = 𝐧2,𝐧3, ⋯ , 𝐧5 Ø 𝐝 = 𝐝2, 𝐝3, ⋯, 𝐝5 Ø Length of 𝐧(, 𝐝(: 𝑙(, 𝑜( Ø Redundancy of 𝐝(: 𝑠

( = 𝑜( − 𝑙(

Notation

5

The set consisting of all neighbors (master nodes) of a node 𝑤( is called the neighborhood of this node, and is denoted by 𝒪

(

vj1 vj2 vj3

(mj1, cj1) (mj2, cj2) (mj3, cj3) (mi, ci)

ei,j1 ei,j2 ei,j3

SLIDE 11

λi,1;∅ λi,1;B1

i

di,0 di,1 λi,2;∅ λi,2;B2

i

di,2 λi,3;∅ λi,3;B3

i

di,3

ECC Hierarchy

6

ECC hierarchy describes the erasure correction (EC) capabilities

f nodes while cooperating with different sets of other nodes

SLIDE 12

λi,1;∅ λi,1;B1

i

di,0 di,1 λi,2;∅ λi,2;B2

i

di,2 λi,3;∅ λi,3;B3

i

di,3

w Cooperation at each node

Ø Sets associated with nodes utilizing 𝑚-th level cooperation

∅ ⊂ 𝒝(

2 ⊂ 𝒝( 3⊂ ⋯ ⊂ 𝒝( >?⊆ 𝑊, ℬ( B 2CBC>? ,𝒝( 2∩ ℬ( B = ∅

ECC Hierarchy

6

ECC hierarchy describes the erasure correction (EC) capabilities

f nodes while cooperating with different sets of other nodes

SLIDE 13

λi,1;∅ λi,1;B1

i

di,0 di,1 λi,2;∅ λi,2;B2

i

di,2 λi,3;∅ λi,3;B3

i

di,3

w Cooperation at each node

Ø Sets associated with nodes utilizing 𝑚-th level cooperation

∅ ⊂ 𝒝(

2 ⊂ 𝒝( 3⊂ ⋯ ⊂ 𝒝( >?⊆ 𝑊, ℬ( B 2CBC>? ,𝒝( 2∩ ℬ( B = ∅

Ø ECC hierarchy at 𝑤(: 𝐞( = 𝑒(,G,𝑒(,2,⋯ , 𝑒(,>? ; depth of 𝐞(: 𝑀(

Nodes in 𝒝(

B are recovered and involved in the decoding process

𝑒(,B : EC capability with the 𝑚-th level cooperation at 𝑤(

ECC Hierarchy

6

ECC hierarchy describes the erasure correction (EC) capabilities

f nodes while cooperating with different sets of other nodes

SLIDE 14

λi,1;∅ λi,1;B1

i

di,0 di,1 λi,2;∅ λi,2;B2

i

di,2 λi,3;∅ λi,3;B3

i

di,3

w Cooperation at each node

Ø Sets associated with nodes utilizing 𝑚-th level cooperation

∅ ⊂ 𝒝(

2 ⊂ 𝒝( 3⊂ ⋯ ⊂ 𝒝( >?⊆ 𝑊, ℬ( B 2CBC>? ,𝒝( 2∩ ℬ( B = ∅

Ø ECC hierarchy at 𝑤(: 𝐞( = 𝑒(,G,𝑒(,2,⋯ , 𝑒(,>? ; depth of 𝐞(: 𝑀(

Nodes in 𝒝(

B are recovered and involved in the decoding process

𝑒(,B : EC capability with the 𝑚-th level cooperation at 𝑤(

Ø Elaborated EC capability: 𝑒(,B = 𝜇(,B;𝒳 ∅⊆𝒳⊆ℬ?

M

𝜇(,B;𝒳: EC capability while nodes in 𝒳 are

also recovered and involved in the 𝑚-th level cooperation at 𝑤(

ECC Hierarchy

6

ECC hierarchy describes the erasure correction (EC) capabilities

f nodes while cooperating with different sets of other nodes

SLIDE 15

w Example

Ø Nodes in 𝒝3

2 are neighbors of 𝑤3 and are required to be locally-

recoverable to remove cross parities from the parity part of 𝐝3

𝑤3 tolerates 𝑠3 erasures in this scenario

ECC Hierarchy

7

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 A1

2

B1

2

SLIDE 16

w Example

Ø Nodes in 𝒝3

2 are neighbors of 𝑤3 and are required to be locally-

recoverable to remove cross parities from the parity part of 𝐝3

𝑤3 tolerates 𝑠3 erasures in this scenario

Ø Nodes in ℬ3

2 are neighbors of nodes in 𝒝3 2 except for 𝑤3 and

nodes in 𝒝3

2 themselves

𝑤3 obtains extra parities from 𝑤N if 𝑤O is also recovered, i.e., 𝑤O ⊆ 𝒳
𝑤3 obtains extra parities from 𝑤P if 𝑤O,𝑤Q,𝑤R ⊆ 𝒳
𝑤3 obtains extra parities from 𝑤2

ECC Hierarchy

7

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 A1

2

B1

2

SLIDE 17

w Example

Ø Nodes in 𝒝3

2 are neighbors of 𝑤3 and are required to be locally-

recoverable to remove cross parities from the parity part of 𝐝3

𝑤3 tolerates 𝑠3 erasures in this scenario

Ø Nodes in ℬ3

2 are neighbors of nodes in 𝒝3 2 except for 𝑤3 and

nodes in 𝒝3

2 themselves

𝑤3 obtains extra parities from 𝑤N if 𝑤O is also recovered, i.e., 𝑤O ⊆ 𝒳
𝑤3 obtains extra parities from 𝑤P if 𝑤O,𝑤Q,𝑤R ⊆ 𝒳
𝑤3 obtains extra parities from 𝑤2

Ø All possible 𝜇3,2;𝒳

𝜇3,2;𝒳 = 𝜇3,2;∅,𝒳 ⊆ 𝑤Q,𝑤R
𝜇3,2;𝒳 = 𝜇3,2; ST ,

l

𝑤O ⊆ 𝒳 ⊂ 𝑤O,𝑤Q,𝑤R

𝜇3,2;𝒳 = 𝜇3,2; ST,SU,SV = 𝑒3,2,

l 𝒳 = 𝑤O,𝑤Q,𝑤R

ECC Hierarchy

7

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 A1

2

B1

2

SLIDE 18

w Parity part of the generator matrix of a single-level accessible

code based on CRS codes

Example: Single-Level Cooperation

8

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12

SLIDE 19

Example: Single-Level Cooperation

8

w Parity part of the generator matrix of a single-level accessible

code based on CRS codes

Ø Diagonal components: 𝐁(,(

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12

SLIDE 20

w Parity part of the generator matrix of a single-level accessible

code based on CRS codes

Ø Diagonal components: 𝐁(,( Ø Non-diagonal components:𝐂(,+𝐕+

Example: Single-Level Cooperation

8

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12

SLIDE 21

w Parity part of the generator matrix of a single-level accessible

code based on CRS codes

Ø Diagonal components: 𝐁(,( (𝑙(×𝑠

()

Ø Non-diagonal components:𝐂(,+𝐕+ (𝑙(×𝜀

+,𝜀 +×𝑠 +)

Ø Components are parts of Cauchy matrices 𝐔(

ℳ(: nodes cooperating with 𝑤( in the 1-st level cooperation

Example: Single-Level Cooperation

8

𝑒(,G = 𝑠( − 𝜀(, 𝑒(,2 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

, 𝜇(,2;𝒳 = 𝑠( + ∑ 𝜀

+ ℳ`\ S𝒋 ⊆ℳ?∪𝒳

SLIDE 22

9

w ECC hierarchy at node 𝑤3

Ø Local decoding: 𝑒3,G = 𝑠

3 − 𝜀3

Example: Hierarchical Erasure Correction

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 A1

2

B1

2

SLIDE 23

9

w ECC hierarchy at node 𝑤3

Ø Local decoding: 𝑒3,G = 𝑠

3 − 𝜀3

Ø Cross parities at node 𝑤3 can be subtracted from its local parity

part if 𝑤2, 𝑤N, 𝑤P are recovered

Example: Hierarchical Erasure Correction

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 A1

2

B1

2

SLIDE 24

w ECC hierarchy at node 𝑤3

Ø Local decoding: 𝑒3,G = 𝑠

3 − 𝜀3

Ø Cross parities at node 𝑤3 can be subtracted from its local parity

part if 𝑤2, 𝑤N, 𝑤P are recovered

Ø Additional cross parities of 𝐝3

can be obtained from 𝑤+ if all other neighbors of 𝑤+ except for 𝑤3 are recovered:

𝜇3,2;𝒳 = 𝑠3 + ∑

𝜀

+ ℳ`\ S- ⊆ℳ?∪𝒳

Example: Hierarchical Erasure Correction

9

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 A1

2

B1

2

SLIDE 25

Example: Fast Recovery Speed

10

w Suppose 𝐝3 has (𝑠

3 + 𝜀2 + 1) erasures

Ø Needs to obtain additional cross parities from 𝑤N and 𝑤P

w Suppose transmission over 𝑓(,+ needs time 𝑢(,+

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 A1

2

B1

2

SLIDE 26

w Suppose 𝐝3 has (𝑠

3 + 𝜀2 + 1) erasures

Ø Needs to obtain additional cross parities from 𝑤N and 𝑤P

w Suppose transmission over 𝑓(,+ needs time 𝑢(,+

Ø Time cost at path 𝑤O → 𝑤N → 𝑤3

𝑢O,N + 𝑢N,3

Example: Fast Recovery Speed

10

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 A1

2

B1

2

SLIDE 27

w Suppose 𝐝3 has (𝑠

3 + 𝜀2 + 1) erasures

Ø Needs to obtain additional cross parities from 𝑤N and 𝑤P

w Suppose transmission over 𝑓(,+ needs time 𝑢(,+

Ø Time cost at path 𝑤O → 𝑤N → 𝑤3

𝑢O,N + 𝑢N,3

Ø Time cost at path 𝑤Q → 𝑤P → 𝑤3

𝑢Q,P + 𝑢P,3

Example: Fast Recovery Speed

10

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 A1

2

B1

2

SLIDE 28

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 A1

2

B1

2

w Suppose 𝐝3 has (𝑠

3 + 𝜀2 + 1) erasures

Ø Needs to obtain additional cross parities from 𝑤N and 𝑤P

w Suppose transmission over 𝑓(,+ needs time 𝑢(,+

Ø Time cost at path 𝑤O → 𝑤N → 𝑤3

𝑢O,N + 𝑢N,3

Ø Time cost at path 𝑤Q → 𝑤P → 𝑤3

𝑢Q,P + 𝑢P,3

Example: Fast Recovery Speed

10

If 𝑢Q,P + 𝑢P,3 > 𝑢O,N + 𝑢N,3, only 4 nodes are required to be recovered

SLIDE 29

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12 A1

2

B1

2

w Suppose 𝐝3 has (𝑠

3 + 𝜀2 + 1) erasures

Ø Needs to obtain additional cross parities from 𝑤N and 𝑤P

w Suppose transmission over 𝑓(,+ needs time 𝑢(,+

Ø Time cost at path 𝑤O → 𝑤N → 𝑤3

𝑢O,N + 𝑢N,3

Ø Time cost at path 𝑤Q → 𝑤P → 𝑤3

𝑢Q,P + 𝑢P,3

Example: Fast Recovery Speed

10

If 𝑢Q,P + 𝑢P,3 > 𝑢O,N + 𝑢N,3, only 4 nodes are required to be recovered

Nodes in the DSN automatically choose the fastest path for recovery

SLIDE 30

Example: Flexible Erasure Pattern

11

w Compare with schemes in centralized storage

Ø If codewords stored at nodes 𝑤3, 𝑤O,𝑤R, 𝑤2G (marked by red) are

not locally recoverable, they can be recovered in our scheme

[1] S. Yang et al., “Hierarchical coding to enable scalability and flexibility in heterogeneous cloud storage”, IEEE CLOBECOM, 2019

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12

SLIDE 31

Example: Flexible Erasure Pattern

11

w Compare with schemes in centralized storage

Ø If codewords stored at nodes 𝑤3, 𝑤O,𝑤R, 𝑤2G (marked by red) are

not locally recoverable, they can be recovered in our scheme

Ø In order to let them be recovered in centralized scheme

specified in [1], nodes need to be divided into 4 groups where each group contains exactly one red node

[1] S. Yang et al., “Hierarchical coding to enable scalability and flexibility in heterogeneous cloud storage”, IEEE CLOBECOM, 2019

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12

SLIDE 32

w Compare with schemes in centralized storage

Ø If codewords stored at nodes 𝑤3, 𝑤O,𝑤R, 𝑤2G (marked by red) are

not locally recoverable, they can be recovered in our scheme

Ø In order to let them be recovered in centralized scheme

specified in [1], nodes need to be divided into 4 groups where each group contains exactly one red node

The EC capability at each node (except for 𝑤2,𝑤23) decreases since

information that flows through edges marked in blue no longer exist

Example: Flexible Erasure Pattern

11 [1] S. Yang et al., “Hierarchical coding to enable scalability and flexibility in heterogeneous cloud storage”, IEEE CLOBECOM, 2019

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12

SLIDE 33

w Compare with schemes in centralized storage

Ø If codewords stored at nodes 𝑤3, 𝑤O,𝑤R, 𝑤2G (marked by red) are

not locally recoverable, they can be recovered in our scheme

Ø In order to let them be recovered in centralized scheme

specified in [1], nodes need to be divided into 4 groups where each group contains exactly one red node

The EC capability at each node (except for 𝑤2,𝑤23) decreases since

information that flows through edges marked in blue no longer exist

Example: Flexible Erasure Pattern

11 [1] S. Yang et al., “Hierarchical coding to enable scalability and flexibility in heterogeneous cloud storage”, IEEE CLOBECOM, 2019

Topology-aware coding enables information flow from more reliable nodes to less reliable nodes, tolerating more flexible erasure patterns

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12

SLIDE 34

Outline

w Introduction

Ø Decentralized storage networks Ø Existing work

w Cooperative Data Protection

Ø Preliminaries Ø ECC hierarchy Ø Single-level cooperation

w Multi-level cooperation

Ø Cooperation graphs and compatible graphs Ø Construction over compatible graphs

w Conclusion

SLIDE 35

Example: Multi-Level Cooperation

12

1

1

1

1 1

1

1

1

1 1 1

1

1 1 1

1

1

1

1 1 1 1

1

1 1

1

1

1

1 1 1

1

1 1

1

1

1

1 1

1

1

1

1 1 1

1

1 1 1

1

1

1

1 1 1 1

1

1 1

1

1

1

1 1 1

1

1 1

1

2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

w Parity part of the generator matrix

Ø Components corresponding to the 1-st level cooperation

SLIDE 36

Example: Multi-Level Cooperation

12

1

1

1

1 1

1

1

1

1 1 1

1

1 1 1

1

1

1

1 1 1 1

1

1 1

1

1

1

1 1 1

1

1 1

1

1

1

1 1

1

1

1

1 1 1

1

1 1 1

1

1

1

1 1 1 1

1

1 1

1

1

1

1 1 1

1

1 1

1

2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

w Parity part of the generator matrix

Ø Components corresponding to the 1-st level cooperation Ø Components corresponding to higher level cooperation

Find different cycles such that nodes representing rows/columns each

cycle spans are mutually connected

SLIDE 37

Example: Multi-Level Cooperation

12

1

1

1

1 1

1

1

1

1 1 1

1

1 1 1

1

1

1

1 1 1 1

1

1 1

1

1

1

1 1 1

1

1 1

1

1

1

1 2 2 3 3 1

1

2 2 3 3 1

1

1 2 2 2 2 1 1

1

1 2 2 1 1

1

2 2 2 2 1

1

1 2 1 2 2 2 2 1 2 1

1

2 2 2 2 2 2 1 1

1

2 2 3 3 3 3 3 3 1

1

1 3 3 3 3 1 3 3 1

1

3 3 3 3 1 1

1

2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

w Parity part of the generator matrix

Ø Components corresponding to the 1-st level cooperation Ø Components corresponding to higher level cooperation

Find different cycles such that nodes representing rows/columns each

cycle spans are mutually connected

Label each cycle with a number

SLIDE 38

Example: Multi-Level Cooperation

12

1

1

1

1 c c d d 1

1

e e f f 1

1

1 g g 1 1

1

1 h h 1 1

1

j j 1

1

1 l 1 l m m y 1 y 1

1

n n

z

z 1 1

1

p p q q r r s s 1

1

1 x x t t 1 u u 1

1

v v w w 1 1

1

1

1

1 2 2 3 3 1

1

2 2 3 3 1

1

1 2 2 2 2 1 1

1

1 2 2 1 1

1

2 2 2 2 1

1

1 2 1 2 2 2 2 1 2 1

1

2 2 2 2 2 2 1 1

1

2 2 3 3 3 3 3 3 1

1

1 3 3 3 3 1 3 3 1

1

3 3 3 3 1 1

β

β α α 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

w Parity part of the generator matrix

Ø Components corresponding to the 1-st level cooperation Ø Components corresponding to higher level cooperation

Find different cycles such that nodes representing rows/columns each

cycle spans are mutually connected

Label each cycle with a number
Label each horizontal edge of

each cycle with a symbol

SLIDE 39

w Parity part of the generator matrix

Ø Components corresponding to the 1-st level cooperation Ø Components corresponding to higher level cooperation

Find different cycles such that nodes representing rows/columns each

cycle spans are mutually connected

Label each cycle with a number
Label each horizontal edge of

each cycle with a symbol

Replace the entry labelled with

number 𝑚 and symbol 𝑡 with 𝐂k𝐖

+;B

Example: Multi-Level Cooperation

12

1

1

1

1 c c d d 1

1

e e f f 1

1

1 g g 1 1

1

1 h h 1 1

1

j j 1

1

1 l 1 l m m y 1 y 1

1

n n

z

z 1 1

1

p p q q r r s s 1

1

1 x x t t 1 u u 1

1

v v w w 1 1

1

1

1

1 2 2 3 3 1

1

2 2 3 3 1

1

1 2 2 2 2 1 1

1

1 2 2 1 1

1

2 2 2 2 1

1

1 2 1 2 2 2 2 1 2 1

1

2 2 2 2 2 2 1 1

1

2 2 3 3 3 3 3 3 1

1

1 3 3 3 3 1 3 3 1

1

3 3 3 3 1 1

β

β α α 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

SLIDE 40

Example: Multi-Level Cooperation

13

w Obtaining parities from multi-level cooperation

Ø If 𝑤2,𝑤N, 𝑤P are recovered, not only the 1-st level cross parities

at 𝑤3 can be obtained, the 2-nd and 3-rd level cross parities can be also obtained

The 2-nd and 3-rd cross parities at 𝑤3,𝑤N are identical, thus recovering

𝑤N is sufficient for 𝑤3 to obtain them

SLIDE 41

Example: Multi-Level Cooperation

13

w Obtaining parities from multi-level cooperation

Ø If 𝑤2,𝑤N, 𝑤P are recovered, not only the 1-st level cross parities

at 𝑤3 can be obtained, the 2-nd and 3-rd level cross parities can be also obtained

The 2-nd and 3-rd cross parities at 𝑤3,𝑤N are identical, thus recovering

𝑤N is sufficient for 𝑤3 to obtain them

Ø Extra cross parities of 𝐧3 because of the 2-nd level cooperation

can be obtained from either 𝑤R or 𝑤m

They can be obtained if any one of 𝑤R,𝑤m is recovered, and both 𝑤O,𝑤Q

are also recovered

SLIDE 42

Cooperation Graphs

14

Co Cooperation Matrix

1

1

1

1 2 2 3 3 1

1

2 2 3 3 1

1

1 2 2 2 2 1 1

1

1 2 2 1 1

1

2 2 2 2 1

1

1 2 1 2 2 2 2 1 2 1

1

2 2 2 2 2 2 1 1

1

2 2 3 3 3 3 3 3 1

1

1 3 3 3 3 1 3 3 1

1

3 3 3 3 1 1

1

2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

w The matrix with numbers is referred to as a cooperation matrix w Each cycle is represented by 𝐷o(𝑌o, 𝑍

, 𝑌o;+ +∈r

s, 𝑍

;( (∈ts, 𝑚o)

SLIDE 43

Cooperation Graphs

14

Co Cooperation Matrix

1

1

1

1 2 2 3 3 1

1

2 2 3 3 1

1

1 2 2 2 2 1 1

1

1 2 2 1 1

1

2 2 2 2 1

1

1 2 1 2 2 2 2 1 2 1

1

2 2 2 2 2 2 1 1

1

2 2 3 3 3 3 3 3 1

1

1 3 3 3 3 1 3 3 1

1

3 3 3 3 1 1

1

2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

Y1 X1

w The matrix with numbers is referred to as a cooperation matrix w Each cycle is represented by 𝐷o(𝑌o, 𝑍

, 𝑌o;+ +∈r

s, 𝑍

;( (∈ts, 𝑚o)

Ø Each 𝑌o,𝑍

represent a pair of edges (𝐷2)

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12

SLIDE 44

Cooperation Graphs

14

Co Cooperation Matrix

1

1

1

1 2 2 3 3 1

1

2 2 3 3 1

1

1 2 2 2 2 1 1

1

1 2 2 1 1

1

2 2 2 2 1

1

1 2 1 2 2 2 2 1 2 1

1

2 2 2 2 2 2 1 1

1

2 2 3 3 3 3 3 3 1

1

1 3 3 3 3 1 3 3 1

1

3 3 3 3 1 1

1

2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

X2 Y2 Y1 X1

w The matrix with numbers is referred to as a cooperation matrix w Each cycle is represented by 𝐷o(𝑌o, 𝑍

, 𝑌o;+ +∈r

s, 𝑍

;( (∈ts, 𝑚o)

Ø Each 𝑌o,𝑍

represent a pair of edges (𝐷2) or triangles (𝐷3)

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12

SLIDE 45

Cooperation Graphs

14

Co Cooperation Matrix

1

1

1

1 2 2 3 3 1

1

2 2 3 3 1

1

1 2 2 2 2 1 1

1

1 2 2 1 1

1

2 2 2 2 1

1

1 2 1 2 2 2 2 1 2 1

1

2 2 2 2 2 2 1 1

1

2 2 3 3 3 3 3 3 1

1

1 3 3 3 3 1 3 3 1

1

3 3 3 3 1 1

1

2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

X2 Y2 Y1 X1 X2;10 Y1;9

w The matrix with numbers is referred to as a cooperation matrix w Each cycle is represented by 𝐷o(𝑌o, 𝑍

, 𝑌o;+ +∈r

s, 𝑍

;( (∈ts, 𝑚o)

Ø Each 𝑌o,𝑍

represent a pair of edges (𝐷2) or triangles (𝐷3)

Ø 𝑌o;+ consists of two row indices of nodes at column 𝑘 in 𝐷o

v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12

SLIDE 46

w The matrix with numbers is referred to as a cooperation matrix w Each cycle is represented by 𝐷o(𝑌o, 𝑍

, 𝑌o;+ +∈r

s, 𝑍

;( (∈ts, 𝑚o)

Ø Each 𝑌o,𝑍

represent a pair of edges (𝐷2) or triangles (𝐷3)

Ø 𝑌o;+ consists of two row indices of nodes at column 𝑘 in 𝐷o

Cooperation Graphs

14

Co Cooperation Matrix

1

1

1

1 2 2 3 3 1

1

2 2 3 3 1

1

1 2 2 2 2 1 1

1

1 2 2 1 1

1

2 2 2 2 1

1

1 2 1 2 2 2 2 1 2 1

1

2 2 2 2 2 2 1 1

1

2 2 3 3 3 3 3 3 1

1

1 3 3 3 3 1 3 3 1

1

3 3 3 3 1 1

1

2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12

Cooperation graph: for each cycle 𝐷o, draw an arrow pointing from 𝑌o to 𝑍

and label the arrow with 𝑚o

Co Cooperation Graph: th the label is simplifi fied into to edge ty types (dashed line fo for 2 and dotte tted-das dashed hed line ne for

r 3

X2 Y2 Y1 X1 X2;10 Y1;9 v1 v2 v3 v4 v5 v6 v7 v8 v9 v10 v11 v12

SLIDE 47

w Not all cooperation graphs can result in an appropriate

cooperation matrix

Compatible Graph

15

Compatible graph: a cooperation graph that results in an “appropriate” cooperation matrix

SLIDE 48

w Not all cooperation graphs can result in an appropriate

cooperation matrix

w Sufficient conditions for a cooperation graph being a compatible

graph

Ø All cycles are disjoint Ø All nodes 𝑤( and 𝑤+ such that there exist two cycles labelled

with the same cooperation level, which intersects with columns 𝑗, 𝑘, respectively, and share at least one column, cooperate in their 1-st level cooperation

Compatible Graph

15

Compatible graph: a cooperation graph that results in an “appropriate” cooperation matrix

SLIDE 49

w Not all cooperation graphs can result in an appropriate

cooperation matrix

w Sufficient conditions for a cooperation graph being a compatible

graph

Ø All cycles are disjoint Ø All nodes 𝑤( and 𝑤+ such that there exist two cycles labelled

with the same cooperation level, which intersects with columns 𝑗, 𝑘, respectively, and share at least one column, cooperate in their 1-st level cooperation

w We next present constructions over compatible graphs

Compatible Graph

15

Compatible graph: a cooperation graph that results in an “appropriate” cooperation matrix

SLIDE 50

w Parity part of the generator matrix of a multi-level code

Ø Diagonal components: 𝐁(,( (𝑙(×𝑠

()

Ø 1-st level cooperation: 𝐂(,+𝐕+ (𝑙(×𝜀

+,𝜀 +×𝑠 +), 𝑘 ∈ ℳ(

ℳ(: nodes cooperates with 𝑤( in the 1-st level cooperation

Construction Over Compatible Graphs

16

SLIDE 51

w Parity part of the generator matrix of a multi-level code

Ø Diagonal components: 𝐁(,( (𝑙(×𝑠

()

Ø 1-st level cooperation: 𝐂(,+𝐕+ (𝑙(×𝜀

+,𝜀 +×𝑠 +), 𝑘 ∈ ℳ(

ℳ(: nodes cooperates with 𝑤( in the 1-st level cooperation

Ø Higher level cooperation: 𝐂(,+𝐖

+;B (𝑙(×𝜃+;B,𝜃+;B×𝑠 +)

𝐅(;B;o (𝑙(×𝛿(;o), 𝐂(,+ = 𝐅(;B;o,𝟏{?×(|`;M}~?;s) , 𝑢 ∈ 𝑈(;B,𝑘 ∈ 𝑍
;(
𝑈(;B: indices of all cycles that provide extra parities in the l-th level

cooperation of node 𝑤(

Construction Over Compatible Graphs

16

SLIDE 52

w Parity part of the generator matrix of a multi-level code

Ø Diagonal components: 𝐁(,( (𝑙(×𝑠

()

Ø 1-st level cooperation: 𝐂(,+𝐕+ (𝑙(×𝜀

+,𝜀 +×𝑠 +), 𝑘 ∈ ℳ(

ℳ(: nodes cooperates with 𝑤( in the 1-st level cooperation

Ø Higher level cooperation: 𝐂(,+𝐖

+;B (𝑙(×𝜃+;B,𝜃+;B×𝑠 +)

𝐅(;B;o (𝑙(×𝛿(;o), 𝐂(,+ = 𝐅(;B;o,𝟏{?×(|`;M}~?;s) , 𝑢 ∈ 𝑈(;B,𝑘 ∈ 𝑍
;(
𝑈(;B: indices of all cycles that provide extra parities in the l-th level

cooperation of node 𝑤(

Ø Components are parts of Cauchy matrices 𝐔( on GF(𝑟)

𝑟 is a power of 2

Construction Over Compatible Graphs

16

SLIDE 53

w ECC hierarchy of the construction

Construction Over Compatible Graphs

17

𝑒(,G = 𝑠( − 𝜀( − ∑ 𝜃(;B

>? B•3

, 𝑒(,2 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

, 𝑒(,3 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

+∑ 𝛿(;o

3CB‚CB,o∈ƒ?;M‚

1 1 … 1 1 1 1

Bl

i

SLIDE 54

w ECC hierarchy of the construction

Ø Suppose ℐ(

B = 𝒝( B\𝒝( B}2

ℐ(

B consists of nodes that are involved in the l-th level cooperation of 𝑤(

Construction Over Compatible Graphs

17

𝑒(,G = 𝑠( − 𝜀( − ∑ 𝜃(;B

>? B•3

, 𝑒(,2 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

, 𝑒(,3 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

+∑ 𝛿(;o

3CB‚CB,o∈ƒ?;M‚

1 1 … 1 1 1 1

Bl

i

SLIDE 55

w ECC hierarchy of the construction

Ø Suppose ℐ(

B = 𝒝( B\𝒝( B}2

ℐ(

B consists of nodes that are involved in the l-th level cooperation of 𝑤(

Restriction requires all cycles labelled with 𝑚

that intersects with column 𝑘,𝑘 ∈ ℐ(

2, to be com-

pletelycontained in columns spanned by ℐ(

2

Construction Over Compatible Graphs

17

𝑒(,G = 𝑠( − 𝜀( − ∑ 𝜃(;B

>? B•3

, 𝑒(,2 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

, 𝑒(,3 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

+∑ 𝛿(;o

3CB‚CB,o∈ƒ?;M‚

1 1 … 1 1 1 1

Bl

i

I1

i

Il

i

j

SLIDE 56

w ECC hierarchy of the construction

Ø Suppose ℐ(

B = 𝒝( B\𝒝( B}2

ℐ(

B consists of nodes that are involved in the l-th level cooperation of 𝑤(

Restriction requires all cycles labelled with 𝑚

that intersects with column 𝑘,𝑘 ∈ ℐ(

2, to be com-

pletelycontained in columns spanned by ℐ(

2

Summation of cross parities generated because
f nodes laying on the intersection of those

cycles with column 𝑘 is denoted by 𝐭

+;B

Construction Over Compatible Graphs

17

𝑒(,G = 𝑠( − 𝜀( − ∑ 𝜃(;B

>? B•3

, 𝑒(,2 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

, 𝑒(,3 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

+∑ 𝛿(;o

3CB‚CB,o∈ƒ?;M‚

1 1 … 1 1 1 1

Bl

i

I1

i

Il

i

⊕sj;l

j

SLIDE 57

w ECC hierarchy of the construction

Ø Suppose ℐ(

B = 𝒝( B\𝒝( B}2

ℐ(

B consists of nodes that are involved in the l-th level cooperation of 𝑤(

Restriction requires all cycles labelled with 𝑚

that intersects with column 𝑘,𝑘 ∈ ℐ(

2, to be com-

pletelycontained in columns spanned by ℐ(

2

Summation of cross parities generated because
f nodes laying on the intersection of those

cycles with column 𝑘 is denoted by 𝐭

+;B

The summation of cross parities generated from

all nodes at each cycle is a zero vector

Construction Over Compatible Graphs

17

𝑒(,G = 𝑠( − 𝜀( − ∑ 𝜃(;B

>? B•3

, 𝑒(,2 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

, 𝑒(,3 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

+∑ 𝛿(;o

3CB‚CB,o∈ƒ?;M‚

1 1 … 1 1 1 1

Bl

i

I1

i

Il

i

t1 ⊕ ⊕

sj;l j

SLIDE 58

w ECC hierarchy of the construction

Ø Suppose ℐ(

B = 𝒝( B\𝒝( B}2

ℐ(

B consists of nodes that are involved in the l-th level cooperation of 𝑤(

Restriction requires all cycles labelled with 𝑚

that intersects with column 𝑘,𝑘 ∈ ℐ(

2, to be com-

pletelycontained in columns spanned by ℐ(

2

Summation of cross parities generated because
f nodes laying on the intersection of those

cycles with column 𝑘 is denoted by 𝐭

+;B

The summation of cross parities generated from

all nodes at each cycle is a zero vector

𝐭(;B is the summation of all 𝐭

+;B’s

Construction Over Compatible Graphs

17

𝑒(,G = 𝑠( − 𝜀( − ∑ 𝜃(;B

>? B•3

, 𝑒(,2 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

, 𝑒(,3 = 𝑠( + ∑ 𝜀

+ S`∈ℳ?

+∑ 𝛿(;o

3CB‚CB,o∈ƒ?;M‚

1 1 … 1 1 1 1

Bl

i

I1

i

Il

i

t1 ⊕ ⊕

sj;l j si;l + X

Vi;l\Vj;l6=∅ sj;l = 0

SLIDE 59

Outline

w Introduction

Ø Decentralized storage networks Ø Existing work

w Cooperative Data Protection

Ø Preliminaries Ø ECC hierarchy Ø Single-level cooperation

w Multi-level cooperation

Ø Cooperation graphs and compatible graphs Ø Construction over compatible graphs

w Conclusion

SLIDE 60

Conclusion

w Main contribution

Ø We presented a topology-aware cooperative data protection

scheme for DSNs, which achieved faster recovery speed and corrects more flexible erasure patterns

Ø We incorporated hierarchical coding into the proposed scheme

for lower latency

w Future work

Ø Analysis of recoverable erasure patterns Ø Algorithms that search for cooperation graphs

18

SLIDE 61