Partial MDS Codes with Local Regeneration Lukas Holzbaur , Sven - - PowerPoint PPT Presentation

Partial MDS Codes with Local Regeneration

Lukas Holzbaur, Sven Puchinger, Eitan Yaakobi, Antonia Wachter-Zeh Technical University of Munich Institute for Communications Engineering

Introduction

• Large scale distributed storage systems need to be

efficiently repairable

• Two important measures of efficiency:

1Kamath, Govinda M., et al. "Codes with local regeneration and erasure correction." IEEE Transactions on Information Theory 60.8 (2014): 4637-4660. 2Rawat, Ankit Singh, et al. "Optimal locally repairable and secure codes for distributed storage systems." IEEE Transactions on Information Theory 60.1

(2013): 212-236.

3Krishnan, M. Nikhil, and P

. Vijay Kumar. "Codes with Combined Locality and Regeneration Having Optimal Rate, dmin and Linear Field Size." 2018 IEEE International Symposium on Information Theory (ISIT). IEEE, 2018.

4Calis, Gokhan, and O. Ozan Koyluoglu. "A general construction for PMDS codes." IEEE Communications Letters 21.3 (2016): 452-455.

Lukas Holzbaur (TUM) 2

Introduction

• Large scale distributed storage systems need to be

efficiently repairable

• Two important measures of efficiency:

◮ Locality: Number of nodes involved in repair → LRCs / PMDS codes

1Kamath, Govinda M., et al. "Codes with local regeneration and erasure correction." IEEE Transactions on Information Theory 60.8 (2014): 4637-4660. 2Rawat, Ankit Singh, et al. "Optimal locally repairable and secure codes for distributed storage systems." IEEE Transactions on Information Theory 60.1

(2013): 212-236.

3Krishnan, M. Nikhil, and P

. Vijay Kumar. "Codes with Combined Locality and Regeneration Having Optimal Rate, dmin and Linear Field Size." 2018 IEEE International Symposium on Information Theory (ISIT). IEEE, 2018.

4Calis, Gokhan, and O. Ozan Koyluoglu. "A general construction for PMDS codes." IEEE Communications Letters 21.3 (2016): 452-455.

Lukas Holzbaur (TUM) 2

Introduction

• Large scale distributed storage systems need to be

efficiently repairable

• Two important measures of efficiency:

◮ Locality: Number of nodes involved in repair → LRCs / PMDS codes ◮ Repair Bandwidth: Amount of data transmitted for repair → Regenerating codes

1Kamath, Govinda M., et al. "Codes with local regeneration and erasure correction." IEEE Transactions on Information Theory 60.8 (2014): 4637-4660. 2Rawat, Ankit Singh, et al. "Optimal locally repairable and secure codes for distributed storage systems." IEEE Transactions on Information Theory 60.1

(2013): 212-236.

3Krishnan, M. Nikhil, and P

. Vijay Kumar. "Codes with Combined Locality and Regeneration Having Optimal Rate, dmin and Linear Field Size." 2018 IEEE International Symposium on Information Theory (ISIT). IEEE, 2018.

4Calis, Gokhan, and O. Ozan Koyluoglu. "A general construction for PMDS codes." IEEE Communications Letters 21.3 (2016): 452-455.

Lukas Holzbaur (TUM) 2

Introduction

• Large scale distributed storage systems need to be

efficiently repairable

• Two important measures of efficiency:

◮ Locality: Number of nodes involved in repair → LRCs / PMDS codes ◮ Repair Bandwidth: Amount of data transmitted for repair → Regenerating codes

• Several constructions of locally regenerating codes are

known 123 But: Only one with PMDS24 property, for which the field size is exponential in the length

1Kamath, Govinda M., et al. "Codes with local regeneration and erasure correction." IEEE Transactions on Information Theory 60.8 (2014): 4637-4660. 2Rawat, Ankit Singh, et al. "Optimal locally repairable and secure codes for distributed storage systems." IEEE Transactions on Information Theory 60.1

(2013): 212-236.

3Krishnan, M. Nikhil, and P

. Vijay Kumar. "Codes with Combined Locality and Regeneration Having Optimal Rate, dmin and Linear Field Size." 2018 IEEE International Symposium on Information Theory (ISIT). IEEE, 2018.

4Calis, Gokhan, and O. Ozan Koyluoglu. "A general construction for PMDS codes." IEEE Communications Letters 21.3 (2016): 452-455.

Lukas Holzbaur (TUM) 2

Minimum Storage Regenerating (MSR) Codes

• Storage system storing ℓ(n − r) data symbols on n nodes using [n, n − r; ℓ] code

⇒ r “redundancy nodes”; Each node stores ℓ symbols

Example: n = 8, r = 3 1 2 3 4 5 6 7 8

5Dimakis, Alexandros G., et al. "Network coding for distributed storage systems." IEEE transactions on information theory 56.9 (2010): 4539-4551.

Lukas Holzbaur (TUM) 3

Minimum Storage Regenerating (MSR) Codes

• Storage system storing ℓ(n − r) data symbols on n nodes using [n, n − r; ℓ] code

⇒ r “redundancy nodes”; Each node stores ℓ symbols

• Minimum Storage Regenerating (MSR) Codes

◮ Maximum Distance Separable (MDS) Code ◮ The repair bandwidth is 5 (n − 1) · ℓ

r Example: n = 8, r = 3 1 2 3 4 5 6 7 8

5Dimakis, Alexandros G., et al. "Network coding for distributed storage systems." IEEE transactions on information theory 56.9 (2010): 4539-4551.

Lukas Holzbaur (TUM) 3

Minimum Storage Regenerating (MSR) Codes

• Storage system storing ℓ(n − r) data symbols on n nodes using [n, n − r; ℓ] code

⇒ r “redundancy nodes”; Each node stores ℓ symbols

• Minimum Storage Regenerating (MSR) Codes

◮ Maximum Distance Separable (MDS) Code ◮ The repair bandwidth is 5 (n − 1) · ℓ

r Example: n = 8, r = 3 1 2 3 4 5 6 7 8

5Dimakis, Alexandros G., et al. "Network coding for distributed storage systems." IEEE transactions on information theory 56.9 (2010): 4539-4551.

Lukas Holzbaur (TUM) 3

Minimum Storage Regenerating (MSR) Codes

• Storage system storing ℓ(n − r) data symbols on n nodes using [n, n − r; ℓ] code

⇒ r “redundancy nodes”; Each node stores ℓ symbols

• Minimum Storage Regenerating (MSR) Codes

◮ Maximum Distance Separable (MDS) Code ◮ The repair bandwidth is 5 (n − 1) · ℓ

r Example: n = 8, r = 3 1 2 3 4 5 6 7 8 4′

5Dimakis, Alexandros G., et al. "Network coding for distributed storage systems." IEEE transactions on information theory 56.9 (2010): 4539-4551.

Lukas Holzbaur (TUM) 3

Minimum Storage Regenerating (MSR) Codes

• Storage system storing ℓ(n − r) data symbols on n nodes using [n, n − r; ℓ] code

⇒ r “redundancy nodes”; Each node stores ℓ symbols

• Minimum Storage Regenerating (MSR) Codes

◮ Maximum Distance Separable (MDS) Code ◮ The repair bandwidth is 5 (n − 1) · ℓ

r Example: n = 8, r = 3 1 2 3 4 5 6 7 8 4′

ℓ

Repair Bandwidth: (n − r) · ℓ

5Dimakis, Alexandros G., et al. "Network coding for distributed storage systems." IEEE transactions on information theory 56.9 (2010): 4539-4551.

Lukas Holzbaur (TUM) 3

Minimum Storage Regenerating (MSR) Codes

• Storage system storing ℓ(n − r) data symbols on n nodes using [n, n − r; ℓ] code

⇒ r “redundancy nodes”; Each node stores ℓ symbols

• Minimum Storage Regenerating (MSR) Codes

◮ Maximum Distance Separable (MDS) Code ◮ The repair bandwidth is 5 (n − 1) · ℓ

r Example: n = 8, r = 3 1 2 3 4 5 6 7 8 4′

ℓ

r Repair Bandwidth: (n − 1) · ℓ

r

5Dimakis, Alexandros G., et al. "Network coding for distributed storage systems." IEEE transactions on information theory 56.9 (2010): 4539-4551.

Lukas Holzbaur (TUM) 3

Ye-Barg MSR Codes 6

Construction: Ye-Barg MSR Codes, Construction 1; Ye et al.6

Let C ⊂ Fq be an [n, n − r; ℓ] array code over Fq, where q ≥ rn. Let {βi,j}i∈[n],j∈[r] be a set of rn distinct elements of Fq. Then each codeword is an array with ℓ = r n rows and n columns, where the a-th row fulfills the parity check equations H(a) =

    

1 1

. . .

1

β1,a1 β2,a2 . . . βn,an

. . . . . . . . .

βr−1

1,a1 βr−1 2,a2 . . . βr−1 n,an

     ,

for a ∈ [0, ℓ − 1] and a = n

i=1 air i−1.

6Ye, Min, and Alexander Barg. "Explicit constructions of high-rate MDS array codes with optimal repair bandwidth." IEEE Transactions on Information

Theory 63.4 (2017): 2001-2014. Lukas Holzbaur (TUM) 4

Ye-Barg MSR Codes 6

Construction: Ye-Barg MSR Codes, Construction 1; Ye et al.6

Let C ⊂ Fq be an [n, n − r; ℓ] array code over Fq, where q ≥ rn. Let {βi,j}i∈[n],j∈[r] be a set of rn distinct elements of Fq. Then each codeword is an array with ℓ = r n rows and n columns, where the a-th row fulfills the parity check equations H(a) =

    

1 1

. . .

1

β1,a1 β2,a2 . . . βn,an

. . . . . . . . .

βr−1

1,a1 βr−1 2,a2 . . . βr−1 n,an

     ,

for a ∈ [0, ℓ − 1] and a = n

i=1 air i−1.

6Ye, Min, and Alexander Barg. "Explicit constructions of high-rate MDS array codes with optimal repair bandwidth." IEEE Transactions on Information

Theory 63.4 (2017): 2001-2014. Lukas Holzbaur (TUM) 4

Ye-Barg MSR Codes6

Parity check matrix of row a H(a) =

    

1 1

. . .

1

β1,a1 β2,a2 . . . βn,an

. . . . . . . . .

βr−1

1,a1 βr−1 2,a2 . . . βr−1 n,an

    

6Ye, Min, and Alexander Barg. "Explicit constructions of high-rate MDS array codes with optimal repair bandwidth." IEEE Transactions on Information

Theory 63.4 (2017): 2001-2014. Lukas Holzbaur (TUM) 5

Ye-Barg MSR Codes6

Parity check matrix of row a H(a) =

    

1 1

. . .

1

β1,a1 β2,a2 . . . βn,an

. . . . . . . . .

βr−1

1,a1 βr−1 2,a2 . . . βr−1 n,an

    

Codeword of a Ye-Barg MSR Code c =

    

c(1) c(2) . . . c(ℓ)

     ∈ GRS(n, n − r, L(1)) ∈ GRS(n, n − r, L(2)) . . . ∈ GRS(n, n − r, L(ℓ)) ⇒ Each individual row is a codeword of an RS code with specific locators

6Ye, Min, and Alexander Barg. "Explicit constructions of high-rate MDS array codes with optimal repair bandwidth." IEEE Transactions on Information

Theory 63.4 (2017): 2001-2014. Lukas Holzbaur (TUM) 5

Partial MDS codes (Maximally Recoverable LRC)

Definition: Partial MDS codes

The code C is a PMDS(µn, µ, n, r, s; ℓ) partial MDS array code if there exists a partition W

= {W1, W2, ..., Wµ} of [µn] with |Wi| = n for all i ∈ [µ] such that

Lukas Holzbaur (TUM) 6

Partial MDS codes (Maximally Recoverable LRC)

Definition: Partial MDS codes

The code C is a PMDS(µn, µ, n, r, s; ℓ) partial MDS array code if there exists a partition W

= {W1, W2, ..., Wµ} of [µn] with |Wi| = n for all i ∈ [µ] such that

• the code C|Wi is an [n, n − r, r + 1; ℓ] MDS code for all i ∈ [µ] and

Lukas Holzbaur (TUM) 6

Partial MDS codes (Maximally Recoverable LRC)

Definition: Partial MDS codes

The code C is a PMDS(µn, µ, n, r, s; ℓ) partial MDS array code if there exists a partition W

= {W1, W2, ..., Wµ} of [µn] with |Wi| = n for all i ∈ [µ] such that

• the code C|Wi is an [n, n − r, r + 1; ℓ] MDS code for all i ∈ [µ] and
• the code C|[µn]\∪µ

i=1Ei is an [µn − rµ, n − rµ − s, s + 1; ℓ] MDS code, where Ei ⊂ Wi with |Ei| = r for

all i ∈ [µ].

Lukas Holzbaur (TUM) 6

Partial MDS codes (Maximally Recoverable LRC)

Definition: Partial MDS codes

The code C is a PMDS(µn, µ, n, r, s; ℓ) partial MDS array code if there exists a partition W

= {W1, W2, ..., Wµ} of [µn] with |Wi| = n for all i ∈ [µ] such that

• the code C|Wi is an [n, n − r, r + 1; ℓ] MDS code for all i ∈ [µ] and
• the code C|[µn]\∪µ

i=1Ei is an [µn − rµ, n − rµ − s, s + 1; ℓ] MDS code, where Ei ⊂ Wi with |Ei| = r for

all i ∈ [µ].

G

k puncture

GMDS

Lukas Holzbaur (TUM) 6

Partial MDS codes (Maximally Recoverable LRC)

Parity-check matrix of each local code: H0 ∈ Fr×n

q

Parity-check matrix of a PMDS code: H =

     

H0

. . .

H0 . . . . . . . . . ... . . .

. . . H0

H1 H2 . . . Hµ

      ∈ Frµ+s×µn

q Lukas Holzbaur (TUM) 7

Locally Regenerating PMDS Codes

PMDS code where each local code is an MSR code

C

ℓ

puncture

CMDS

Lukas Holzbaur (TUM) 8

Locally Regenerating PMDS Codes

PMDS code where each local code is an MSR code

C

ℓ

puncture

CMDS CMSR

local codeword

Lukas Holzbaur (TUM) 8

PMDS codes for s = 2; Blaum et al.7

Construction: PMDS codes with s = 2; Blaum et al.7

Let O(β) ≥ µN. The [µn, µ(n − r) − 2] code C(µ, n, r, 2, N) is given by the (rµ + 2) × µn parity-check matrix

7Blaum, Mario, et al. "Construction of partial MDS and sector-disk codes with two global parity symbols." IEEE Transactions on Information Theory 62.5

(2016): 2673-2681. Lukas Holzbaur (TUM) 9

PMDS codes for s = 2; Blaum et al.7

Construction: PMDS codes with s = 2; Blaum et al.7

Let O(β) ≥ µN. The [µn, µ(n − r) − 2] code C(µ, n, r, 2, N) is given by the (rµ + 2) × µn parity-check matrix H =

  

H0 0

...

0 H0 ...

. . . . . . ... . . .

... H0

H1 H2 ... Hµ

   ,

H0 =

  

1 1

...

1 1

β ... βn−1

1

β2 ... β2(n−1)

. . . . . . ... . . .

1 βr−1 ... β(r−1)(n−1)

  

7Blaum, Mario, et al. "Construction of partial MDS and sector-disk codes with two global parity symbols." IEEE Transactions on Information Theory 62.5

(2016): 2673-2681. Lukas Holzbaur (TUM) 9

PMDS codes for s = 2; Blaum et al.7

Construction: PMDS codes with s = 2; Blaum et al.7

Let O(β) ≥ µN. The [µn, µ(n − r) − 2] code C(µ, n, r, 2, N) is given by the (rµ + 2) × µn parity-check matrix H =

  

H0 0

...

0 H0 ...

. . . . . . ... . . .

... H0

H1 H2 ... Hµ

   ,

H0 =

  

1 1

...

1 1

β ... βn−1

1

β2 ... β2(n−1)

. . . . . . ... . . .

1 βr−1 ... β(r−1)(n−1)

  

and for 0 ≤ j ≤ µ − 1, Hj+1 =

• 1

βr ... βr(n−1) β−jN β−jN−1 ... β−jN−n+1

• .

For N = (r + 1)(n − 1 − r) + 1 the code C is an PMDS code.

7Blaum, Mario, et al. "Construction of partial MDS and sector-disk codes with two global parity symbols." IEEE Transactions on Information Theory 62.5

(2016): 2673-2681. Lukas Holzbaur (TUM) 9

PMDS codes with s = 2 and local MSR codes

Construction 1

Let O(β) ≥ µN and βi,j = βi−1+(j−1)n. We define the array code C(µ, n, r, 2, N, d; ℓ)q as

  C =  

c(0) c(1)

. . .

c(ℓ−1)

  ∈ Fℓ×µn

q

: H(a)c(a) = 0 ∀ a = 0, . . . , ℓ − 1    ,

Lukas Holzbaur (TUM) 10

PMDS codes with s = 2 and local MSR codes

Construction 1

Let O(β) ≥ µN and βi,j = βi−1+(j−1)n. We define the array code C(µ, n, r, 2, N, d; ℓ)q as

  C =  

c(0) c(1)

. . .

c(ℓ−1)

  ∈ Fℓ×µn

q

: H(a)c(a) = 0 ∀ a = 0, . . . , ℓ − 1    ,

The matrix H(a) is defined as H(a) =

    

H(a)

...

H(a)

...

. . . . . . ...

··· ... H(a)

H(a)

1

H(a)

2

... H(a)

µ

     ∈ Frµ+2×µn

q

,

H(a)

=  

1 1

...

1

β1,a1 β2,a2 ... βn,an

. . . . . . . . .

βr−1

1,a1 βr−1 2,a2 ... βr−1 n,an

  ∈ Fr×n

q

,

with a ∈ [0, ℓ − 1] and a = n

i=1 air i−1. Lukas Holzbaur (TUM) 10

PMDS codes with s = 2 and local MSR codes

Construction 1

Let O(β) ≥ µN and βi,j = βi−1+(j−1)n. We define the array code C(µ, n, r, 2, N, d; ℓ)q as

  C =  

c(0) c(1)

. . .

c(ℓ−1)

  ∈ Fℓ×µn

q

: H(a)c(a) = 0 ∀ a = 0, . . . , ℓ − 1    ,

The matrix H(a) is defined as H(a) =

    

H(a)

...

H(a)

...

. . . . . . ...

··· ... H(a)

H(a)

1

H(a)

2

... H(a)

µ

     ∈ Frµ+2×µn

q

,

H(a)

=  

1 1

...

1

β1,a1 β2,a2 ... βn,an

. . . . . . . . .

βr−1

1,a1 βr−1 2,a2 ... βr−1 n,an

  ∈ Fr×n

q

,

with a ∈ [0, ℓ − 1] and a = n

i=1 air i−1. For 0 ≤ j ≤ µ − 1 let

H(a)

j+1 =

• βr

1,a1

βr

2,a2

... βr

n,an

β−jNβ−1

1,a1 β−jNβ−1 2,a2 ... β−jNβ−1 n,an

• ∈ F2×n

q

.

Lukas Holzbaur (TUM) 10

PMDS codes with s = 2 and local MSR codes

Construction 1

Let O(β) ≥ µN and βi,j = βi−1+(j−1)n. We define the array code C(µ, n, r, 2, N, d; ℓ)q as

  C =  

c(0) c(1)

. . .

c(ℓ−1)

  ∈ Fℓ×µn

q

: H(a)c(a) = 0 ∀ a = 0, . . . , ℓ − 1    ,

The matrix H(a) is defined as H(a) =

    

H(a)

...

H(a)

...

. . . . . . ...

··· ... H(a)

H(a)

1

H(a)

2

... H(a)

µ

     ∈ Frµ+2×µn

q

,

H(a)

=  

1 1

...

1

β1,a1 β2,a2 ... βn,an

. . . . . . . . .

βr−1

1,a1 βr−1 2,a2 ... βr−1 n,an

  ∈ Fr×n

q

,

with a ∈ [0, ℓ − 1] and a = n

i=1 air i−1. For 0 ≤ j ≤ µ − 1 let

H(a)

j+1 =

• βr

1,a1

βr

2,a2

... βr

n,an

β−jNβ−1

1,a1 β−jNβ−1 2,a2 ... β−jNβ−1 n,an

• ∈ F2×n

q

.

Lukas Holzbaur (TUM) 10

PMDS codes with s = 2 and local MSR codes

Recall: PMDS codes over Fq with q ≥ Nµ with N = (r + 1)(n − 1 − r) + 1 .

Lemma

The code is a locally regenerating PMDS code for any N

≥ (r + 1)(max

i,j logβ(βi,j) − r) + 1

≥ (r + 1)(rn − 1 − r) + 1 .

Lukas Holzbaur (TUM) 11

PMDS codes over small fields; Gabrys et al. 8

Construction: PMDS codes for any s; Gabrys et al. 8

The [µn, µ(n − r) − s] code C(µ, n, r, s, H0, Γ) is given by the (rµ + s) × µn parity-check matrix H =

  

H0 0

...

0 H0 ...

. . . . . . ... . . .

... H0

H1 H2 ... Hµ

  

where for 1 ≤ j ≤ µ, Hj =

  

αj,1 αj,2 ... αj,n αq

j,1

αq

j,2

... αq

j,n

. . . . . . ... . . .

αqs−1

j,1

αqs−1

j,2

... αqs−1

j,n

   .

Whether this code is PMDS/SD depends on the choice of the αj,i.

8Gabrys, Ryan, et al. "Constructions of partial MDS codes over small fields." IEEE Transactions on Information Theory 65.6 (2018): 3692-3701.

Lukas Holzbaur (TUM) 12

Locally Regenerating PMDS Codes for any s

Lemma

The code construction by Gabrys et al.8 is a PMDS code for any H0 that is a parity-check matrix of an

[n, n − r]q MDS code.

Theorem

There exists a locally regenerating (µ, n, r, s; ℓ)qM PMDS code with qM ≤ 2nr(2nµ)s(r+1)−1 .

8Gabrys, Ryan, et al. "Constructions of partial MDS codes over small fields." IEEE Transactions on Information Theory 65.6 (2018): 3692-3701.

Lukas Holzbaur (TUM) 13

Conclusion and Future Work

Conclusion

• Constructed PMDS codes where every local code is an MSR code
• The field size is significantly smaller than in previously known constructions

Future Work

• Apply to other constructions of PMDS and MSR codes
• Construct globally PMDS codes with global regeneration

Thank you for your attention!

Lukas Holzbaur (TUM) 14