Coding Schemes for Distributed Storage Systems Min Ye Advisor: Alexander Barg ECE/ISR, University of Maryland
August 11, 2017
Motivation: Distributed Storage Systems (DSS)
• DSS spread data across thousands of storage nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
2/1
Motivation: Distributed Storage Systems (DSS)
• DSS spread data across thousands of storage nodes • Individual storage nodes fail frequently
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
2/1
Motivation: Distributed Storage Systems (DSS)
• DSS spread data across thousands of storage nodes • Individual storage nodes fail frequently • Need redundancy to protect data
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
2/1
Replication: large storage overhead
Can tolerate any 2 node failures Storage overhead = 3×
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
3/1
Erasure-Correcting Codes: optimal storage efficiency
Add 2 parity nodes to every 3 data nodes Form an (n = 5, k = 3) code
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
4/1
Erasure-Correcting codes: optimal storage efficiency
Can tolerate any 2 node failures Storage overhead = 1.67×
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
5/1
Disk I/O and network flow of the repair process Repair a failed node
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
6/1
Disk I/O and network flow of the repair process Repair a failed node
Access more nodes ⇒ greater number of disk I/O operations
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
6/1
Disk I/O and network flow of the repair process Repair a failed node
Access more nodes ⇒ greater number of disk I/O operations Download more data from functioning nodes ⇒ larger network flow Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
6/1
Compare Erasure Codes to Replication
Storage overhead Number of disk I/O (access) Network flow (bandwidth)
Min Ye, Ph.D. Dissertation Defense
Erasure Codes Small Large Large
Coding Schemes for Distributed Storage Systems
Replication Large Small Small
August 11, 2017
7/1
Compare Erasure Codes to Replication
Storage overhead Number of disk I/O (access) Network flow (bandwidth)
Erasure Codes Small Large Large
Replication Large Small Small
We prefer Erasure Codes due to their small storage overhead
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
7/1
Compare Erasure Codes to Replication
Storage overhead Number of disk I/O (access) Network flow (bandwidth)
Erasure Codes Small Large Large
Replication Large Small Small
We prefer Erasure Codes due to their small storage overhead
Want to improve the access and bandwidth of Erasure Codes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
7/1
Important metrics and operations Metric
Operation
locality/ access
repair
bandwidth/ communication
error correction
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
8/1
Important metrics and operations Operation
Metric 1
locality/ access bandwidth/ communication
repair
error correction
1 Locally Recoverable codes (local recovery)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
8/1
Important metrics and operations Operation
Metric 1
locality/ access
repair 2
bandwidth/ communication
error correction
1 Locally Recoverable codes (local recovery) 2 Regenerating codes (local recovery)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
8/1
Important metrics and operations Operation
Metric 1
locality/ access
repair
3 2
bandwidth/ communication
error correction
1 Locally Recoverable codes (local recovery) 2 Regenerating codes (local recovery) 3 LDPC codes (global recovery)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
8/1
Important metrics and operations Operation
Metric 1
locality/ access
repair
3 2
bandwidth/ communication
4
error correction
1 Locally Recoverable codes (local recovery) 2 Regenerating codes (local recovery) 3 LDPC codes (global recovery) 4 Fractional decoding (global recovery) Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
8/1
MDS codes
• (n, k, `) MDS array code (vector codes) over field F:
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
9/1
MDS codes
• (n, k, `) MDS array code (vector codes) over field F: • code length n
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
9/1
MDS codes
• (n, k, `) MDS array code (vector codes) over field F: • code length n • k data nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
9/1
MDS codes
• (n, k, `) MDS array code (vector codes) over field F: • code length n • k data nodes • r = n − k parity nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
9/1
MDS codes
• (n, k, `) MDS array code (vector codes) over field F: • code length n • k data nodes • r = n − k parity nodes • each codeword coordinate is a vector of dimension ` over F
Each Ci is a vector in F `
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
9/1
MDS codes
• (n, k, `) MDS array code (vector codes) over field F: • code length n • k data nodes • r = n − k parity nodes • each codeword coordinate is a vector of dimension ` over F
Each Ci is a vector in F ` • contents of any r nodes can be determined by the other k nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
9/1
MDS codes
• (n, k, `) MDS array code (vector codes) over field F: • code length n • k data nodes • r = n − k parity nodes • each codeword coordinate is a vector of dimension ` over F
Each Ci is a vector in F ` • contents of any r nodes can be determined by the other k nodes
• MDS scalar codes: each coordinate is a scalar in F
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
9/1
MDS codes
• (n, k, `) MDS array code (vector codes) over field F: • code length n • k data nodes • r = n − k parity nodes • each codeword coordinate is a vector of dimension ` over F
Each Ci is a vector in F ` • contents of any r nodes can be determined by the other k nodes
• MDS scalar codes: each coordinate is a scalar in F • View scalar codes as vector codes over some subfield of F
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
9/1
MDS codes
• (n, k, `) MDS array code (vector codes) over field F: • code length n • k data nodes • r = n − k parity nodes • each codeword coordinate is a vector of dimension ` over F
Each Ci is a vector in F ` • contents of any r nodes can be determined by the other k nodes
• MDS scalar codes: each coordinate is a scalar in F • View scalar codes as vector codes over some subfield of F • ` is the degree of the field extension
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
9/1
Main focus and background
• Main focus: The bandwidth of MDS codes for the repair task and error correction
task.
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
10 / 1
Main focus and background
• Main focus: The bandwidth of MDS codes for the repair task and error correction
task. • Repair bandwidth: the minimum amount of data we need to download in order to
recover failed nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
10 / 1
Main focus and background
• Main focus: The bandwidth of MDS codes for the repair task and error correction
task. • Repair bandwidth: the minimum amount of data we need to download in order to
recover failed nodes • MDS codes with smallest possible repair bandwidth are called Minimum Storage
Regenerating (MSR) Codes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
10 / 1
Main contributions
• We give explicit constructions of MSR codes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
11 / 1
Main contributions
• We give explicit constructions of MSR codes • MSR codes exist, but explicit constructions were only available for some particular
parameters.
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
11 / 1
Main contributions
• We give explicit constructions of MSR codes • MSR codes exist, but explicit constructions were only available for some particular
parameters. • Explicit construction of MSR codes for general parameters was an open problem.
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
11 / 1
Main contributions
• We give explicit constructions of MSR codes • MSR codes exist, but explicit constructions were only available for some particular
parameters. • Explicit construction of MSR codes for general parameters was an open problem. • We give various explicit constructions of such codes for any given parameters.
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
11 / 1
Main contributions
• We give explicit constructions of MSR codes • MSR codes exist, but explicit constructions were only available for some particular
parameters. • Explicit construction of MSR codes for general parameters was an open problem. • We give various explicit constructions of such codes for any given parameters. • The first explicit constructions for general parameters in the literature
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
11 / 1
Main contributions
• We give explicit constructions of MSR codes • MSR codes exist, but explicit constructions were only available for some particular
parameters. • Explicit construction of MSR codes for general parameters was an open problem. • We give various explicit constructions of such codes for any given parameters. • The first explicit constructions for general parameters in the literature
• We introduce the “Fractional decoding” problem: study the optimal bandwidth of
MDS codes for error correction task.
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
11 / 1
Main contributions
• We give explicit constructions of MSR codes • MSR codes exist, but explicit constructions were only available for some particular
parameters. • Explicit construction of MSR codes for general parameters was an open problem. • We give various explicit constructions of such codes for any given parameters. • The first explicit constructions for general parameters in the literature
• We introduce the “Fractional decoding” problem: study the optimal bandwidth of
MDS codes for error correction task. • We give lower bound and explicit constructions that achieve the lower bound
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
11 / 1
Cut-set bound for MDS array codes • Cut-set bound for the repair of single node failure • (n, k, `) MDS array code over field F
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
12 / 1
Cut-set bound for MDS array codes • Cut-set bound for the repair of single node failure • (n, k, `) MDS array code over field F • connect to d surviving (helper) nodes to recover the failed node
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
12 / 1
Cut-set bound for MDS array codes • Cut-set bound for the repair of single node failure • (n, k, `) MDS array code over field F • connect to d surviving (helper) nodes to recover the failed node • the number of ‘downloaded’ symbols in F is called the repair bandwidth β
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
12 / 1
Cut-set bound for MDS array codes • Cut-set bound for the repair of single node failure • (n, k, `) MDS array code over field F • connect to d surviving (helper) nodes to recover the failed node • the number of ‘downloaded’ symbols in F is called the repair bandwidth β β≥
Min Ye, Ph.D. Dissertation Defense
d` d+1−k
(Dimakis et al., 2010)
Coding Schemes for Distributed Storage Systems
August 11, 2017
12 / 1
Cut-set bound for MDS array codes • Cut-set bound for the repair of single node failure • (n, k, `) MDS array code over field F • connect to d surviving (helper) nodes to recover the failed node • the number of ‘downloaded’ symbols in F is called the repair bandwidth β β≥
d` d+1−k
(Dimakis et al., 2010)
• The right-hand side is a decreasing function of d, thus β → min if d = n − 1
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
12 / 1
Cut-set bound for MDS array codes • Cut-set bound for the repair of single node failure • (n, k, `) MDS array code over field F • connect to d surviving (helper) nodes to recover the failed node • the number of ‘downloaded’ symbols in F is called the repair bandwidth β β≥
d` d+1−k
(Dimakis et al., 2010)
• The right-hand side is a decreasing function of d, thus β → min if d = n − 1
• Cut-set bound for the repair of multiple node failures • h failed nodes, d helper nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
12 / 1
Cut-set bound for MDS array codes • Cut-set bound for the repair of single node failure • (n, k, `) MDS array code over field F • connect to d surviving (helper) nodes to recover the failed node • the number of ‘downloaded’ symbols in F is called the repair bandwidth β β≥
d` d+1−k
(Dimakis et al., 2010)
• The right-hand side is a decreasing function of d, thus β → min if d = n − 1
• Cut-set bound for the repair of multiple node failures • h failed nodes, d helper nodes β≥
Min Ye, Ph.D. Dissertation Defense
dh` d+h−k
(Cadambe et al., 2013)
Coding Schemes for Distributed Storage Systems
August 11, 2017
12 / 1
Cut-set bound for MDS array codes • Cut-set bound for the repair of single node failure • (n, k, `) MDS array code over field F • connect to d surviving (helper) nodes to recover the failed node • the number of ‘downloaded’ symbols in F is called the repair bandwidth β β≥
d` d+1−k
(Dimakis et al., 2010)
• The right-hand side is a decreasing function of d, thus β → min if d = n − 1
• Cut-set bound for the repair of multiple node failures • h failed nodes, d helper nodes β≥
dh` d+h−k
(Cadambe et al., 2013)
• (h, d)-optimal repair property: if the lower bound can be achieved for the recovery
of any h failed nodes from any d helper nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
12 / 1
(1, d)-optimal repair property • Most common scenario: single node failure
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
13 / 1
(1, d)-optimal repair property • Most common scenario: single node failure • Most works devoted to MDS array codes with (1, d)-optimal repair property (MSR
codes)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
13 / 1
(1, d)-optimal repair property • Most common scenario: single node failure • Most works devoted to MDS array codes with (1, d)-optimal repair property (MSR
codes) • Explicit low rate (≤ 1/2) construction is known (Rashmi et al., ’11, Suh et al., ’11)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
13 / 1
(1, d)-optimal repair property • Most common scenario: single node failure • Most works devoted to MDS array codes with (1, d)-optimal repair property (MSR
codes) • Explicit low rate (≤ 1/2) construction is known (Rashmi et al., ’11, Suh et al., ’11) • High rate regime (> 1/2) • r ≤ 3: explicit construction of MDS codes with (1, n − 1)-optimal repair property available (Wang et al., ’11, Tamo et al., ’13, Papailiopoulos et al., ’13, Raviv et al., ’15) • r > 3: only existence proofs, over large enough finite fields (Tamo et al., ’13, etc.)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
13 / 1
(1, d)-optimal repair property • Most common scenario: single node failure • Most works devoted to MDS array codes with (1, d)-optimal repair property (MSR
codes) • Explicit low rate (≤ 1/2) construction is known (Rashmi et al., ’11, Suh et al., ’11) • High rate regime (> 1/2) • r ≤ 3: explicit construction of MDS codes with (1, n − 1)-optimal repair property available (Wang et al., ’11, Tamo et al., ’13, Papailiopoulos et al., ’13, Raviv et al., ’15) • r > 3: only existence proofs, over large enough finite fields (Tamo et al., ’13, etc.) • Various papers relaxed the condition of (1, d)-optimal repair property – only
require optimal repair of systematic nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
13 / 1
(1, d)-optimal repair property • Most common scenario: single node failure • Most works devoted to MDS array codes with (1, d)-optimal repair property (MSR
codes) • Explicit low rate (≤ 1/2) construction is known (Rashmi et al., ’11, Suh et al., ’11) • High rate regime (> 1/2) • r ≤ 3: explicit construction of MDS codes with (1, n − 1)-optimal repair property available (Wang et al., ’11, Tamo et al., ’13, Papailiopoulos et al., ’13, Raviv et al., ’15) • r > 3: only existence proofs, over large enough finite fields (Tamo et al., ’13, etc.) • Various papers relaxed the condition of (1, d)-optimal repair property – only
require optimal repair of systematic nodes • Even under this relaxed requirement, no explicit construction are known for rate larger
than 1/2 and r > 3
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
13 / 1
(1, d)-optimal repair property • Most common scenario: single node failure • Most works devoted to MDS array codes with (1, d)-optimal repair property (MSR
codes) • Explicit low rate (≤ 1/2) construction is known (Rashmi et al., ’11, Suh et al., ’11) • High rate regime (> 1/2) • r ≤ 3: explicit construction of MDS codes with (1, n − 1)-optimal repair property available (Wang et al., ’11, Tamo et al., ’13, Papailiopoulos et al., ’13, Raviv et al., ’15) • r > 3: only existence proofs, over large enough finite fields (Tamo et al., ’13, etc.) • Various papers relaxed the condition of (1, d)-optimal repair property – only
require optimal repair of systematic nodes • Even under this relaxed requirement, no explicit construction are known for rate larger
than 1/2 and r > 3 • High-rate constructions are the most important in practice: Facebook employs
MDS code with rate 0.71 and 4 parity nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
13 / 1
Our results: Explicit high-rate constructions • We solve the open problem of constructing explicit high-rate MSR codes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
14 / 1
Our results: Explicit high-rate constructions • We solve the open problem of constructing explicit high-rate MSR codes • Start from basic case: explicit constructions of MDS codes with (1, n − 1)-optimal
repair property (the first explicit high-rate construction for r > 3 in the literature)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
14 / 1
Our results: Explicit high-rate constructions • We solve the open problem of constructing explicit high-rate MSR codes • Start from basic case: explicit constructions of MDS codes with (1, n − 1)-optimal
repair property (the first explicit high-rate construction for r > 3 in the literature) • Simple extension: explicit constructions of MDS codes with (1, d)-optimal repair
property, k ≤ d ≤ n − 1
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
14 / 1
Our results: Explicit high-rate constructions • We solve the open problem of constructing explicit high-rate MSR codes • Start from basic case: explicit constructions of MDS codes with (1, n − 1)-optimal
repair property (the first explicit high-rate construction for r > 3 in the literature) • Simple extension: explicit constructions of MDS codes with (1, d)-optimal repair
property, k ≤ d ≤ n − 1 • Further extension: Given any n and r, we present explicit (n, k = n − r, `) MDS
array codes with universal (h, d)-optimal repair property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously.
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
14 / 1
Our results: Explicit high-rate constructions • We solve the open problem of constructing explicit high-rate MSR codes • Start from basic case: explicit constructions of MDS codes with (1, n − 1)-optimal
repair property (the first explicit high-rate construction for r > 3 in the literature) • Simple extension: explicit constructions of MDS codes with (1, d)-optimal repair
property, k ≤ d ≤ n − 1 • Further extension: Given any n and r, we present explicit (n, k = n − r, `) MDS
array codes with universal (h, d)-optimal repair property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously. • Our construction is available for any rate, any number of parity nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
14 / 1
Our results: Explicit high-rate constructions • We solve the open problem of constructing explicit high-rate MSR codes • Start from basic case: explicit constructions of MDS codes with (1, n − 1)-optimal
repair property (the first explicit high-rate construction for r > 3 in the literature) • Simple extension: explicit constructions of MDS codes with (1, d)-optimal repair
property, k ≤ d ≤ n − 1 • Further extension: Given any n and r, we present explicit (n, k = n − r, `) MDS
array codes with universal (h, d)-optimal repair property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously. • Our construction is available for any rate, any number of parity nodes • The only explicit construction for h > 1 in the literature
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
14 / 1
Our results: Explicit high-rate constructions • We solve the open problem of constructing explicit high-rate MSR codes • Start from basic case: explicit constructions of MDS codes with (1, n − 1)-optimal
repair property (the first explicit high-rate construction for r > 3 in the literature) • Simple extension: explicit constructions of MDS codes with (1, d)-optimal repair
property, k ≤ d ≤ n − 1 • Further extension: Given any n and r, we present explicit (n, k = n − r, `) MDS
array codes with universal (h, d)-optimal repair property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously. • Our construction is available for any rate, any number of parity nodes • The only explicit construction for h > 1 in the literature • Can optimally recover any number of erasures from any set of helper nodes. Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
14 / 1
Construction 1: (1, n − 1)-optimal repair MDS codes • (n, k = n − r, ` = rn ) MDS array code which can optimally repair any single node
failure from all the other surviving nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
15 / 1
Construction 1: (1, n − 1)-optimal repair MDS codes • (n, k = n − r, ` = rn ) MDS array code which can optimally repair any single node
failure from all the other surviving nodes • Ci = (ci,0 , ci,1 , . . . , ci.`−1 )T
Min Ye, Ph.D. Dissertation Defense
c1,0 c1,1 .. .
c2,0 c2,1 .. .
c1,`−1
c2,`−1
... ... .. . ...
cn,0 cn,1 .. . cn,`−1
Coding Schemes for Distributed Storage Systems
August 11, 2017
15 / 1
Construction 1: (1, n − 1)-optimal repair MDS codes • (n, k = n − r, ` = rn ) MDS array code which can optimally repair any single node
failure from all the other surviving nodes • Ci = (ci,0 , ci,1 , . . . , ci.`−1 )T
c1,0 c1,1 .. .
c2,0 c2,1 .. .
c1,`−1
c2,`−1
... ... .. . ...
cn,0 cn,1 .. . cn,`−1
• Each row forms a Generalized Reed-Solomon code with different evaluation points
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
15 / 1
Construction 1: (1, n − 1)-optimal repair MDS codes • (n, k = n − r, ` = rn ) MDS array code which can optimally repair any single node
failure from all the other surviving nodes • Ci = (ci,0 , ci,1 , . . . , ci.`−1 )T
c1,0 c1,1 .. .
c2,0 c2,1 .. .
c1,`−1
c2,`−1
... ... .. . ...
cn,0 cn,1 .. . cn,`−1
• Each row forms a Generalized Reed-Solomon code with different evaluation points
• Evaluation points λi,j , 1 ≤ i ≤ n, 0 ≤ j ≤ r − 1: rn distinct elements in F
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
15 / 1
Construction 1: (1, n − 1)-optimal repair MDS codes • (n, k = n − r, ` = rn ) MDS array code which can optimally repair any single node
failure from all the other surviving nodes • Ci = (ci,0 , ci,1 , . . . , ci.`−1 )T
c1,0 c1,1 .. .
c2,0 c2,1 .. .
c1,`−1
c2,`−1
... ... .. . ...
cn,0 cn,1 .. . cn,`−1
• Each row forms a Generalized Reed-Solomon code with different evaluation points
• Evaluation points λi,j , 1 ≤ i ≤ n, 0 ≤ j ≤ r − 1: rn distinct elements in F • F: any finite field with size |F| ≥ rn
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
15 / 1
Construction 1: (1, n − 1)-optimal repair MDS codes
• ` = rn (r-ary expansion idea, Cadambe et al., ’11, Tamo et al., ’13)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
16 / 1
Construction 1: (1, n − 1)-optimal repair MDS codes
• ` = rn (r-ary expansion idea, Cadambe et al., ’11, Tamo et al., ’13) • For a = 0, 1, . . . , ` − 1, write r-ary expansion a = (a1 , a2 , . . . , an )
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
16 / 1
Construction 1: (1, n − 1)-optimal repair MDS codes
• ` = rn (r-ary expansion idea, Cadambe et al., ’11, Tamo et al., ’13) • For a = 0, 1, . . . , ` − 1, write r-ary expansion a = (a1 , a2 , . . . , an ) • Evaluation points for a-th row: (λ1,a1 , λ2,a2 , . . . , λn,an )
λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
t = 0, 1, . . . , r − 1
August 11, 2017
16 / 1
Construction 1: (1, n − 1)-optimal repair MDS codes
• ` = rn (r-ary expansion idea, Cadambe et al., ’11, Tamo et al., ’13) • For a = 0, 1, . . . , ` − 1, write r-ary expansion a = (a1 , a2 , . . . , an ) • Evaluation points for a-th row: (λ1,a1 , λ2,a2 , . . . , λn,an )
λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,
t = 0, 1, . . . , r − 1
• MDS property follows from the fact that each row is an MDS code
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
16 / 1
Construction 1: (1, n − 1)-optimal repair MDS codes
• ` = rn (r-ary expansion idea, Cadambe et al., ’11, Tamo et al., ’13) • For a = 0, 1, . . . , ` − 1, write r-ary expansion a = (a1 , a2 , . . . , an ) • Evaluation points for a-th row: (λ1,a1 , λ2,a2 , . . . , λn,an )
λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,
t = 0, 1, . . . , r − 1
• MDS property follows from the fact that each row is an MDS code • Low-complexity encoding, update and repair procedures
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
16 / 1
Construction 1: (1, n − 1)-optimal repair MDS codes
• ` = rn (r-ary expansion idea, Cadambe et al., ’11, Tamo et al., ’13) • For a = 0, 1, . . . , ` − 1, write r-ary expansion a = (a1 , a2 , . . . , an ) • Evaluation points for a-th row: (λ1,a1 , λ2,a2 , . . . , λn,an )
λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,
t = 0, 1, . . . , r − 1
• MDS property follows from the fact that each row is an MDS code • Low-complexity encoding, update and repair procedures • (1, n − 1)-optimal repair: can recover a column by downloading `/r elements of F
from each of the other columns
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
16 / 1
(1, n − 1)-optimal repair property
• a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an )
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
17 / 1
(1, n − 1)-optimal repair property
• a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •
λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
t = 0, 1, . . . , r − 1
August 11, 2017
17 / 1
(1, n − 1)-optimal repair property
• a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •
λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,
t = 0, 1, . . . , r − 1
•
λti,ai ci,a +
X
λtj,aj cj,a = 0
j6=i
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
17 / 1
(1, n − 1)-optimal repair property
• a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •
λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,
t = 0, 1, . . . , r − 1
•
λti,ai ci,a +
X
λtj,aj cj,a = 0
j6=i
•
λti,u ci,a(i,u) +
X
λtj,aj cj,a(i,u) = 0,
u = 0, 1, . . . , r − 1
j6=i
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
17 / 1
(1, n − 1)-optimal repair property
• a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •
λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,
t = 0, 1, . . . , r − 1
•
λti,ai ci,a +
X
λtj,aj cj,a = 0
j6=i
•
λti,u ci,a(i,u) +
X
λtj,aj cj,a(i,u) = 0,
u = 0, 1, . . . , r − 1
j6=i
• r−1 X u=0
Min Ye, Ph.D. Dissertation Defense
λti,u ci,a(i,u) +
r−1 X X
λtj,aj cj,a(i,u) = 0
u=0 j6=i
Coding Schemes for Distributed Storage Systems
August 11, 2017
17 / 1
(1, n − 1)-optimal repair property
r−1 X
λti,u ci,a(i,u) +
r−1 XX ( λtj,aj cj,a(i,u) ) = 0, j6=i
u=0
| +
X j6=i
|
Min Ye, Ph.D. Dissertation Defense
t = 0, 1, . . . , r − 1
u=0
1 λi,0 .. . λr−1 i,0
... ... .. . ...
1 λi,1 .. . λr−1 i,1
1
λi,r−1 .. . λr−1 i,r−1
{z
Vandermonde, rank = r
1 λj,aj .. . λr−1 j,aj
1 λj,aj .. . λr−1 j,aj {z
... ... .. . ...
aligned, rank = 1
1 λj,aj .. . λr−1 j,aj
}|
ci,a(i,0) ci,a(i,1) .. .
ci,a(i,r−1) {z }
desired information
cj,a(i,0) cj,a(i,1) .. .
=0
cj,a(i,r−1) }
Coding Schemes for Distributed Storage Systems
August 11, 2017
18 / 1
(1, n − 1)-optimal repair property
r−1 X
λti,u ci,a(i,u) +
j6=i
u=0
r−1 X
r−1 XX ( λtj,aj cj,a(i,u) ) = 0,
cj,a(i,u)
|
u=0
+
X j6=i
|
Min Ye, Ph.D. Dissertation Defense
t = 0, 1, . . . , r − 1
u=0
1 λi,0 .. . λr−1 i,0
... ... .. . ...
1 λi,1 .. . λr−1 i,1
1
λi,r−1 .. . λr−1 i,r−1
{z
Vandermonde, rank = r
1 λj,aj .. . λr−1 j,aj
1 λj,aj .. . λr−1 j,aj {z
... ... .. . ...
aligned, rank = 1
1 λj,aj .. . λr−1 j,aj
}|
ci,a(i,0) ci,a(i,1) .. .
ci,a(i,r−1) {z }
desired information
cj,a(i,0) cj,a(i,1) .. .
=0
cj,a(i,r−1) }
Coding Schemes for Distributed Storage Systems
August 11, 2017
19 / 1
(1, n − 1)-optimal repair property
• ci,a(i,0) , ci,a(i,1) , . . . , ci,a(i,r−1) can be determined by {
Min Ye, Ph.D. Dissertation Defense
Pr−1
Coding Schemes for Distributed Storage Systems
u=0
cj,a(i,u) }j6=i
August 11, 2017
20 / 1
(1, n − 1)-optimal repair property
• ci,a(i,0) , ci,a(i,1) , . . . , ci,a(i,r−1) can be determined by {
Pr−1 u=0
cj,a(i,u) }j6=i
• ` elements in Ci : partition into `/r groups of size r
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
20 / 1
(1, n − 1)-optimal repair property
• ci,a(i,0) , ci,a(i,1) , . . . , ci,a(i,r−1) can be determined by {
Pr−1 u=0
cj,a(i,u) }j6=i
• ` elements in Ci : partition into `/r groups of size r • Each group can be determined by downloading a scalar in F from each of the
other nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
20 / 1
(1, n − 1)-optimal repair property
• ci,a(i,0) , ci,a(i,1) , . . . , ci,a(i,r−1) can be determined by {
Pr−1 u=0
cj,a(i,u) }j6=i
• ` elements in Ci : partition into `/r groups of size r • Each group can be determined by downloading a scalar in F from each of the
other nodes • In total: only need to download `/r elements of F from each of the other nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
20 / 1
Construction 2: (1, d)-optimal repair MDS codes • s = d + 1 − k, Construction 1 is a special case of Construction 2
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
21 / 1
Construction 2: (1, d)-optimal repair MDS codes • s = d + 1 − k, Construction 1 is a special case of Construction 2 • (n, k = n − r, ` = sn ) MDS array code which can optimally repair any single erasure
from any d helper nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
21 / 1
Construction 2: (1, d)-optimal repair MDS codes • s = d + 1 − k, Construction 1 is a special case of Construction 2 • (n, k = n − r, ` = sn ) MDS array code which can optimally repair any single erasure
from any d helper nodes • Ci = (ci,0 , ci,1 , . . . , ci.`−1 )T
Min Ye, Ph.D. Dissertation Defense
c1,0 c1,1 .. .
c2,0 c2,1 .. .
c1,`−1
c2,`−1
... ... .. . ...
cn,0 cn,1 .. . cn,`−1
Coding Schemes for Distributed Storage Systems
August 11, 2017
21 / 1
Construction 2: (1, d)-optimal repair MDS codes • s = d + 1 − k, Construction 1 is a special case of Construction 2 • (n, k = n − r, ` = sn ) MDS array code which can optimally repair any single erasure
from any d helper nodes • Ci = (ci,0 , ci,1 , . . . , ci.`−1 )T
c1,0 c1,1 .. .
c2,0 c2,1 .. .
c1,`−1
c2,`−1
... ... .. . ...
cn,0 cn,1 .. . cn,`−1
• Each row forms a Generalized Reed-Solomon code with different evaluation points
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
21 / 1
Construction 2: (1, d)-optimal repair MDS codes • s = d + 1 − k, Construction 1 is a special case of Construction 2 • (n, k = n − r, ` = sn ) MDS array code which can optimally repair any single erasure
from any d helper nodes • Ci = (ci,0 , ci,1 , . . . , ci.`−1 )T
c1,0 c1,1 .. .
c2,0 c2,1 .. .
c1,`−1
c2,`−1
... ... .. . ...
cn,0 cn,1 .. . cn,`−1
• Each row forms a Generalized Reed-Solomon code with different evaluation points • Evaluation points λi,j , 1 ≤ i ≤ n, 0 ≤ j ≤ s − 1: sn distinct elements in F
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
21 / 1
Construction 2: (1, d)-optimal repair MDS codes • s = d + 1 − k, Construction 1 is a special case of Construction 2 • (n, k = n − r, ` = sn ) MDS array code which can optimally repair any single erasure
from any d helper nodes • Ci = (ci,0 , ci,1 , . . . , ci.`−1 )T
c1,0 c1,1 .. .
c2,0 c2,1 .. .
c1,`−1
c2,`−1
... ... .. . ...
cn,0 cn,1 .. . cn,`−1
• Each row forms a Generalized Reed-Solomon code with different evaluation points • Evaluation points λi,j , 1 ≤ i ≤ n, 0 ≤ j ≤ s − 1: sn distinct elements in F • F: any finite field with size |F| ≥ sn
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
21 / 1
Construction 2: (1, d)-optimal repair MDS codes
• s = d + 1 − k, ` = sn
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
22 / 1
Construction 2: (1, d)-optimal repair MDS codes
• s = d + 1 − k, ` = sn • For a = 0, 1, . . . , ` − 1, write s-ary expansion a = (a1 , a2 , . . . , an )
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
22 / 1
Construction 2: (1, d)-optimal repair MDS codes
• s = d + 1 − k, ` = sn • For a = 0, 1, . . . , ` − 1, write s-ary expansion a = (a1 , a2 , . . . , an ) • Evaluation points for a-th row: (λ1,a1 , λ2,a2 , . . . , λn,an )
λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
t = 0, 1, . . . , r − 1
August 11, 2017
22 / 1
Construction 2: (1, d)-optimal repair MDS codes
• s = d + 1 − k, ` = sn • For a = 0, 1, . . . , ` − 1, write s-ary expansion a = (a1 , a2 , . . . , an ) • Evaluation points for a-th row: (λ1,a1 , λ2,a2 , . . . , λn,an )
λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,
t = 0, 1, . . . , r − 1
• MDS property follows from the fact that each row is an MDS code
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
22 / 1
Construction 2: (1, d)-optimal repair MDS codes
• s = d + 1 − k, ` = sn • For a = 0, 1, . . . , ` − 1, write s-ary expansion a = (a1 , a2 , . . . , an ) • Evaluation points for a-th row: (λ1,a1 , λ2,a2 , . . . , λn,an )
λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,
t = 0, 1, . . . , r − 1
• MDS property follows from the fact that each row is an MDS code • Low-complexity encoding, update and repair procedures
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
22 / 1
Construction 2: (1, d)-optimal repair MDS codes
• s = d + 1 − k, ` = sn • For a = 0, 1, . . . , ` − 1, write s-ary expansion a = (a1 , a2 , . . . , an ) • Evaluation points for a-th row: (λ1,a1 , λ2,a2 , . . . , λn,an )
λt1,a1 c1,a + λt2,a2 c2,a + · · · + λtn,an cn,a = 0,
t = 0, 1, . . . , r − 1
• MDS property follows from the fact that each row is an MDS code • Low-complexity encoding, update and repair procedures • (1, d)-optimal repair property: • given any set of d helper nodes • can recover a column by downloading `/s elements of F from each of the d helper nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
22 / 1
(1, d)-optimal repair property • a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an )
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
23 / 1
(1, d)-optimal repair property • a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •
λt1,a1 c1,a +
n X
λtj,aj cj,a = 0,
t = 0, 1, . . . , r − 1
j=2
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
23 / 1
(1, d)-optimal repair property • a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •
λt1,a1 c1,a +
n X
λtj,aj cj,a = 0,
t = 0, 1, . . . , r − 1
j=2
•
λt1,u c1,a(1,u) +
n X
λtj,aj cj,a(1,u) = 0,
u = 0, 1, . . . , s − 1
j=2
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
23 / 1
(1, d)-optimal repair property • a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •
λt1,a1 c1,a +
n X
λtj,aj cj,a = 0,
t = 0, 1, . . . , r − 1
j=2
•
λt1,u c1,a(1,u) +
n X
λtj,aj cj,a(1,u) = 0,
u = 0, 1, . . . , s − 1
j=2
• s−1 X u=0
Min Ye, Ph.D. Dissertation Defense
λt1,u c1,a(1,u) +
s−1 X n X
λtj,aj cj,a(1,u) = 0
u=0 j=2
Coding Schemes for Distributed Storage Systems
August 11, 2017
23 / 1
(1, d)-optimal repair property • a(i, u) = (a1 , a2 , . . . , ai−1 , u, ai+1 , ai+2 , . . . , an ) •
λt1,a1 c1,a +
n X
λtj,aj cj,a = 0,
t = 0, 1, . . . , r − 1
j=2
•
λt1,u c1,a(1,u) +
n X
λtj,aj cj,a(1,u) = 0,
u = 0, 1, . . . , s − 1
j=2
• s−1 X
λt1,u c1,a(1,u) +
u=0
s−1 X n X
λtj,aj cj,a(1,u) = 0
u=0 j=2
• s−1 X
λt1,u c1,a(1,u) +
u=0
n X j=2
s−1 X λtj,aj ( cj,a(1,u) ) = 0,
t = 0, 1, . . . , r − 1
u=0
|
{z
}
,µj,a
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
23 / 1
(1, d)-optimal repair property
s−1 X
λt1,u c1,a(1,u) +
u=0
Min Ye, Ph.D. Dissertation Defense
n X
λtj,aj µj,a = 0,
t = 0, 1, . . . , r − 1
j=2
Coding Schemes for Distributed Storage Systems
August 11, 2017
24 / 1
(1, d)-optimal repair property
s−1 X
λt1,u c1,a(1,u) +
u=0
n X
λtj,aj µj,a = 0,
t = 0, 1, . . . , r − 1
j=2
1 λ1,0 .. . λr−1 1,0
|
1
λ1,s−1 .. . λr−1 1,s−1
{z
1
1
λ2,a2 + . .. λr−1 2,a2
λ3,a3 .. . λr−1 3,a3
... ... .. . ...
c1,a(1,0) c1,a(1,1) .. .
c1,a(1,s−1) }
r≥s, full column rank
Min Ye, Ph.D. Dissertation Defense
... ... .. . ...
1 λ1,1 .. . λr−1 1,1
1
λn,an .. . λr−1 n,an
µ2,a µ3,a .. . µn,a
Coding Schemes for Distributed Storage Systems
=0
August 11, 2017
24 / 1
(1, d)-optimal repair property
s−1 X
λt1,u c1,a(1,u) +
u=0
n X
λtj,aj µj,a = 0,
t = 0, 1, . . . , r − 1
j=2
1 λ1,0 .. . λr−1 1,0
|
... ... .. . ...
1 λ1,1 .. . λr−1 1,1
1
λ1,s−1 .. . λr−1 1,s−1
{z
1
1
λ2,a2 + . .. λr−1 2,a2
λ3,a3 .. . λr−1 3,a3
... ... .. . ...
c1,a(1,s−1) }
r≥s, full column rank
c1,a(1,0) c1,a(1,1) .. .
1
λn,an .. . λr−1 n,an
µ2,a µ3,a .. . µn,a
=0
Want to show that (µ2,a , µ3,a , . . . , µn,a ) forms an MDS code with dimension d.
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
24 / 1
(1, d)-optimal repair property • r−s=n−1−d
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
25 / 1
(1, d)-optimal repair property • r−s=n−1−d • Want to find an (r − s) × r matrix P such that
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
25 / 1
(1, d)-optimal repair property • r−s=n−1−d • Want to find an (r − s) × r matrix P such that •
P
1 λ1,0 .. . λr−1 1,0
|
1 ... λ1,1 . . . .. .. . . λr−1 ... 1,1 {z
1
λ1,s−1 .. . λr−1 1,s−1
1
λ2,a2 +P . .. λr−1 2,a2 |
1 λ3,a3 .. . λr−1 3,a3 {z
... ... .. . ...
1 λn,an .. . λr−1 n,an
parity check matrix of an MDS code
Min Ye, Ph.D. Dissertation Defense
c1,a(1,s−1) }
=0
c1,a(1,0) c1,a(1,1) .. .
µ2,a µ3,a . .. µn,a }
Coding Schemes for Distributed Storage Systems
=0
August 11, 2017
25 / 1
(1, d)-optimal repair property
• Each row of P: coefficients of a polynomial of degree less than r
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
26 / 1
(1, d)-optimal repair property
• Each row of P: coefficients of a polynomial of degree less than r •
p0 (x) =
s−1 Y (x − λ1,u ),
degree s
u=0
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
26 / 1
(1, d)-optimal repair property
• Each row of P: coefficients of a polynomial of degree less than r •
p0 (x) =
s−1 Y (x − λ1,u ),
degree s
u=0
p1 (x) = xp0 (x),
Min Ye, Ph.D. Dissertation Defense
degree s + 1
Coding Schemes for Distributed Storage Systems
August 11, 2017
26 / 1
(1, d)-optimal repair property
• Each row of P: coefficients of a polynomial of degree less than r •
p0 (x) =
s−1 Y (x − λ1,u ),
degree s
u=0
Min Ye, Ph.D. Dissertation Defense
p1 (x) = xp0 (x),
degree s + 1
p2 (x) = x2 p0 (x),
degree s + 2
Coding Schemes for Distributed Storage Systems
August 11, 2017
26 / 1
(1, d)-optimal repair property
• Each row of P: coefficients of a polynomial of degree less than r •
p0 (x) =
s−1 Y (x − λ1,u ),
degree s
u=0
p1 (x) = xp0 (x),
degree s + 1
p2 (x) = x2 p0 (x),
degree s + 2 .. .
pr−s−1 (x) = xr−s−1 p0 (x),
Min Ye, Ph.D. Dissertation Defense
degree r − 1
Coding Schemes for Distributed Storage Systems
August 11, 2017
26 / 1
(1, d)-optimal repair property
• Each row of P: coefficients of a polynomial of degree less than r •
p0 (x) =
s−1 Y (x − λ1,u ),
degree s
u=0
p1 (x) = xp0 (x),
degree s + 1
p2 (x) = x2 p0 (x),
degree s + 2 .. .
pr−s−1 (x) = xr−s−1 p0 (x),
degree r − 1
• Write
pi (x) = pi,0 + pi,1 x + pi,2 x2 + · · · + pi,r−1 xr−1
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
26 / 1
(1, d)-optimal repair property
•
P=
Min Ye, Ph.D. Dissertation Defense
p0,0 p1,0 .. .
p0,1 p1,1 .. .
p0,2 p1,2 .. .
pr−s−1,0
pr−s−1,1
pr−s−1,2
... ... .. . ...
Coding Schemes for Distributed Storage Systems
p0,r−1 p1,r−1 .. .
pr−s−1,r−1
August 11, 2017
27 / 1
(1, d)-optimal repair property
•
P=
p0,0 p1,0 .. .
p0,1 p1,1 .. .
p0,2 p1,2 .. .
pr−s−1,0
pr−s−1,1
pr−s−1,2
... ... .. . ...
p0,r−1 p1,r−1 .. .
pr−s−1,r−1
•
P
1 x x2 .. .
=
xr−1
Min Ye, Ph.D. Dissertation Defense
p0 (x) p1 (x) p2 (x) .. . pr−s−1 (x)
=
p0 (x) xp0 (x) x2 p0 (x) .. . xr−s−1 p0 (x)
Coding Schemes for Distributed Storage Systems
August 11, 2017
27 / 1
(1, d)-optimal repair property •
p0 (x) =
s−1 Y (x − λ1,u ) u=0
•
P =
1 λ1,0 .. . λr−1 1,0
1 λ1,1 .. . λr−1 1,1
p0 (λ1,0 ) λ1,0 p0 (λ1,0 ) .. . λr−s−1 p0 (λ1,0 ) 1,0
... ... .. . ...
1
λ1,s−1 .. . λr−1 1,s−1
p0 (λ1,1 ) λ1,1 p0 (λ1,1 ) .. . r−s−1 λ1,1 p0 (λ1,1 )
... ... .. . ...
p0 (λ1,s−1 ) λ1,s−1 p0 (λ1,s−1 ) .. . r−s−1 λ1,s−1 p0 (λ1,s−1 )
=0
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
28 / 1
(1, d)-optimal repair property •
p0 (x) =
s−1 Y (x − λ1,u ) u=0
•
1
1
λ2,a2 λ3,a3 P . .. .. . λr−1 λr−1 2,a2 3,a3 p0 (λ2,a2 ) λ2,a2 p0 (λ2,a2 ) = .. . λr−s−1 p0 (λ2,a2 ) 2,a2 |
... ... .. . ...
1
λn,an .. . λr−1 n,an
p0 (λ3,a3 ) λ3,a3 p0 (λ3,a3 ) .. . r−s−1 λ3,a p0 (λ3,a3 ) 3 {z
... ... .. . ...
p0 (λn,an ) λn,an p0 (λn,an ) .. . r−s−1 λn,a p0 (λn,an ) n
the parity matrix of a GRS code with length n − 1 and dimension d
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
}
August 11, 2017
29 / 1
(1, d)-optimal repair property •
P
1 λ1,0 .. . λr−1 1,0
|
1 ... λ1,1 . . . .. .. . . λr−1 ... 1,1 {z
1
λ1,s−1 .. . λr−1 1,s−1
1
λ2,a2 +P . .. λr−1 2,a2 |
1 λ3,a3 .. . λr−1 3,a3 {z
... ... .. . ...
c1,a(1,s−1)
1
λn,an .. . λr−1 n,an
parity check matrix of a GRS code
Min Ye, Ph.D. Dissertation Defense
}
=0
c1,a(1,0) c1,a(1,1) .. .
µ2,a µ3,a .. . µn,a
=0
}
Coding Schemes for Distributed Storage Systems
August 11, 2017
30 / 1
(1, d)-optimal repair property •
P
1 λ1,0 .. . λr−1 1,0
|
1 ... λ1,1 . . . .. .. . . λr−1 ... 1,1 {z
1
λ1,s−1 .. . λr−1 1,s−1
1
λ2,a2 +P . .. λr−1 2,a2 |
1 λ3,a3 .. . λr−1 3,a3 {z
... ... .. . ...
c1,a(1,s−1) }
=0
c1,a(1,0) c1,a(1,1) .. .
1
λn,an .. . λr−1 n,an
parity check matrix of a GRS code
µ2,a µ3,a .. . µn,a
=0
}
• (µ2,a , µ3,a , . . . , µn,a ) forms an MDS code: can correct errors
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
30 / 1
(1, d)-optimal repair property •
P
1 λ1,0 .. . λr−1 1,0
|
1 ... λ1,1 . . . .. .. . . λr−1 ... 1,1 {z
1
λ1,s−1 .. . λr−1 1,s−1
1
λ2,a2 +P . .. λr−1 2,a2 |
1 λ3,a3 .. . λr−1 3,a3 {z
... ... .. . ...
c1,a(1,s−1) }
=0
c1,a(1,0) c1,a(1,1) .. .
1
λn,an .. . λr−1 n,an
parity check matrix of a GRS code
µ2,a µ3,a .. . µn,a
=0
}
• (µ2,a , µ3,a , . . . , µn,a ) forms an MDS code: can correct errors • Optimal error resilience capability
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
30 / 1
Construction 3: Optimally recover any erasure pattern
• Set s = lcm(1, 2, . . . , n − k) in Construction 2
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
31 / 1
Construction 3: Optimally recover any erasure pattern
• Set s = lcm(1, 2, . . . , n − k) in Construction 2 • (n, k, ` = sn ) MDS array codes with the (h, d)-optimal repair property for all
1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
31 / 1
Construction 3: Optimally recover any erasure pattern
• Set s = lcm(1, 2, . . . , n − k) in Construction 2 • (n, k, ` = sn ) MDS array codes with the (h, d)-optimal repair property for all
1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can optimally recover any erasure pattern from any set of helper nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
31 / 1
Construction 3: Optimally recover any erasure pattern
• Set s = lcm(1, 2, . . . , n − k) in Construction 2 • (n, k, ` = sn ) MDS array codes with the (h, d)-optimal repair property for all
1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can optimally recover any erasure pattern from any set of helper nodes • Can be constructed over any field F with size |F| ≥ sn
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
31 / 1
Construction 3: Optimally recover any erasure pattern
• Set s = lcm(1, 2, . . . , n − k) in Construction 2 • (n, k, ` = sn ) MDS array codes with the (h, d)-optimal repair property for all
1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can optimally recover any erasure pattern from any set of helper nodes • Can be constructed over any field F with size |F| ≥ sn • Low-complexity encoding, update and repair procedures
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
31 / 1
Construction 3: Optimally recover any erasure pattern
• Set s = lcm(1, 2, . . . , n − k) in Construction 2 • (n, k, ` = sn ) MDS array codes with the (h, d)-optimal repair property for all
1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can optimally recover any erasure pattern from any set of helper nodes • Can be constructed over any field F with size |F| ≥ sn • Low-complexity encoding, update and repair procedures • Optimal error resilience in the repair process
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
31 / 1
Systematic generator VS. non-systematic parity check
• k × n systematic generator matrix
Min Ye, Ph.D. Dissertation Defense
I 0 .. . 0
0 I .. . 0
... ... .. . ...
0 0 .. . I
A1,1 A1,2 .. . A1,k
A2,1 A2,2 .. . A2,k
... ... .. . ...
Coding Schemes for Distributed Storage Systems
Ar,1 Ar,2 .. . Ar,k
August 11, 2017
32 / 1
Systematic generator VS. non-systematic parity check
• k × n systematic generator matrix
I 0 .. . 0
0 I .. . 0
... ... .. . ...
0 0 .. . I
A1,1 A1,2 .. . A1,k
A2,1 A2,2 .. . A2,k
... ... .. . ...
Ar,1 Ar,2 .. . Ar,k
• Every square submatrix needs to be invertible. Optimally repair systematic nodes.
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
32 / 1
Systematic generator VS. non-systematic parity check
• k × n systematic generator matrix
I 0 .. . 0
0 I .. . 0
... ... .. . ...
0 0 .. . I
A1,1 A1,2 .. . A1,k
A2,1 A2,2 .. . A2,k
... ... .. . ...
Ar,1 Ar,2 .. . Ar,k
• Every square submatrix needs to be invertible. Optimally repair systematic nodes. • r × n parity check matrix
Min Ye, Ph.D. Dissertation Defense
A1,1 A2,1 .. . Ar,1
A1,2 A2,2 .. . Ar,2
A1,3 A2,3 .. . Ar,3
... ... .. . ...
A1,n A2,n .. . Ar,n
Coding Schemes for Distributed Storage Systems
August 11, 2017
32 / 1
Systematic generator VS. non-systematic parity check
• k × n systematic generator matrix
I 0 .. . 0
0 I .. . 0
... ... .. . ...
0 0 .. . I
A1,1 A1,2 .. . A1,k
A2,1 A2,2 .. . A2,k
... ... .. . ...
Ar,1 Ar,2 .. . Ar,k
• Every square submatrix needs to be invertible. Optimally repair systematic nodes. • r × n parity check matrix
A1,1 A2,1 .. . Ar,1
A1,2 A2,2 .. . Ar,2
A1,3 A2,3 .. . Ar,3
... ... .. . ...
A1,n A2,n .. . Ar,n
• Only require every r × r submatrix to be invertible. Optimally repair all nodes.
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
32 / 1
Block Vandermonde structure
•
Min Ye, Ph.D. Dissertation Defense
I A1 A21 .. . Ar−1 1
I A2 A22 .. . Ar−1 2
I A3 A23 .. . Ar−1 3
... ... ... .. . ...
I An A2n .. . Ar−1 n
Coding Schemes for Distributed Storage Systems
August 11, 2017
33 / 1
Block Vandermonde structure
•
I A1 A21 .. . Ar−1 1
I A2 A22 .. . Ar−1 2
I A3 A23 .. . Ar−1 3
... ... ... .. . ...
I An A2n .. . Ar−1 n
• Commuting: Ai Aj = Aj Ai
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
33 / 1
Block Vandermonde structure
•
I A1 A21 .. . Ar−1 1
I A2 A22 .. . Ar−1 2
I A3 A23 .. . Ar−1 3
... ... ... .. . ...
I An A2n .. . Ar−1 n
• Commuting: Ai Aj = Aj Ai • Ai − Aj invertible
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
33 / 1
Block Vandermonde structure
•
I A1 A21 .. . Ar−1 1
I A2 A22 .. . Ar−1 2
I A3 A23 .. . Ar−1 3
... ... ... .. . ...
I An A2n .. . Ar−1 n
• Commuting: Ai Aj = Aj Ai • Ai − Aj invertible • Construction 1-3: diagonal matrices naturally commute
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
33 / 1
Permutation matrices and optimal access
• Choose Ai to be permutation matrices (Cadambe et al., ’11, Tamo et al., ’13)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
34 / 1
Permutation matrices and optimal access
• Choose Ai to be permutation matrices (Cadambe et al., ’11, Tamo et al., ’13) • Optimal access property: reduce the amount of data read, the disk I/O
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
34 / 1
Permutation matrices and optimal access
• Choose Ai to be permutation matrices (Cadambe et al., ’11, Tamo et al., ’13) • Optimal access property: reduce the amount of data read, the disk I/O • For any n and r, explicit (n, k = n − r, ` = sn ) MDS array codes with the
(h, d)-optimal access property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
34 / 1
Permutation matrices and optimal access
• Choose Ai to be permutation matrices (Cadambe et al., ’11, Tamo et al., ’13) • Optimal access property: reduce the amount of data read, the disk I/O • For any n and r, explicit (n, k = n − r, ` = sn ) MDS array codes with the
(h, d)-optimal access property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can be constructed over any field F with size |F| ≥ n + 1
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
34 / 1
Permutation matrices and optimal access
• Choose Ai to be permutation matrices (Cadambe et al., ’11, Tamo et al., ’13) • Optimal access property: reduce the amount of data read, the disk I/O • For any n and r, explicit (n, k = n − r, ` = sn ) MDS array codes with the
(h, d)-optimal access property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can be constructed over any field F with size |F| ≥ n + 1 • Optimal error resilience in the repair process
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
34 / 1
Permutation matrices and optimal access
• Choose Ai to be permutation matrices (Cadambe et al., ’11, Tamo et al., ’13) • Optimal access property: reduce the amount of data read, the disk I/O • For any n and r, explicit (n, k = n − r, ` = sn ) MDS array codes with the
(h, d)-optimal access property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can be constructed over any field F with size |F| ≥ n + 1 • Optimal error resilience in the repair process • Encoding and update complexity is higher
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
34 / 1
Permutation matrices and optimal access
• Choose Ai to be permutation matrices (Cadambe et al., ’11, Tamo et al., ’13) • Optimal access property: reduce the amount of data read, the disk I/O • For any n and r, explicit (n, k = n − r, ` = sn ) MDS array codes with the
(h, d)-optimal access property for all 1 ≤ h ≤ r and k ≤ d ≤ n − h simultaneously • Can be constructed over any field F with size |F| ≥ n + 1 • Optimal error resilience in the repair process • Encoding and update complexity is higher • M. Ye and A. Barg, “Explicit constructions of high-rate MDS array codes with
optimal repair bandwidth,” IEEE Transactions on Information Theory, vol. 63, no. 4, pp. 2001–2014, April 2017.
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
34 / 1
Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
35 / 1
Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
35 / 1
Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices • For any n and r, explicit (n, k = n − r, ` = rdn/re ) optimal-access MDS array code
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
35 / 1
Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices • For any n and r, explicit (n, k = n − r, ` = rdn/re ) optimal-access MDS array code • Best known sub-packetization `, nearly optimal
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
35 / 1
Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices • For any n and r, explicit (n, k = n − r, ` = rdn/re ) optimal-access MDS array code • Best known sub-packetization `, nearly optimal • Can be constructed over any field F with size |F| ≥ n + r
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
35 / 1
Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices • For any n and r, explicit (n, k = n − r, ` = rdn/re ) optimal-access MDS array code • Best known sub-packetization `, nearly optimal • Can be constructed over any field F with size |F| ≥ n + r • Low complexity encoding and repair procedures
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
35 / 1
Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices • For any n and r, explicit (n, k = n − r, ` = rdn/re ) optimal-access MDS array code • Best known sub-packetization `, nearly optimal • Can be constructed over any field F with size |F| ≥ n + r • Low complexity encoding and repair procedures • M. Ye and A. Barg, “Explicit constructions of optimal-access MDS codes with
nearly optimal sub-packetization,” IEEE Transactions on Information Theory, 2017 (arXiv:1605.08630).
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
35 / 1
Small sub-packetization ` • Smaller ` ⇒ smaller node size ⇒ lower system complexity • Beyond block Vandermonde structure and commuting matrices • For any n and r, explicit (n, k = n − r, ` = rdn/re ) optimal-access MDS array code • Best known sub-packetization `, nearly optimal • Can be constructed over any field F with size |F| ≥ n + r • Low complexity encoding and repair procedures • M. Ye and A. Barg, “Explicit constructions of optimal-access MDS codes with
nearly optimal sub-packetization,” IEEE Transactions on Information Theory, 2017 (arXiv:1605.08630). • Sasidharan et al., 2016, Li et al., ISIT 2017: Same parameters, very similar
constructions, appeared after our paper was posted on arXiv
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
35 / 1
Evaluation in Amazon Cluster
M. Ye et al., “Pairing up for regeneration: The Mantra for fast and efficient node repair in distributed storage,” poster presented at USENIX Annual Technical Conference, July 12-14, 2017, Santa Clara, CA. Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
36 / 1
Repairing Reed-Solomon codes • Guruswami-Wootters (STOC-2016, IEEE-TIT-2017)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
37 / 1
Repairing Reed-Solomon codes • Guruswami-Wootters (STOC-2016, IEEE-TIT-2017) • Linear repair scheme: trace repair framework
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
37 / 1
Repairing Reed-Solomon codes • Guruswami-Wootters (STOC-2016, IEEE-TIT-2017) • Linear repair scheme: trace repair framework • View RS codes as array codes over some subfield of the symbol field
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
37 / 1
Repairing Reed-Solomon codes • Guruswami-Wootters (STOC-2016, IEEE-TIT-2017) • Linear repair scheme: trace repair framework • View RS codes as array codes over some subfield of the symbol field • Repair bandwidth is smaller than under the trivial approach, but far away from the
cut-set bound
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
37 / 1
Repairing Reed-Solomon codes • Guruswami-Wootters (STOC-2016, IEEE-TIT-2017) • Linear repair scheme: trace repair framework • View RS codes as array codes over some subfield of the symbol field • Repair bandwidth is smaller than under the trivial approach, but far away from the
cut-set bound
• Dau-Milenkovic (ISIT-2017)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
37 / 1
Repairing Reed-Solomon codes • Guruswami-Wootters (STOC-2016, IEEE-TIT-2017) • Linear repair scheme: trace repair framework • View RS codes as array codes over some subfield of the symbol field • Repair bandwidth is smaller than under the trivial approach, but far away from the
cut-set bound
• Dau-Milenkovic (ISIT-2017) • Generalize G-W results to a larger set of parameters
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
37 / 1
Repairing Reed-Solomon codes • Guruswami-Wootters (STOC-2016, IEEE-TIT-2017) • Linear repair scheme: trace repair framework • View RS codes as array codes over some subfield of the symbol field • Repair bandwidth is smaller than under the trivial approach, but far away from the
cut-set bound
• Dau-Milenkovic (ISIT-2017) • Generalize G-W results to a larger set of parameters • Repair bandwidth is still far away from the cut-set bound
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
37 / 1
Repairing Reed-Solomon codes • Guruswami-Wootters (STOC-2016, IEEE-TIT-2017) • Linear repair scheme: trace repair framework • View RS codes as array codes over some subfield of the symbol field • Repair bandwidth is smaller than under the trivial approach, but far away from the
cut-set bound
• Dau-Milenkovic (ISIT-2017) • Generalize G-W results to a larger set of parameters • Repair bandwidth is still far away from the cut-set bound
• Open problem: Can RS codes (or any scalar MDS codes) achieve the cut-set
bound?
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
37 / 1
Our results • Explicit construction of RS codes achieving the cut set bound
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
38 / 1
Our results • Explicit construction of RS codes achieving the cut set bound • Given any k < d < n, we construct explicit (n, k) RS codes with (1, d)-optimal repair
property
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
38 / 1
Our results • Explicit construction of RS codes achieving the cut set bound • Given any k < d < n, we construct explicit (n, k) RS codes with (1, d)-optimal repair
property • ` = e(1+o(1))n log n , super-exponential
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
38 / 1
Our results • Explicit construction of RS codes achieving the cut set bound • Given any k < d < n, we construct explicit (n, k) RS codes with (1, d)-optimal repair
property • ` = e(1+o(1))n log n , super-exponential • For MSR array codes, ` = rn/(r+1) (Wang et al., ’16): exponential is enough
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
38 / 1
Our results • Explicit construction of RS codes achieving the cut set bound • Given any k < d < n, we construct explicit (n, k) RS codes with (1, d)-optimal repair
property • ` = e(1+o(1))n log n , super-exponential • For MSR array codes, ` = rn/(r+1) (Wang et al., ’16): exponential is enough • Is this gap a necessary penalty for scalar codes to achieve cut-set bound?
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
38 / 1
Our results • Explicit construction of RS codes achieving the cut set bound • Given any k < d < n, we construct explicit (n, k) RS codes with (1, d)-optimal repair
property • ` = e(1+o(1))n log n , super-exponential • For MSR array codes, ` = rn/(r+1) (Wang et al., ’16): exponential is enough • Is this gap a necessary penalty for scalar codes to achieve cut-set bound?
• An almost matching lower bound on ` of scalar linear MSR codes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
38 / 1
Our results • Explicit construction of RS codes achieving the cut set bound • Given any k < d < n, we construct explicit (n, k) RS codes with (1, d)-optimal repair
property • ` = e(1+o(1))n log n , super-exponential • For MSR array codes, ` = rn/(r+1) (Wang et al., ’16): exponential is enough • Is this gap a necessary penalty for scalar codes to achieve cut-set bound?
• An almost matching lower bound on ` of scalar linear MSR codes • ` ≥ e(1+o(1))k log k
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
38 / 1
Our results • Explicit construction of RS codes achieving the cut set bound • Given any k < d < n, we construct explicit (n, k) RS codes with (1, d)-optimal repair
property • ` = e(1+o(1))n log n , super-exponential • For MSR array codes, ` = rn/(r+1) (Wang et al., ’16): exponential is enough • Is this gap a necessary penalty for scalar codes to achieve cut-set bound?
• An almost matching lower bound on ` of scalar linear MSR codes • ` ≥ e(1+o(1))k log k
• I. Tamo, M. Ye and A. Barg, “Optimal repair of Reed-Solomon codes: Achieving the cut-set bound,” IEEE Symposium on Foundations of Computer Science (FOCS), 2017.
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
38 / 1
Our results • Explicit construction of RS codes achieving the cut set bound • Given any k < d < n, we construct explicit (n, k) RS codes with (1, d)-optimal repair
property • ` = e(1+o(1))n log n , super-exponential • For MSR array codes, ` = rn/(r+1) (Wang et al., ’16): exponential is enough • Is this gap a necessary penalty for scalar codes to achieve cut-set bound?
• An almost matching lower bound on ` of scalar linear MSR codes • ` ≥ e(1+o(1))k log k
• I. Tamo, M. Ye and A. Barg, “Optimal repair of Reed-Solomon codes: Achieving the cut-set bound,” IEEE Symposium on Foundations of Computer Science (FOCS), 2017. • Explicit construction of RS codes achieving the cut set bound asymptotically
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
38 / 1
Our results • Explicit construction of RS codes achieving the cut set bound • Given any k < d < n, we construct explicit (n, k) RS codes with (1, d)-optimal repair
property • ` = e(1+o(1))n log n , super-exponential • For MSR array codes, ` = rn/(r+1) (Wang et al., ’16): exponential is enough • Is this gap a necessary penalty for scalar codes to achieve cut-set bound?
• An almost matching lower bound on ` of scalar linear MSR codes • ` ≥ e(1+o(1))k log k
• I. Tamo, M. Ye and A. Barg, “Optimal repair of Reed-Solomon codes: Achieving the cut-set bound,” IEEE Symposium on Foundations of Computer Science (FOCS), 2017. • Explicit construction of RS codes achieving the cut set bound asymptotically • available for any n and r; repair bandwidth ≤ (n + 1)`/r
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
38 / 1
Our results • Explicit construction of RS codes achieving the cut set bound • Given any k < d < n, we construct explicit (n, k) RS codes with (1, d)-optimal repair
property • ` = e(1+o(1))n log n , super-exponential • For MSR array codes, ` = rn/(r+1) (Wang et al., ’16): exponential is enough • Is this gap a necessary penalty for scalar codes to achieve cut-set bound?
• An almost matching lower bound on ` of scalar linear MSR codes • ` ≥ e(1+o(1))k log k
• I. Tamo, M. Ye and A. Barg, “Optimal repair of Reed-Solomon codes: Achieving the cut-set bound,” IEEE Symposium on Foundations of Computer Science (FOCS), 2017. • Explicit construction of RS codes achieving the cut set bound asymptotically • available for any n and r; repair bandwidth ≤ (n + 1)`/r • Cut-set bound is (n − 1)`/r, ratio goes to 1, asymptotically optimal
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
38 / 1
Our results • Explicit construction of RS codes achieving the cut set bound • Given any k < d < n, we construct explicit (n, k) RS codes with (1, d)-optimal repair
property • ` = e(1+o(1))n log n , super-exponential • For MSR array codes, ` = rn/(r+1) (Wang et al., ’16): exponential is enough • Is this gap a necessary penalty for scalar codes to achieve cut-set bound?
• An almost matching lower bound on ` of scalar linear MSR codes • ` ≥ e(1+o(1))k log k
• I. Tamo, M. Ye and A. Barg, “Optimal repair of Reed-Solomon codes: Achieving the cut-set bound,” IEEE Symposium on Foundations of Computer Science (FOCS), 2017. • Explicit construction of RS codes achieving the cut set bound asymptotically • available for any n and r; repair bandwidth ≤ (n + 1)`/r • Cut-set bound is (n − 1)`/r, ratio goes to 1, asymptotically optimal • M. Ye and A. Barg, “Explicit constructions of MDS array codes and RS codes with
optimal repair bandwidth,” ISIT 2016, pp.1202-1206. Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
38 / 1
Fractional decoding: Global decoding from partial information Code of length n over Fq
C3
C2
C1
)
) C2
C1
f 2(
f1 (
Cn
...
f3 (C3 )
f n(
) Cn
decoder Suppose that Size of (f1 (C1 ), f2 (C2 ), . . . , fn (Cn )) ≤ αn`,
α≤1
How many errors can the decoder correct?
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
39 / 1
Fractional decoding: Global decoding from partial information
1 Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
40 / 1
Fractional decoding: Global decoding from partial information
• (n, k) MDS code C = {C = (C1 , . . . , Cn )}
1 Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
40 / 1
Fractional decoding: Global decoding from partial information
• (n, k) MDS code C = {C = (C1 , . . . , Cn )} • Each Ci is a vector of dimension ` over F
1 Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
40 / 1
Fractional decoding: Global decoding from partial information
• (n, k) MDS code C = {C = (C1 , . . . , Cn )} • Each Ci is a vector of dimension ` over F • Suppose that on average in each coordinate we download an α fraction of the
codeword symbol (possibly, distorted by noise), α < 1
1 Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
40 / 1
Fractional decoding: Global decoding from partial information
• (n, k) MDS code C = {C = (C1 , . . . , Cn )} • Each Ci is a vector of dimension ` over F • Suppose that on average in each coordinate we download an α fraction of the
codeword symbol (possibly, distorted by noise), α < 1 • The total number of symbols of F available to the decoder1 is αn`. What can be
said about the number of (worst-case) correctable errors?
1
these symbols may be functions of the received symbols Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
40 / 1
Fractional decoding: Global decoding from partial information
• (n, k) MDS code C = {C = (C1 , . . . , Cn )} • Each Ci is a vector of dimension ` over F • Suppose that on average in each coordinate we download an α fraction of the
codeword symbol (possibly, distorted by noise), α < 1 • The total number of symbols of F available to the decoder1 is αn`. What can be
said about the number of (worst-case) correctable errors? Call this number the α-decoding radius
1
these symbols may be functions of the received symbols Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
40 / 1
Fractional decoding: Global decoding from partial information
• (n, k) MDS code C = {C = (C1 , . . . , Cn )} • Each Ci is a vector of dimension ` over F • Suppose that on average in each coordinate we download an α fraction of the
codeword symbol (possibly, distorted by noise), α < 1 • The total number of symbols of F available to the decoder1 is αn`. What can be
said about the number of (worst-case) correctable errors? Call this number the α-decoding radius • Main results:
1
these symbols may be functions of the received symbols Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
40 / 1
Fractional decoding: Global decoding from partial information
• (n, k) MDS code C = {C = (C1 , . . . , Cn )} • Each Ci is a vector of dimension ` over F • Suppose that on average in each coordinate we download an α fraction of the
codeword symbol (possibly, distorted by noise), α < 1 • The total number of symbols of F available to the decoder1 is αn`. What can be
said about the number of (worst-case) correctable errors? Call this number the α-decoding radius • Main results: • A bound on the α-decoding radius
1
these symbols may be functions of the received symbols Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
40 / 1
Fractional decoding: Global decoding from partial information
• (n, k) MDS code C = {C = (C1 , . . . , Cn )} • Each Ci is a vector of dimension ` over F • Suppose that on average in each coordinate we download an α fraction of the
codeword symbol (possibly, distorted by noise), α < 1 • The total number of symbols of F available to the decoder1 is αn`. What can be
said about the number of (worst-case) correctable errors? Call this number the α-decoding radius • Main results: • A bound on the α-decoding radius • A matching code construction
1
these symbols may be functions of the received symbols Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
40 / 1
Decoding radius as a function of α
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
41 / 1
Decoding radius as a function of α • r1 (n, k) = b(n − k)/2c
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
41 / 1
Decoding radius as a function of α • r1 (n, k) = b(n − k)/2c • A trivial lower bound: Puncturing an (n, k) MDS code, we obtain
rα (n, k) ≥ b(αn − k)/2c for any k/n ≤ α ≤ 1
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
41 / 1
Decoding radius as a function of α • r1 (n, k) = b(n − k)/2c • A trivial lower bound: Puncturing an (n, k) MDS code, we obtain
rα (n, k) ≥ b(αn − k)/2c for any k/n ≤ α ≤ 1 • Singleton-like upper bound:
rα (n, k) ≤ b(n − k/α)/2c
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
41 / 1
Decoding radius as a function of α • r1 (n, k) = b(n − k)/2c • A trivial lower bound: Puncturing an (n, k) MDS code, we obtain
rα (n, k) ≥ b(αn − k)/2c for any k/n ≤ α ≤ 1 • Singleton-like upper bound:
rα (n, k) ≤ b(n − k/α)/2c Main result: We construct codes with α-decoding radius that matches the upper bound
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
41 / 1
Decoding radius as a function of α • r1 (n, k) = b(n − k)/2c • A trivial lower bound: Puncturing an (n, k) MDS code, we obtain
rα (n, k) ≥ b(αn − k)/2c for any k/n ≤ α ≤ 1 • Singleton-like upper bound:
rα (n, k) ≤ b(n − k/α)/2c Main result: We construct codes with α-decoding radius that matches the upper bound
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
41 / 1
Summary • We constructed various families of MSR array codes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
42 / 1
Summary • We constructed various families of MSR array codes • Optimally recover any number of erasures from any set of helper nodes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
42 / 1
Summary • We constructed various families of MSR array codes • Optimally recover any number of erasures from any set of helper nodes • Optimal error resilience capability during the repair process
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
42 / 1
Summary • We constructed various families of MSR array codes • Optimally recover any number of erasures from any set of helper nodes • Optimal error resilience capability during the repair process • Optimal access property and optimal sub-packetization `
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
42 / 1
Summary • We constructed various families of MSR array codes • Optimally recover any number of erasures from any set of helper nodes • Optimal error resilience capability during the repair process • Optimal access property and optimal sub-packetization `
• We constructed explicit RS codes with optimal repair property
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
42 / 1
Summary • We constructed various families of MSR array codes • Optimally recover any number of erasures from any set of helper nodes • Optimal error resilience capability during the repair process • Optimal access property and optimal sub-packetization `
• We constructed explicit RS codes with optimal repair property • Show that there exist scalar MSR codes by constructing explicit RS codes with optimal
repair property
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
42 / 1
Summary • We constructed various families of MSR array codes • Optimally recover any number of erasures from any set of helper nodes • Optimal error resilience capability during the repair process • Optimal access property and optimal sub-packetization `
• We constructed explicit RS codes with optimal repair property • Show that there exist scalar MSR codes by constructing explicit RS codes with optimal
repair property • An almost matching lower bound on the sub-packetization of scalar linear MSR codes
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
42 / 1
Summary • We constructed various families of MSR array codes • Optimally recover any number of erasures from any set of helper nodes • Optimal error resilience capability during the repair process • Optimal access property and optimal sub-packetization `
• We constructed explicit RS codes with optimal repair property • Show that there exist scalar MSR codes by constructing explicit RS codes with optimal
repair property • An almost matching lower bound on the sub-packetization of scalar linear MSR codes
• We introduced a new problem of fractional decoding (decoding from an αn
proportion of the received vector)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
42 / 1
Summary • We constructed various families of MSR array codes • Optimally recover any number of erasures from any set of helper nodes • Optimal error resilience capability during the repair process • Optimal access property and optimal sub-packetization `
• We constructed explicit RS codes with optimal repair property • Show that there exist scalar MSR codes by constructing explicit RS codes with optimal
repair property • An almost matching lower bound on the sub-packetization of scalar linear MSR codes
• We introduced a new problem of fractional decoding (decoding from an αn
proportion of the received vector) • The decoding radius rα (n, k) is a function of α and (n, k)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
42 / 1
Summary • We constructed various families of MSR array codes • Optimally recover any number of erasures from any set of helper nodes • Optimal error resilience capability during the repair process • Optimal access property and optimal sub-packetization `
• We constructed explicit RS codes with optimal repair property • Show that there exist scalar MSR codes by constructing explicit RS codes with optimal
repair property • An almost matching lower bound on the sub-packetization of scalar linear MSR codes
• We introduced a new problem of fractional decoding (decoding from an αn
proportion of the received vector) • The decoding radius rα (n, k) is a function of α and (n, k) • We give the exact expression of the decoding radius rα (n, k).
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
42 / 1
Some open problems
• Construct explicit MSR codes with sub-packetization ` = rn/(r+1)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
43 / 1
Some open problems
• Construct explicit MSR codes with sub-packetization ` = rn/(r+1) • ` = rn/(r+1) is the best known sub-packetization value of MSR codes (Wang et al., ’16)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
43 / 1
Some open problems
• Construct explicit MSR codes with sub-packetization ` = rn/(r+1) • ` = rn/(r+1) is the best known sub-packetization value of MSR codes (Wang et al., ’16) • only existence proof, no constructions
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
43 / 1
Some open problems
• Construct explicit MSR codes with sub-packetization ` = rn/(r+1) • ` = rn/(r+1) is the best known sub-packetization value of MSR codes (Wang et al., ’16) • only existence proof, no constructions
• Close the gap between achievable sub-packetization value and the lower bound
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
43 / 1
Some open problems
• Construct explicit MSR codes with sub-packetization ` = rn/(r+1) • ` = rn/(r+1) is the best known sub-packetization value of MSR codes (Wang et al., ’16) • only existence proof, no constructions
• Close the gap between achievable sub-packetization value and the lower bound • the best known lower bound (Goparaju et al., ’14):
2 log2 `(logr/(r−1) ` + 1) ≥ k.
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
43 / 1
Some open problems
• Construct explicit MSR codes with sub-packetization ` = rn/(r+1) • ` = rn/(r+1) is the best known sub-packetization value of MSR codes (Wang et al., ’16) • only existence proof, no constructions
• Close the gap between achievable sub-packetization value and the lower bound • the best known lower bound (Goparaju et al., ’14):
2 log2 `(logr/(r−1) ` + 1) ≥ k. • Sub-packetization dependent bound on repair bandwidth (Cadambe-Mazumdar,
’15)
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
43 / 1
Some open problems
• Construct explicit MSR codes with sub-packetization ` = rn/(r+1) • ` = rn/(r+1) is the best known sub-packetization value of MSR codes (Wang et al., ’16) • only existence proof, no constructions
• Close the gap between achievable sub-packetization value and the lower bound • the best known lower bound (Goparaju et al., ’14):
2 log2 `(logr/(r−1) ` + 1) ≥ k. • Sub-packetization dependent bound on repair bandwidth (Cadambe-Mazumdar,
’15) • Cut-set bound is obtained without any restrictions on `
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
43 / 1
Some open problems
• Construct explicit MSR codes with sub-packetization ` = rn/(r+1) • ` = rn/(r+1) is the best known sub-packetization value of MSR codes (Wang et al., ’16) • only existence proof, no constructions
• Close the gap between achievable sub-packetization value and the lower bound • the best known lower bound (Goparaju et al., ’14):
2 log2 `(logr/(r−1) ` + 1) ≥ k. • Sub-packetization dependent bound on repair bandwidth (Cadambe-Mazumdar,
’15) • Cut-set bound is obtained without any restrictions on ` • What if we require ` < L? How does the lower bound change as L decreases?
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
43 / 1
Publication list: Journal papers Coding for distributed storage: 1. Min Ye and A. Barg, “Explicit constructions of optimal-access MDS codes with nearly optimal sub-packetization,” IEEE Transactions on Information Theory, 2017, arXiv:1605.08630. 2. Min Ye and A. Barg, “Explicit constructions of high-rate MDS array codes with optimal repair bandwidth,” IEEE Transactions on Information Theory, vol. 63, no. 4, pp. 2001–2014, April 2017, arXiv:1605.00454. Statistics 1. Min Ye and A. Barg, “Optimal Schemes for Discrete Distribution Estimation under Locally Differential Privacy,” submitted to IEEE Transactions on Information Theory, May 2017, arXiv:1702.00610. 2. Min Ye and A. Barg, “Asymptotically optimal private estimation under mean square loss,” arXiv:1708.00059. Polar codes 1. T. C. Gulcu, Min Ye and A. Barg, “Construction of polar codes for arbitrary discrete memoryless channels,” submitted to IEEE Transactions on Information Theory, March 2016, arXiv:1603.05736. 2. Min Ye and A. Barg, “Polar codes for distributed hierarchical source coding” Advances in Mathematics of Communications, vol. 9, no. 1, pp. 87–103, 2015, arXiv:1404.5501. Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
44 / 1
Publication list: Conference papers Coding for distributed storage: 1. I. Tamo, Min Ye and A. Barg, “Optimal repair of Reed-Solomon codes: Achieving the cut-set bound,” 58th Annual IEEE Symposium on Foundations of Computer Science (FOCS), Berkeley, CA, 2017. 2. I. Tamo, Min Ye and A. Barg, “Fractional decoding: Error correction from partial information,” in IEEE International Symposium on Information Theory (ISIT), Aachen, Germany, 2017, pp. 998–1002 (arXiv:1701.06969). 3. M. Vajha, G. Kini, B. Puranik, V. Ramkumar, E. Lobo, B. Sasidharan, P.V. Kumar, Min Ye, A. Barg, S. Hussain, S. Narayanamurthy, and S. Nandi, “Pairing up for regeneration: The Mantra for fast and efficient node repair in distributed storage,” poster presented at USENIX Annual Technical Conference, July 12-14, 2017, Santa Clara, CA. 4. Min Ye and A. Barg, “Explicit constructions of MDS array codes and RS codes with optimal repair bandwidth,” in IEEE International Symposium on Information Theory, Barcelona, Spain, 2016, pp. 1202–1206.
Statistics: 5. Min Ye and A. Barg, “Optimal Schemes for Discrete Distribution Estimation under Local Differential Privacy,” in IEEE International Symposium on Information Theory, Aachen, Germany, 2017, pp. 759–763.
Polar codes: 6. T. C. Gulcu, M. Ye, and A. Barg, “Construction of polar codes for arbitrary discrete memoryless channels,” in IEEE International Symposium on Information Theory (ISIT), Barcelona, Spain, July 11-15, 2016, pp. 51–55 7. Min Ye and A. Barg, “Polar codes using dynamic kernels,” in IEEE International Symposium on Information Theory (ISIT), Hong Kong, 2015, pp. 231–235. 8. Min Ye and A. Barg, “Universal source polarization and an application to a multi-user problem” in 52nd Annual Allerton Conference on Communication Control and Computing, 2014, pp. 805–812. Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
45 / 1
Thank you
Min Ye, Ph.D. Dissertation Defense
Coding Schemes for Distributed Storage Systems
August 11, 2017
46 / 1
