[PPT] - Pruning Neural Belief Propagation Decoders Andreas Buchberger, 1 PowerPoint Presentation

SLIDE 1

Pruning Neural Belief Propagation Decoders

Andreas Buchberger,1 Christian H¨ ager,1 Henry D. Pfister,2 Laurent Schmalen3 and, Alexandre Graell i Amat,1

1 Chalmers University of Technology, G¨

teborg, Sweden

2 Duke University, Durham, North Carolina, USA 3 Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany

ISIT 2020

SLIDE 2

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Motivation

4 4.5 5 5.5 6 6.5 10−4 10−3 10−2 10−1 Eb/N0 in dB BLER ML BP

Achieving near-ML performance for

algebraic codes such as Reed-Muller or BCH codes is computationally complex.

Belief propagation decoding offers low

complexity and good performance for sparse graph codes.

For dense parity-check matrices, belief

propagation decoding without modifications is not competitive.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 1 / 15

SLIDE 3

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Belief Propagation Decoding

λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 ˆ λ0 ˆ λ1 ˆ λ2 ˆ λ3 ˆ λ4 ˆ λ5 ˆ λ6

Parity-check matrix represented as a Tanner

graph.

Iterative decoding by passing extrinsic

messages along the edges.

Instead of iterating between the nodes, we

can unroll the graph.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 2 / 15

SLIDE 4

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Neural Networks

x0 x1 x2 x3 x4 x5 x6 b(1) b(1)

1

b(1)

2

b(1)

3

b(1)

4

b(1)

5

b(1)

6

b(3) b(3)

1

b(3)

2

b(3)

3

b(3)

4

b(3)

5

b(5) b(5)

1

b(5)

2

b(5)

3

b(5)

4

b(7) b(7)

1

b(7)

2

b(7)

3

b(7)

4

b(7)

5

b(7)

6

y0 y1 y2 y3 y4 y5 y6

Neural networks can be used to approximate

an unknown function y = f(x).

It consists of layers of

neurons connected by edges with a multiplicative weight wi and a bias bi.

The neuron sums the

weighted inputs and the bias and applies a nonlinear activation function.

tanh((·)) w(3)

5,1

w(3)

5,2

w(3)

5,3

b(3)

5

Training set: A large number of pairs (x, y)
Define a loss function Γ(y, fNN(x)).
Optimize the weights wi and bias bi by

gradient descent on the loss function Γ.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 3 / 15

SLIDE 5

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Neural Network vs. Belief Propagation Decoding

x0 x1 x2 x3 x4 x5 x6 b(1) b(1)

1

b(1)

2

b(1)

3

b(1)

4

b(1)

5

b(1)

6

b(3) b(3)

1

b(3)

2

b(3)

3

b(3)

4

b(3)

5

b(5) b(5)

1

b(5)

2

b(5)

3

b(5)

4

b(7) b(7)

1

b(7)

2

b(7)

3

b(7)

4

b(7)

5

b(7)

6

y0 y1 y2 y3 y4 y5 y6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 ˆ λ0 ˆ λ1 ˆ λ2 ˆ λ3 ˆ λ4 ˆ λ5 ˆ λ6

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 4 / 15

SLIDE 6

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Neural Belief Propagation Decoding

λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 ˆ λ0 ˆ λ1 ˆ λ2 ˆ λ3 ˆ λ4 ˆ λ5 ˆ λ6

Unroll the Tanner graph.
Transmitted symbols x.
Channel LLRs λ.
Variable node output LLRs ˆ

λ.

Augment edges

by weights w.

VN w(2)

0,ch

w(2)

0,1

w(2)

0,2

Define a loss function Γ(x, ˆ

λ, w).

Optimize the weights,

wopt = arg min

w Γ(x, ˆ

λ, w), using gradient descent.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 5 / 15

SLIDE 7

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Optimizing the Weights

ˆ λ(0) ˆ λ(1) ˆ λ(2) ˆ λ(3) λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 ˆ λ0 ˆ λ1 ˆ λ2 ˆ λ3 ˆ λ4 ˆ λ5 ˆ λ6

Binary classification for each of the n bits.
Loss function: Binary cross-entropy or soft

bit error rate

Training data: Transmission of the all-zero

codeword over the channel.

Convergence is improved by using

¯ Γ =

i

Γ(x, ˆ λ(i))

Contribution of the intermediate layers

decays over the training.

[1] E. Nachmani, Y. Be’ery, and D. Burshtein, “Learning to decode linear codes using deep learning,” in Proc. Annu. Allerton Conf. Commun., Control, Comput., Allerton, IL, USA, Sep. 2016, pp. 341—346. Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 6 / 15

SLIDE 8

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

4 4.5 5 5.5 6 6.5 10−4 10−3 10−2 10−1 Eb/N0 in dB BLER ML BP WBP [1] MBBP

Neural belief propagation decoding improves

upon conventional belief propagation decoding since the weights compensate for cycles in the Tanner graph.

It does not account for the fact that the

parity-check matrix may be ill suited for belief propagation decoding.

Decode multiple parity-check matrices in

parallel and choose the best result - multiple bases belief propagation.

[2] T. Hehn, J. Huber, O. Milenkovic, and S. Laendner, “Multiple-bases belief-propagation decoding of high-density cyclic codes,” IEEE Trans. Commun., vol. 58, no. 1, pp. 1-–8, Jan. 2010. Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 7 / 15

SLIDE 9

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Our Contribution

Contribution

We consider neural belief propagation decoder.
We introduce a method to optimize the parity-check matrix based on pruning.
We demonstrate via numerical simulations for Reed-Muller and LDPC codes that our approach
utperforms conventional and the original neural belief propagation decoder while allowing

lower complexity.

Main idea

Starting from a large, overcomplete parity-check matrix, exploit the fact that some parity-check equations contribute less to decoding than others.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 8 / 15

SLIDE 10

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Pruning the Neural Belief Propagation Decoder

λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 ˆ λ0 ˆ λ1 ˆ λ2 ˆ λ3 ˆ λ4 ˆ λ5 ˆ λ6

Start with a large overcomplete

parity-check matrix.

Tie the weights at each check node.

CN w(1)

4

w(1)

4

w(1)

4

w(1)

4

The weights indicate the contribution of

the respective check node.

Schedule:
1. Train the network.
2. Find the least important check node and

remove it.

3. Train the network.
4. If the performance starts to degrade - stop,
therwise go to step 2.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 9 / 15

SLIDE 11

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Pruning the Neural Belief Propagation Decoder

λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 ˆ λ0 ˆ λ1 ˆ λ2 ˆ λ3 ˆ λ4 ˆ λ5 ˆ λ6

Start with a large overcomplete

parity-check matrix.

Tie the weights at each check node.

CN w(1)

4

w(1)

4

w(1)

4

w(1)

4

The weights indicate the contribution of

the respective check node.

Schedule:
1. Train the network.
2. Find the least important check node and

remove it.

3. Train the network.
4. If the performance starts to degrade - stop,
therwise go to step 2.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 9 / 15

SLIDE 12

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Pruning the Neural Belief Propagation Decoder

λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 ˆ λ0 ˆ λ1 ˆ λ2 ˆ λ3 ˆ λ4 ˆ λ5 ˆ λ6

Start with a large overcomplete

parity-check matrix.

Tie the weights at each check node.

CN w(1)

4

w(1)

4

w(1)

4

w(1)

4

The weights indicate the contribution of

the respective check node.

Schedule:
1. Train the network.
2. Find the least important check node and

remove it.

3. Train the network.
4. If the performance starts to degrade - stop,
therwise go to step 2.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 9 / 15

SLIDE 13

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Pruning the Neural Belief Propagation Decoder

λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 ˆ λ0 ˆ λ1 ˆ λ2 ˆ λ3 ˆ λ4 ˆ λ5 ˆ λ6

Start with a large overcomplete

parity-check matrix.

Tie the weights at each check node.

CN w(1)

4

w(1)

4

w(1)

4

w(1)

4

The weights indicate the contribution of

the respective check node.

Schedule:
1. Train the network.
2. Find the least important check node and

remove it.

3. Train the network.
4. If the performance starts to degrade - stop,
therwise go to step 2.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 9 / 15

SLIDE 14

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Pruning the Neural Belief Propagation Decoder

λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 ˆ λ0 ˆ λ1 ˆ λ2 ˆ λ3 ˆ λ4 ˆ λ5 ˆ λ6

Start with a large overcomplete

parity-check matrix.

Tie the weights at each check node.

CN w(1)

4

w(1)

4

w(1)

4

w(1)

4

The weights indicate the contribution of

the respective check node.

Schedule:
1. Train the network.
2. Find the least important check node and

remove it.

3. Train the network.
4. If the performance starts to degrade - stop,
therwise go to step 2.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 9 / 15

SLIDE 15

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Pruning the Neural Belief Propagation Decoder

λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 ˆ λ0 ˆ λ1 ˆ λ2 ˆ λ3 ˆ λ4 ˆ λ5 ˆ λ6

Start with a large overcomplete

parity-check matrix.

Tie the weights at each check node.

CN w(1)

4

w(1)

4

w(1)

4

w(1)

4

The weights indicate the contribution of

the respective check node.

Schedule:
1. Train the network.
2. Find the least important check node and

remove it.

3. Train the network.
4. If the performance starts to degrade - stop,
therwise go to step 2.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 9 / 15

SLIDE 16

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Pruning the Neural Belief Propagation Decoder

λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 ˆ λ0 ˆ λ1 ˆ λ2 ˆ λ3 ˆ λ4 ˆ λ5 ˆ λ6

Start with a large overcomplete

parity-check matrix.

Tie the weights at each check node.

CN w(1)

4

w(1)

4

w(1)

4

w(1)

4

The weights indicate the contribution of

the respective check node.

Schedule:
1. Train the network.
2. Find the least important check node and

remove it.

3. Train the network.
4. If the performance starts to degrade - stop,
therwise go to step 2.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 9 / 15

SLIDE 17

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Interpretation

λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 λch,0 λch,1 λch,2 λch,3 λch,4 λch,5 λch,6 ˆ λ0 ˆ λ1 ˆ λ2 ˆ λ3 ˆ λ4 ˆ λ5 ˆ λ6

Removing check nodes ↔ removing rows in

the parity-check matrix.

Removing different check nodes in different

layers ↔ using a different parity-check matrix in each decoding iteration.

Define a set of parity-check matrices

Hopt = {H1, . . . , HL}.

Define a set of optimized weights Wopt.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 10 / 15

SLIDE 18

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Variations of the Pruned Decoder

Decoder D1

Using the result from the optimization directly, i.e., Hopt and Wopt.

Decoder D2

Using the optimized set of parity-check matrices, i.e., Hopt, but sets all weights to one, i.e., neglects Wopt.

Decoder D3

Using the optimized set of parity-check matrices, i.e., Hopt, and additionally untied optimized weights over all iterations and edges.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 11 / 15

SLIDE 19

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

The Reed-Muller Code RM(2, 5)

4 4.5 5 5.5 6 10−4 10−3 10−2 10−1 Eb/N0 in dB BLER ML WBP H MBBP D1 D2 D3 Random Pruning WBP Hoc RNN Hoc

n = 32, k = 16, six iterations
Overcomplete parity-check matrix:

All 620 minimum-weight codewords

f the dual code.
MBBP [2]: 15 randomly chosen

parity-check matrices with six iterations.

Number of CN evaluations:
D1, D2, D3, random pruning:

620 · 6 · 0.31 = 1170

WBP, RNN Hoc: 620 · 6 = 3720
MBBP: 15 · 6 · 16 = 1440
WBP H: 16 · 6 = 96

[3] E. Nachmani, E. Marciano, L. Lugosch, W. J. Gross,

D. Burshtein, and Y. Be’ery, “Deep learning methods for

improved decoding of linear codes,” IEEE J. Sel. Topics Signal Process., vol. 12, no. 1, pp. 119-–131, Feb. 2018. Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 12 / 15

SLIDE 20

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

The Reed-Muller Code RM(2, 5)

1 2 3 4 5 6 0.1 0.2 0.3 0.4 Decoding iteration Fraction of CNs

n = 32, k = 16, six iterations
Overcomplete parity-check matrix:

All 620 minimum-weight codewords

f the dual code.
MBBP [2]: 15 randomly chosen

parity-check matrices with six iterations.

Number of CN evaluations:
D1, D2, D3, random pruning:

620 · 6 · 0.31 = 1170

WBP, RNN Hoc: 620 · 6 = 3720
MBBP: 15 · 6 · 16 = 1440
WBP H: 16 · 6 = 96

[3] E. Nachmani, E. Marciano, L. Lugosch, W. J. Gross,

D. Burshtein, and Y. Be’ery, “Deep learning methods for

improved decoding of linear codes,” IEEE J. Sel. Topics Signal Process., vol. 12, no. 1, pp. 119-–131, Feb. 2018. Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 12 / 15

SLIDE 21

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

The Reed-Muller Code RM(3, 7)

2.5 3 3.5 4 4.5 5 10−4 10−3 10−2 10−1 Eb/N0 in dB BLER ML MBBP D1 ˜ D1 D2 D3 WBP Hoc RNN Hoc

n = 128, k = 64
Six iterations.
Overcomplete parity-check matrix:

All 94488 minimum-weight codewords of the dual code.

MBBP [2]: 60 randomly chosen

parity-check matrices with six iterations.

Number of CN evaluations:
D1, D2, D3:

94488 · 6 · 0.03 = 19842

WBP, RNN Hoc:

94488 · 6 = 566928

MBBP [2]: 60 · 6 · 64 = 23440

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 13 / 15

SLIDE 22

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

A Short LDPC Code

2 2.5 3 3.5 4 4.5 5 5.5 10−4 10−3 10−2 10−1 Eb/N0 in dB BLER ML D1 D3 BP, 25 iterations BP, 100 iterations WBP

Short LDPC code standardized by

CCSDS.

n = 128, k = 64
Overcomplete parity-check matrix:

10000 randomly chosen codewords

f low weight of the dual code.
Number of CN evaluations:
D1, D3: 10000 · 6 · 0.027 = 1600
BP, 25 it.: 64 · 25 = 1600
BP, 100 it.: 64 · 100 = 6400
WBP: 64 · 6 = 384

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 14 / 15

SLIDE 23

Introduction Pruning Neural BP Decoders Numerical Results Conclusion

Conclusion

Take-home message

Starting with a large, overcomplete parity-check matrix, we can tailor it to (neural) belief

propagation decoding by pruning check nodes.

For RM codes we demonstrated a performance close to ML decoding with practical complexity.
For a short LDPC code, we improve upon BP decoding with practical complexity.
This concept can be applied to any other linear block code and similar gains can be expected.

Pruning Neural BP Decoders |

A. Buchberger, C. H¨

ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 15 / 15