Pruning Neural Belief Propagation Decoders Andreas Buchberger, 1 - PowerPoint PPT Presentation

Pruning Neural Belief Propagation Decoders Andreas Buchberger, 1 Christian H¨ ager, 1 Henry D. Pfister, 2 Laurent Schmalen 3 and, Alexandre Graell i Amat, 1 1 Chalmers University of Technology, G¨ oteborg, Sweden 2 Duke University, Durham, North Carolina, USA 3 Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany ISIT 2020

Introduction Pruning Neural BP Decoders Numerical Results Conclusion Motivation 10 − 1 • Achieving near-ML performance for ML BP algebraic codes such as Reed-Muller or BCH codes is computationally complex . 10 − 2 • Belief propagation decoding offers low complexity and good performance for sparse BLER graph codes. • For dense parity-check matrices , belief 10 − 3 propagation decoding without modifications is not competitive . 10 − 4 4 . 5 5 . 5 6 . 5 4 5 6 E b /N 0 in dB Pruning Neural BP Decoders | A. Buchberger, C. H¨ ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 1 / 15

Introduction Pruning Neural BP Decoders Numerical Results Conclusion Belief Propagation Decoding • Parity-check matrix represented as a Tanner λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 graph . • Iterative decoding by passing extrinsic messages along the edges. λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 • Instead of iterating between the nodes, we can unroll the graph . λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 ˆ ˆ ˆ ˆ ˆ ˆ ˆ λ 0 λ 1 λ 2 λ 3 λ 4 λ 5 λ 6 Pruning Neural BP Decoders | A. Buchberger, C. H¨ ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 2 / 15

Introduction Pruning Neural BP Decoders Numerical Results Conclusion Neural Networks • Neural networks can be used to approximate x 0 x 1 x 2 x 3 x 4 x 5 x 6 an unknown function y = f ( x ) . b (1) b (1) b (1) b (1) b (1) b (1) b (1) • It consists of layers of w (3) w (3) w (3) 0 1 2 3 4 5 6 5 , 1 5 , 2 5 , 3 neurons connected by edges with a multiplicative b (3) b (3) b (3) b (3) b (3) b (3) weight w i and a bias b i . 0 1 2 3 4 5 b (3) tanh( � ( · )) • The neuron sums the 5 weighted inputs and the bias and applies a nonlinear b (5) b (5) b (5) b (5) b (5) activation function . 0 1 2 3 4 • Training set: A large number of pairs ( x , y ) • Define a loss function Γ( y , f NN ( x )) . • Optimize the weights w i and bias b i by b (7) b (7) b (7) b (7) b (7) b (7) b (7) gradient descent on the loss function Γ . 0 1 2 3 4 5 6 y 0 y 1 y 2 y 3 y 4 y 5 y 6 Pruning Neural BP Decoders | A. Buchberger, C. H¨ ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 3 / 15

Introduction Pruning Neural BP Decoders Numerical Results Conclusion Neural Network vs. Belief Propagation Decoding x 0 x 1 x 2 x 3 x 4 x 5 x 6 λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 b (1) b (1) b (1) b (1) b (1) b (1) b (1) 0 1 2 3 4 5 6 b (3) b (3) b (3) b (3) b (3) b (3) λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 0 1 2 3 4 5 b (5) b (5) b (5) b (5) b (5) λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 0 1 2 3 4 b (7) b (7) b (7) b (7) b (7) b (7) b (7) λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 0 1 2 3 4 5 6 ˆ ˆ ˆ ˆ ˆ ˆ ˆ y 0 y 1 y 2 y 3 y 4 y 5 y 6 λ 0 λ 1 λ 2 λ 3 λ 4 λ 5 λ 6 Pruning Neural BP Decoders | A. Buchberger, C. H¨ ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 4 / 15

Introduction Pruning Neural BP Decoders Numerical Results Conclusion Neural Belief Propagation Decoding • Unroll the Tanner graph. λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 • Transmitted symbols x . • Channel LLRs λ . • Variable node output LLRs ˆ λ . λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 w (2) w (2) 0 , 1 0 , 2 • Augment edges w (2) VN by weights w . 0 , ch λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 • Define a loss function Γ( x , ˆ λ , w ) . • Optimize the weights, w Γ( x , ˆ w opt = arg min λ , w ) , λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 ˆ ˆ ˆ ˆ ˆ ˆ ˆ λ 0 λ 1 λ 2 λ 3 λ 4 λ 5 λ 6 using gradient descent. Pruning Neural BP Decoders | A. Buchberger, C. H¨ ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 5 / 15

Introduction Pruning Neural BP Decoders Numerical Results Conclusion Optimizing the Weights • Binary classification for each of the n bits. λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 • Loss function: Binary cross-entropy or soft λ (0) ˆ bit error rate • Training data: Transmission of the all-zero codeword over the channel. λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 • Convergence is improved by using λ (1) ˆ ¯ � λ ( i ) ) Γ( x , ˆ Γ = i λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 • Contribution of the intermediate layers decays over the training. λ (2) ˆ [1] E. Nachmani, Y. Be’ery, and D. Burshtein, “Learning to decode linear λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 codes using deep learning,” in Proc. Annu. Allerton Conf. Commun., Control, Comput., Allerton, IL, USA, Sep. 2016, pp. 341—346. ˆ ˆ ˆ ˆ ˆ ˆ ˆ λ (3) ˆ λ 0 λ 1 λ 2 λ 3 λ 4 λ 5 λ 6 Pruning Neural BP Decoders | A. Buchberger, C. H¨ ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 6 / 15

Introduction Pruning Neural BP Decoders Numerical Results Conclusion 10 − 1 • Neural belief propagation decoding improves ML BP upon conventional belief propagation WBP [1] decoding since the weights compensate for MBBP cycles in the Tanner graph. 10 − 2 • It does not account for the fact that the parity-check matrix may be ill suited for BLER belief propagation decoding. • Decode multiple parity-check matrices in parallel and choose the best result - multiple 10 − 3 bases belief propagation . [2] T. Hehn, J. Huber, O. Milenkovic, and S. Laendner, “Multiple-bases belief-propagation decoding of high-density cyclic codes,” IEEE Trans. 10 − 4 Commun., vol. 58, no. 1, pp. 1-–8, Jan. 2010. 4 . 5 5 . 5 6 . 5 4 5 6 E b /N 0 in dB Pruning Neural BP Decoders | A. Buchberger, C. H¨ ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 7 / 15

Introduction Pruning Neural BP Decoders Numerical Results Conclusion Our Contribution Contribution • We consider neural belief propagation decoder . • We introduce a method to optimize the parity-check matrix based on pruning . • We demonstrate via numerical simulations for Reed-Muller and LDPC codes that our approach outperforms conventional and the original neural belief propagation decoder while allowing lower complexity . Main idea Starting from a large, overcomplete parity-check matrix , exploit the fact that some parity-check equations contribute less to decoding than others. Pruning Neural BP Decoders | A. Buchberger, C. H¨ ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 8 / 15

Introduction Pruning Neural BP Decoders Numerical Results Conclusion Pruning the Neural Belief Propagation Decoder • Start with a large overcomplete λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 parity-check matrix . • Tie the weights at each check node. λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 CN w (1) w (1) w (1) w (1) 4 4 4 4 • The weights indicate the contribution of λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 the respective check node. • Schedule : 1. Train the network. 2. Find the least important check node and remove it. λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 3. Train the network. 4. If the performance starts to degrade - stop, ˆ ˆ ˆ ˆ ˆ ˆ ˆ λ 0 λ 1 λ 2 λ 3 λ 4 λ 5 λ 6 otherwise go to step 2. Pruning Neural BP Decoders | A. Buchberger, C. H¨ ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 9 / 15

Introduction Pruning Neural BP Decoders Numerical Results Conclusion Pruning the Neural Belief Propagation Decoder • Start with a large overcomplete λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 parity-check matrix . • Tie the weights at each check node. λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 CN w (1) w (1) w (1) w (1) 4 4 4 4 • The weights indicate the contribution of λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 the respective check node. • Schedule : 1. Train the network. 2. Find the least important check node and remove it. λ ch , 0 λ ch , 1 λ ch , 2 λ ch , 3 λ ch , 4 λ ch , 5 λ ch , 6 3. Train the network. 4. If the performance starts to degrade - stop, ˆ ˆ ˆ ˆ ˆ ˆ ˆ λ 0 λ 1 λ 2 λ 3 λ 4 λ 5 λ 6 otherwise go to step 2. Pruning Neural BP Decoders | A. Buchberger, C. H¨ ager, H. D. Pfister, L. Schmalen, A. Graell i Amat 9 / 15