Fast Polarization for Processes with Memory Boaz Shuval and Ido Tal - PowerPoint PPT Presentation

Fast Polarization for Processes with Memory Boaz Shuval and Ido Tal Andrew and Erna Viterbi Department of Electrical Engineering Technion — Israel Institute of Technology Haifa, 32000, Israel June 2018 1/21

In this talk Setting : binary-input, symmetric, memoryless channel 2/21

In this talk Setting : binary-input, symmetric, memoryless channel 2/21

In this talk Setting : binary-input, symmetric, memoryless channel 2/21

In this talk Setting : binary-input, symmetric, memoryless channel 2/21

In this talk Setting : binary-input, symmetric, ✭✭✭✭✭✭ ❤❤❤❤❤❤ ✭ memoryless channel ❤ 2/21

Polar codes: [Arıkan:09], [ArıkanTelatar:09], [S ¸as ¸o˘ glu+:09], [KoradaUrbanke:10], [HondaYamamoto:13] ◮ Setting : Memoryless i.i.d. process ( X i , Y i ) N i = 1 ◮ For simplicity : Assume X i binary 1 = X N 1 · G N ◮ Polar transform : U N ◮ Index sets : � 1 ) < 2 − N β � i : H ( U i | U i − 1 Λ N = , Y N Low entropy: 1 � 1 ) > 1 − 2 − N β � i : H ( U i | U i − 1 Ω N = , Y N High entropy: 1 ◮ Polarization : 1 lim N | Λ N | = 1 − H ( X 1 | Y 1 ) N →∞ 1 lim N | Ω N | = H ( X 1 | Y 1 ) N →∞ 3/21

Polar codes: Optimal rate for: ◮ Coding for non-symmetric memoryless channels ◮ Coding for memoryless channels with non-binary inputs ◮ (Lossy) compression of memoryless sources Question ◮ How to handle memory? 4/21

A framework for memory ◮ Process : ( X i , Y i , S i ) N i = 1 ◮ Finite number of states : S i ∈ S , where |S| < ∞ ◮ Hidden state : S i is unknown to encoder and decoder ◮ Probability distribution : P ( x i , y i , s i | s i − 1 ) ◮ Stationary : same for all i ◮ Markov : P ( x i , y i , s i | s i − 1 ) = P ( x i , y i , s i |{ x j , y j , s j } j < i ) ◮ State sequence : aperiodic and irreducible Markov chain 5/21

Example 1 ◮ Model : Finite state channel P s ( y | x ) , s ∈ S ◮ Input distribution : X i i.i.d. and independent of state ◮ State transition : π ( s i | s i − 1 ) ◮ Distribution : P ( x i , y i , s i | s i − 1 ) = P ( x i ) π ( s i | s i − 1 ) P s i ( y i | x i ) 6/21

Example 2 ◮ Model : ISI + noise Y i = h 0 X i + h 1 X i − 1 + · · · + h m X i − m + noise ◮ Input : X i has memory P ( x i | x i − 1 , x i − 2 , . . . , x i − m , x i − m − 1 ) ◮ State : � � S i = · · · X i X i − 1 X i − m ◮ Distribution : For x i , s i , s i − 1 compatible, P ( x i , y i , s i | s i − 1 ) = P noise ( y i | h T s i ) · P ( x i | s i − 1 ) 7/21

Example 3 ◮ Model : ( d , k ) -RLL constrained system with noise ( 1 , ∞ ) -RLL Constraint 1 0 0 8/21

Example 3 ◮ Model : ( d , k ) -RLL constrained system with noise ( 1 , ∞ ) -RLL Constraint 1 X N Y N 1 1 BSC ( p ) 0 0 8/21

Example 3 ◮ Model : ( d , k ) -RLL constrained system with noise ( 1 , ∞ ) -RLL Constraint 1 X N Y N 1 − α 1 1 BSC ( p ) α 0 1 0 state Markov chain 0 / 0 1 / 0 α ( 1 − p ) ( 1 − α )( p ) 1 / 1 ( 1 − α ) ( 1 − ) p P ( x i , y i , s i | s i − 1 ) ( 1 1 − ) p 0 / 0 1 ( p ) α ( p ) 1 / 0 0 / 1 8/21

Polar codes: [S ¸as ¸o˘ glu:11], [S ¸as ¸o˘ gluTal:16], [ShuvalTal:17] ◮ Setting : Process ( X i , Y i , S i ) N i = 1 with memory, as above ◮ Hidden state : State unknown to encoder and decoder 1 = X N 1 · G N ◮ Polar transform : U N U N 1 are neither independent, nor identically distributed ◮ Index sets : � 1 ) < 2 − N β � i : H ( U i | U i − 1 Λ N = , Y N Low entropy: 1 � 1 ) > 1 − 2 − N β � i : H ( U i | U i − 1 Ω N = , Y N High entropy: 1 ◮ Polarization : 1 lim N | Λ N | = 1 − H ⋆ ( X | Y ) N →∞ 1 lim N | Ω N | = H ⋆ ( X | Y ) N →∞ lim N →∞ 1 N H ( X N 1 | Y N 1 ) 9/21

Achievable rate ◮ Achievable rate : In all examples, R approaches 1 I ⋆ ( X ; Y ) = lim NI ( X N 1 ; Y N 1 ) N →∞ ◮ Also lossy compression of a source with memory ◮ Successive cancellation : [Wang+:15] ◮ Without state estimation! 10/21

Three parameters ◮ Joint distribution P ( x , y ) ◮ For simplicity: X ∈ { 0 , 1 } ◮ Parameters : H ( X | Y ) = − � x , y P ( x , y ) log P ( x | y ) Entropy � Z ( X | Y ) = 2 � P ( 0 , y ) P ( 1 , y ) Bhattacharyya y K ( X | Y ) = � y | P ( 0 , y ) − P ( 1 , y ) | T.V. distance ◮ Connections : H ≈ 0 ⇐ ⇒ Z ≈ 0 ⇐ ⇒ K ≈ 1 H ≈ 1 ⇐ ⇒ Z ≈ 1 ⇐ ⇒ K ≈ 0 11/21

Three processes For n = 1 , 2 , . . . { X i , Y i , S i } ◮ N = 2 n 1 = X N ◮ U N 1 G N { X i , Y i } ◮ Pick B n ∈ { 0 , 1 } uniform, i.i.d. ◮ Random index from { 1 , 2 , . . . , N } i = 1 + � B 1 B 2 · · · B n � 2 ◮ Processes : H n = H ( U i | U i − 1 , Y N 1 ) Entropy 1 Z n = Z ( U i | U i − 1 , Y N 1 ) Bhattacharyya 1 K n = K ( U i | U i − 1 , Y N 1 ) T.V. distance 1 12/21

Proof — memoryless case Slow polarization Fast polarization � B n + 1 = 0 2 Z n H n ∈ ( ǫ, 1 − ǫ ) Z n + 1 ≤ B n + 1 = 1 Z 2 n | H n + 1 − H n | > 0 1 N | Λ N | − n →∞ 1 − H ( X 1 | Y 1 ) − − → Low entropy set New � B n + 1 = 0 K 2 K n + 1 ≤ n B n + 1 = 1 2 K n 1 N | Ω N | − n →∞ H ( X 1 | Y 1 ) − − → High entropy set 13/21

❍❍ ✟ Proof — memory ✟✟ less case [S ❍ ¸as ¸o˘ gluTal:16], [ShuvalTal:17] { X i , Y i , S i } Slow polarization Fast polarization { X i , Y i } � 2 ψ Z n B n + 1 = 0 H n ∈ ( ǫ, 1 − ǫ ) Z n + 1 ≤ ψ Z 2 B n + 1 = 1 n | H n + 1 − H n | > 0 1 N | Λ N | − n →∞ 1 − H ⋆ ( X | Y ) − − → Low entropy set New � ψ ˆ B n + 1 = 0 K 2 1 ˆ ψ = ψ ( 0 ) = max K n + 1 ≤ n π ( s ) 2 ˆ B n + 1 = 1 s K n π : stationary state distribution 1 N | Ω N | − n →∞ H ⋆ ( X | Y ) − − → High entropy set 14/21

Fast polarization to high entropy set Ω N ◮ Memoryless case : ◮ Parameter evolution inequality hinges on independence: P ( x 2 N 1 , y 2 N 1 ) = P ( x N 1 , y N 1 ) · P ( x 2 N N + 1 , y 2 N N + 1 ) ◮ Memory case : ◮ Force independence: condition on middle state S N P ( x 2 N 1 , y 2 N 1 | s N ) = P ( x N 1 , y N 1 | s N ) · P ( x 2 N N + 1 , y 2 N N + 1 | s N ) ◮ New processes : ˆ H n = H ( U i | U i − 1 , Y N 1 , S 0 , S N ) 1 ˆ K n = K ( U i | U i − 1 , Y N 1 , S 0 , S N ) 1 15/21

Polarization of K n (memoryless case) 1 = X N U N 1 · G N V N 1 = X 2 N N + 1 · G N Q i = ( U i − 1 1 ) , Y N ◮ Memoryless assumption : 1 R i = ( V i − 1 N + 1 ) , Y 2 N 1 P ( u i , v i , q i , r i ) = P ( u i , q i ) · P ( v i , r i ) ◮ Notation : T i = U i + V i ◮ One step polarization : � K ( T i | Q i , R i ) B n + 1 = 0 ‘ − ’ transform K n + 1 = K ( V i | T i , Q i , R i ) B n + 1 = 1 ‘ + ’ transform ◮ Recall : � K ( X | Y ) = | P ( 0 , y ) − P ( 1 , y ) | y 16/21

Polarization of K n (memoryless case), ‘ − ’ transform � K n + 1 = | P T i , Q i , R i ( 0 , q , r ) − P T i , Q i , R i ( 1 , q , r ) | q , r � � 1 � � � � = P ( v , r )( P ( v , q ) − P ( v + 1 , q )) � � � � � � q , r v = 0 � �� � �� � � � = P ( 0 , q ) − P ( 1 , q ) P ( 0 , r ) − P ( 1 , r ) � � � � q , r � = | P ( 0 , q ) − P ( 1 , q ) | · | P ( 0 , r ) − P ( 1 , r ) | q , r � � = | P ( 0 , q ) − P ( 1 , q ) | · | P ( 0 , r ) − P ( 1 , r ) | q r = K 2 n , 17/21

Polarization of K n (memoryless case), ‘ + ’ transform � � � K n + 1 = � P T i , V i , Q i , R i ( t , 0 , q , r ) − P T i , V i , Q i , R i ( t , 1 , q , r ) � t , q , r � = | P ( t , q ) P ( 0 , r ) − P ( t + 1 , q ) P ( 1 , r ) | t , q , r ( ∗ ) ≤ 1 � P ( q ) | P ( 0 , r ) − P ( 1 , r ) | + P ( r ) | P ( t , q ) − P ( t + 1 , q ) | 2 t , q , r = 1 | P ( 0 , r ) − P ( 1 , r ) | + 1 � � | P ( t , q ) − P ( t + 1 , q ) | 2 2 t , r t , q = 2K n , Identity for ( ∗ ) : For any a , b , c , d : ab − cd = ( a + c )( b − d ) + ( b + d )( a − c ) 2 18/21

Polarization of ˆ K n (memory) ◮ Follows steps of memoryless case ◮ Requires additional inequalities ◮ Inequality I : For states s 0 , s N , s 2 N ∈ S , P ( s 0 , s N , s 2 N ) = P ( s 0 , s N ) · P ( s N , s 2 N ) P ( s N ) ≤ ψ · P ( s 0 , s N ) · P ( s N , s 2 N ) where 1 ψ = max π ( s ) s ◮ Inequality II : For f , g ≥ 0, � � � g ( s ′ f ( s N ) g ( s N ) ≤ f ( s N ) N ) s ′ s N s N N 19/21

Connections Extreme Values Ordering ˆ H ≈ 0 ⇔ Z ≈ 0 ⇔ K ≈ 1 H n ≤ H n ˆ Z n ≤ Z n H ≈ 1 ⇔ Z ≈ 1 ⇔ K ≈ 0 ˆ K n ≥ K n ˆ ( · ) processes also for All six processes ( H n , ˆ H n , Z n , ˆ Z n , K n , ˆ K n ) polarize fast both to 0 and 1 with any β < 1 / 2 20/21

Summary ◮ A general framework for memory: P ( x i , y i , s i | s i − 1 ) ◮ Memory allowed in both source and channel ◮ State sequence S i ◮ Hidden ◮ Stationary ◮ Finite state Markov ◮ Aperiodic and irreducible ◮ Achieve rate I ⋆ ( X ; Y ) through polar codes ◮ No change to polarization exponent ( β < 1 / 2) 21/21