Lecture 4 Matthieu Bloch 1 Source coding As shown in Fig. 1, the - PDF document

1 Figure 1: Source coding with side information. assignments. can then analyze the probability of reconstruction error averaged over the set of all possible random sequences by assigning them a label drawn uniformly at random from a set of possible labels; one bution of the source is fixed as part of the problem. Tie crux of random binning is to “bin” source Tiis technique is the natural approach when studying source coding problems, in which the distri- channel coding: instead of generating codewords at random, one labels all sequences at random. random binning , which can be viewed as a counterpart to the random coding used in the proof of Tie study of source coding with side information presented next relies on a technique known as X . lim sup and lim sup is measured in terms of the average probability of error Revised December 5, 2019 Information Theoretic Security Lecture 4 Matthieu Bloch 1 Source coding As shown in Fig. 1, the problem of source coding with side information consists in compressing sequences X emitted by a discrete source ( X × Y , { p X n Y n } n ⩾ 1 ) into messages W , in such a way that a decoder having access the message W and to a correlated sequence Y can perform a reliable estimation ˆ source P X n Y n ! ! M � ENCODER DECODER ! An ( M, n ) source code C with side information at the decoder consists of two stochastic maps f : X n → � 1 , M � , to compress source sequences, and g : � 1 , M � × Y n → X n ∪ { ? } , which outputs an estimate of the encoded sequence or an error symbol “?.” Tie performance of a code C ( ) ˆ P e ( C ) ≜ P X � = X |C = P ( g ( f ( X ) , Y ) � = X ) . Definition 1.1 (Achievable source coding rate) . A rate R is an achievable source coding rate if there exists a sequence of ( M n , n ) codes {C n } n ⩾ 1 such that log M n ⩽ R P e ( C n ) = 0 . n n →∞ n →∞ Tie infimum of all achievable source coding rates is called the source coding capacity and denoted S ( { P X n Y n } n ⩾ 1 ) . 1.1 Random binning for source reliability Formally, we consider a generic source ( U × V , p UV ) in which |U| < ∞ . For each sequence u ∈ U , let ϕ ( u ) be a bin index drawn independently according to a uniform distribution on � 1 , M � .

2 averaged over the randomly generated bin indices satisfies the following. sive applications of the union bound, we obtain Lemma 1.2 (Random binning for source reliability) . Revised December 5, 2019 Information Theoretic Security For simplicity set K ≜ |U| and U = { u i : i ∈ � 1 , K � } . Tien, if Φ( u ) denotes the random variable representing the random mapping of sequence u to an index, we have ( 1 K ) K ∏ ∀{ j i } K i =1 ∈ � 1 , M � K P (Φ( u 1 ) = j 1 , . . . , Φ( u K ) = j K ) = P (Φ( u i ) = j i ) = . M i =1 For γ > 0 , let { 1 } B γ = ( u, v ) ∈ U × V : log P U | V ( u | v ) < γ . Define the encoder as the mapping f : U → � 1 , M � : u �→ ϕ ( u ) . Define the decoder g : V × � 1 , M � : ( v, w ) �→ u ∗ , where u ∗ = u if u is the unique sequence such that ( u, v ) ∈ B γ and f ( u ) = w ; otherwise, an error u ∗ =? is declared. Tien, the probability of decoding error P e ∈ B γ ) + 2 γ E C ( P e ( C )) ⩽ P P UV (( U, V ) / M . Proof. Let F denote the random variable representing the random indexing function. With succes- (∑ ) ∈ B γ or ∃ u ′ � = u s.t. ( u ′ , v ) ∈ B γ and F ( u ′ ) = F ( u ) } ∑ E ( P e ) = E P UV ( u, v ) 1 { ( u, v ) / u v (∑ ) ∑ ⩽ E P UV ( u, v ) 1 { ( u, v ) / ∈ B γ } u v (∑ ) P UV ( u, v ) 1 {∃ u ′ � = u s.t. ( u ′ , v ) ∈ B γ and F ( u ′ ) = F ( u ) } ∑ + E u v   ∑ ∑ ∑ ⩽ P P UV (( u, v ) / P UV ( u, v ) 1 { ( u ′ , v ) ∈ B γ and F ( u ′ ) = F ( u ) } ∈ B γ ) + E  u v u ′ ̸ = u ∑ ∑ ∑ P UV ( u, v ) 1 { ( u ′ , v ) ∈ B γ } E ( 1 { F ( u ′ ) = F ( u ) } ) . = P P UV (( u, v ) / ∈ B γ ) + u v u ′ ̸ = u For any u ′ � = u ∈ U , the random binning procedures guarantees that E ( 1 { F ( u ′ ) = F ( u ) } ) = P ( F ( u ) = F ( u ′ )) = 1 M . For any ( u, v ) ∈ B γ , the definition of B γ ensures that 1 ⩽ P U | V ( u | v )2 γ . Consequently, P U | V ( u | v )2 γ = 2 γ . ∑ ∑ 1 { ( u, v ) ∈ B γ } ⩽ ∀ y ∈ Y u u Notice that the term ∑ u 1 { ( u, v ) ∈ B γ } is well-defined because |U| < ∞ . Tierefore, ∑ ∑ 1 { ( u ′ , v ) ∈ B γ } E ( 1 { F ( u ′ ) = F ( u ) } ) E ( P e ) ⩽ P P UV (( U, V ) / ∈ B γ ) + P V ( v ) v u ′ ∈ B γ ) + 2 γ ⩽ P P UV (( U, V ) / M . ■

3 such that Using Markov’s inequality, we obtain the following result. achievable. log log By direct application, we obtain lim sup Revised December 5, 2019 Information Theoretic Security Proposition 1.3. Let ( U × V , p UV ) be a source. For any M ∈ N ∗ and γ > 0 , there exists an ( M, 1) source code C with deterministic encoder and decoder such that ∈ B γ ) + 2 γ P e ( C ) ⩽ P P UV (( U, V ) / M . 1.2 Source coding with side information for discrete memoryless sources Definition 1.4 (Discrete Memoryless Source (DMS)) . A DMS ( X , P X ) is a source ( X , { P X n } n ⩾ 1 ) for which P X n is a product distribution, i.e., for any sequence x ∈ X n , we have n ∏ P X n ( x ) = P X ( x i ) . i =1 ⊗ n X in place of P X n and S ( P X ) in place of S ( { P X n } n ⩾ 1 ) . For simplicity, we write P Proposition 1.5 (Achievability for source coding with side information) . For a DMS ( X × Y , P XY ) and any R > H ( X | Y ) , there exists a sequence {C n } n ⩾ 1 of ( M n , n ) source codes and a constant α > 0 log M n ⩽ R and P e ( C n ) ⩽ 2 − αn . n n →∞ ⊗ n Proof. Tie result follows by applying Lemma 1.2 to the product distribution P XY in place of P UV . + 2 γ E ( P e ( C n )) ⩽ P P ( ∈ B n ) ( X , Y ) / M . ⊗ n γ XY ⊗ n Since P XY is a product distribution, ( n ) 1 ∑ ∈ B n ( ) ( X , Y ) / = P P P X | Y ( X i | Y i ) > γ . P P ⊗ n ⊗ n γ XY XY i =1 For any δ > 0 , upon choosing γ ≜ (1 + δ ) H ( X | Y ) , we obtain ( n ) 1 ∑ ⩽ 2 − βn P X | Y ( X i | Y i ) > (1 + δ ) H ( X | Y ) P P ⊗ n XY i =1 for some β > 0 . Hence, choosing the number of bins M n ≜ ⌊ 2 n (1+2 δ ) H ( X | Y ) ⌋ , we obtain E ( P e ( C n )) ⩽ 2 − αn for some appropriate choice of α > 0 . Finally, we have P ( P e ( C n ) > E ( P e ( C n ))) < 1 by Markov’s inequality, so that there exists at least one specific sequence of codes {C n } n ⩾ 1 of ( M n , n ) codes for which 1 n log M n ⩽ (1 + 2 δ ) H ( X | Y ) . Hence, (1 + 2 δ ) H ( X | Y ) is an achievable rate, and the result follows by setting ξ ≜ 2 δ . Since ξ > 0 may be chosen arbitrarily small, any rate R > H ( X | Y ) is ■

4 Tieorem 1.7 (Source coding with side information for a DMS) . For a DMS with two component It is also possible to establish a converse result. lim any achievable rate must satisfy and (1) lim sup are used uniformly at random, the corresponding output statistics are x characterized by Tie problem of channel output approximation (channel approximation for short) is illustrated in where the last inequality follows from Fano’s inequality. Consequently, Revised December 5, 2019 Information Theoretic Security Proposition 1.6 (Converse for source coding with side information) . For a DMS ( X × Y , P XY ) , R ⩾ H ( X | Y ) . Proof. Consider an ( M n , n ) code C n with probability of error ϵ n . Tien, log M n ⩾ H ( M ) ⩾ H ( M | Y ) ⩾ I ( M ; X | Y ) = H ( X | Y ) − H ( X | M Y ) ⩾ n H ( X | Y ) − 1 − ϵ n n log |X| , n log M n ⩾ H ( X | Y ) − 1 1 n − ϵ n log |X| . Taking the limit as n → ∞ and ϵ n → 0 yields 1 n log M n ⩾ H ( X | Y ) . ■ Combining Proposition 1.5 with Proposition 1.6, we obtain the following. ( X × Y , P XY ) , we have S ( P XY ) = H ( X | Y ) . 2 Channel output approximation Figure 2. Consider a source ( X , { P X n } n ⩾ 1 ) and a channel ( X , { W Z n | X n } n ⩾ 1 , Z ) . If the output of the source is used as the input to the channel, the output of the channel is another source ( Z , Q Z n ) ∑ ∀ z ∈ Z n Q Z n ( z ) = P X n ( x ) W Z n | X n ( z | x ) . Tie objective of channel approximation is to approximate the output statistics Q Z n using an ( M n , n ) channel code in place of the source ( X , P X n ) . If the codewords { x i } i ∈ � 1 ,M n � in the channel code M n W Z n | X n ( z | x m ) 1 ∀ z ∈ Z n ∑ P Z n ( z ) = . M n m =1 Any distance δ on the space of distributions over Z n is a potential metric to measure how well P Z n approximates Q Z n . We focus here specifically on D ( P Z n � Q Z n ) . Definition 2.1 (Achievable channel approximation rate and channel resolvability) . A rate R is an achievable approximation rate if there exists a sequence of ( M n , n ) channel codes {C n } n ⩾ 1 such that 1 n log M n ⩽ R n →∞ δ ( P Z n , Q Z n ) = 0 . n →∞ Tie infimum of all achievable rates is the channel resolvability C r ( { W Z n | X n } n ⩾ 1 , { P X n } n ⩾ 1 ) .