ECEN 5022 Cryptography Introduction to Information Theory Peter - - PowerPoint PPT Presentation

Entropy Mutual Information Perfect Cryptosystem

ECEN 5022 Cryptography

Introduction to Information Theory Peter Mathys

University of Colorado

Spring 2008

Peter Mathys ECEN 5022 Cryptography

Entropy Mutual Information Perfect Cryptosystem Entropy Binary Entropy Function Conditional Entropy

Entropy

◮ Definition: The entropy H(X) of a discrete RV X with

alphabet A is defined as H(X) = −

• x∈A

pX(x) log pX(x) . H(X) is bounded as 0 ≤ H(X) ≤ log |A|. If log2 is used then the units of H(X) are bits.

◮ Definition: The joint entropy H(X, Y ) of discrete RVs X

and Y with alphabets A and B is defined as H(X, Y ) = −

• x∈A
• y∈B

pX,Y (x, y) log pX,Y (x, y) . Bounds: 0 ≤ H(X, Y ) ≤ H(X) + H(Y ) ≤ log |A| + log |B|.

Peter Mathys ECEN 5022 Cryptography

Entropy Mutual Information Perfect Cryptosystem Entropy Binary Entropy Function Conditional Entropy

Binary Entropy Function

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 p H(p) [bits] Binary Entropy Function H(p) = − p*log2p − (1−p)*log2(1−p)

◮ Definition: The binary entropy function is defined as

H(p) = −p log p − (1 − p) log(1 − p). If log2 is used the result is in bits.

Peter Mathys ECEN 5022 Cryptography

Entropy Mutual Information Perfect Cryptosystem Entropy Binary Entropy Function Conditional Entropy

Conditional Entropy

◮ Definition: The conditional entropy H(Y |X) of discrete RV

Y given discrete RV X (with alphabets B and A) is defined as H(Y |X) =

x∈A pX(x) H(Y |X = x)

= −

x∈A pX(x) y∈B pY |X(y|x) log pY |X(y|x)

= −

x∈A

• y∈B pX,Y (x, y) log pY |X(y|x)

Bounds: 0 ≤ H(Y |X) ≤ H(Y ) ≤ log |B|.

◮ H(Y |X) = H(Y ) (and H(X|Y ) = H(X)) iff X and Y are

statistically independent.

◮ Chain rule: H(X, Y ) = H(X) + H(Y |X) = H(Y ) + H(X|Y ).

This follows from Bayes rule.

Peter Mathys ECEN 5022 Cryptography

Entropy Mutual Information Perfect Cryptosystem Mutual Information Conditional Mutual Information

Mutual Information

◮ Definition: The mutual information I(X; Y ) between two

discrete RVs X and Y with alphabets A and B is defined as I(X; Y ) =

• x∈A
• y∈B

pX,Y (x, y) log pX,Y (x, y) pX(x) pY (y) = H(X) − H(X|Y ) = H(Y ) − H(Y |X) = H(X) + H(Y ) − H(X, Y ) Bounds: 0 ≤ I(X; Y ) ≤ max{log |A|, log |B|}.

Peter Mathys ECEN 5022 Cryptography

Entropy Mutual Information Perfect Cryptosystem Mutual Information Conditional Mutual Information

Conditional Mutual Information

◮ Definition: The conditional mutual information I(X; Y |Z)

between two discrete RVs X and Y with alphabets A and B, given discrete RV Z with alphabet C is defined as I(X; Y |Z) =

• x∈A
• y∈B
• z∈C

pX,Y ,Z(x, y, z) log pX,Y |Z(x, y|z) pX|Z(x|z) pY |Z(y|z) = H(X|Z) − H(X|YZ) = H(Y |Z) − H(Y |XZ) = H(X|Z) + H(Y |Z) − H(X, Y |Z)

Peter Mathys ECEN 5022 Cryptography

Entropy Mutual Information Perfect Cryptosystem Perfect Cryptosystem Entropy of Language Unicity Distance

Perfect Cryptosystem

E(K, M) Key K M C ◮ A perfect cryptosystem should have the following properties

I(M; C) = 0 , I(M; C|K) = H(M) .

◮ From the first statement I(M; C) = H(M) − H(M|C) = 0

and thus H(M|C) = H(M), i.e., M, C must be statistically independent.

◮ From the second statement

I(M; C|K) = H(M|K) − H(M|CK) = H(M|K) = H(M) (2’nd equality follows since H(M|CK) = 0) and thus M and K must be statistically independent.

Peter Mathys ECEN 5022 Cryptography

Entropy Mutual Information Perfect Cryptosystem Perfect Cryptosystem Entropy of Language Unicity Distance

Key Equivocation

◮ Let the discrete RVs M, K, C denote the plaintext, key,

ciphertext, respectively, of a cryptosystem.

◮ Definition. H(K|C) is called the key equivocation. ◮ Theorem. If M, K are statistically independent then

H(K|C) = H(K) + H(M) − H(C).

◮ Proof: Express H(M, K, C) in two ways as

H(M, K, C) = H(M, K) + H(C|M, K)

• =0

= H(M, K) = H(M) + H(K|M)

• =H(K)

= H(M) + H(K) , and as H(M, K, C) = H(K, C) + H(M|K, C)

• =0

= H(K, C). = ⇒ H(K|C) = H(K, C) − H(C) = H(M, K, C) − H(C) = H(M) + H(K) − H(C) .

Peter Mathys ECEN 5022 Cryptography

Entropy Mutual Information Perfect Cryptosystem Perfect Cryptosystem Entropy of Language Unicity Distance

Entropy of Language

◮ Plaintext RV: Mn = (M1, M2, . . . , Mn). ◮ Ciphertext RV: C n = (C1, C2, . . . , Cn). ◮ Question: How much ciphertext is needed to dermine key

uniquely?

◮ Definition: The rate or entropy of a language L is defined as

HL = lim

n→∞

H(Mn) n . For English 1.0 ≤ HL ≤ 1.5.

◮ Definition: The redundancy of a language L with alphabet M

is defined as RL = 1 − HL log |M| . English has redundancy ≈ 0.75.

Peter Mathys ECEN 5022 Cryptography

Entropy Mutual Information Perfect Cryptosystem Perfect Cryptosystem Entropy of Language Unicity Distance

Spurious Keys

◮ Definition. The set of possible keys k for encryption function

Ek given some ciphertext c ∈ Cn of length n is K(c) =

• k ∈ K
• ∃ m ∈ Mn with Pr(m) > 0 and c = Ek(m)
• “Set of keys that yield plausible plaintexts given ciphertext c.”

◮ Definition. Only one member of K(c) is correct. The rest are

called spurious keys and denoted Sn.

◮ The expected number of spurious keys, averaged over all

ciphertexts c ∈ Cn of length n, is E[Sn] =

• c∈Cn
• |K(c)| − 1
• Pr(c) .

Peter Mathys ECEN 5022 Cryptography

Entropy Mutual Information Perfect Cryptosystem Perfect Cryptosystem Entropy of Language Unicity Distance

Unicity Distance

◮ Theorem. If |C| = |M| and keys are chosen equiprobably from

the keyspace K then, given a ciphertext of sufficiently large length n, the expected number of spurious keys satisfies E[Sn] ≥ |K| |M|nRL − 1 , where |K| is the size of the keyspace, |M| is the size of the plaintext alphabet, and RL is the redundancy of the plaintext language.

◮ Setting E[Sn] = 0 yields

nRL log |M| ≥ log |K| and thus n ≥ n0 = log |K| RL log |M| . The quantity n0 is called the unicity distance of a cryptosystem.

Peter Mathys ECEN 5022 Cryptography

Entropy Mutual Information Perfect Cryptosystem Perfect Cryptosystem Entropy of Language Unicity Distance

Examples

◮ For all examples |M| = 26 =

⇒ log2 |M| = 4.70 and RL = 0.75.

◮ Shift (Caesar) cipher. |K| = 26 =

⇒ log2 |K| = 4.70 Unicity distance: n0 = 4.70 0.75 × 4.70 = 1.333 .

◮ Simple substitution cipher. |K| = 26! =

⇒ log2 |K| = 88.38 Unicity distance: n0 = 88.38 0.75 × 4.70 = 25.07 .

◮ One-time pad. |K| = 26n =

⇒ log2 |K| = 4.70 n Unicity distance: n0 = 4.70 n 0.75 × 4.70 = 1.333 n . The only solution to n ≥ n0 = 1.333 n is n = 0.

Peter Mathys ECEN 5022 Cryptography