[PPT] - Applied Information Theory Daniel Bosk Department of Information PowerPoint Presentation

SLIDE 1

Introduction Shannon entropy Applications References

Applied Information Theory

Daniel Bosk

Department of Information and Communication Systems, Mid Sweden University, Sundsvall.

14th March 2019

1

SLIDE 2

Introduction Shannon entropy Applications References

1 Introduction

History

2 Shannon entropy

Definition of Shannon Entropy Properties for Shannon entropy Conditional entropy Information density and redundancy Information gain

3 Application in security

Passwords Research about human chosen passwords Identifying information

2

SLIDE 3

Introduction Shannon entropy Applications References

1 Introduction

History

2 Shannon entropy

Definition of Shannon Entropy Properties for Shannon entropy Conditional entropy Information density and redundancy Information gain

3 Application in security

Passwords Research about human chosen passwords Identifying information

3

SLIDE 4

Introduction Shannon entropy Applications References History

Created 1948 by Shannon’s paper ‘A Mathematical Theory of Communication’ [Sha48]. He starts using the term ‘entropy’ as a measure for information.

In physics entropy measures the disorder of molecules. Shannon’s entropy measures disorder of information.

He used this theory to analyse communication.

What are the theoretical limits for different channels? How much redundancy is needed for certain noise?

4

SLIDE 5

Introduction Shannon entropy Applications References History

Created 1948 by Shannon’s paper ‘A Mathematical Theory of Communication’ [Sha48]. He starts using the term ‘entropy’ as a measure for information.

In physics entropy measures the disorder of molecules. Shannon’s entropy measures disorder of information.

He used this theory to analyse communication.

What are the theoretical limits for different channels? How much redundancy is needed for certain noise?

4

SLIDE 6

Introduction Shannon entropy Applications References History

Created 1948 by Shannon’s paper ‘A Mathematical Theory of Communication’ [Sha48]. He starts using the term ‘entropy’ as a measure for information.

In physics entropy measures the disorder of molecules. Shannon’s entropy measures disorder of information.

He used this theory to analyse communication.

What are the theoretical limits for different channels? How much redundancy is needed for certain noise?

4

SLIDE 7

Introduction Shannon entropy Applications References History

This theory is interesting on the physical layer of networking. It’s also interesting for security.

Field of Information Theoretic Security ‘Efficiency’ of passwords Measure identifiability . . .

5

SLIDE 8

Introduction Shannon entropy Applications References History

This theory is interesting on the physical layer of networking. It’s also interesting for security.

Field of Information Theoretic Security ‘Efficiency’ of passwords Measure identifiability . . .

5

SLIDE 9

Introduction Shannon entropy Applications References

1 Introduction

History

2 Shannon entropy

Definition of Shannon Entropy Properties for Shannon entropy Conditional entropy Information density and redundancy Information gain

3 Application in security

Passwords Research about human chosen passwords Identifying information

6

SLIDE 10

Introduction Shannon entropy Applications References Definition of Shannon Entropy

Definition (Shannon entropy) Stochastic variable X assumes values from X. Shannon entropy H(X) defined as H(X) = −K

x∈X

Pr(X = x) log Pr(X = x), Usually K =

1 log 2 to give entropy in unit bits (bit).

7

SLIDE 11

Introduction Shannon entropy Applications References Definition of Shannon Entropy

Shannon entropy can be seen as . . . . . . how much choice in each event. . . . the uncertainty of each event. . . . how many bits to store each event. . . . how much information it produces.

8

SLIDE 12

Introduction Shannon entropy Applications References Definition of Shannon Entropy

Example (Toss a coin) Stochastic variable S takes values from S = {h, t}. We have Pr(S = h) = Pr(S = t) = 1

2.

This gives H(S) as follows: H(S) = − (Pr(S = h) log Pr(S = h) + Pr(S = t) log Pr(S = t)) = −2 × 1 2 log 1 2 = log 2 = 1.

9

SLIDE 13

Introduction Shannon entropy Applications References Definition of Shannon Entropy

Example (Roll a die) Stochastic variable D takes values from D = { q , q q, q

q q, q q q q, q q q q q, q q q q q q}.

We have Pr(D = d) = 1

6 for all d ∈ D.

The entropy H(D) is as follows: H(D) = −

d∈D

Pr(D = d) log Pr(D = d) = −6 × 1 6 log 1 6 = log 6 ≈ 2.585.

10

SLIDE 14

Introduction Shannon entropy Applications References Definition of Shannon Entropy

Remark If we didn’t know already, we now know that a roll of a die . . .

contains more ‘choice’ than a coin toss. is more uncertain to predict than a coin toss. requires more bits to store than a coin toss. produces more information than a coin toss.

What if we modify the die a bit?

11

SLIDE 15

Introduction Shannon entropy Applications References Definition of Shannon Entropy

Example (Roll of a modified die) Stochastic variable D′ taking values from D. We now have Pr(D′ = q q

q q q q) = 9

10 and Pr(D′ = d) = 1 10 × 1 5 for

d = q q

q q q q.

This yields H(D′) = −   9 10 log 9 10 +

d=6

1 50 log 1 50   = − 9 10 log 9 10 − 5 × 1 50 log 1 50 = − 9 10 log 9 10 − 1 10 log 1 50 ≈ 0.701. Note that the log function is the logarithm in base 2 (i.e. log2).

12

SLIDE 16

Introduction Shannon entropy Applications References Definition of Shannon Entropy

Remark This die is much easier to predict. It produces much less information — less than a coin toss! Requires less data for storage etc.

13

SLIDE 17

Introduction Shannon entropy Applications References Properties for Shannon entropy

Definition Function f : R → R such that tf (x) + (1 − t)f (y) ≤ f (tx + (1 − t)y), Then f is concave. With strict inequality for x = y we say that f is strictly concave. Example log: R → R is strictly concave.

14

SLIDE 18

Introduction Shannon entropy Applications References Properties for Shannon entropy

1 2 3 4 5 0.5 1 1.5 x log x

15

SLIDE 19

Introduction Shannon entropy Applications References Properties for Shannon entropy

Theorem (Jensen’s inequality) Strictly concave function f : R → R. Real numbers a1, a2, . . . , an > 0 such that n

i=1 ai = 1.

Then we have

n

i=1

aif (xi) ≤ f n

i=1

aixi

.

We have equality iff x1 = x2 = · · · = xn.

16

SLIDE 20

Introduction Shannon entropy Applications References Properties for Shannon entropy

Theorem Stochastic variable X with probability distribution p1, p2, . . . , pn, where pi > 0 for 1 ≤ i ≤ n. Then H(X) ≤ log n. Equality iff p1 = p2 = · · · = pn = 1/n.

17

SLIDE 21

Introduction Shannon entropy Applications References Properties for Shannon entropy

Proof. The theorem follows directly from Jensen’s inequality: H(X) = −

n

i=1

pi log pi =

n

i=1

pi log 1 pi ≤ log

n

i=1

pi 1 pi = log n. With equality iff p1 = p2 = · · · = pn. Q.E.D.

18

SLIDE 22

Introduction Shannon entropy Applications References Properties for Shannon entropy

Corollary H(X) = 0 iff Pr(X = x) = 1 for some x ∈ X and Pr(X = x′) = 0 for all x = x′ ∈ X. Proof. If Pr(X = x) = 1, then n = 1 and thus H(X) = log n = 0. If H(X) = 0, then H(X) ≤ log n = 0. Thus n = 1. Q.E.D.

19

SLIDE 23

Introduction Shannon entropy Applications References Properties for Shannon entropy

Lemma Stochastic variables X and Y. Then we have H(X, Y) ≤ H(X) + H(Y). Equality iff X and Y are independent.

20

SLIDE 24

Introduction Shannon entropy Applications References Conditional entropy

Definition (Conditional entropy) Define conditional entropy H(Y | X) as H(Y | X) = −

y
x

Pr(Y = y) Pr(X = x | y) log Pr(X = x | y). Remark This is the uncertainty in Y which is not revealed by X.

21

SLIDE 25

Introduction Shannon entropy Applications References Conditional entropy

Definition (Conditional entropy) Define conditional entropy H(Y | X) as H(Y | X) = −

y
x

Pr(Y = y) Pr(X = x | y) log Pr(X = x | y). Remark This is the uncertainty in Y which is not revealed by X.

21

SLIDE 26

Introduction Shannon entropy Applications References Conditional entropy

Theorem H(X, Y) = H(X) + H(Y | X) H(X) H(Y | X)

22

SLIDE 27

Introduction Shannon entropy Applications References Conditional entropy

Corollary H(X | Y) ≤ H(X). Corollary H(X | Y) = H(X) iff X and Y independent.

23

SLIDE 28

Introduction Shannon entropy Applications References Information density and redundancy

Definition Natural language L. Stochastic variable Pn

L of strings of length n.

(Alphabet PL.) Entropy of L defined as HL = lim

n→∞

H(Pn

L)

n . Redundancy in L is RL = 1 − HL log |PL|.

24

SLIDE 29

Introduction Shannon entropy Applications References Information density and redundancy

Remark Meaning we have HL bits per character in L. Example ([Sha48]) Entropy of 1–1.5 bits per character in English. Redundancy of approximately 1 − 1.25

log 26 ≈ 0.73.

25

SLIDE 30

Introduction Shannon entropy Applications References Information density and redundancy

Example ([Sha48]) Two-dimensional cross-word puzzles requires redundancy of approximately 0.5. Example Redundancy of ‘SMS languages’ is lower than for ‘non-SMS languages’. Compare ‘också’ and ‘oxå’. Remark Lower redundancy is more space-efficient. Incurs more errors.

26

SLIDE 31

Introduction Shannon entropy Applications References Information gain

Definition Set U of possible outcomes. Probability of outcome u ∈ U denoted pu. We learn that some unknown outcome is in A ⊂ U. Then the information gain G(A | U) is defined as G(A | U) = log 1 Pr(A) = − log Pr(A), where Pr(A) =

i∈A pi.

27

SLIDE 32

Introduction Shannon entropy Applications References Information gain

Example (Roll of dice again) Someone rolls and we should guess the result, 1

6 chance.

We learn that it was an even number, we gain − log 1 6 + 1 6 + 1 6

= − log 3

6 = log 6 3 = log 2 = 1. The remaining uncertainty is 1.58 bit. Remark X ′ = { q q, q q

q q, q q q q q q}

H(X′) = −

x∈X ′ Pr(X′ = x) log Pr(X′ = x)

I.e. −3 × 1

3 log 1 3 ≈ 1.58.

28

SLIDE 33

Introduction Shannon entropy Applications References Information gain

Example (Roll of dice again) Someone rolls and we should guess the result, 1

6 chance.

We learn that it was an even number, we gain − log 1 6 + 1 6 + 1 6

= − log 3

6 = log 6 3 = log 2 = 1. The remaining uncertainty is 1.58 bit. Remark X ′ = { q q, q q

q q, q q q q q q}

H(X′) = −

x∈X ′ Pr(X′ = x) log Pr(X′ = x)

I.e. −3 × 1

3 log 1 3 ≈ 1.58.

28

SLIDE 34

Introduction Shannon entropy Applications References Information gain

Example (Dice yet again) We learn the die show less than five, i.e. not q

q q q q nor q q q q q q.

This yields − log

4 × 1

6

= log 6

4 ≈ 0.58

29

SLIDE 35

Introduction Shannon entropy Applications References

1 Introduction

History

2 Shannon entropy

Definition of Shannon Entropy Properties for Shannon entropy Conditional entropy Information density and redundancy Information gain