entropy and shannon s theorem
play

Entropy and Shannons Theorem Lecture 28 December 10, 2013 Sariel - PowerPoint PPT Presentation

Apr 12, 2023 •332 likes •1.28k views

CS 573: Algorithms, Fall 2013 Entropy and Shannons Theorem Lecture 28 December 10, 2013 Sariel (UIUC) CS573 1 Fall 2013 1 / 24 Part I . Entropy . Sariel (UIUC) CS573 2 Fall 2013 2 / 24 Part II . Extracting randomness .

  1. CS 573: Algorithms, Fall 2013 Entropy and Shannon’s Theorem Lecture 28 December 10, 2013 Sariel (UIUC) CS573 1 Fall 2013 1 / 24

  2. Part I . Entropy . Sariel (UIUC) CS573 2 Fall 2013 2 / 24

  3. Part II . Extracting randomness . Sariel (UIUC) CS573 3 Fall 2013 3 / 24

  4. Storing all strings of length n and j bits on . . S n , j : set of all strings of length n with j ones in them. 1 . . T n , j : prefix tree storing all T n , j . 2 T 0 , 0 T 1 , 0 T 1 , 1 Sariel (UIUC) CS573 4 Fall 2013 4 / 24

  5. Prefix tree ∀ binary strings of length n with j ones T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 T n,n T n, 0 0 1 T n − 1 , 0 T n − 1 ,n − 1 Sariel (UIUC) CS573 5 Fall 2013 5 / 24

  6. Prefix tree ∀ binary strings of length n with j ones T n,j # of leafs: 0 1 | T n , j | = | T n − 1 , j | + | T n − 1 , j − 1 | T n − 1 ,j T n − 1 ,j − 1 T n,n T n, 0 0 1 T n − 1 , 0 T n − 1 ,n − 1 Sariel (UIUC) CS573 5 Fall 2013 5 / 24

  7. Prefix tree ∀ binary strings of length n with j ones T n,j # of leafs: | T n , j | = | T n − 1 , j | + | T n − 1 , j − 1 | 0 1 ( n ) ( n − 1 ) ( n − 1 ) = + j j j − 1 T n − 1 ,j T n − 1 ,j − 1 T n,n T n, 0 0 1 T n − 1 , 0 T n − 1 ,n − 1 Sariel (UIUC) CS573 5 Fall 2013 5 / 24

  8. Prefix tree ∀ binary strings of length n with j ones T n,j # of leafs: | T n , j | = | T n − 1 , j | + | T n − 1 , j − 1 | 0 1 ( n ) ( n − 1 ) ( n − 1 ) = + j − 1 j j ( n ) = ⇒ | T n , j | = . T n − 1 ,j T n − 1 ,j − 1 j T n,n T n, 0 0 1 T n − 1 , 0 T n − 1 ,n − 1 Sariel (UIUC) CS573 5 Fall 2013 5 / 24

  9. Encoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 . . Input: s ∈ S n , j : compute index of s in sorted set S n , j . 4 . . EncodeBinomCoeff ( s ) denote this polytime procedure. 5 Sariel (UIUC) CS573 6 Fall 2013 6 / 24

  10. Encoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 . . Input: s ∈ S n , j : compute index of s in sorted set S n , j . 4 . . EncodeBinomCoeff ( s ) denote this polytime procedure. 5 Sariel (UIUC) CS573 6 Fall 2013 6 / 24

  11. Encoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 . Input: s ∈ S n , j : compute index of s in sorted set S n , j . 4 . . EncodeBinomCoeff ( s ) denote this polytime procedure. 5 Sariel (UIUC) CS573 6 Fall 2013 6 / 24

  12. Encoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 . Input: s ∈ S n , j : compute index of s in sorted set S n , j . 4 . . EncodeBinomCoeff ( s ) denote this polytime procedure. 5 Sariel (UIUC) CS573 6 Fall 2013 6 / 24

  13. Encoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 . Input: s ∈ S n , j : compute index of s in sorted set S n , j . 4 . . EncodeBinomCoeff ( s ) denote this polytime procedure. 5 Sariel (UIUC) CS573 6 Fall 2013 6 / 24

  14. Decoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 . . { ( n )} x ∈ 1 , . . . , : compute x th string in S n , j in polytime. 4 j . . DecodeBinomCoeff ( x ) denote this procedure. 5 Sariel (UIUC) CS573 7 Fall 2013 7 / 24

  15. Decoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 . . { ( n )} x ∈ 1 , . . . , : compute x th string in S n , j in polytime. 4 j . . DecodeBinomCoeff ( x ) denote this procedure. 5 Sariel (UIUC) CS573 7 Fall 2013 7 / 24

  16. Decoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 . { ( n )} x ∈ 1 , . . . , : compute x th string in S n , j in polytime. 4 j . . DecodeBinomCoeff ( x ) denote this procedure. 5 Sariel (UIUC) CS573 7 Fall 2013 7 / 24

  17. Decoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 . { ( n )} x ∈ 1 , . . . , : compute x th string in S n , j in polytime. 4 j . . DecodeBinomCoeff ( x ) denote this procedure. 5 Sariel (UIUC) CS573 7 Fall 2013 7 / 24

  18. Decoding a string in S n , j . . T n , j leafs corresponds to strings of S n , j . 1 . . Order all strings of S n , j order in lexicographical ordering 2 . . ≡ ordering leafs of T n , j from left to right. 3 T n,j 0 1 T n − 1 ,j T n − 1 ,j − 1 . { ( n )} x ∈ 1 , . . . , : compute x th string in S n , j in polytime. 4 j . . DecodeBinomCoeff ( x ) denote this procedure. 5 Sariel (UIUC) CS573 7 Fall 2013 7 / 24

  19. Encoding/decoding strings of S n , j . Lemma . S n , j : Set of binary strings of length n with j ones, sorted lexicographically. . . EncodeBinomCoeff ( α ): Input is string α ∈ S n , j , compute 1 index x of α in S n , j in polynomial time in n . . . { ( n )} DecodeBinomCoeff ( x ): Input index x ∈ 1 , . . . , . 2 j Output x th string α in S n , j , in time O (polylog n + n ) . . Sariel (UIUC) CS573 8 Fall 2013 8 / 24

  20. Extracting randomness . Theorem . Consider a coin that comes up heads with probability p > 1 / 2 . For any constant δ > 0 and for n sufficiently large: (A) One can extract, from an input of a sequence of n flips, an output sequence of (1 − δ ) n H ( p ) (unbiased) independent random bits. (B) One can not extract more than n H ( p ) bits from such a sequence. . Sariel (UIUC) CS573 9 Fall 2013 9 / 24

  21. Proof... . . ( n ) There are input strings with exactly j heads. 1 j . . each has probability p j (1 − p ) n − j . 2 . . map string s like that to index number in the set 3 { ( n )} S j = 1 , . . . , . j . . Given that input string s has j ones (out of n bits) defines a 4 uniform distribution on S n , j . . . x ← EncodeBinomCoeff ( s ) 5 . . ( n ) x uniform distributed in { 1 , . . . , N } , N = . 6 j . . Seen in previous lecture... 7 . . ... extract in expectation, ⌊ lg N ⌋ − 1 bits from uniform 8 random variable in the range 1 , . . . , N . . . Extract bits using ExtractRandomness ( x , N ):. 9 Sariel (UIUC) CS573 10 Fall 2013 10 / 24

  22. Proof... . . ( n ) There are input strings with exactly j heads. 1 j . . each has probability p j (1 − p ) n − j . 2 . . map string s like that to index number in the set 3 { ( n )} S j = 1 , . . . , . j . . Given that input string s has j ones (out of n bits) defines a 4 uniform distribution on S n , j . . . x ← EncodeBinomCoeff ( s ) 5 . . ( n ) x uniform distributed in { 1 , . . . , N } , N = . 6 j . . Seen in previous lecture... 7 . . ... extract in expectation, ⌊ lg N ⌋ − 1 bits from uniform 8 random variable in the range 1 , . . . , N . . . Extract bits using ExtractRandomness ( x , N ):. 9 Sariel (UIUC) CS573 10 Fall 2013 10 / 24

  23. Proof... . . ( n ) There are input strings with exactly j heads. 1 j . . each has probability p j (1 − p ) n − j . 2 . . map string s like that to index number in the set 3 { ( n )} S j = 1 , . . . , . j . . Given that input string s has j ones (out of n bits) defines a 4 uniform distribution on S n , j . . . x ← EncodeBinomCoeff ( s ) 5 . . ( n ) x uniform distributed in { 1 , . . . , N } , N = . 6 j . . Seen in previous lecture... 7 . . ... extract in expectation, ⌊ lg N ⌋ − 1 bits from uniform 8 random variable in the range 1 , . . . , N . . . Extract bits using ExtractRandomness ( x , N ):. 9 Sariel (UIUC) CS573 10 Fall 2013 10 / 24

  24. Proof... . . ( n ) There are input strings with exactly j heads. 1 j . . each has probability p j (1 − p ) n − j . 2 . . map string s like that to index number in the set 3 { ( n )} S j = 1 , . . . , . j . . Given that input string s has j ones (out of n bits) defines a 4 uniform distribution on S n , j . . . x ← EncodeBinomCoeff ( s ) 5 . . ( n ) x uniform distributed in { 1 , . . . , N } , N = . 6 j . . Seen in previous lecture... 7 . . ... extract in expectation, ⌊ lg N ⌋ − 1 bits from uniform 8 random variable in the range 1 , . . . , N . . . Extract bits using ExtractRandomness ( x , N ):. 9 Sariel (UIUC) CS573 10 Fall 2013 10 / 24

Download Presentation
Download Policy: The content available on the website is offered to you 'AS IS' for your personal information and use only. It cannot be commercialized, licensed, or distributed on other websites without prior consent from the author. To download a presentation, simply click this link. If you encounter any difficulties during the download process, it's possible that the publisher has removed the file from their server.

Recommend

Entropy and Shannon information Entropy and Shannon information For a random variable X with

Entropy and Shannon information Entropy and Shannon information For a random variable X with

Entropy and Shannon information Entropy and Shannon information For a random variable X with distribution p(x), the entropy is H[ X ] = - S x p( x ) log 2 p( x ) Information is defined as I[ X ] = - log 2 p( x ) Mutual information Typically,

695 views • 47 slides
ARE THE SHANNON ENTROPY AND RESIDUAL ENTROPY SYNONYMS? H = R 0 ? MARKO POPOVIC DEPARTMENT OF

ARE THE SHANNON ENTROPY AND RESIDUAL ENTROPY SYNONYMS? H = R 0 ? MARKO POPOVIC DEPARTMENT OF

ARE THE SHANNON ENTROPY AND RESIDUAL ENTROPY SYNONYMS? H = R 0 ? MARKO POPOVIC DEPARTMENT OF CHEMISTRY AND BIOCHEMISTRY, BYU, PROVO, UT 84602, POPOVIC.PASA@GMAIL.COM Thermodynamic entropy - S (J/K) At T=0K ideal crystal imperfect

560 views • 21 slides
Learning Outcomes I can implement logic for any truth table by using Shannon's theorem to

Learning Outcomes I can implement logic for any truth table by using Shannon's theorem to

2-4.1 2-4.2 Learning Outcomes I can implement logic for any truth table by using Shannon's theorem to decompose the function to create two smaller functions and a 2-to-1 mux Spiral 2-4 I can recursively apply Shannon's theorem k times

409 views • 8 slides
Asymptotic behaviour of the weighted Shannon differential entropy in a Bayesian problem Mark

Asymptotic behaviour of the weighted Shannon differential entropy in a Bayesian problem Mark

Asymptotic behaviour of the weighted Shannon differential entropy in a Bayesian problem Mark Kelbert, Pavel Mozgunov 2nd International Electronic Conference on Entropy and Its Applications November 2015 Introduction Let U U [0 , 1]. Given a

428 views • 22 slides
Spiral 2-4 Function synthesis with: Muxes (Shannon's Theorem) Memories 2-4.2 Learning Outcomes

Spiral 2-4 Function synthesis with: Muxes (Shannon's Theorem) Memories 2-4.2 Learning Outcomes

2-4.1 Spiral 2-4 Function synthesis with: Muxes (Shannon's Theorem) Memories 2-4.2 Learning Outcomes I can implement logic for any truth table by using Shannon's theorem to decompose the function to create two smaller functions and a

455 views • 33 slides
Topological entropy and algebraic entropy on locally compact abelian groups - The Bridge Theorem

Topological entropy and algebraic entropy on locally compact abelian groups - The Bridge Theorem

Topological entropy and algebraic entropy on locally compact abelian groups - The Bridge Theorem - Topological entropy and algebraic entropy on locally compact abelian groups - The Bridge Theorem - Anna Giordano Bruno - University of Udine

212 views • 17 slides
Quantum Lecture 6 Shannon information Quantum information Distance measures Mikael

Quantum Lecture 6 Shannon information Quantum information Distance measures Mikael

Quantum Lecture 6 Shannon information Quantum information Distance measures Mikael Skoglund, Quantum Info 1/16 Shannon Entropy and Information The Shannon entropy for a discrete variable X with alphabet X and pmf p ( x ) = Pr( X = x

440 views • 8 slides
H ( C ) = (1 . 0 log 2 1 . 0 + 0 . 0 log 2 0 . 0) = 0 . 0 4 Wednesday, 11 Sep. 2019 Machine

H ( C ) = (1 . 0 log 2 1 . 0 + 0 . 0 log 2 0 . 0) = 0 . 0 4 Wednesday, 11 Sep. 2019 Machine

Review: Entropy } Shannon defined the entropy of a probability distribution as the average amount of information carried by events: P = { p 1 , p 2 , . . . , p k } 1 X X H ( P ) = p i log 2 = p i log 2 p i p i i i } This can be thought of

100 views • 8 slides
Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of

Entropy, Relative Entropy, Cross Entropy Entropy Entropy, H(x) is a measure of the uncertainty of a discrete random variable. Properties: H(x) >= 0 Entropy Entropy Lesser the probability for an event, larger the entropy.

471 views • 21 slides
Dynamics and algebraic integers: Perspectives on Thurstons last theorem is a weak Perron

Dynamics and algebraic integers: Perspectives on Thurstons last theorem is a weak Perron

Theorem ( Thurston, Entropy in dimension one, 2014 ) There exists a post-critically finite map f : I I with entropy h(f) = log( ) Dynamics and algebraic integers: Perspectives on Thurstons last theorem is a weak Perron number. 2

426 views • 8 slides
Shannon entropy as leitmotiv for string model building Sven Krippendorf Workshop on Big Data

Shannon entropy as leitmotiv for string model building Sven Krippendorf Workshop on Big Data

Shannon entropy as leitmotiv for string model building Sven Krippendorf Workshop on Big Data in String Theory Boston, 02.12.2017 The role of naturalness in the sense of aesthetic beauty is a powerful guiding principle as they

856 views • 37 slides
Welding Torch Problem Nyquist-Shannon Sampling Theorem Welding Torch Problem Model Solution

Welding Torch Problem Nyquist-Shannon Sampling Theorem Welding Torch Problem Model Solution

Welding Torch Problem Nyquist-Shannon Sampling Theorem Welding Torch Problem Model Solution Example 1 Example 2 Nyquist-Shannon Sampling Theorem Statement Whittaker-Shannon Interpolation Formula Welding Torch Problem Consider a long

175 views • 6 slides
Welding Torch Problem Nyquist-Shannon Sampling Theorem Welding Torch Problem Model Solution

Welding Torch Problem Nyquist-Shannon Sampling Theorem Welding Torch Problem Model Solution

Welding Torch Problem Nyquist-Shannon Sampling Theorem Welding Torch Problem Model Solution Example 1 Example 2 Nyquist-Shannon Sampling Theorem Statement Whittaker-Shannon Interpolation Formula Welding Torch Problem Consider a long

228 views • 7 slides
Exploring Idiomaticity with Variant-based Distributional Measures and Shannon Entropy Marco S. G.

Exploring Idiomaticity with Variant-based Distributional Measures and Shannon Entropy Marco S. G.

Exploring Idiomaticity with Variant-based Distributional Measures and Shannon Entropy Marco S. G. Senaldi 1 Gianluca E. Lebani 2 Alessandro Lenci 2 1 Scuola Normale Superiore, Pisa 2 University of Pisa DGfS 2017 Saarbrcken | 9 th March 2017

736 views • 38 slides
Chapter 28 Entropy and Shannons Theorem CS 573: Algorithms, Fall 2013 December 10, 2013 28.1

Chapter 28 Entropy and Shannons Theorem CS 573: Algorithms, Fall 2013 December 10, 2013 28.1

Chapter 28 Entropy and Shannons Theorem CS 573: Algorithms, Fall 2013 December 10, 2013 28.1 Entropy 28.2 Extracting randomness 28.2.1 Enumerating binary strings with j ones 28.2.1.1 Storing all strings of length n and j bits on (A) S

180 views • 6 slides
INPUTS We made MNIST images

INPUTS We made MNIST images

INPUTS We made MNIST images binary (white or black) and computed shannon entropy for Max Entropy each pixel Entropy is close to maximum (50% white) for most pixels in

434 views • 26 slides
Lecture 4 Hidden Markov Models Michael Picheny, Bhuvana Ramabhadran, Stanley F . Chen, Markus

Lecture 4 Hidden Markov Models Michael Picheny, Bhuvana Ramabhadran, Stanley F . Chen, Markus

Lecture 4 Hidden Markov Models Michael Picheny, Bhuvana Ramabhadran, Stanley F . Chen, Markus Nussbaum-Thom Watson Group IBM T.J. Watson Research Center Yorktown Heights, New York, USA {picheny,bhuvana,stanchen,nussbaum}@us.ibm.com 10

1.79k views • 157 slides
Lecture 8 Barna Saha AT&T-Labs Research October 3, 2013 Outline Clustering K-Center

Lecture 8 Barna Saha AT&T-Labs Research October 3, 2013 Outline Clustering K-Center

Lecture 8 Barna Saha AT&T-Labs Research October 3, 2013 Outline Clustering K-Center K-Center Given a set of distinct points P = { p 1 , p 2 , . . . , p n } find a set of k points Q P , | Q | = k , that minimizes max min q Q

514 views • 9 slides
CSE101: Design and Analysis of Algorithms Ragesh Jaiswal, CSE, UCSD Ragesh Jaiswal, CSE, UCSD

CSE101: Design and Analysis of Algorithms Ragesh Jaiswal, CSE, UCSD Ragesh Jaiswal, CSE, UCSD

CSE101: Design and Analysis of Algorithms Ragesh Jaiswal, CSE, UCSD Ragesh Jaiswal, CSE, UCSD CSE101: Design and Analysis of Algorithms Administrative Information Ragesh Jaiswal, CSE, UCSD CSE101: Design and Analysis of Algorithms

641 views • 25 slides
1 Choose your language 4 Exercise sessions (bungsgruppen) are available in German (5) and

1 Choose your language 4 Exercise sessions (bungsgruppen) are available in German (5) and

1 Introduction to Programming Bertrand Meyer ETH Zurich, October 2005 February 2006 Chair of Softw are Engineering I ntro Lecture 1 German version of lecture slides 2 Folien fr diese und alle weiteren Vorlesungseinheiten werden von

869 views • 22 slides
CS101 Lecture 01: Introduction Aaron Stevens (azs@bu.edu) 16 January 2013 Computer Science What

CS101 Lecture 01: Introduction Aaron Stevens (azs@bu.edu) 16 January 2013 Computer Science What

1/16/13 CS101 Lecture 01: Introduction Aaron Stevens (azs@bu.edu) 16 January 2013 Computer Science What Youll Learn Today Computer Science What is computer science? What are data and information? What is a computer? What are

278 views • 15 slides
Faculty-Peer Partnerships for Teaching Feedback 2) Share what happened in my department 3)

Faculty-Peer Partnerships for Teaching Feedback 2) Share what happened in my department 3)

Goal of this presentation 1) Start conversations about peer-review of teaching Faculty-Peer Partnerships for Teaching Feedback 2) Share what happened in my department 3) Brainstorm research directions for teaching reviews Nancy Aguilar-Roca

170 views • 5 slides
Challenge and Solutions for { Peta | Exa }-scale Programming WPSE09 panel discussion Raymond

Challenge and Solutions for { Peta | Exa }-scale Programming WPSE09 panel discussion Raymond

Challenge and Solutions for { Peta | Exa }-scale Programming WPSE09 panel discussion Raymond Namyst Runtime group INRIA Bordeaux Research Center University of Bordeaux 1 France Legal Disclaimer This presentation represents EXCLUSIVELY

48 views • 4 slides
Disclaimer As we continue to monitor the developments surrounding Coronavirus (COVID-19), this

Disclaimer As we continue to monitor the developments surrounding Coronavirus (COVID-19), this

Texas Apartment Association, Inc. Title slide design to come FAQs on COVID-19 and the Texas Rental Housing Industry March 27, 2020 1 Disclaimer As we continue to monitor the developments surrounding Coronavirus (COVID-19), this seminar

259 views • 5 slides

More recommend