Lecture 0 Introduction I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw September 13, 2016 1 / 50 I-Hsiang Wang IT Lecture 0
What is Information Theory about? It is a mathematical theory of information . Information is usually obtained by getting some "messages" (speech, text, images, etc.) from others. When obtaining information from a message, you may care about: What is the meaning of a message? How important is the message? How much information can I get from the message? 2 / 50 I-Hsiang Wang IT Lecture 0
What is Information Theory about? It is a mathematical theory of information . Information is usually obtained by getting some "messages" (speech, text, images, etc.) from others. When obtaining information from a message, you may care about: What is the meaning of a message? How important is the message? How much information can I get from the message? Information theory is about the quantification of information. 3 / 50 I-Hsiang Wang IT Lecture 0
What is Information Theory about? It is a mathematical theory of information (primarily for communication systems) Establishes the fundamental limits of various types of information processing, including storage, compression, transmission, exchange, etc. Built upon probability theory and statistical decision theory Main Focus : ultimate performance limit (usually the efficiency of information processing) as certain resources (usually the total amount of time) scales to an asymptotic regime, given that the desired task is accomplished at a certain "satisfactory" level. 4 / 50 I-Hsiang Wang IT Lecture 0
In this course, we will 1 Establish solid foundations and intuitions of information theory, 2 Introduce explicit methods to achieve information theoretic limits, 3 Demonstrate further applications of information theory beyond communications. Later, we begin with a brief overview of information theory and the materials to be covered in this course. 5 / 50 I-Hsiang Wang IT Lecture 0
Course Information Course Information 1 Course Overview 2 General Overview of Information Theory First Part of this Course: Shannon Theory Second Part of this Course: Information Theory and Statistics 6 / 50 I-Hsiang Wang IT Lecture 0
Course Information Logistics 1 Instructor: I-Hsiang Wang ⺩奕翔 Email: ihwang@ntu.edu.tw Website: http://cc.ee.ntu.edu.tw/~ihsiangw/ Office: MD-524 明達館 524 室 Office Hours: 17:00 – 18:00, Monday and Tuesday 2 Lecture Time: 09:10 – 12:10 (234) Wednesday 3 Lecture Location: BL-114 博理館 114 室 4 Course Website: http://homepage.ntu.edu.tw/~ihwang/Teaching/Fa16/IT.html 5 Prerequisites: Probability, Linear Algebra. 7 / 50 I-Hsiang Wang IT Lecture 0
Course Information Logistics 6 Grading: Homework (30%), Midterm (30%), Final (40%) 7 References T. Cover and J. Thomas, Elements of Information Theory, 2nd Edition, Wiley-Interscience, 2006. R. Gallager, Information Theory and Reliable Communications, Wiley, 1968. I. Csiszar and J. Korner, Information Theory: Coding Theorems for Discrete Memoryless Systems, 2nd Edition, Cambridge University Press, 2011. S. M. Moser, Information Theory (Lecture Notes), 4th edition, ISI Lab, ETH Zürich, Switzerland, 2014. R. Yeung, Information Theory and Network Coding, Springer, 2008. A. El Gamal and Y.-H. Kim, Network Information Theory, Cambridge University Press, 2011. Y. Polyanskiy and Y. Wu, Lecture notes on Information Theory, MIT (6.441), UIUC (ECE 563), 2012-2016. 8 / 50 I-Hsiang Wang IT Lecture 0
Course Information Homework 1 Roughly 4 – 5 problems every 2 – 3 weeks, in total 6 times. 2 Homework (HW) is usually released on Monday. Deadline of submission is usually on the next Wednesday in class. 3 Late homework = 0 points. (Let me know in advance if you have difficulties.) 4 Everyone has to develop detailed solution for one HW problem, documented in L A T EX and submitted 1 week after the HW due. (L A T EX template will be provided) You should discuss with the instructor and TA about the homework problem that you are in charge of, making sure the solution is correct. 5 This additional effort accounts for part of your homework grades. 9 / 50 I-Hsiang Wang IT Lecture 0
Course Information Reading and Lecture Notes 1 Slides are usually released/updated every Sunday evening. 2 Each lecture has assigned readings. Reading is required: it is not enough to learn from the slides! 3 Go through the slides and the assigned readings before our lectures. It helps you learn better. 4 I recommend you get a copy of the textbook by Cover and Thomas. It is a good reference, and we will often assign readings and exercises in the book. 5 Other assigned readings could be Moser's lecture note, Polyanskiy-Wu lecture note, and relevant papers (all can be obtained online) . 10 / 50 I-Hsiang Wang IT Lecture 0
Course Information Interaction 1 In-class: Language : This class is taught in English. However, to encourage interaction, feel free to ask questions in Mandarin. I will repeat your question in English (if necessary), and answer it in English. Exercises : We put some exercises on the slides to help you learn and understand. Occasionally, I will call for volunteer to solve the exercises in class. Volunteers get bonus. 2 Out-of-class: Office Hours : Both TA and myself have 2-hour office hours per week. You are more than welcome to come visit us and ask questions, discuss about research, chat, complain, etc. If you cannot make it to the regular office hours, send us emails to schedule a time slot. My schedule can be found on my website. Send us emails with a subject starting with "[NTU Fall16 IT]". Feedback : There will be online polls during the semester to collect your feedback anonymously. 11 / 50 I-Hsiang Wang IT Lecture 0
Course Information Course Outline Measures of Information : entropy, conditional entropy, mutual information, differential entropy, information divergence. Lossless Source Coding : lossless source coding theorem, discrete memoryless sources, asymptotic equipartition property, typical sequences, Fano's inequality, converse proof, ergodic sources, entropy rate. Noisy Channel Coding : noisy channel coding theorem, discrete memoryless channels, random coding, typicality decoder, threshold decoder, error probability analysis, converse proof, channel with feedback, channel coding with cost constraints, Gaussian channel capacity. Lossy Source Coding (Rate Distortion Theory) : distortion, rate-distortion tradeoff, typicality encoder, converse proof. Capacity Achieving Code : polar coding. 12 / 50 I-Hsiang Wang IT Lecture 0
Course Information Course Outline Statistical Decision Theory : hypothesis testing, estimation, minimax risk, Bayes risk. Information Theory and Statistics : method of types, Sanov's theorem, large deviation, large sample asymptotics, Cramér-Rao lower bound, high-dimensional estimation problems. Advanced Topics (if time allowed): compressed sensing, community detection, non-asymptotic information theory, etc. 13 / 50 I-Hsiang Wang IT Lecture 0
Course Information Tentative Schedule Week Date Content Remark 1 09/14 Introduction; Measures of Information 2 09/21 Measures of Information HW1 out 3 09/28 No Lecture (I-Hsiang out of town) 4 10/05 Lossless Source Coding HW1 due 5 10/12 Lossless Source Coding HW2 out 6 10/19 Noisy Channel Coding 7 10/26 Noisy Channel Coding HW2 due; HW3 out 8 11/02 Lossy Source Coding 9 11/09 Lossy Source Coding HW3 due; HW4 out 14 / 50 I-Hsiang Wang IT Lecture 0
Course Information Tentative Schedule Week Date Content Remark 10 11/16 Polar Coding 11 11/23 Midterm Exam HW4 due 12 11/30 Statistical Decision Theory 13 12/07 Statistical Decision Theory HW5 out 14 12/14 Information Theory and Statistics 15 12/21 Information Theory and Statistics HW5 due; HW6 out 16 12/28 Information Theory and Statistics 17 01/04 Advanced Topics HW6 due 18 01/11 Final Exam 15 / 50 I-Hsiang Wang IT Lecture 0
Course Overview Course Information 1 Course Overview 2 General Overview of Information Theory First Part of this Course: Shannon Theory Second Part of this Course: Information Theory and Statistics 16 / 50 I-Hsiang Wang IT Lecture 0
Course Overview Claude E. Shannon (1916 – 2001) 17 / 50 I-Hsiang Wang IT Lecture 0
Course Overview Information theory – a mathematical theory of communication 18 / 50 I-Hsiang Wang IT Lecture 0
Course Overview Information theory – the mathematical theory of communication 19 / 50 I-Hsiang Wang IT Lecture 0
Course Overview Origin of Information Theory 20 / 50 I-Hsiang Wang IT Lecture 0
Course Overview Origin of Information Theory Shannon's 1948 paper is generally considered as the "birth" of information theory and (modern) digital communication. It was set clear that information theory is about the quantification of information. In particular, it focuses on characterizing the necessary and sufficient condition of whether or not a destination terminal can reproduce a message generated by a source terminal. 21 / 50 I-Hsiang Wang IT Lecture 0
Course Overview General Overview of Information Theory Course Information 1 Course Overview 2 General Overview of Information Theory First Part of this Course: Shannon Theory Second Part of this Course: Information Theory and Statistics 22 / 50 I-Hsiang Wang IT Lecture 0
Recommend
More recommend
