Lecture 1 Introduction I-Hsiang Wang Department of Electrical - PowerPoint PPT Presentation

Course Information Overview Lecture 1 Introduction I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw September 17, 2014 1 / 45 I-Hsiang Wang NIT Lecture 1

Course Information Overview Network Information Theory Information is usually obtained by getting some “messages” (speech, text, images, etc.) from others. What one may care about a piece of information? What is the meaning of a message? How important is the message? How much information can I get from the message? 2 / 45 I-Hsiang Wang NIT Lecture 1 Information Theory is a mathematical theory of information

Course Information Overview Network Information Theory Information is usually obtained by getting some “messages” (speech, text, images, etc.) from others. What one may care about a piece of information? What is the meaning of a message? How important is the message? How much information can I get from the message? Information theory is about the quantification of information. 3 / 45 I-Hsiang Wang NIT Lecture 1 Information Theory is a mathematical theory of information

Course Information Overview Network Information Theory for communication systems) that Establishes the fundamental limits of communication systems (Quantifies the amount of information that can be delivered from a party to another) Built upon probability theory and statistics Main concern: ultimate performance limit (usually the rate of scales to the asymptotic regime, given that the desired information is delivered “reliably”. 4 / 45 I-Hsiang Wang NIT Lecture 1 Information Theory is a mathematical theory of information (primarily information) as certain resources (usually the total amount of time)

Course Information Overview Network Information Theory Network is a collection of terminals connected through communication links that Exchange information and communicate with one another, so that certain goals are met Use graph theoretic quantities/properties to model terminal interaction Extend the architecture of point-to-point communication systems 5 / 45 I-Hsiang Wang NIT Lecture 1

Course Information Overview In this course, we will 1 Establish solid foundations and intuitions of information theory, and 2 Explore various topics extending the legacy of information theory to network settings Later, we begin with a brief overview of information theory and how the theory extends to multi-user networks. 6 / 45 I-Hsiang Wang NIT Lecture 1

Course Information Overview 1 Course Information 2 Overview 7 / 45 I-Hsiang Wang NIT Lecture 1

Course Information Overview Logistics Email: ihwang@ntu.edu.tw Office Hours: 17:30 – 18:30, Tuesday and Wednesday 2 Lecture Time: 13:20 – 14:10 (5) Tuesday, and 10:20 – 12:10 (34) Wednesday 4 Course Website: http://homepage.ntu.edu.tw/~ihwang/Teaching/Fa14/NIT.html 8 / 45 I-Hsiang Wang NIT Lecture 1 1 Instructor: I-Hsiang Wang 王奕翔 Office: MD-524 明達館 524 室 3 Lecture Location: EE2-225 電機二館 225 室

Course Information Overview Logistics 5 References A. El Gamal and Y.-H. Kim, Network Information Theory, Cambridge University Press, 2011. T. Cover and J. Thomas, Elements of Information Theory, Wiley-Interscience, 2006. R. Yeung, Information Theory and Network Coding, Springer, 2008. R. Gallager, Information Theory and Reliable Communications, Wiley, 1968. 6 Prerequisites: Probability, Linear Algebra, Principles of Communications. 9 / 45 I-Hsiang Wang NIT Lecture 1

Course Information T I-Hsiang Wang 10 / 45 This additional effort accounts for part of your homework grades. are in charge of, so that we can make sure the solution is correct. you should discuss with me about the homework problem that you EX templates, and A T EX. We will provide L A Overview Everyone has to provide the detailed solution for one homework Roughly 3–4 problems every two weeks; late homework = 0 points. 3 Homework: Roughly 2 quizzes in the whole semester In order to make sure that you understand the lecture materials 2 Quiz: 1 Grading: Quiz (30%), Homework (40%), Project (30%) Grading Policy NIT Lecture 1 problem, documented in L

Course Information become part of the final project presentation grades I-Hsiang Wang 11 / 45 EX T A report, documented in L 4 Final report: Each team has to summarize their project in a written Other people have to give grades on the presentation, which will Overview Each team has to present the results in 20 minutes to the audience Scheduled in the last two weeks 3 Final presentation: Meeting with the instructor to decide the topic A list of potential topics will be announced 2 Topic: 1 Team: 2 people per team Project NIT Lecture 1

Course Information Network information flow : cut-set bound, network coding I-Hsiang Wang 12 / 45 degraded BC, Marton coding scheme Broadcast channel (BC) : superposition coding, capacity of cancellation, time-sharing, joint decoding Multiple access channel (MAC) : successive interference states, dirty-paper coding Overview Channel with states : compound channel, channel with random coding theorem, source-channel separation Point-to-point communication : source coding theorem, channel divergence (relative entropy), typical sequences Measures of information : entropy, mutual information, KL Course Outline NIT Lecture 1

Course Information Overview Course Outline Interference channel (IC) : Han-Kobayashi coding scheme, capacity of Gaussian IC to within 1 bit, capacity of deterministic IC, interference alignment. Distributed source coding : Slepian-Wolf theorem, Wyner-Ziv theorem, CEO problem Feedback (if time allows): Schalkwijk-Kailath scheme, feedback in Gaussian IC Advanced topics (if time allows): error exponents, non-asymptotic information theory, etc. 13 / 45 I-Hsiang Wang NIT Lecture 1

Course Information 11/4, 5 10/21, 22 Channel Coding, Source-Channel separa- tion 7 10/28, 29 Channel with States HW3 8 Graphical Networks, Max-Flow Min-Cut HW2 Theorem Quiz 1 (11/5) 9 11/11, 12 Max-Flow Min-Cut Theorem, Network Coding HW4 14 / 45 I-Hsiang Wang 6 Channel Coding Overview 2 Tentative Schedule Week Date Content Remark 1 9/16, 17 Introduction 9/23, 24 10/14, 15 Measure of Information 3 9/30, 10/1 Measure of Information, Source Coding HW1 4 10/7, 8 Source Coding 5 NIT Lecture 1

Course Information 12/30, 31 12/16, 17 Interference Channel, Distributed Source Coding 15 12/23, 24 No class Quiz 2 (12/24) 16 Distributed Source Coding HW6 HW7 17 1/6, 7 Project Presentation 18 1/13, 14 Project Presentation 15 / 45 I-Hsiang Wang 14 Interference Channel Overview 11/25, 26 Tentative Schedule Week Date Content Remark 10 11/18, 19 Multiple Access Channel 11 Multiple 12/9, 10 Access Channel, Broadcast Channel HW5 12 12/2, 3 Broadcast Channel 13 NIT Lecture 1

Course Information Overview 1 Course Information 2 Overview 16 / 45 I-Hsiang Wang NIT Lecture 1

Course Information Overview Claude Elwood Shannon (1916 – 2001) 17 / 45 I-Hsiang Wang NIT Lecture 1

Course Information Overview Shannon’s landmark paper in 1948 is generally considered as the “birth” of information theory. In the paper, Shannon set it clear that information theory is about the quantification of information in a communication system. In particular, it focuses on characterizing the necessary and sufficient message generated by a source terminal. 18 / 45 I-Hsiang Wang NIT Lecture 1 condition of whether or not a destination terminal can reproduce a

Course Information 2 The channel is the physical medium that connects the source and I-Hsiang Wang 19 / 45 reproduce the source message. 4 The decoder can carry out any processing of the channel output to including compression, modulation, insertion of redundancy, etc. 3 The encoder can carry out any processing of the source output, is usually subject to certain noise disturbances. the destination, such as cable, optical fiber, EM radiation, etc., and where the message includes speech, image, video, audio, text, etc. Overview 1 The source would like to deliver some message to the destination, Above is an abstract model of communication system: Communication System NIT Lecture 1 Noise Source Encoder Channel Decoder Destination

Course Information Overview A primary concern of information theory is on the encoder and the decoder, both in terms of How the encoder and the decoder function, and The existence or nonexistence of encoders and decoders that achieve a given level of performance 20 / 45 I-Hsiang Wang NIT Lecture 1

Course Information Prior to Shannon, theorist and engineers were able to analyze the I-Hsiang Wang 21 / 45 the message sent from the source? sufficient condition for the destination to be able to reconstruct For all possible encoders/decoders, what is the necessary and Shannon asked: knowledge about what is the ultimate limit. performance of certain choice of encoders/decoders, but had little than the entire waveform? Overview to be communicated, why not simply send the frequency rather If the receiver knows that a sine wave of unknown frequency is Shannon asked: based on Fourier analysis and gave birth to sampling theory. to extract useful information (usually, voice). This line of research was the destination should try its best to reconstruct this waveform, in order analog paradigm – if the source produces a electromagnetic waveform, Prior to the 1948 paper, design of communication systems followed the NIT Lecture 1