Course Information Overview
Lecture 1 Introduction
I-Hsiang Wang
Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw
September 17, 2014
1 / 45 I-Hsiang Wang NIT Lecture 1
Lecture 1 Introduction I-Hsiang Wang Department of Electrical - - PowerPoint PPT Presentation
Course Information Overview Lecture 1 Introduction I-Hsiang Wang Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw September 17, 2014 1 / 45 I-Hsiang Wang NIT Lecture 1 Course Information Overview
Course Information Overview
Department of Electrical Engineering National Taiwan University ihwang@ntu.edu.tw
1 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
2 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
3 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
4 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
5 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
1 Establish solid foundations and intuitions of information theory, and 2 Explore various topics extending the legacy of information theory to
6 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
7 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
1 Instructor: I-Hsiang Wang 王奕翔
2 Lecture Time:
3 Lecture Location: EE2-225 電機二館 225 室 4 Course Website:
8 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
5 References
6 Prerequisites:
9 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
1 Grading: Quiz (30%), Homework (40%), Project (30%) 2 Quiz:
3 Homework:
A
AT
10 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
1 Team: 2 people per team 2 Topic:
3 Final presentation:
4 Final report: Each team has to summarize their project in a written
A
11 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
12 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
13 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
14 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
15 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
16 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
17 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
18 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Encoder Channel Decoder Source Destination Noise
1 The source would like to deliver some message to the destination,
2 The channel is the physical medium that connects the source and
3 The encoder can carry out any processing of the source output,
4 The decoder can carry out any processing of the channel output to
19 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
20 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
21 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Encoder Channel Decoder Source Destination Noise
22 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Encoder Channel Decoder Source Destination Noise
23 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Source Encoder Source Noisy Channel Channel Encoder Destination Source Decoder Channel Decoder Binary Interface Bits Bits
24 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Noisy Channel Channel Encoder Channel Decoder Source Encoder Source Destination Source Decoder
25 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Noisy Channel Channel Encoder Channel Decoder Source Encoder Source Destination Source Decoder
s[1], . . . , s[N] b[1], . . . , b[K] b s[1], . . . , b s[N]
26 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Noisy Channel Channel Encoder Channel Decoder Source Encoder Source Destination Source Decoder
s[1], . . . , s[N] b[1], . . . , b[K] b s[1], . . . , b s[N]
N > the entropy rate of the source, H(S)
27 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Source Encoder Source Noisy Channel Channel Encoder Destination Source Decoder Channel Decoder
28 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Source Encoder Source Noisy Channel Channel Encoder Destination Source Decoder Channel Decoder
b[1], . . . , b[K] b b[1], . . . ,b b[K] x[1], . . . , x[N] y[1], . . . , y[N]
p (y|x)
29 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Source Encoder Source Noisy Channel Channel Encoder Destination Source Decoder Channel Decoder
b[1], . . . , b[K] b b[1], . . . ,b b[K] x[1], . . . , x[N] y[1], . . . , y[N]
p (y|x)
N < the channel capacity of the channel, C
30 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Source Destination Encoder Noisy Channel Decoder C(e) Edge e Point-to-point communication link represented by edge e
31 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
1956 IRE TRANXACTIONX ON INFORiMATION THEORY 117
AND
Summary--This note discusses the problem
the rate of flow from one terminal to another, through a network which consists of a number of branches, each of which has a !imited capa-
The maximum possible flow from left to right through a network is equal to the minimum value among all simple cut-sets. This theorem is applied to solve a more general problem, in which a number
nodes are used. c ONSIDER a two-terminal network such as that
might represent communication channels,
more generally, any conveying system of limited capacity as, for example, a railroad system, a power feeding system,
to assign a definite maximum allowed rate of flow over a given branch. The links may be of two types, either one directional (indicated by arrows) or two directional, in which case flow is allowed in either direction at anything up to maximum capacity. At the nodes or junction points
any redistribution
the outgoing flow is allowed, subject
striction
law that the total (algebraic) flow into a node be zero. Note that in the case of infor- mation flow, this may require arbitrarily large delays at each node to permit recoding of the output signals from that
is to evaluate the maximum possible flow through the network as a whole, entering at the left terminal and emerging at the right terminal. 7
3 b 5 cl I f
The answer can be given in terms of cut-sets of the network. A cut-set of a two-terminal network is a set of branches such that when deleted from the network, the network falls into two or more unconnected parts with the two terminals in different
* Manuscript received by the PGIT, July 11, 1956. t Elec. Ena. Deot. and Res. Lab.
Tech., CambrTdge, -Mass. 1 Lincoln Lab., M.I.T., Lexington! Mass. 5 Bell Telephone Labs., Murray Hill,
Cam- bridge, Mass.
from one terminal to the other in the original network passes through at least one branch in the cut-set. In the network above, some examples of cut-sets are (d, e, f), and (b, c, e, g, h), (d, g, h, i) . By a simple cut-set we will mean a cut-set such that if any branch is omitted it is no longer a cut-set. Thus (d, e, f) and (b, c, e, g, h) are simple cut-sets while (d, g, h, ;) is not. When a simple cut-set is deleted from a connected two-terminal network, the net- work falls into exactly two parts, a left part containing the left terminal and a right part containing the right terminal. We assign a value to a simple cut-set by taking the sum of capacities
in the cut-set,
counting capacities, however, from the left part to the right part for branches that are unidirectional. Note that the direction
branch cannot be deduced from its appearance in the graph of the network. A branch is directed from left to right in a minimal cut-set if, and
left part of the network to a node in the right part. Thus, in the example, the cut-set (d, e, f) has the value 5 + 1 = 6, the cut-set (b, c, e, g, h) has value 3 + 2 + 3 + 2 = 10. Theorem: The maximum possible flow from left to right through a net,work is equal to the minimum value among all simple cut-sets. This theorem may appear almost obvious on physical grounds and appears to have been accepted without proof for some time by workers in communication theory. However, while the fact that this flow cannot be exceeded is indeed almost trivial, the fact that it can actually be achieved is by no means obvious. We understand that proofs of the theorem have been given by Ford and Fulkerson’ and Fulkerson and Dantzig.2 The following proof is relatively simple, and we believe different in principle. To prove first that the minimum cut-set flow cannot be exceeded, consider any given flow pattern and a minimum- valued cut-set C. Take the algebraic sum X of flows from left to right across this cut-set. This is clearly less than or equal to the value V of the cut-set, since the latter would result if all paths from left to right in C were carrying full capacity, and those in the reverse direction were carrying
flows into all nodes in the right-hand group for the cut- set C. This sum is zero because of the Kirchhoff law constraint at each node. Viewed another way, however, we see that it cancels out each flow contributing to S, and also that each flow on a branch with both ends in the
1 L. Ford, Jr. and D. R. Fulkerson, Can.
* G. B. Dantsig and D. R. Fulkerson, “On the Max-Flow Min- Cut Theorem
in “Linear Inequalities,”
Studies, no. 38, Princeton, New Jersey, 1956.
1956 IRE TRANXACTIONX ON INFORiMATION THEORY 117
AND
Summary--This note discusses the problem
the rate of flow from one terminal to another, through a network which consists of a number of branches, each of which has a !imited capa-
The maximum possible flow from left to right through a network is equal to the minimum value among all simple cut-sets. This theorem is applied to solve a more general problem, in which a number
nodes are used. c ONSIDER a two-terminal network such as that
might represent communication channels,
more generally, any conveying system of limited capacity as, for example, a railroad system, a power feeding system,
to assign a definite maximum allowed rate of flow over a given branch. The links may be of two types, either one directional (indicated by arrows) or two directional, in which case flow is allowed in either direction at anything up to maximum capacity. At the nodes or junction points
any redistribution
the outgoing flow is allowed, subject
striction
law that the total (algebraic) flow into a node be zero. Note that in the case of infor- mation flow, this may require arbitrarily large delays at each node to permit recoding of the output signals from that
is to evaluate the maximum possible flow through the network as a whole, entering at the left terminal and emerging at the right terminal.
7
3 b
5 cl I
f
The answer can be given in terms of cut-sets of the network. A cut-set of a two-terminal network is a set of branches such that when deleted from the network, the network falls into two or more unconnected parts with the two terminals in different
* Manuscript received by the PGIT, July 11, 1956. t Elec. Ena. Deot. and Res. Lab.
Tech., CambrTdge, -Mass. 1 Lincoln Lab., M.I.T., Lexington! Mass. 5 Bell Telephone Labs., Murray Hill,
Cam- bridge, Mass.
from one terminal to the other in the original network passes through at least one branch in the cut-set. In the network above, some examples of cut-sets are (d, e, f), and (b, c, e, g, h), (d, g, h, i) . By a simple cut-set we will mean a cut-set such that if any branch is omitted it is no longer a cut-set. Thus (d, e, f) and (b, c, e, g, h) are simple cut-sets while (d, g, h, ;) is not. When a simple cut-set is deleted from a connected two-terminal network, the net- work falls into exactly two parts, a left part containing the left terminal and a right part containing the right terminal. We assign a value to a simple cut-set by taking the sum of capacities
in the cut-set,
counting capacities, however, from the left part to the right part for branches that are unidirectional. Note that the direction
branch cannot be deduced from its appearance in the graph of the network. A branch is directed from left to right in a minimal cut-set if, and
left part of the network to a node in the right part. Thus, in the example, the cut-set (d, e, f) has the value 5 + 1 = 6, the cut-set (b, c, e, g, h) has value 3 + 2 + 3 + 2 = 10. Theorem: The maximum possible flow from left to right through a net,work is equal to the minimum value among all simple cut-sets. This theorem may appear almost obvious on physical grounds and appears to have been accepted without proof for some time by workers in communication theory. However, while the fact that this flow cannot be exceeded is indeed almost trivial, the fact that it can actually be achieved is by no means obvious. We understand that proofs of the theorem have been given by Ford and Fulkerson’ and Fulkerson and Dantzig.2 The following proof is relatively simple, and we believe different in principle. To prove first that the minimum cut-set flow cannot be exceeded, consider any given flow pattern and a minimum- valued cut-set C. Take the algebraic sum X of flows from left to right across this cut-set. This is clearly less than or equal to the value V of the cut-set, since the latter would result if all paths from left to right in C were carrying full capacity, and those in the reverse direction were carrying
flows into all nodes in the right-hand group for the cut- set C. This sum is zero because of the Kirchhoff law constraint at each node. Viewed another way, however, we see that it cancels out each flow contributing to S, and also that each flow on a branch with both ends in the
1 L. Ford, Jr. and D. R. Fulkerson, Can.
* G. B. Dantsig and D. R. Fulkerson, “On the Max-Flow Min- Cut Theorem
in “Linear Inequalities,”
Studies, no. 38, Princeton, New Jersey, 1956.
32 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
33 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
34 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
35 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
36 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
37 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
1204 IEEE TRANSACTIONS ON INFORMATION THEORY, VOL. 46, NO. 4, JULY 2000
Rudolf Ahlswede, Ning Cai, Shuo-Yen Robert Li, Senior Member, IEEE, and Raymond W. Yeung, Senior Member, IEEE
Abstract—We introduce a new class of problems called network information flow which is inspired by computer network applica-
a number of information sources are to be mulitcast to certain sets
tually independent. The problem is to characterize the admissible coding rate region. This model subsumes all previously studied models along the same line. In this paper, we study the problem with one information source, and we have obtained a simple char- acterization of the admissible coding rate region. Our result can be regarded as the Max-flow Min-cut Theorem for network informa- tion flow. Contrary to one’s intuition, our work reveals that it is in general not optimal to regard the information to be multicast as a “fluid” which can simply be routed or replicated. Rather, by em- ploying coding at the nodes, which we refer to as network coding, bandwidth can in general be saved. This finding may have signifi- cant impact on future design of switching systems. Index Terms—Diversity coding, multicast, network coding,
38 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Source Destination 2 Destination 1 Source Destination 2 Destination 1
39 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
40 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
41 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Encoder 1 Multiple Access Channel Decoder Encoder 2 Encoder K X1 X2 XK Source 1 Source 2 Source K p (y|x1, . . . , xK) Y Destination
42 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Broadcast Channel Decoder 2 Encoder Source Destination 2 X Y1 Decoder 1 Destination1 Decoder K Destination K Y2 YK p (y1, . . . , yK|x)
43 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
Interference Channel Decoder 2 Destination 2 Y1 Decoder 1 Destination1 Decoder K Destination K Y2 YK Encoder 1 Encoder 2 Encoder K Source 1 Source 2 Source K p
44 / 45 I-Hsiang Wang NIT Lecture 1
Course Information Overview
45 / 45 I-Hsiang Wang NIT Lecture 1