SLIDE 1
http://www.ee.unlv.edu/~b1morris/ee361
EE361: SIGNALS AND SYSTEMS II
CH1: PROBABILITY
1
SAMPLE SPACE AND EVENTS
CHAPTER 1.1-1.2 2
EE361: SIGNALS AND SYSTEMS II CH1: PROBABILITY - - PowerPoint PPT Presentation
EE361: SIGNALS AND SYSTEMS II CH1: PROBABILITY - - PowerPoint PPT Presentation
1 EE361: SIGNALS AND SYSTEMS II CH1: PROBABILITY http://www.ee.unlv.edu/~b1morris/ee361 2 SAMPLE SPACE AND EVENTS CHAPTER 1.1-1.2 3 INTRODUCTION Very important mathematical framework that is used in every ECE discipline (and science and
SLIDE 2
SLIDE 3
INTRODUCTION
Very important mathematical framework that is
used in every ECE discipline (and science and engineering in general)
The world is not perfect and some variability occurs
in everything we do professionally
The recorded sound of a voice saying a number multiple
times
The pixel values of successive images
3
SLIDE 4
(RANDOM) EXPERIMENT
Process of observation
(Random if outcome cannot be determined with certainty)
Outcome – the result of an observation of an experiment
Will use 𝜊 (xi) for an outcome
Think of Schrödinger's cat paradox from quantum
mechanics an experiment has many possible outcomes but only after observation can it be known
4
SLIDE 5
EXAMPLE RANDOM EXPERIMENTS
Roll of a dice Toss of a coin Drawing of a card from a deck
5
SLIDE 6
SAMPLE SPACE
The universal set of sample space 𝑇 – the set of all
possible outcomes of a random experiment
Sample point – an element in set 𝑇 (a possible
- utcome)
Cardinality – the number of elements in a set
6
SLIDE 7
EXAMPLES
Single coin toss
𝑇 = 𝐼, 𝑈 = {Heads, Tails} 𝑇 = 2 for two possible outcomes
Two coin tosses
𝑇 = {𝐼𝐼, 𝐼𝑈, 𝑈𝐼, 𝑈𝑈} 𝑇 = 4
7
SLIDE 8
SAMPLE SPACE
Discrete 𝑇 – finite number of sample points (outcomes)
- r countably infinite
Countable set – elements can be placed in one-to-one
correspondence with positive integers Continuous 𝑇 – sample points constitute a continuum
Example: transistor lifetime in hours Transistor can live until it dies at time 𝜐 𝑇 = {𝜐: 0 ≤ 𝜐 < ∞}
8
SLIDE 9
Set notations 𝜊 ∈ 𝑌 - 𝜊 is an element of
(belongs to) set 𝑇
𝜊 ∉ 𝑇 - 𝜊 is not an element of 𝑇 𝐵 ⊂ 𝐶 – 𝐵 contained in 𝐶
𝐵 is a subset of 𝐶 if every element
- f 𝐵 is contained in 𝐶
Event – subset of sample space
𝑇
Even roll of die 𝐵 = {2, 4, 6} Sum of rolls greater than 6
Elementary event – sample
point of 𝑇 (single outcome of 𝑇)
Certain event – 𝑇 ⊂ 𝑇
9
EVENTS
SLIDE 10
ALGEBRA OF SETS
CHAPTER 1.3 10
SLIDE 11
SET ALGEBRA I
1 Equality – 𝐵 = 𝐶 if 𝐵 ⊂ 𝐶 and 𝐶 ⊂ 𝐵 2 Complement – suppose 𝐵 ⊂ 𝑇,
ҧ 𝐵 = {𝜊: 𝜊 ∈ 𝑇 and 𝜊 ∉ 𝐵}
All elements in 𝑇 but not in 𝐵
ҧ 𝐵 is event 𝐵 did not occur
3 Union – set containing all elements in either 𝐵 or 𝐶
- r both
𝐵 ∪ 𝐶 = {𝜊: 𝜊 ∈ 𝐵 or 𝜊 ∈ 𝐶}
Event either 𝐵 of 𝐶 occurred
4 Intersection – set containing all elements in both 𝐵
and 𝐶
𝐵 ∩ 𝐶 = {𝜊: 𝜊 ∈ 𝐵 and 𝜊 ∈ 𝐶}
Both event 𝐵 and 𝐶 occurred
11
https://www.onlinemathlearning.com/venn-diagram.html
Venn Diagram
SLIDE 12
SET ALGEBRA II
Null set – set containing no elements
𝑇 = 𝜚 or 𝑇 = {𝜚}
Disjoint sets (mutually exclusive)
Sets 𝐵 and 𝐶 have no common elements
𝐵 ∩ 𝐶 = 𝜚 Partition of 𝑇 – a way to divide space 𝑇
up completely
If 𝐵𝑗 ∩ 𝐵𝑘 = 𝜚 (𝑗 ≠ 𝑘) and ڂ𝑗=1
𝑙
𝐵𝑗 = 𝑇
Then {𝐵𝑗: 1 ≤ 𝑗 ≤ 𝑙} is a partition
12
SLIDE 13
Size (cardinality) of set
|𝐵| - number of elements in 𝐵 (when
countable)
Properties
If 𝐵 ∩ 𝐶 = 𝜚, then 𝐵 ∪ 𝐶 = 𝐵 + 𝐶
𝜚 = 0
If 𝐵 ⊂ 𝐶, then 𝐵 ≤ |𝐶|
𝐵 ∪ 𝐶 + 𝐵 ∩ 𝐶 = 𝐵 + |𝐶|
(Cartesian) product of sets
𝐷 = 𝐵 × 𝐶 =
𝑏, 𝑐 : 𝑏 ∈ 𝐵, 𝑐 ∈ 𝐶
Set of ordered pairs of elements
Example 𝐵 = 𝑏1, 𝑏2, 𝑏3 , 𝐶 = {𝑐1, 𝑐2} 𝐷 = 𝐵 × 𝐶 =
{ 𝑏1, 𝑐1 , 𝑏1, 𝑐2 , 𝑏2, 𝑐1 , 𝑏2, 𝑐2 , 𝑏3, 𝑐1 , 𝑏3, 𝑐2 }
𝐸 = 𝐶 × 𝐵 =
{ 𝑐1, 𝑏1 , 𝑐1, 𝑏2 , 𝑐1, 𝑏3 , 𝑐2, 𝑏1 , 𝑐2, 𝑏2 , 𝑐2, 𝑏3 }
13
SET ALGEBRA III
SLIDE 14
LAWS OF SETS
Commutative
𝐵 ∪ 𝐶 = 𝐶 ∪ 𝐵
𝐵 ∩ 𝐶 = 𝐶 ∩ 𝐵
Associativity
𝐵 ∪ 𝐶 ∪ 𝐷 = 𝐵 ∪ 𝐶 ∪ 𝐷
𝐵 ∩ 𝐶 ∩ 𝐷 = 𝐵 ∩ 𝐶 ∩ 𝐷
Distributivity
𝐵 ∩ 𝐶 ∪ 𝐷 = 𝐵 ∩ 𝐶 ∪ (𝐵 ∩ 𝐷) 𝐵 ∪ 𝐶 ∩ 𝐷 = 𝐵 ∪ 𝐶 ∩ (𝐵 ∪ 𝐷)
De Morgan’s
𝐵 ∪ 𝐶 =
ҧ 𝐵 ∩ ത 𝐶 𝐵 ∩ 𝐶 = ҧ 𝐵 ∪ ത 𝐶
14
SLIDE 15
EVENT SPACE
Collection 𝐺 of subsets of sample space 𝑇
Also known as a sigma field
A set of subsets 𝐵 such that
𝑇 ∈ 𝐺 If 𝐵 ∈ 𝐺, then ҧ
𝐵 ∈ 𝐺
If 𝐵𝑗 ∈ 𝐺 for 𝑗 ≥ 1, then ڂ𝑗=1
∞ 𝐵𝑗 ∈ 𝐺
Example coin toss – 𝑇 = {𝐼, 𝑈}
𝐺
1 = {𝑇, 𝜚},
𝐺2 = {𝑇, 𝜚, 𝐼, 𝑈}
𝐺3 = {𝑇, 𝜚, 𝐼} – not event space since ഥ
𝐼 = 𝑈 not included
15
SLIDE 16
PROBABILITY SPACE/EQUALLY LIKELY EVENTS
CHAPTER 1.4-1.5 16
SLIDE 17
PROBABILITY SPACE
Assigns a real number to events in event space 𝐺
Known as probability measure
Given a random experiment with sample space 𝑇
𝐵 is an event defined in 𝐺 Probability of event 𝐵 𝑄(𝐵)
Probability space is defined over an event space
Triple: (𝑇, 𝐺, 𝑄) = (sample space, event space,
probability measure)
17
SLIDE 18
PROBABILITY MEASURE
Two methods to define
Classical definition Relative frequency definition
18
SLIDE 19
CLASSICAL (FINITE OUTCOMES) DEFINITION
Defined a priori, without need for experimentation
Only possible to use for simple problems with finite and
equally likely outcomes 𝑄 𝐵 =
𝐵 𝑇
- probability of event 𝐵
𝑄
ҧ 𝐵 =
ҧ 𝐵 𝑇 = 𝑇 − 𝐵 |𝑡|
= 1 −
𝐵 𝑇 = 1 − 𝑄 𝐵
Given disjoint sets 𝐵 ∩ 𝐶 = 𝜚
𝑄 𝐵 ∪ 𝐶 = 𝑄 𝐵 + 𝑄(𝐶)
19
SLIDE 20
EXAMPLE: DICE ROLL
𝑇 = {1, 2, 3, 4, 5, 6} Define events
A: Roll (outcome) is even
𝐵 = {2,4,6}
B: Roll odd
𝐶 = {1,3,5}
C: Roll prime
𝐷 = {1,2,3,5}
Probability of events
𝑄 𝐵 = 3
6 = 1 2
𝑄 𝐶 = 1
2
𝑄 𝐷 = 4
6 = 2 3
20
SLIDE 21
RELATIVE FREQUENCY DEFINITION
Repeated experiment definition Random experiment repeated 𝑜 times
𝑄 𝐵 = lim𝑜→∞
𝑜(𝐵) 𝑜
relative frequency of event A
𝑜(𝐵) – number of times event A occurs
0 ≤
𝑜 𝐵 𝑜
≤ 1
0 – never occurs
1 – occurs every time
Like classical, 𝐵 ∩ 𝐶 = 𝜚 ⇒ 𝑄 𝐵 ∪ 𝐶 = 𝑄 𝐵 + 𝑄(𝐶)
21
SLIDE 22
AXIOMATIC DEFINITION
Given probability space (𝑇, 𝐺, 𝑄) and event 𝐵 ∈ 𝐺 Axiom 1: 𝑄 𝐵 ≥ 0 Axiom 2: 𝑄 𝑇 = 1
One outcome certainly happens
Axiom 3: 𝑄 𝐵 ∪ 𝐶 = 𝑄 𝐵 + 𝑄(𝐶), if 𝐵 ∩ 𝐶 = 𝜚
22
SLIDE 23
ELEMENTARY PROPERTIES OF PROB.
1 𝑄
ҧ 𝐵 = 1 − 𝑄 𝐵
2 𝑄 𝜚 = 0 3 𝑄 𝐵 ≤ 𝑄 𝐶 if 𝐵 ⊂ 𝐶 4 𝑄 𝐵 ≤ 1 ⇒ 0 ≤ 𝑄 𝐵 ≤ 1 5 𝑄 𝐵 ∪ 𝐶 = 𝑄 𝐵 + 𝑄 𝐶 − 𝑄 𝐵 ∩ 𝐶
≤ 𝑄 𝐵 + 𝑄(𝐶)
8 Given 𝐵1, 𝐵2, … , 𝐵𝑜 finite sequence of mutually exclusive
events (𝐵𝑗 ∩ 𝐵𝑘 = 𝜚, 𝑗 ≠ 𝑘), 𝑄ڂ𝑗=1
𝑜
𝐵𝑗 = σ𝑗=1
𝑜
𝑄(𝐵𝑗)
23
SLIDE 24
EQUALLY LIKELY EVENTS
CHAPTER 1.5 24
SLIDE 25
EQUALLY LIKELY EVENTS
Finite sample space (n-finite)
𝑇 = {𝜊1, 𝜊2, … , 𝜊𝑜} and 𝑄 𝜊𝑗 = 𝑞𝑗
0 ≤ 𝑞𝑗 ≤ 1
σ𝑗=1
𝑜
𝑞𝑗 = 1
𝐵 =ڂ𝑗∈𝐽 𝜊𝑗 ⇒ σ𝜊𝑗∈𝐵 𝑞 𝜊𝑗 = σ𝑗∈𝐽 𝑞𝑗
Equally likely events
𝑞1 = 𝑞2 = ⋯ = 𝑞𝑜 𝑞𝑗 = 1
𝑜 , 𝑗 = 1, 2, … , 𝑜 and 𝑄 𝐵 = 𝑜 𝐵 /𝑜
25
SLIDE 26
CONDITIONAL PROBABILITY
CHAPTER 1.6 26
SLIDE 27
CONDITIONAL PROBABILITY
Probability of an event A given event B has
- ccurred
𝑄 𝐵 𝐶 =
𝑄 𝐵∩𝐶 𝑄 𝐶
, 𝑄 𝐶 > 0
𝑄(𝐵 ∩ 𝐶) – joint probability of A and B
𝑄 𝐶 𝐵 = 𝑄 𝐵∩𝐶
𝑄(𝐵)
⇒ 𝑄 𝐵 ∩ 𝐶 = 𝑄 𝐵 𝐶 𝑄 𝐶 = 𝑄 𝐶 𝐵 𝑄 𝐵
27
SLIDE 28
BAYES RULE
Incredibly important relationship from joint probability
𝑄 𝐵 𝐶 - posterior (what we want to estimate) 𝑄(𝐶|𝐵) – likelihood (probability of observing B given A) 𝑄(𝐵) – prior (probability of A with no extra information) 𝑄(𝐶) – marginal likelihood (total probability)
Tells how to update our believes based on the arrival of
new, relevant pieces of evidence
28
𝑄 𝐵 𝐶 = 𝑄 𝐶 𝐵 𝑄 𝐵 𝑄(𝐶)
SLIDE 29
TOTAL PROBABILITY/INDEPENDENT EVENTS
CHAPTER 1.7-1.8 29
SLIDE 30
TOTAL PROBABILITY
Let B be an event in S
𝑄 𝐶 = σ𝑗=1
𝑜
𝑄 𝐶 ∩ 𝐵𝑗 = σ𝑗=1
𝑜
𝑄 𝐶 𝐵𝑗 𝑄(𝐵𝑗)
Exhaustive
ڂ𝑗=1
𝑜
𝐵𝑗 = 𝑇
Mutually exclusive
𝐵𝑗 ∩ 𝐵𝑘 = 𝜚
From Bayes
𝑄 𝐵𝑗 𝐶 =
𝑄 𝐶 𝐵𝑗 𝑄 𝐵𝑗 σ𝑗=1
𝑜
𝑄 𝐶 𝐵𝑗 𝑄(𝐵𝑗)
30
SLIDE 31
INDEPENDENT EVENTS
Two events are statistically independent iff 𝑄 𝐵 ∩ 𝐶 = 𝑄 𝐵 𝑄 𝐶 ⇒ 𝑄 𝐵 𝐶 = 𝑄(𝐵) and 𝑄 𝐶 𝐵 = 𝑄(𝐶) If independent, probability of event A is not changed by knowledge
- f event B
Note:
Mutually exclusive
𝑄(ڂ𝑗=1
𝑜
𝐵𝑗) = σ𝑗=1
𝑜
𝑄 𝐵𝑗
Independent
𝑄ځ𝑗=1
𝑜
𝐵𝑗 = ς𝑗=1
𝑜
𝑄(𝐵𝑗)
31