Review of probability Nuno Vasconcelos UCSD Probability - - PDF document
Review of probability Nuno Vasconcelos UCSD Probability probability is the language to deal with processes that are non-deterministic examples: if I flip a coin 100 times, how many can I expect to see heads? what is the
– if I flip a coin 100 times, how many can I expect to see heads? – what is the weather going to be like tomorrow? – are my stocks going to be up or down? – am I in front of a classroom or is this just a picture of it?
– these are the outcomes or states of the process
– we roll a pair of dice – call the value on the up face at the nth toss xn – note that possible events such as
two sixes x1 = 2 and x2 = 6
– can all be expressed as combinations
1 6 6 1
x2 x1
– finest grain: all possible distinguishable events are listed separately – mutually exclusive: if one event happens the other does not (if x1 = 5 it cannot be anything else) – collectively exhaustive: any possible
sample space events
1 6 6 1
x2 x1
– number expressing the chance that the event will be the outcome
– P(A) ≥ 0 for any event A – P(universal event) = 1 – if A ∩ B = ∅, then P(A+B) = P(A) + P(B)
– P(x1 ≥ 0) = 1 – P(x1 even U x1 odd) = P(x1 even)+ P(x1 odd)
1 6 6 1
x2 x1
– combined with the mutually exclusive property of the sample set – allows us to easily assign probabilities to all possible events
– suppose that the probability of any pair (x1,x2) is 1/36 – we can compute probabilities of all “union” events – P(x2 odd) = 18x1/36 = 1 – P(U) = 36x1/36 = 1 – P(two sixes) = 1/36 – P(x1 = 2 and x2 = 6) = 1/36
1 6 6 1
x2 x1
– e.g. A = {x2 odd}, B = {x2 even}, U = A U B – on the other hand U = (1,1) U (1,2) U (1,3) U … U (6,6) – the fact that the sample space is finest grain, exhaustive, and mutually exclusive and the measure axioms – make the whole procedure consistent
1 6 6 1
x2 x1
– is a function that assigns a real value to each sample space event – we have already seen one such function: PX(x1,x2) = 1/36 for all (x1,x2)
– specify both the random variable and the value that it takes in your probability statements – we do this by specifying the random variable as subscript PX and the value as argument PX (x1,x2) = 1/36 means Prob[X=(x1,x2)] = 1/36 – without this, probability statements can be hopelessly confusing
– discrete and continuous – really means what types of values the RV can take
– this is the dice example we saw, there are only 36 possibilities
1 6 6 1
x2 x1
– we know that it could be any number between -50 and 150 degrees – random variable T ∈ [-50,150] – note that the extremes do not have to be very precise, we can just say that P(T < -45o) = 0
– this can be thought of as a normalized histogram – satisfies the following properties
– X ∈ {1,2,3, …, 20} where X = i if the grade of student z on class is between 5i and 5(i+1) – we see that PX(14) = α
α
1 ) ( , 1 ) ( = ∀ ≤ ≤
a X X
a P a a P
– this is just a continuous function – satisfies the following properties
1 ) ( ) ( = ∀ ≤
da a P a a P
X X
⎭ ⎬ ⎫ ⎩ ⎨ ⎧ − − =
2 2
2 ) ( exp 2 1 ) ( σ µ σ π a a PX
– this is probability per unit – the probability of a particular event is always zero (unless there is a discontinuity) – we can only talk about – note also that PDFs are not upper bounded – e.g. Gaussian goes to Dirac when variance goes to zero
= ≤ ≤
b a X
dt t P b X a ) ( ) Pr(
– e.g. when in the doctor, you are mostly a collection of random variables
x1: temperature x2: blood pressure x3: weight x4: cough …
– a vector X = (x1, …, xn) of n random variables – PX(x1, …, xn) is the joint probability distribution
– e.g. having a cold does not depend on blood pressure and weight – all that matters are fever and cough – that is, we need to know PX1,X4(a,b)
– (in this case X1 and X4) – this is done by summing (or integrating) the others out
3 2 4 3 2 1 , , , 4 1 ,
4 3 2 1 4 1
X X X X X X
4 3 4 3 2 1 4 1
, 4 3 2 1 , , , 4 1 ,
x x X X X X X X
– so far, doctor has PX1,X4(fever,cough) – still does not allow a diagnostic – for this we need a new variable Y with two states Y ∈ {sick, not sick} – doctor measures fever and cough levels, these are no longer unknowns, or even random quantities – the question of interest is “what is the probability that patient is sick given the measured values of fever and cough?”
– what is the probability that Y takes a given value given
4 1,
|
X X Y
|
Y
– what is the probability that you will be sick and cough a lot?
,
Y
– what is the probability that you are sick given that you cough a lot? – “given” is the key word here – conditional probability is very important because it allows us to structure our thinking – shows up again and again in design of intelligent systems
|
Y
– we simply normalize the joint by the probability of what we know – note that this makes sense since – and, by the marginalization equation, – the definition of conditional probability
just makes these two statements coherent simply says that, given what we know, we still have a valid probability measure universal event {sick} U {not sick} still probability 1 after observation
1 1 1
, | X X Y X Y
1 1
| |
X Y X Y
1 1 1
, , X X Y X Y
– note that, from this definition, – more generally, it has the form
1 1 | 1 ,
1 1 1
X X Y X Y
2 1 ,..., | 2 1 ,..., ,
2 1 2 1
n X X X n X X X
n n
3 2 ..., |
3 2
n X X X
n
1 |
1
n X n n X X
n n n
−
−
– breaks down a hard question (prob of sick and 104) into two easier questions – Prob (sick|104): everyone knows that this is close to one
1 1 1
| , X X Y X Y
You have a cold!
|
Y
– Prob(104): still hard, but easier than P(sick,104) since we know
– does not depend on sickness, it is just the question “what is the probability that someone will have 104o?”
gather a number of people, measure their temperatures and make an histogram that everyone can use after that
1 1 1
| , X X Y X Y
– e.g. – in this way we can get away with knowing
PX1(t), which we may know because it was needed for some other problem PY|X1(sick|t), we can ask a doctor or approximate with rule of thumb
X Y Y
1
,
X X Y
1 1
|
1
|
Y
(marginalization)
– two variables are independent if the joint is the product of the marginals – note: implies that – which captures the intuitive notion: – “if X1 is independent of X2, knowing X2 does not change the probability of X1”
e.g. knowing that it is sunny does not change the probability that it will rain in three months
2 1 2 1,
X X X X
1 2 2 1 2 1
, |
X X X X X X
– frequently, knowing X makes Y independent of Z – e.g. consider the shivering symptom:
if you have temperature you sometimes shiver it is a symptom of having a cold but once you measure the temperature, the two become independent
, | , ,
1 1
S X Y S X Y
1 1
| X X S
1
|
Y
1 1
| X X S
– i.e. if z = x + y and x,y are independent then where * is the convolution operator – for discrete random variables – for continuous random variables
y X Z
k Y X Z
Y X Z
– summarize the distribution
– mean: µ = E[x] – variance: σ2 = E[(x-µ)2] – various distributions are completely specified by a small number
µ σ2
discrete continuous mean variance
k X
X
2 2
k X
X
2 2
– if Z = X + Y, then E[Z] = E[X] + E[Y] – this does not require any special relation between X and Y – always holds
– nth order moment is E[Xn] – nth central moments is E[(X-µ)n] discrete continuous mean
k X
X
µ σ2
– it is the second central moment
– if Z = X + Y, then Var[Z] = Var[X] + Var[Y] – only holds if X and Y are independent
discrete continuous variance
2 2
k X
X
2 2
2 2 2 2 2 2 2 2
µ σ2