Gov 2000: 3. Multiple Random Variables Matthew Blackwell Fall 2016 - - PowerPoint PPT Presentation

Gov 2000: 3. Multiple Random Variables

Matthew Blackwell

Fall 2016

1 / 57

• 1. Distributions of Multiple Random Variables
• 2. Properties of Joint Distributions
• 3. Conditional Distributions
• 4. Wrap-up

2 / 57

Where are we? Where are we going?

• Distributions of one variable: how to describe and summarize

uncertainty about one variable.

• Today: distributions of multiple variables to describe

relationships between variables.

• Later: use data to learn about probability distributions.

3 / 57

Why multiple random variables?

1 2 3 4 5 6 7 8 6 7 8 9 10 Log Settler Mortality Log GDP/pop growth

• 1. How do we summarize the relationship between two variables,

𝑌 and 𝑍?

• 2. What if we have many observations of the same variable,

𝑌1, 𝑌2, … , 𝑌𝑜?

4 / 57

1/ Distributions of Multiple Random Variables

5 / 57

Joint distributions

• 2
• 1

1 2

• 2
• 1

1 2 x y

• 2
• 1

1 2

• 2
• 1

1 2 x y

• 2
• 1

1 2

• 15
• 10
• 5

5 x y

• The joint distribution of two r.v.s, 𝑌 and 𝑍, describes what

pairs of observations, (𝑦, 𝑧) are more likely than others.

▶ Settler mortality (𝑌) and GDP per capita (𝑍) for the same

country.

• Shape of the joint distribution now includes the relationship

between 𝑌 and 𝑍

6 / 57

Discrete r.v.s

Joint probability mass function

The joint p.m.f. of a pair of discrete r.v.s, (𝑌, 𝑍) describes the probability of any pair of values: 𝑔𝑌,𝑍(𝑦, 𝑧) = ℙ(𝑌 = 𝑦, 𝑍 = 𝑧)

• Properties of a joint p.m.f.:

▶ 𝑔𝑌,𝑍(𝑦, 𝑧) ≥ 0 (probabilities can’t be negative) ▶ ∑𝑦 ∑𝑧 𝑔𝑌,𝑍(𝑦, 𝑧) = 1 (something must happen) ▶ ∑𝑦 is shorthand for sum over all possible values of 𝑌 7 / 57

Example: Gay marriage and gender

Favor Gay Oppose Gay Marriage Marriage 𝑍 = 1 𝑍 = 0 Female 𝑌 = 1 0.3 0.21 Male 𝑌 = 0 0.22 0.27

• Joint p.m.f. can be summarized in a cross-tab:

▶ Each cell is the probability of that combination, 𝑔𝑌,𝑍(𝑦, 𝑧)

• Probability that we randomly select a woman who favors gay

marriage? 𝑔𝑌,𝑍(1, 1) = ℙ(𝑌 = 1, 𝑍 = 1) = 0.3

8 / 57

Marginal distributions

• Often need to fjgure out the distribution of just one of the

r.v.s

▶ Called the marginal distribution in this context.

• Computing marginals from the joint p.m.f.:

𝑔𝑍(𝑧) = ℙ(𝑍 = 𝑧) = ∑

𝑦

𝑔𝑌,𝑍(𝑦, 𝑧)

• Intuition: sum over the probability that 𝑍 = 𝑧 for all possible

values of 𝑦

▶ Works because these are mutually exclusive events that

partition the space of 𝑌

9 / 57

Example: marginals for gay marriage

Favor Gay Oppose Gay Marriage Marriage Marginal 𝑍 = 1 𝑍 = 0 Female 𝑌 = 1 0.3 0.21 0.51 Male 𝑌 = 0 0.22 0.27 0.49 Marginal 0.52 0.48

• What’s the 𝑔𝑍(1) = ℙ(𝑍 = 1)?

▶ Probability that a man favors gay marriage plus the probability

that a woman favors gay marriage. 𝑔𝑍(1) = 𝑔𝑌,𝑍(1, 1) + 𝑔𝑌,𝑍(0, 1) = 0.3 + 0.22 = 0.52

• Works for all marginals.

10 / 57

Continuous r.v.s

𝑌 𝑍 𝐵

• We will focus on getting the probability of being in some

subset of the 2-dimensional plane.

11 / 57

Continuous joint p.d.f.

Continuous joint distribution

Two continuous r.v.s 𝑌 and 𝑍 have a continuous joint distribution if there is a nonnegative function 𝑔𝑌,𝑍(𝑦, 𝑧) such that for any subset 𝐵 of the 𝑦𝑧-plane, ℙ((𝑌, 𝑍) ∈ 𝐵) = ∬

(𝑦,𝑧)∈𝐵 𝑔𝑌,𝑍(𝑦, 𝑧)𝑒𝑦𝑒𝑧.

• 𝑔𝑌,𝑍(𝑦, 𝑧) is the joint probability density function.
• {(𝑦, 𝑧) ∶ 𝑔𝑌,𝑍(𝑦, 𝑧) > 0} is called the support of the distribution.
• Joint p.d.f. must meet the following conditions:
• 1. 𝑔𝑌,𝑍(𝑦, 𝑧) ≥ 0 for all values of (𝑦, 𝑧), (nonnegative)
• 2. ∫∞

−∞ ∫∞ −∞ 𝑔𝑌,𝑍(𝑦, 𝑧)𝑒𝑦𝑒𝑧 = 1, (probabilities “sum” to 1)

• ℙ(𝑌 = 𝑦, 𝑍 = 𝑧) = 0 for similar reasons as with single r.v.s.

12 / 57

Joint densities are 3D

0.00 0.05 0.10 0.15

• 4
• 2

2 4

• 4
• 2

2 4 x y

• 𝑌 and 𝑍 axes are on the “fmoor,” height is the value of

𝑔𝑌,𝑍(𝑦, 𝑧).

• Remember 𝑔𝑌,𝑍(𝑦, 𝑧) ≠ ℙ(𝑌 = 𝑦, 𝑍 = 𝑧).

13 / 57

Probability = volume

0.00 0.05 0.10 0.15

• 4
• 2

2 4

• 4
• 2

2 4 x y

• ℙ((𝑌, 𝑍) ∈ 𝐵) = ∬

(𝑦,𝑧)∈𝐵 𝑔𝑌,𝑍(𝑦, 𝑧)𝑒𝑦𝑒𝑧

• Probability = volume above a specifjc region.

14 / 57

Working with joint p.d.f.s

• Suppose we have the following form of a joint p.d.f.:

𝑔𝑌,𝑍(𝑦, 𝑧) = ⎧ { ⎨ { ⎩ 𝑑(𝑦 + 𝑧) for 0 < 𝑦 < 2 and 0 < 𝑧 < 2

• therwise
• What does 𝑑 have to be for this to be a valid p.d.f.?

1 = ∫

∞ −∞ ∫ ∞ −∞ 𝑔𝑌,𝑍(𝑦, 𝑧)𝑒𝑦𝑒𝑧

= ∫

2 0 ∫ 2 0 𝑑(𝑦 + 𝑧)𝑒𝑦𝑒𝑧

= 𝑑 ∫

2 0 (𝑦2

2 + 𝑦𝑧)∣

𝑦=2 𝑦=0

𝑒𝑧 = 𝑑 ∫

2 0 (2 + 2𝑧)𝑒𝑧

= (2𝑑𝑧 + 𝑑𝑧2)∣

2 0 = 8𝑑

• Thus to be a valid p.d.f., 𝑑 = 1/8

15 / 57

Example continuous distribution

0.0 0.1 0.2 0.3 0.4 0.5 0.0 0.5 1.0 1.5 2.0 0.0 0.5 1.0 1.5 2.0 x y x y z p l u s

𝑔𝑌,𝑍(𝑦, 𝑧) = ⎧ { ⎨ { ⎩ (𝑦 + 𝑧)/8 for 0 < 𝑦 < 2 and 0 < 𝑧 < 2

• therwise

16 / 57

Continuous marginal distributions

• We can recover the marginal PDF of one of the variables by

integrating over the distribution of the other variable: 𝑔𝑍(𝑧) = ∫

∞ −∞ 𝑔𝑌,𝑍(𝑦, 𝑧)𝑒𝑦

• Works for either variable:

𝑔𝑌(𝑦) = ∫

∞ −∞ 𝑔𝑌,𝑍(𝑦, 𝑧)𝑒𝑧

17 / 57

Visualizing continuous marginals

y x z

• Marginal integrates (sums, basically) over other r.v.:

𝑔𝑍(𝑧) = ∫∞

−∞ 𝑔𝑌,𝑍(𝑦, 𝑧)𝑒𝑦

• Pile up/fmatten all of the joint density onto a single dimension.

18 / 57

Deriving continuous marginals

𝑔𝑌,𝑍(𝑦, 𝑧) = ⎧ { ⎨ { ⎩ (𝑦 + 𝑧)/8 for 0 < 𝑦 < 2 and 0 < 𝑧 < 2

• therwise
• Let’s calculate the marginals for this p.d.f.:

𝑔𝑌(𝑦) = ∫

2

1 8(𝑦 + 𝑧)𝑒𝑧 = (𝑦𝑧 8 + 𝑧2 16)∣

𝑧=2 𝑧=0

= 𝑦 4 + 1 4 = 𝑦 + 1 4

• By symmetry we have the same for 𝑧:

𝑔𝑍(𝑧) = (𝑧 + 1)/4

19 / 57

Joint c.d.f.s

Joint cumulative distribution function

For two r.v.s 𝑌 and 𝑍, the joint cumulative distribution function or joint c.d.f. 𝐺𝑌,𝑍(𝑦, 𝑧) is a function such that for fjnite values 𝑦 and 𝑧, 𝐺𝑌,𝑍(𝑦, 𝑧) = ℙ(𝑌 ≤ 𝑦, 𝑍 ≤ 𝑧).

• Deriving p.d.f. from c.d.f.: 𝑔𝑌,𝑍(𝑦,𝑧) = 𝜖2𝐺𝑌,𝑍(𝑦,𝑧)

𝜖𝑦𝜖𝑧

• Deriving c.d.f. from p.d.f: 𝐺𝑌,𝑍(𝑦, 𝑧) = ∫𝑧

−∞ ∫𝑦 −∞ 𝑔𝑌,𝑍(𝑠, 𝑡)𝑒𝑠𝑒𝑡

20 / 57

2/ Properties of Joint Distributions

21 / 57

Properties of joint distributions

• Single r.v.: summarized 𝑔𝑌(𝑦) with 𝔽[𝑌] and 𝕎[𝑌]
• With 2 r.v.s, we can additionally measure how strong the

dependence is between the variables.

• First: expectations over joint distributions and independence

22 / 57

Expectations over multiple r.v.s

• 2-d LOTUS: take expectations over the joint distribution.
• With discrete 𝑌 and 𝑍:

𝔽[𝑕(𝑌, 𝑍)] = ∑

𝑦

∑

𝑧

𝑕(𝑦, 𝑧) 𝑔𝑌,𝑍(𝑦, 𝑧)

• With continuous 𝑌 and 𝑍:

𝔽[𝑕(𝑌, 𝑍)] = ∫

𝑦 ∫ 𝑧 𝑕(𝑦, 𝑧) 𝑔𝑌,𝑍(𝑦, 𝑧)𝑒𝑦𝑒𝑧

• Marginal expectations:

𝔽[𝑍] = ∑

𝑦

∑

𝑧

𝑧 𝑔𝑌,𝑍(𝑦, 𝑧)

• Example: expectation of the product:

𝔽[𝑌𝑍] = ∑

𝑦

∑

𝑧

𝑦𝑧 𝑔𝑌,𝑍(𝑦, 𝑧)

23 / 57

Marginal expectations from joint

𝑔𝑌,𝑍(𝑦, 𝑧) = ⎧ { ⎨ { ⎩ (𝑦 + 𝑧)/8 for 0 < 𝑦 < 2 and 0 < 𝑧 < 2

• therwise
• Marginal expectation of 𝑍:

𝔽[𝑍] = ∫

2 0 ∫ 2 0 𝑧1

8(𝑦 + 𝑧)𝑒𝑦𝑒𝑧 = ∫

2 0 𝑧 ∫ 2

1 8(𝑦 + 𝑧)𝑒𝑦𝑒𝑧 = ∫

2 0 𝑧1

4(𝑧 + 1)𝑒𝑧 = ( 𝑧3 12 + 𝑧2 8 )∣

2

= 2 3 + 1 2 = 7 6

• By symmetry, 𝔽[𝑌] = 𝔽[𝑍] = 7/6

24 / 57

Independence

Independence

Two r.v.s 𝑍 and 𝑌 are independent (which we write 𝑌 ⟂ ⟂ 𝑍) if for all sets 𝐵 and 𝐶: ℙ(𝑌 ∈ 𝐵, 𝑍 ∈ 𝐶) = ℙ(𝑌 ∈ 𝐵)ℙ(𝑍 ∈ 𝐶).

• Knowing the value of 𝑌 gives us no information about the

value of 𝑍.

• If 𝑌 and 𝑍 are independent, then:

▶ 𝑔𝑌,𝑍(𝑦, 𝑧) = 𝑔𝑌(𝑦)𝑔𝑍(𝑧) (joint is the product of marginals) ▶ 𝐺𝑌,𝑍(𝑦, 𝑧) = 𝐺𝑌(𝑦)𝐺𝑍(𝑧) ▶ ℎ(𝑌) ⟂

⟂ 𝑕(𝑍) for any functions ℎ() and 𝑕() (functions of independent r.v.s are independent)

25 / 57

Key properties of independent r.v.s

• Theorem If 𝑌 and 𝑍 are independent r.v.s, then

𝔽[𝑌𝑍] = 𝔽[𝑌]𝔽[𝑍].

• Proof for discrete 𝑌 and 𝑍:

𝔽[𝑌𝑍] = ∑

𝑦

∑

𝑧

𝑦𝑧 𝑔𝑌,𝑍(𝑦, 𝑧) = ∑

𝑦

∑

𝑧

𝑦𝑧 𝑔𝑌(𝑦)𝑔𝑍(𝑧) = (∑

𝑦

𝑦 𝑔𝑌(𝑦)) ⎛ ⎜ ⎝ ∑

𝑧

𝑧 𝑔𝑍(𝑧)⎞ ⎟ ⎠ = 𝔽[𝑌]𝔽[𝑍]

26 / 57

Why independence?

• Independence assumptions are everywhere in theoretical and

applied statistics.

▶ Each response in a poll is considered independent of all other

responses.

▶ In a randomized control trial, treatment assignment is

independent of background characteristics.

• Lack of independence is a blessing or a curse:

▶ Two variables not independent ⇝ potentially interesting

relationship.

▶ In observational studies, treatment assignment is usually not

independent of background characteristics.

27 / 57

Covariance

• If two variables are not independent, how do we measure the

strength of their dependence?

▶ Covariance ▶ Correlation

• Covariance: how do two r.v.s vary together?

▶ How often do high values of 𝑌 occur with high values of 𝑍? 28 / 57

Defining covariance

• If two variables are not independent, how do we measure the

strength of their dependence?

Covariance

The covariance between two r.v.s, 𝑌 and 𝑍 is defjned as: Cov[𝑌, 𝑍] = 𝔽[(𝑌 − 𝔽[𝑌])(𝑍 − 𝔽[𝑍])]

• How often do high values of 𝑌 occur with high values of 𝑍?
• Properties of covariances:

▶ Cov[𝑌, 𝑍] = 𝔽[𝑌𝑍] − 𝔽[𝑌]𝔽[𝑍] ▶ If 𝑌 ⟂

⟂ 𝑍, Cov[𝑌, 𝑍] = 𝔽[𝑌𝑍] − 𝔽[𝑌]𝔽[𝑍] = 𝔽[𝑌]𝔽[𝑍] − 𝔽[𝑌]𝔽[𝑍] = 0

29 / 57

Covariance intuition

• 4
• 2

2 4

• 4
• 2

2 4 x y E[X] E[Y]

30 / 57

Covariance intuition

• 4
• 2

2 4

• 4
• 2

2 4 x y E[X] E[Y] Y > E[Y] X > E[X] Y > E[Y] X < E[X] Y < E[Y] X < E[X] Y < E[Y] X > E[X]

• Large values of 𝑌 tend to occur with large values of 𝑍:

▶ (𝑌 − 𝔽[𝑌])(𝑍 − 𝔽[𝑍]) = (pos. num.) × (pos. num) = +

• Small values of 𝑌 tend to occur with small values of 𝑍:

▶ (𝑌 − 𝔽[𝑌])(𝑍 − 𝔽[𝑍]) = (neg. num.) × (neg. num) = +

• If these dominate ⇝ positive covariance.

31 / 57

Covariance intuition

• 4
• 2

2 4

• 4
• 2

2 4 x y E[X] E[Y] Y > E[Y] X > E[X] Y > E[Y] X < E[X] Y < E[Y] X < E[X] Y < E[Y] X > E[X]

• Large values of 𝑌 tend to occur with small values of 𝑍:

▶ (𝑌 − 𝔽[𝑌])(𝑍 − 𝔽[𝑍]) = (pos. num.) × (neg. num) = −

• Small values of 𝑌 tend to occur with large values of 𝑍:

▶ (𝑌 − 𝔽[𝑌])(𝑍 − 𝔽[𝑍]) = (neg. num.) × (pos. num) = −

• If these dominate ⇝ negative covariance.

32 / 57

Covariance from joint p.d.f.

• Using our running example of 𝑔𝑌,𝑍(𝑦, 𝑧) = (𝑦 + 𝑧)/8
• From earlier: 𝔽[𝑌] = 𝔽[𝑍] = 7/6
• Expectation of the product:

𝔽[𝑌𝑍] = ∫

2 0 ∫ 2 0 𝑦𝑧1

8(𝑦 + 𝑧)𝑒𝑦𝑒𝑧 = ∫

2 0 ∫ 2

1 8(𝑦2𝑧 + 𝑦𝑧2)𝑒𝑦𝑒𝑧 = ∫

2 0 (𝑦3𝑧

24 + 𝑦2𝑧2 16 )∣

𝑦=2 𝑦=0

𝑒𝑧 = ∫

2 0 (𝑧

3 + 𝑧2 4 ) 𝑒𝑧 = (𝑧2 6 + 𝑧3 12)∣

2

= 2 3 + 2 3 = 4 3

• Covariance:

Cov[𝑌, 𝑍] = 𝔽[𝑌𝑍] − 𝔽[𝑌]𝔽[𝑍] = 4 3 − (7 6)

2

= − 1 36

33 / 57

Zero covariance doesn’t imply independence

• We saw that 𝑌 ⟂

⟂ 𝑍 ⇝ Cov[𝑌, 𝑍] = 0.

• Does Cov[𝑌, 𝑍] = 0 imply that 𝑌⟂

⟂𝑍? No!

• Counterexample: 𝑌 ∈ {−1, 0, 1} with equal probability and

𝑍 = 𝑌2.

• Covariance is a measure of linear dependence, so it can miss

non-linear dependence.

34 / 57

Properties of variances and covariances

• Properties of covariances:
• 1. Cov[𝑏𝑌 + 𝑐, 𝑑𝑍 + 𝑒] = 𝑏𝑑Cov[𝑌, 𝑍].
• 2. Cov[𝑌, 𝑌] = 𝕎[𝑌]
• Properties of variances that we can state now that we know

covariance:

• 1. 𝕎[𝑏𝑌 + 𝑐𝑍 + 𝑑] = 𝑏2𝕎[𝑌] + 𝑐2𝕎[𝑍] + 2𝑏𝑐Cov[𝑌, 𝑍]
• 2. If 𝑌 and 𝑍 independent, 𝕎[𝑌 + 𝑍] = 𝕎[𝑌] + 𝕎[𝑍].

35 / 57

Using properties of covariance

• Rescale our running example: 𝑎 = 2𝑌, 𝑋 = 2𝑍.
• What’s the covariance of (𝑎, 𝑋)?

▶ Ugh, let’s avoid more integrals.

• Use properties of covariances:

Cov[𝑎, 𝑋] = Cov[2𝑌, 2𝑍] = 2 × 2 × Cov[𝑌, 𝑍] = −1 9

36 / 57

Correlation

• Covariance is not scale-free: Cov[2𝑌, 𝑍] = 2Cov[𝑌, 𝑍]

▶ ⇝ hard to compare covriances across difgerent r.v.s ▶ Is a relationship stronger? Or just do to rescaling?

• Correlation is a scale-free measure of linear dependence.

Correlation

The correlation between two r.v.s 𝑌 and 𝑍 is defjned as: 𝜍 = 𝜍(𝑌, 𝑍) = Cov[𝑌, 𝑍] √𝕎[𝑌]𝕎[𝑍]

• Covariance after dividing out the scales of the respective

variables.

• Correlation properties:

▶ −1 ≤ 𝜍 ≤ 1 ▶ if |𝜍(𝑌, 𝑍)| = 1, then 𝑌 and 𝑍 are perfectly correlated with a

deterministic linear relationship: 𝑍 = 𝑏 + 𝑐𝑌.

37 / 57

3/ Conditional Distributions

38 / 57

Conditional distributions

• Conditional distribution: distribution of 𝑍 if we know 𝑌 = 𝑦.

Conditional probability mass function

The conditional probability mass function or conditional p.m.f. of 𝑍 conditional on 𝑌 is 𝑔𝑍|𝑌(𝑧|𝑦) = ℙ(𝑌 = 𝑦, 𝑍 = 𝑧) ℙ(𝑌 = 𝑦) = 𝑔𝑌,𝑍(𝑦, 𝑧) 𝑔𝑌(𝑦)

• Intuitive defjnition:

𝑔𝑍|𝑌(𝑧|𝑦) = Probability that 𝑌 = 𝑦 and 𝑍 = 𝑧 Probability that 𝑌 = 𝑦

• This is a valid univariate probability distribution!

▶ 𝑔𝑍|𝑌(𝑧|𝑦) ≥ 0 and ∑𝑧 𝑔𝑍|𝑌(𝑧|𝑦) = 1

• If 𝑌 ⟂

⟂ 𝑍 then 𝑔𝑍|𝑌(𝑧|𝑦) = 𝑔𝑍(𝑧) (conditional is the marginal)

39 / 57

Example: conditionals for gay marriage

Favor Gay Oppose Gay Marriage Marriage Marginal 𝑍 = 1 𝑍 = 0 Female 𝑌 = 1 0.3 0.21 0.51 Male 𝑌 = 0 0.22 0.27 0.49 Marginal 0.52 0.48

• Probability of favoring gay marriage conditional on being a

man? 𝑔𝑍|𝑌(𝑧 = 1|𝑦 = 0) = ℙ(𝑌 = 0, 𝑍 = 1) ℙ(𝑌 = 0) = 0.22 0.22 + 0.27 = 0.44

40 / 57

Example: conditionals for gay marriage

1

Men

Gay marriage support (Y) Conditional Prob. 0.0 0.2 0.4 0.6 0.8 1.0 1

Women

Gay marriage support (Y) Conditional Prob. 0.0 0.2 0.4 0.6 0.8 1.0

• Two values of 𝑌 ⇝ two univariate conditional distribution of 𝑍

41 / 57

Continuous conditional distributions

Conditional probability density function

The conditional p.d.f. of a continuous random variable is 𝑔𝑍|𝑌(𝑧|𝑦) = 𝑔𝑌,𝑍(𝑦, 𝑧) 𝑔𝑌(𝑦) assuming that 𝑔𝑌(𝑦) > 0.

• Implies

ℙ(𝑏 < 𝑍 < 𝑐|𝑌 = 𝑦) = ∫

𝑐 𝑏 𝑔𝑍|𝑌(𝑧|𝑦)𝑒𝑧.

• Based on the defjnition of the conditional p.m.f./p.d.f., we

have the following factorization: 𝑔𝑌,𝑍(𝑦, 𝑧) = 𝑔𝑍|𝑌(𝑧|𝑦)𝑔𝑌(𝑦)

42 / 57

Conditional distributions as slices

• 𝑔𝑍|𝑌(𝑧|𝑦0) is the conditional p.d.f. of 𝑍 when 𝑌 = 𝑦0
• 𝑔𝑍|𝑌(𝑧|𝑦0) is proportional to joint p.d.f. along 𝑦0: 𝑔𝑌,𝑍(𝑧, 𝑦0)
• Normalize by dividing by 𝑔𝑌(𝑦0) to ensure proper p.d.f.

43 / 57

Continuous conditional example

• Using our running example of 𝑔𝑌,𝑍(𝑦, 𝑧) = (𝑦 + 𝑧)/8
• Earlier we calculated 𝑔𝑌(𝑦) = (𝑦 + 1)/4
• Calculate conditional:

𝑔𝑍|𝑌(𝑧|𝑦) = 𝑔𝑌,𝑍(𝑦, 𝑧) 𝑔𝑌(𝑦) = (𝑦 + 𝑧)/8 (𝑦 + 1)/4 = 𝑦 + 𝑧 2(𝑦 + 1)

• Remember the limits: 0 < 𝑧 < 2, 0 otherwise

44 / 57

Conditional Independence

Conditional independence

Two r.v.s 𝑌 and 𝑍 are conditionally independent given 𝑎 (written 𝑌 ⟂ ⟂ 𝑍|𝑎) if 𝑔𝑌,𝑍|𝑎(𝑦, 𝑧|𝑨) = 𝑔𝑌|𝑎(𝑦|𝑨)𝑔𝑍|𝑎(𝑧|𝑨).

• 𝑌 and 𝑍 are independent within levels of 𝑎.
• Massively important for regression, causal inference.
• Example:

▶ 𝑌 = swimming accidents, 𝑍 = number of ice cream cones sold. ▶ In general, dependent. ▶ Conditional on 𝑎 = temperature, independent. 45 / 57

Summarizing conditional distributions

• 4
• 2

2 4 y f(y|0) f(y|1)

• Conditional distributions are also univariate distribution and

so we can summarize them with its mean and variance.

• Gives us insight into a key question:

▶ How does the mean of 𝑍 change as we change 𝑌? 46 / 57

Defining condition expectations

Conditional expectation

The conditional expectation of 𝑍 conditional on 𝑌 = 𝑦 is: 𝔽[𝑍|𝑌 = 𝑦] = ⎧ { { ⎨ { { ⎩ ∑

𝑧

𝑧 𝑔𝑍|𝑌(𝑧|𝑦) discrete 𝑍 ∫

∞ −∞ 𝑧 𝑔𝑍|𝑌(𝑧|𝑦)𝑒𝑧

continuous 𝑍

• Intuition: exactly the same defjnition of the expected value

with 𝑔𝑍|𝑌(𝑧|𝑦) in place of 𝑔𝑍(𝑧)

• The expected value of the (univariate) conditional distribution.
• This is a function of 𝑦!

47 / 57

Calculating conditional expectations

Favor Gay Oppose Gay Marriage Marriage Marginal 𝑍 = 1 𝑍 = 0 Female 𝑌 = 1 0.3 0.21 0.51 Male 𝑌 = 0 0.22 0.27 0.49 Marginal 0.52 0.48

• What’s the conditional expectation of support for gay

marriage 𝑍 given someone is a man 𝑌 = 0? 𝔽[𝑍|𝑌 = 0] = ∑

𝑧

𝑧 𝑔𝑍|𝑌(𝑧|𝑦 = 0) = 0 × 𝑔 (𝑧 = 0|𝑦 = 0) + 1 × 𝑔 (𝑧 = 1|𝑦 = 0) = 1 × 0.22 0.22 + 0.27 = 0.44

48 / 57

Conditional expectations are random variables

• For a particular 𝑦, 𝔽[𝑍|𝑌 = 𝑦] is a number.
• But 𝑌 takes on many possible values with uncertainty

⇝ 𝔽[𝑍|𝑌] takes on many possible values with uncertainty.

• ⇝ Conditional expectations are random variables!
• Binary 𝑌:

𝔽[𝑍|𝑌] = ⎧ { ⎨ { ⎩ 𝔽[𝑍|𝑌 = 0] with prob. ℙ(𝑌 = 0) 𝔽[𝑍|𝑌 = 1] with prob. ℙ(𝑌 = 1)

• Has an expectation, 𝔽[𝔽[𝑍|𝑌]], and a variance, 𝕎[𝔽[𝑍|𝑌]].

49 / 57

Law of iterated expectations

• Average/mean of the conditional expectations: 𝔽[𝔽[𝑍|𝑌]].

▶ Can we connect this to the marginal (overall) expectation?

• Theorem (The Law of Iterated Expectations): If the

expectation exist and for discrete 𝑌, 𝔽[𝑍] = 𝔽 [𝔽[𝑍|𝑌]] = ∑

𝑦

𝔽[𝑍|𝑌 = 𝑦]𝑔𝑌(𝑦)

50 / 57

Example: law of iterated expectations

Favor Gay Oppose Gay Marriage Marriage Marginal 𝑍 = 1 𝑍 = 0 Female 𝑌 = 1 0.3 0.21 0.51 Male 𝑌 = 0 0.22 0.27 0.49 Marginal 0.52 0.48 1

• 𝔽[𝑍|𝑌 = 1] = 0.59 and 𝔽[𝑍|𝑌 = 0] = 0.44.
• 𝑔𝑌(1) = 0.51 (females) and 𝑔𝑌(0) = 0.49 (males).
• Plug into the iterated expectations:

𝔽[𝔽[𝑍|𝑌]] = 𝔽[𝑍|𝑌 = 0]𝑔𝑌(0) + 𝔽[𝑍|𝑌 = 1]𝑔𝑌(1) = 0.44 × 0.49 + 0.59 × 0.51 = 0.52 = 𝔽[𝑍]

51 / 57

Properties of conditional expectations

• 1. 𝔽[𝑑(𝑌)|𝑌] = 𝑑(𝑌) for any function 𝑑(𝑌).

▶ Example: 𝔽[𝑌2|𝑌] = 𝑌2 (If we know 𝑌, then we also know 𝑌2)

• 2. If 𝑌 and 𝑍 are independent r.v.s, then

𝔽[𝑍|𝑌 = 𝑦] = 𝔽[𝑍].

• 3. If 𝑌 ⟂

⟂ 𝑍|𝑎, then 𝔽[𝑍|𝑌 = 𝑦, 𝑎 = 𝑨] = 𝔽[𝑍|𝑎 = 𝑨].

52 / 57

Conditional Variance

Conditional expectation

The conditional variance of a 𝑍 given 𝑌 = 𝑦 is defjned as: 𝕎[𝑍|𝑌 = 𝑦] = 𝔽 [(𝑍 − 𝔽[𝑍|𝑌 = 𝑦])2|𝑌 = 𝑦]

• Conditional variance describes the spread of the conditional

distribution around the conditional expectation.

• Usual variance formula applied to conditional distribution.
• Using LOTUS:

▶ Discrete 𝑍:

𝕎[𝑍|𝑌 = 𝑦] = ∑

𝑧

(𝑧 − 𝔽[𝑍|𝑌 = 𝑦])2𝑔𝑍|𝑌(𝑧|𝑦)

▶ Continuous 𝑍:

𝕎[𝑍|𝑌 = 𝑦] = ∫

𝑧(𝑧 − 𝔽[𝑍|𝑌 = 𝑦])2𝑔𝑍|𝑌(𝑧|𝑦)𝑒𝑧

53 / 57

Conditional variance is a random variable

• Again, 𝕎[𝑍|𝑌] is a random variable and a function of 𝑌, just

like 𝔽[𝑍|𝑌]. With a binary 𝑌: 𝕎[𝑍|𝑌] = ⎧ { ⎨ { ⎩ 𝕎[𝑍|𝑌 = 0] with prob. ℙ(𝑌 = 0) 𝕎[𝑍|𝑌 = 1] with prob. ℙ(𝑌 = 1)

54 / 57

Law of total variance

• We can also relate the marginal variance to the conditional

variance and the conditional expectation.

• Theorem (Law of Total Variance/EVE’s law):

𝕎[𝑍] = 𝔽[𝕎[𝑍|𝑌]] + 𝕎[𝔽[𝑍|𝑌]]

• The total variance can be decomposed into:
• 1. the average of the within group variance (𝔽[𝕎[𝑍|𝑌]]) and
• 2. how much the average varies between groups (𝕎[𝔽[𝑍|𝑌]]).

55 / 57

4/ Wrap-up

56 / 57

Review

• Multiple r.v.s require joint p.m.f.s and joint p.d.f.s
• Multiple r.v.s can have distributions that exhibit dependence

as measured by covariance and correlation.

• The conditional expectation of one variable on the other is an

important quantity that we’ll see over and over again.

57 / 57