Chapter 4: Multiple Random Variables STK4011/9011: Statistical - - PowerPoint PPT Presentation

Chapter 4: Multiple Random Variables

STK4011/9011: Statistical Inference Theory

Johan Pensar

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 1 / 20

Overview

1

Joint and Marginal Distributions

2

Conditional Distributions and Independence

3

Covariance and Correlation

4

Bivariate Transformations Covers Sec 4.1–4.3 and 4.5 in CB.

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 2 / 25

Random vectors

In most experiments one is interested in more than one random variable. Multivariate probability models: models over multiple random variables (random vector). An n-dimensional random vector X = (X1, . . . , Xn) is a function from a sample space S to Rn, that is, X : S → Rn. In the following we focus on the bivariate case (two random variables), but the results generalize to the multivariate case with more than two variables.

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 3 / 20

Joint Probability Mass Function (PMF)

The joint pmf of a discrete random vector (X, Y ) is defined by f (x, y) = P(X = x, Y = y). For any subset (or event) A ⊆ R2: P((X, Y ) ∈ A) = X

(x,y)∈A

f (x, y). Properties of the joint pmf:

f (x, y) ≥ 0 for all (x, y) ∈ R2 (and f (x, y) > 0 for a countable number of outcomes), P

(x,y)∈R2 f (x, y) = 1.

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 4 / 20

Example: Tossing two fair dice

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 5 / 20

Joint Probability Density Function (PDF)

The joint pdf of a random continuous random vector (X, Y ) is a function f (x, y) : R2 → R that satisfies P

• (X, Y ) ∈ A
• =

Z Z

A

f (x, y) dx dy, for A ⊂ R2. Properties of the joint pdf:

f (x, y) ≥ 0 for all (x, y) ∈ R2, R ∞

−∞

R ∞

−∞ f (x, y) dx dy = 1.

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 6 / 20

Example: Calculating joint probabilities in the continuous case

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 7 / 20

Joint Cumulative Distribution Function (CDF)

The joint distribution of (X, Y ) can be described by the joint cdf F(x, y) = P(X ≤ x, Y ≤ y), for all (x, y) ∈ R2. For a continuous random vector (X, Y ):

F(x, y) = R x

−∞

R y

−∞ f (s, t) dt ds,

∂2F(x, y) ∂x ∂y = f (x, y) at continuity points of f (x, y).

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 8 / 20

Expectations of Functions of Random vectors

For a real-valued function g(x, y), defined for all values (x, y) of the random vector (X, Y ):

g(X, Y ) is a random variable, If (X, Y ) is discrete: E

• g(X, Y )
• = P

(x,y)∈R2 g(x, y)f (x, y),

If (X, Y ) is continuous E

• g(X, Y )
• =

R ∞

−∞

R ∞

−∞ g(x, y)f (x, y) dx dy.

The expectation operator has the same properties as in the univariate case, e.g. E

• ag1(X, Y ) + bg2(X, Y ) + c
• = aE
• g1(X, Y )
• + bE
• g2(X, Y )
• + c.

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 9 / 20

Marginal Distributions

From the joint pmf/pdf fX,Y (x, y) we can calculate the marginal (in this case, univariate) pmfs/pdfs fX(x) and fY (y):

If (X, Y ) is discrete: fX(x) = P

y∈R fX,Y (x, y) and fY (y) = P x∈R fX,Y (x, y).

If (X, Y ) is continuous: fX(x) = R ∞

−∞ fX,Y (x, y) dy and fY (y) =

R ∞

−∞ fX,Y (x, y) dx.

Note: the marginal distributions of X and Y do not (in general) determine the joint distribution of (X, Y ), i.e., we cannot obtain fX,Y (x, y) based on only fX(x) and fY (y).

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 10 / 20

Conditional Distributions

If (X, Y ) is discrete: for any x such that P(X = x) = fX(x) > 0, the conditional pmf of Y given X = x is f (y | x) = P(Y = y | X = x) = fX,Y (x, y) fX(x) . If (X, Y ) is continuous: for any x such that fX(x) > 0, the conditional pdf of Y given X = x is f (y | x) = fX,Y (x, y) fX(x) .

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 11 / 20

Conditional Expectation and Variance

The conditional expected value of Y given X = x is E(Y | x) = X

y

yf (y | x)

• r

E(Y | x) = Z ∞

−∞

yf (y | x) dy. The conditional variance of Y given X = x is Var(Y | x) = E(Y 2 | x) − E(Y | x)2.

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 12 / 20

Example: Conditional Continuous Distributions

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 13 / 20

Law of Total Expectation

Thm 4.4.3: For two random variables X and Y : E(X) = E

• E(X | Y )
• ,

provided that the expectation exists.

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 14 / 25

Independence

The random variables X and Y are called independent if f (x, y) = f (x)f (y), for all (x, y) ∈ R2. If X and Y are independent, then f (y | x) = f (y). We can use the above definition of independence to:

Check if X and Y are independent given their joint pmf/pdf. Construct a model in which X and Y are independent.

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 14 / 20

Checking Independence

Verifying that X and Y are independent by direct use of the definition requires knowledge

• f fX(x) and fY (y).

Lemma 4.2.7: Let (X, Y ) be a random vector with joint pmf or pdf f (x, y). Then, X and Y are independent iff there exist functions g(x) and h(y) such that f (x, y) = g(x)h(y), for all (x, y) ∈ R2.

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 15 / 20

Proof of Lemma 4.2.7

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 16 / 20

Some Useful Properties for Independent Variables

Thm 4.2.10 & 4.2.12: For two independent variables X and Y , we have that:

For any A ⊂ R and B ⊂ R, P(X ∈ A, Y ∈ B) = P(X ∈ A)P(Y ∈ B). For functions g(x) and h(y), E

• g(X)h(Y )
• = E
• g(X)
• E
• h(Y )
• .

Assuming mgfs MX(t) and MY (t), the mgf of Z = X + Y is MZ(t) = MX(t)MY (t).

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 17 / 20

Covariance and Correlation

Covariance and correlation are measures for quantifying the strength of the (linear) relationship between nonindependent variables. The covariance between X and Y is defined by Cov(X, Y ) = E

• (X − µX)(Y − µY )
• = E(XY ) − µXµY .

The correlation between X and Y is defined by ρXY = Cov(X, Y ) σXσY , where −1 ≤ ρXY ≤ 1 and |ρXY | = 1 iff there is a perfect linear relationship (Thm 4.5.7).

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 18 / 20

Example: Correlation

<https://en.wikipedia.org/wiki/Correlation_and_dependence>

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 19 / 20

Some Properties

Thm 4.5.5: If X and Y are independent, then Cov(X, Y ) = ρXY = 0. Thm 4.5.6: For any two constants a and b: Var(aX + bY ) = a2Var(X) + b2Var(Y ) + 2abCov(X, Y ), and if X and Y are independent: Var(aX + bY ) = a2Var(X) + b2Var(Y ).

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 20 / 20

Bivariate Transformations - Discrete Case

Let (X, Y ) be a random vector with pmf fX,Y (x, y). Let (U, V ) be a new random vector where U = g1(X, Y ) and V = g2(X, Y ). Define the sets:

A = {(x, y) : fX,Y (x, y) > 0}, B = {(u, v) : u = g1(x, y) and v = g2(x, y) for some (x, y) ∈ A}, Au,v = {(x, y) : u = g1(x, y) and v = g2(x, y)}.

The joint pmf of (U, V ) can be computed from the joint pmf of (X, Y ): fU,V (u, v) = P

• (X, Y ) ∈ Au,v
• =

X

(x,y)∈Au,v

fX,Y (x, y).

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 22 / 25

Bivariate Transformations - Continuous Case

Let (X, Y ), (U, V ), and the sets A and B be defined as in the discrete case. Assume that u = g1(x, y) and v = g2(x, y) define a one-to-one transformation of A onto B ) We can calculate the inverse transformation x = h1(u, v) and y = h2(u, v). The joint pdf of (U, V ) is given by fU,V (u, v) = fX,Y

• h1(u, v), h2(u, v)
• |J| , for (u, v) 2 B,

where J =

• ∂x

∂u ∂x ∂v ∂y ∂u ∂y ∂v

• = ∂x

∂u ∂y ∂v ∂x ∂v ∂y ∂u , provided that J 6= 0 on B.

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 23 / 25

Example: Sum of two independent U(0, 1) variables

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 24 / 25

Example (cont.): Sum of two independent U(0, 1) variables

STK4011/9011: Statistical Inference Theory Chapter 4: Multiple Random Variables 25 / 25