1 Product of Expectations The Dance of the Covariance Let X and Y - - PDF document

▶
1
SMART_READER_LITE
LIVE PREVIEW

1 Product of Expectations The Dance of the Covariance Let X and Y - - PDF document

Jun 28, 2023 410 likes •460 views

Indicators: Now With Pair-wise Flavor! From Event Pairs to Variance Recall I i is indicator variable for event A i when: Expected number of pairs of events: 1 if A occurs X

slide-1
SLIDE 1

1 Indicators: Now With Pair-wise Flavor!

  • Recall Ii is indicator variable for event Ai when:
  • Let X = # of events that occur:
  • Now consider pair of events Ai Aj occurring
  • Ii Ij = 1 if both events Ai and Aj occur, 0 otherwise
  • Number of pairs of events that occur is

   

  • therwise
  • ccurs

if 1

i i

A I

  

  

        

n i i n i i n i i

A P I E I E X E

1 1 1

) ( ] [ ] [

n i i

I X

1

      

j i j i I

I

X 2

From Event Pairs to Variance

  • Expected number of pairs of events:
  • Recall: Var(X) = E[X2] – (E[X])2

  

  

              

       

j i j i j i j i j i j i

A A P I I E I I E E

X

) ( ] [

2

 

  

j i j i A

A P X E X E E

X X

) ( ]) [ ] [ (

2

2 1 2 ) 1 (

] [ ) ( 2 ] [ ) ( 2 ] [ ] [

2 2

X E A A P X E A A P X E X E

j i j i j i j i

    

 

  2

]) [ ( ] [ ) ( 2 ) ( Var X E X E A A P X

j i j i

   

 2 1 1

) ( ) ( ) ( 2         

  

   n i i n i i j i j i

A P A P A A P

Let’s Try It With the Binomial

  • X ~ Bin(n, p)
  • Each trial: Xi ~ Ber(p)
  • Let event Ai = trial i is success (i.e., Xi = 1)

2 2

2 2

) ( ] [ p p A A P X X E E

n X

j i j i j i j i j i

               

  

  

       

np A P X E

n i i 



1

) ( ] [ p X E

i 

] [

2 2

) 1 ( ] [ ] [ )] 1 ( [ p n n X E X E X X E     

2 2 2 2

]) [ ( ] [ ]) [ ] [ ( ]) [ ( ] [ ) ( Var X E X E X E X E X E X E X      

2 2 2 2 2 2 2

) ( ) 1 ( p n np np p n np np p n n         ) 1 ( p np  

Computer Cluster Utilization

  • Computer cluster with N servers
  • Requests independently go to server i with probability pi
  • Let event Ai = server i receives no requests
  • X = # of events A1, A2, … An that occur
  • Y = # servers that receive ≥ 1 request = N – X
  • E[Y] after first n requests?
  • Since requests independent:

n i i

p A P ) 1 ( ) (  

 

 

  

N i n i N i i

p A P X E

1 1

) 1 ( ) ( ] [

    

N i n i

p N X E N Y E

1

) 1 ( ] [ ] [

 

n N i n i

N N N

N N Y E N i p ) 1 ( 1 ) 1 ( ] [ , 1

1 1 1

1

        

for when

Computer Cluster Utilization (cont.)

  • Computer cluster with N servers
  • Requests independently go to server i with probability pi
  • Let event Ai = server i receives no requests
  • X = # of events A1, A2, … An that occur
  • Y = # servers that receive ≥ 1 request = N – X
  • Var(Y) after first n requests?
  • Independent requests:

j i p p A A P

n j i j i

    , ) 1 ( ) (

 

 

      

j i n j i j i j i

p p A A P X E X E X X E ) 1 ( 2 ) ( 2 ] [ ] [ )] 1 ( [

2 2

]) [ ( ] [ ) 1 ( 2 ) ( Var X E X E p p X

j i n j i

     

 

N i n i

p X E

1

) 1 ( ] [ ) ( Var ) 1 ( ) 1 ( ) 1 ( 2

2 1 1

Y p p p p

N i n i N i n i j i n j i

             

  

  

( = (-1)2 Var(X) = Var(X) )

Computer Cluster = Coupon Collecting

  • Computer cluster with N servers
  • Requests independently go to server i with probability pi
  • Let event Ai = server i receives no requests
  • X = # of events A1, A2, … An that occur
  • Y = # servers that receive ≥ 1 request = N – X
  • This is really another “Coupon Collector” problem
  • Each server is a “coupon type”
  • Request to server = collecting a coupon of that type
  • Hash table version
  • Each server is a bucket in table
  • Request to server = string gets hashed to that bucket
slide-2
SLIDE 2

2 Product of Expectations

  • Let X and Y are independent random variables,

and g() and h() are real-valued functions

  • Proof:

)] ( [ )] ( [ )] ( ) ( [ Y h E X g E Y h X g E  dy dx y x f y h x g Y h X g E

y x Y X

) , ( ) ( ) ( )] ( ) ( [

,

 

     

 dy y f y h dx x f x g

y Y x X

) ( ) ( ) ( ) (

 

     

  dy dx y f x f y h x g

y x Y X

) ( ) ( ) ( ) (

 

     

 )] ( [ )] ( [ Y h E X g E 

The Dance of the Covariance

  • Say X and Y are arbitrary random variables
  • Covariance of X and Y:
  • Equivalently:
  • X and Y independent, E[XY] = E[X]E[Y]  Cov(X,Y) = 0
  • But Cov(X,Y) = 0 does not imply X and Y independent!

])] [ ])( [ [( ) , ( Cov Y E Y X E X E Y X    ]] [ ] [ ] [ ] [ [ ) , ( Cov X E Y E Y XE Y X E XY E Y X     ] [ ] [ ] [ ] [ ] [ ] [ ] [ Y E X E Y E X E Y E X E XY E     ] [ ] [ ] [ Y E X E XY E  

Dependence and Covariance

  • X and Y are random variables with PMF:
  • E[X] = 0, E[Y] = 1/3
  • Since XY = 0, E[XY] = 0
  • Cov(X, Y) = E[XY] – E[X]E[Y] = 0 – 0 = 0
  • But, X and Y are clearly dependent

X Y

  • 1

1 pY(y) 1/3 1/3 2/3 1 1/3 1/3 pX(x) 1/3 1/3 1/3 1     

  • therwise

1 if X Y

Example of Covariance

  • Consider rolling a 6-sided die
  • Let indicator variable X = 1 if roll is 1, 2, 3, or 4
  • Let indicator variable Y = 1 if roll is 3, 4, 5, or 6
  • What is Cov(X, Y)?
  • E[X] = 2/3 and E[Y] = 2/3
  • E[XY]

= = (0 * 0) + (0 * 1/3) + (0 * 1/3) + (1 * 1/3) = 1/3

  • Cov(X, Y) = E[XY] – E[X]E[Y] = 1/3 – 4/9 = -1/9
  • Consider: P(X = 1) = 2/3 and P(X = 1 | Y = 1) = 1/2
  • Observing Y = 1 makes X = 1 less likely



x y

y x p xy ) , (

Another Example of Covariance

  • Consider the following data:

Weight Height Weight * Height 64 57 3648 71 59 4189 53 49 2597 67 62 4154 55 51 2805 58 50 2900 77 55 4235 57 48 2736 56 42 2352 51 42 2142 76 61 4636 68 57 3876 E[W] = 62.75 E[H] = 52.75 E[W*H] = 3355.83

30 35 40 45 50 55 60 65 40 45 50 55 60 65 70 75 80 Height Weight

Cov(W, H) = E[W*H] – E[W]E[H] = 3355.83 – (62.75)(52.75) = 45.77

Properties of Covariance

  • Say X and Y are arbitrary random variables
  • Covariance of sums of random variables
  • X1, X2, …, Xn and Y1, Y2, …, Ym are random variables
  • )

, ( Cov ) , ( Cov X Y Y X  ) ( Var ] [ ] [ ] [ ) , ( Cov

2

X X E X E X E X X    ) , ( Cov ) , ( Cov Y X a Y b aX  

  

   

        

n i m j j i m j j n i i

Y X Y , X

1 1 1 1

) , ( Cov Cov

slide-3
SLIDE 3

3 Variance of Sum of Variables

  • Proof:
  • If all Xi and Xj independent (i  j):

              

  

   n j j n i i n i i

X , X X

1 1 1

Cov Var

   

    

       

n i n i j j i n i i n i i

X X X X

1 1 1 1

) , ( Cov 2 ) ( Var Var



 

n i n j j i X

X

1 1

) , ( Cov

  

   

 

n i n i j j j i n i i

X X X

1 , 1 1

) , ( Cov ) ( Var

  

   

 

n i n i j j i n i i

X X X

1 1 1

) , ( Cov 2 ) ( Var ) ( Var ) , ( Cov X X X  Note:

 

 

      

n i i n i i

X X

1 1

) ( Var Var

) , ( Cov ) , ( Cov

i j j i

X X X X 

By symmetry:

Hola Compadre: La Distribución Binomial

  • Let Y ~ Bin(n, p)
  • n independent trials
  • Let Xi = 1 if i-th trial is “success”, 0 otherwise
  • Xi ~ Ber(p)

E[Xi] = p

  • Var(Y) = Var(X1) + Var(X2) + ... + Var(Xn)
  • Var(Xi) = E[Xi

2] – (E[Xi])2

= E[Xi] – (E[Xi])2 since Xi

2 = Xi

= p – p2 = p(1 – p)

  • Var(Y) = nVar(Xi) = np(1 – p)

Variance of Sample Mean

  • Consider n I.I.D. random variables X1, X2, ... Xn
  • Xi have distribution F with E[Xi] = m and Var(Xi) = s2
  • We call sequence of Xi a sample from distribution F
  • Recall sample mean: where
  • What is ?

n i i

n X X

1

) ( Var X                    

 

  n i i n i i

X n n X X

1 2 1

Var 1 Var ) ( Var

m  ] [X E

2 2 1 2 2 1 2

1 1 ) ( Var 1 s s n n n X n

n i n i i

                    

 

 

n

2

s 

Sample Variance

  • Consider n I.I.D. random variables X1, X2, ... Xn
  • Xi have distribution F with E[Xi] = m and Var(Xi) = s2
  • We call sequence of Xi a sample from distribution F
  • Recall sample mean: where
  • Sample deviation: for i = 1, 2, ..., n
  • Sample variance:
  • What is E[S2]?
  • E[S2] = s 2
  • We say S2 is “unbiased estimate” of s 2

n i i

n X X

1 i

X X 

  

n i i

n X X S

1 2 2

1 ) (

m  ] [X E

Proof that E[S2] = s 2 (just for reference)

  • So, E[S2] = s 2

                  

 

  n i i n i i

X X E S E n n X X E S E

1 2 2 1 2 2

) ( ] [ ) 1 ( 1 ) ( ] [                   

 

  n i i n i i

X X E X X E S E n

1 2 1 2 2

)) ( ) (( ) ( ] [ ) 1 ( m m             

  

   n i i n i n i i

X X X X E

1 1 2 1 2

) )( ( 2 ) ( ) ( m m m m             

 

  n i i n i i

X X X n X E

1 2 1 2

) ( ) ( 2 ) ( ) ( m m m m             

) ( ) ( 2 ) ( ) (

2 1 2

m m m m X n X X n X E

n i i

] ) [( ] ) [( ) ( ) (

2 1 2 2 1 2

X nE X E X n X E

n i i n i i

             

 

 

m m m m

2 2 2 2 2 2

) 1 ( ) ( Var s s s s s s         n n n n n X n n