Probability and Statistics for Computer Science In sta(s(cs we - PowerPoint PPT Presentation

Probability and Statistics ì for Computer Science “In sta(s(cs we apply probability to draw conclusions from data.” ---Prof. J. Orloff Credit: wikipedia Hongye Liu, Teaching Assistant Prof, CS361, UIUC, 10.13.2020

Last time ✺ Exponen(al Distribu(on ✺ Sample mean and confidence interval

Objectives ✺ Bootstrap simula(on ✺ Hypothesis test

Motivation of sampling: the poll example Source: FiveThirtyEight.com ✺ This senate elec(on poll tells us: ✺ The sample has 1211 likely voters ✺ Ms. Hyde-Smith has realized sample mean equal to 51% ✺ What is the es(mate of the percentage of votes for Hyde-smith? ✺ How confident is that es(mate?

Expected value of one random sample is the population mean ✺ Since each sample is drawn uniformly from the popula(on E [ X (1) ] = popmean ( { X } ) therefore E [ X ( N ) ] = popmean ( { X } ) ✺ We say that is an unbiased es(mator of the X ( N ) popula(on mean.

Standard deviation of the sample mean ✺ We can also rewrite another result from the lecture on the weak law of large numbers var [ X ( N ) ] = popvar ( { X } ) N ✺ The standard devia(on of the sample mean std [ X ( N ) ] = popsd ( { X } ) √ N ✺ But we need the popula(on standard devia(on in order to calculate the ! std [ X ( N ) ]

Unbiased estimate of population standard deviation & Stderr ✺ The unbiased es(mate of is popsd ( { X } ) defined as � 1 � stdunbiased ( { x } ) = ( x i − mean ( { x i } )) 2 N − 1 x i ∈ sample ✺ So the standard error is an es(mate of std [ X ( N ) ] std [ X ( N ) ] = popsd ( { X } ) √ N popsd ( { X } ) = stdunbiased ( { x } ) . = stderr ( { x } ) x √ √ N N

Standard error: election poll 51% ✺ What is the es(mate of the percentage of votes 51% for Hyde-smith? Number of sampled voters who selected Ms. Smith is: 1211(0.51) ≅ 618 Number of sampled voters who didn’t selected Ms. Smith was 1211(0.49) ≅ 593

Standard error: election poll ✺ stdunbiased ( { x } ) � 1 1211 − 1(618(1 − 0 . 51) 2 + 593(0 − 0 . 51) 2 ) = 0 . 5001001 = ✺ stderr ( { x } ) 0 . 5 1211 ≃ 0 . 0144 √ ≃

Interpreting the standard error ✺ Sample mean is a random variable and has its own probability distribu(on, stderr is an es(mate of sample mean’s standard devia(on ✺ When N is very large, according to the Central Limit Theorem , sample mean is approaching a normal distribu(on with σ = popsd ( { X } ) . µ = popmean ( { X } ) ; = stderr ( { x } ) x √ N stderr ( { x } ) = stdunbiased ( { x } ) √ N

Interpreting the standard error Probability 99.7% distribu(on 95% of sample 68% mean tends normal when N is large Credit: wikipedia Popula(on μ+Standard error mean

Confidence intervals ✺ Confidence interval 95% 0.5 for a popula(on mean 0.4 is defined by frac(on 0.3 dnorm(x) ✺ Given a percentage, 0.2 find how many units of 0.1 strerr it covers. 0.0 − 4 − 2 0 2 4 -2 2 x For 95% of the realized sample means , the popula(on mean lies in [sample mean-2 stderr, sample mean+2 stderr]

Confidence intervals when N is large ✺ For about 68% of realized sample means mean ( { x } ) − stderr ( { x } ) ≤ popmean ( { X } ) ≤ mean ( { x } ) + stderr ( { x } ) ✺ For about 95% of realized sample means mean ( { x } ) − 2 stderr ( { x } ) ≤ popmean ( { X } ) ≤ mean ( { x } )+2 stderr ( { x } ) ✺ For about 99.7% of realized sample means mean ( { x } ) − 3 stderr ( { x } ) ≤ popmean ( { X } ) ≤ mean ( { x } )+3 stderr ( { x } )

Q. Confidence intervals ✺ What is the 68% confidence interval for a popula(on mean? A. [sample mean-2stderr, sample mean+2stderr] B. [sample mean-stderr, sample mean+stderr] C. [sample mean-std, sample mean+std]

Standard error: election poll 51% ✺ We es(mate the popula(on mean as 51% with stderr 1.44% ✺ The 95% confidence interval is [51%-2×1.44%, 51%+2×1.44%]= [48.12%, 53.88%]

Q. ✺ A store staff mixed their fuji and gala apples and they were individually wrapped, so they are indis(nguishable. if I pick 30 apples and found 21 fuji , what is my 95% confidence interval to es(mate the popmean is 70% for fuji? (hint: strerr > 0.05) A. [0.7-0.17, 0.7+0.17] B. [0.7-0.056, 0.7+0.056]

What if N is small? When is N large enough? ✺ If samples are taken from normal distributed popula(on, the following variable is a random variable whose distribu(on is Student’s t - distribu(on with N -1 degree of freedom. T = mean ( { x } ) − popmean ( { X } ) stderr ( { x } ) Degree of freedom is N -1 due to this constraint: � ( x i − mean ( { x } )) = 0 i

t-distribution is a family of distri. with different degrees of freedom t-distribu(on with N=5 pdf of t − distribution and N=30 0.5 degree = 4, N=5 degree = 29, N=30 0.4 0.3 density 0.2 0.1 Credit : wikipedia 0.0 William Sealy Gosset 1876-1937 − 10 − 5 0 5 10 X

When N=30, t-distribution is almost Normal pdf of t (n=30) and normal distribution 0.5 t-distribu(on looks very degree = 29, N=30 standard normal similar to normal 0.4 when N=30. 0.3 So N=30 is a rule of density thumb to decide N is 0.2 large or not 0.1 0.0 − 10 − 5 0 5 10 X

Confidence intervals when N< 30 ✺ If the sample size N< 30, we should use t- distribu(on with its parameter (the degrees of freedom) set to N-1

Centered Confidence intervals ✺ Centered Confidence 0.5 interval for a 0.4 popula(on mean by 0.3 dnorm(x) α value, where 0.2 0.1 P ( T ≥ b ) = α 0.0 − 4 − 2 0 2 4 α α x For 1-2α of the realized sample means, the popula(on mean lies in [sample mean- b ×stderr, sample mean+ b ×stderr]

Centered Confidence intervals ✺ Centered Confidence 0.5 interval for a 0.4 popula(on mean by 0.3 dnorm(x) α value, where 0.2 0.1 P ( T ≥ b ) = α 0.0 − 4 − 2 0 2 4 α α x For 1-2α of the realized sample means, the popula(on mean lies in [sample mean- b ×stderr, sample mean+ b ×stderr]

Q. ✺ The 95% confidence interval for a popula(on mean is equivalent to what 1-2α interval? A. α= 0.05 B. α= 0.025 C. α= 0.1

Sample statistic ✺ A staQsQc is a func(on of a dataset ✺ For example, the mean or median of a dataset is a sta(s(c ✺ Sample staQsQc ✺ Is a sta(s(c of the data set that is formed by the realized sample ✺ For example, the realized sample mean

Q. Is this a sample statistic? ✺ The largest integer that is smaller than or equal to the mean of a sample A. Yes B. No.

Q. Is this a sample statistic? ✺ The interquar(le range of a sample A. Yes B. No.

Confidence intervals for other sample statistics ✺ Sample staQsQc such as median and others are also interes(ng for drawing conclusion about the popula(on ✺ It’s osen difficult to derive the analy(cal expression in terms of stderr for the corresponding random variable ✺ So we can use simula(on…

Bootstrap for confidence interval of other sample statistics ✺ Bootstrap is a method to construct confidence interval for any * sample staQsQcs using resampling of the sample data set ✺ Bootstrapping is essen(ally uniform random sampling with replacement on the sample of size N

Bootstrap for confidence interval of other sample statistics Credit: E S. Banjanovic and J. W. Osborne, 2016, PAREonline

Example of Bootstrap for confidence interval of sample median ✺ The realized sample of student awendance {12,10,9,8,10,11,12,7,5,10}, N =10, median=10 ✺ Generate a random index uniformly from [1,10] that correspond to the 10 numbers in the sample, ie. if index=6, the bootstrap sample’s number will be 11. ✺ Repeat the process 10 (mes to get one bootstrap sample Bootstrap replicate Sample median {11, 11, 12, 10, 10, 10, 12, 10, 7, 10} 10

Example of Bootstrap for confidence interval of sample median ✺ The realized sample of student awendance {12,10,9,8,10,11,12,7,5,10}, N =10, median=10 Bootstrap replicate Sample median {11, 11, 12, 10, 10, 10, 12, 10, 7, 10} 10 {7, 10, 10, 10, 9, 7, 9, 10, 12, 10} 10 {9, 7, 10, 8, 5, 10, 7, 10, 12, 8} 8.5 … …

Q. How many possible bootstrap replicates? ✺ A. 10 10 B.10! C. e 10 Bootstrap replicate Sample median {11, 11, 12, 10, 10, 10, 12, 10, 7, 10} 10 {7, 10, 10, 10, 9, 7, 9, 10, 12, 10} 10 {9, 7, 10, 8, 5, 10, 7, 10, 12, 8} 8.5 … …

Example of Bootstrap for confidence interval of sample median ✺ Do the bootstrapping for r = 10000 (mes, then draw the histogram and also find the stderr of sample median) Bootstrap replicate Sample median {11, 11, 12, 10, 10, 10, 12, 10, 7, 10} 10 {7, 10, 10, 10, 9, 7, 9, 10, 12, 10} 10 {9, 7, 10, 8, 5, 10, 7, 10, 12, 8} 8.5 … …

Example of Bootstrap for confidence interval of sample median ✺ Bootstrapping Histogram of sample_median for r = 10000 5000 (mes, then draw Is this similar to 4000 the histogram Normal? and also find the 3000 Frequency stderr of sample 2000 median. 1000 �� i [ S ( { x } i ) − S ] 2 stderr ( { S } ) = r − 1 0 mean(Sample Median) = 9.73625 5 6 7 8 9 10 11 12 sample_median stderr(Sample Median) = 0.7724446