Business Statistics CONTENTS Two types of error The power of a - - PowerPoint PPT Presentation

POWER AND DESIGN

Business Statistics

Two types of error The power of a test Experimental design Choosing sample size Power curves Power and “big data” Old exam question Further study CONTENTS

What is the precise meaning of the significance level 𝛽? ▪ 𝛽 is the maximum acceptable probability of rejecting the null hypothesis when it is in fact true ▪ so 𝑄 reject 𝐼0 𝐼0 There are two possible decisions: ▪ reject 𝐼0 ▪ do not reject 𝐼0 And there are two possible realities: ▪ 𝐼0 is true ▪ 𝐼0 is not true TWO TYPES OF ERROR

To be read as: “the conditional probability

• f rejecting 𝐼0, given that

𝐼0 is true”

Organize the situations in a 2 × 2-table: Correct decision ▪ no further concern, because OK Wrong decision ▪ type I error ▪ type II error TWO TYPES OF ERROR

Probability of a type I error: 𝑄 reject 𝐼0 𝐼0 ≤ 𝛽

▪ conventionally 𝛽 = 5% ▪ but you may choose another significance level if you think that is appropriate ▪ e.g., aircraft safety: better use 𝛽 = 1% or even 𝛽 = 0.1% ▪ e.g., choosing a colour of your shampoo flask: 𝛽 = 10% is OK

▪ Anyhow, you control the maximum type I error by choosing 𝛽 in advance

▪ and accept that you will once in a while reject 𝐼0 while it is true

TWO TYPES OF ERROR

Probability of a type II error: 𝑄 do not reject 𝐼0 specific 𝐼1 = 𝛾

▪ how to choose 𝛾?

▪ You cannot simply control the maximum type II error 𝛾

▪ because it depends on the true value of the unknown (!) parameter (that is to be tested itself) ▪ as well as on 𝛽 ▪ but it could be good to know it, at least ▪ usually, this is done through the power concept

TWO TYPES OF ERROR

for instance, 𝐼0: 𝜈 = 10 vs. 𝐼1: 𝜈 = 12

Controlling 𝛽 and 𝛾 is important ▪ Example: airport security

▪ 𝐼0: bag does not contain a weapon

▪ Type I error:

▪ the bag did not contain a weapon, but the machine said it did ▪ loss of time in manual search, offended clients, delayed flights ▪ management will try to minimize the probability of this error type

▪ Type II error:

▪ the bag did contain a weapon, but the machine did not detect it ▪ hijacks, loss of crew and aircraft, liability claims, loss of credibility ▪ management will try to minimize the probability of this error type

TWO TYPES OF ERROR

There is a trade-off: ▪ minimizing 𝛽 typically leads to increasing 𝛾 ▪ minimizing 𝛾 typically leads to increasing 𝛽 𝛽 and 𝛾 can only be decreased simultaneously by changing the set-up of the research ▪ most importantly, by increasing sample size TWO TYPES OF ERROR

What error do we make?

• a. A population has 𝜈 = 120. We reject 𝐼0: 𝜈 ≤ 125.
• b. A population has 𝜌 = 0.3. We do not reject 𝐼0: 𝜌 ≤ 0.4.
• c. A population has 𝜏2 = 2.5. We do not reject 𝐼0: 𝜏2 ≤ 2.

EXERCISE 1

In business and politics, a lot depends on what clients, the market and the public require So you do experiments: ▪ market surveys ▪ questionnaires ▪ customer cards ▪ polls ▪ website analysis (tracking cookies, etc) ▪ etc. EXPERIMENTAL DESIGN

How to set up such experiments ▪ qualitative research methods (interview techniques, etc.) ▪ quantitative research methods (choosing sample size, etc.) Here we will focus on sample size for 𝜈 and 𝜌 EXPERIMENTAL DESIGN

Recall that the confidence interval of a mean 𝜈 is ҧ 𝑦 − 𝑨𝛽/2 𝜏 𝑜 , ҧ 𝑦 + 𝑨𝛽/2 𝜏 𝑜 ▪ This means that the width of the confidence interval scales with a factor

1 𝑜

If we need to estimate 𝜈 with a (minimal) precision of ±𝐹 (the allowable error), you would need a certain (minimal) sample size CHOOSING SAMPLE SIZE

This gives 𝐹 = 𝑨𝛽/2 𝜏 𝑜

▪ so

𝑜 = 𝑨𝛽/2 𝜏 𝐹

2

Example ▪ to realize a 95% confidence interval for the mean of a population with 𝜏 = 3 with precision 𝐹 = 1, 𝑜 = 34.5, so use 𝑜 = 35 CHOOSING SAMPLE SIZE

always round to the higher value in determining sample size

Observe that we need to know 𝜏 ▪ how to know it? Three suggestions: ▪ take a small preliminary sample and use the sample 𝑡 instead of 𝜏 in the formula ▪ estimate rough upper and lower limits 𝑏 and 𝑐 and set 𝜏 =

𝑐−𝑏 12

▪ estimate rough upper and lower limits 𝑏 and 𝑐 and set 𝜏 =

𝑐−𝑏 4

CHOOSING SAMPLE SIZE

based on the fact that most of the values od a normal distribution are between 𝜈 − 2𝜏 and 𝜈 + 2𝜏 based on a uniform distribution

Likewise, the confidence interval of a proportion 𝜌 is 𝑞 − 𝑨𝛽/2 𝜌 1 − 𝜌 𝑜 , 𝑞 + 𝑨𝛽/2 𝜌 1 − 𝜌 𝑜 This gives 𝐹 = 𝑨𝛽/2 𝜌 1 − 𝜌 𝑜 ▪ so 𝑜 = 𝜌 1 − 𝜌 𝑨𝛽/2 𝐹

2

CHOOSING SAMPLE SIZE

Observe that we need to know 𝜌 ▪ so to determine the sample size to estimate 𝜌, you need 𝜌 Four suggestions ▪ take a small preliminary sample and use the sample 𝑞 instead of 𝜌 in the sample size formula ▪ take a small preliminary sample, find a confidence interval and from this interval use the value closest to 0.5 instead

• f 𝜌 in the sample size formula

▪ use a prior sample or historical data ▪ assume that 𝜌 = 0.50 CHOOSING SAMPLE SIZE

this conservative method ensures the desired precision (𝜌 1 − 𝜌 has a maximum at 𝜌 = 0.50)

We ask a sample of persons if they are in favor or against

• Brexit. We want to deduce the proportion in favor, with a

margin of no more than ±2%. What sample size to use? EXERCISE 2

▪ Suppose we test a one-sample mean

▪ the null hypothesis is 𝐼0: 𝜈 = 𝜈ℎ𝑧𝑞 = 3 ▪ at a significance level 𝛽 = 0.05

▪ If the true parameter is 𝜈 = 𝜈𝑢𝑠𝑣𝑓 = 3

▪ there is a probability 𝛽 = 0.05 to reject 𝐼0 ▪ so this is the probability to make a type I error

POWER

▪ But if the true parameter is 𝜈 = 𝜈𝑢𝑠𝑣𝑓 = 3.1 instead

▪ there is a larger probability to reject 𝐼0 ▪ which is the correct decision ▪ how large depends on 𝜏 and 𝑜 (recall 𝑨𝛽/2

𝜏 𝑜)

▪ And if the true parameter is 𝜈 = 𝜈𝑢𝑠𝑣𝑓 = 10 instead

▪ there is an even larger probability to reject 𝐼0 ▪ which is the correct decision

POWER

So, the probability of a rejecting an incorrect 𝐼0 on the mean depends on ▪ the pre-defined probability 𝛽 of not rejecting a correct 𝐼0 ▪ the sample size 𝑜 ▪ the standard deviation of the popolution 𝜏 ▪ the difference between the hypothesized 𝜈 (𝜈ℎ𝑧𝑞) and the true 𝜈 (𝜈𝑢𝑠𝑣𝑓) POWER

Power is defined as the probability of rejecting 𝐼0 when it should indeed be rejected

▪ when a specific 𝐼1 is true

So: power = 𝑄 reject 𝐼0 specific 𝐼1 Therefore: power = 1 − 𝑄 do not reject 𝐼0 specific 𝐼1 = 1 − 𝛾 POWER

To calculate the power of a test for the mean, you need ▪ the significance level 𝛽 (you choose it) ▪ the sample size 𝑜 (you choose it) ▪ the standard deviation 𝜏 ▪ the hypothesized mean 𝜈ℎ𝑧𝑞 (you choose it) ▪ the true mean 𝜈𝑢𝑠𝑣𝑓 (you have no clue) Therefore, we typically do not calculate power ▪ but rather calculate a power function or power curves for different values of 𝜈𝑢𝑠𝑣𝑓 POWER

For a fixed 𝐼0: 𝜈 = 𝜈ℎ𝑧𝑞 what happens for different values

• f 𝜈𝑢𝑠𝑣𝑓

POWER CURVES

1 0    1 P (R e j e c t H o ) 

 = P (t y p e I I e r r o r )

𝜈0 = 𝜈ℎ𝑧𝑞

• ne specific

𝜈1 = 𝜈𝑢𝑠𝑣𝑓 𝜈-axis: different

• ptions for 𝜈𝑢𝑠𝑣𝑓

Effect of different values of 𝜈𝑢𝑠𝑣𝑓: 𝜈𝑢𝑠𝑣𝑓 = 6 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4

 =3

 =6

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.05

Effect of different values of 𝜈𝑢𝑠𝑣𝑓: 𝜈𝑢𝑠𝑣𝑓 = 5 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4

 =3

 =5

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.05

Effect of different values of 𝜈𝑢𝑠𝑣𝑓: 𝜈𝑢𝑠𝑣𝑓 = 4 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4

 =3

 =4

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.05

Effect of different values of 𝜈𝑢𝑠𝑣𝑓: 𝜈𝑢𝑠𝑣𝑓 = 3.4 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4

 =3 =3.4

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.05

Effect of different values of 𝛽: 𝛽 = 0.05 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4

 =3

 =5.5

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.05

Effect of different values of 𝛽: 𝛽 = 0.1 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4

 =3

 =5.5

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.1

Effect of different values of 𝛽: 𝛽 = 0.27 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4

 =3

 =5.5

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.27

Effect of different values of 𝛽: 𝛽 = 0.05 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4

 =3

 =5.5

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.05

Effect of different values of 𝑜: 𝑜 = 100 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4

 =3

 =5

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.05

Effect of different values of 𝑜: 𝑜 = 200 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4

 =3

 =5

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.05

Effect of different values of 𝑜: 𝑜 = 300 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4 0.5

 =3

 =5

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.05

Effect of different values of 𝑜: 𝑜 = 400 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.1 0.2 0.3 0.4 0.5

 =3

 =5

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.05

Effect of different values of 𝑜: 𝑜 = 500 POWER CURVES

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6

 =3

 =5

• 1

1 2 3 4 5 6 7 8 9 10 0.2 0.4 0.6 0.8 1 p o w e r T W O - S I D E D T E S T I N G

 =0.05

We perform a 𝑨-test for 𝐼0: 𝜈 = 10 (𝛽 = 0.05, 𝑜 = 100, 𝜏 = 2).

• a. What is the rejection region?
• b. What is the power of this test when 𝜈𝑢𝑠𝑣𝑓 = 10.1?

EXERCISE 3

▪ If sample size 𝑜 gets very large (“big data”)

▪ the standard error of the estimate gets very small ▪ confidence intervals get very narrow ▪ an exact null hypothesis (e.g., 𝜈 = 5 or 𝜌 = 0.50) is easily rejected ▪ an exact two-sample test (e.g., 𝜈1 = 𝜈2) is easily rejected ▪ every regression model becomes easily significant, even with low 𝑆2 ▪ etc.

▪ So, “statistical significance” becomes almost meaningless

▪ once more, also consider effect size ▪ which is more like practical signficance

POWER AND “BIG DATA”

26 March 2015, Q1n OLD EXAM QUESTION

Doane & Seward 5/E 8.8-8.9, 9.2, 9.7 Tutorial exercises week 6 errors and power power of 𝑨 test sample size FURTHER STUDY