Sample Size and Power Calculations IPA/JPAL/CMF Training Limuru, - PowerPoint PPT Presentation

Motivation Probability basics Power calculation Exercises Sample Size and Power Calculations IPA/JPAL/CMF Training Limuru, Kenya 28 July 2010 Owen Ozier Department of Economics University of California at Berkeley Slides revised 14 September 2010 Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Thanks and Introduction Thanks to everyone from JPAL/IPA who made this happen! Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Thanks and Introduction Thanks to everyone from JPAL/IPA who made this happen! My background: randomized evaluations in Busia, Kenya. Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Motivation Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Motivation Program evaluation: bringing the scientific method to social science Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Motivation Program evaluation: bringing the scientific method to social science First steps: Propose a hypothesis Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Motivation Program evaluation: bringing the scientific method to social science First steps: Propose a hypothesis Design an experiment to test the hypothesis Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Motivation Program evaluation: bringing the scientific method to social science First steps: Propose a hypothesis Design an experiment to test the hypothesis This involves gathering data... Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Motivation Program evaluation: bringing the scientific method to social science First steps: Propose a hypothesis Design an experiment to test the hypothesis This involves gathering data... ...but how much data will we need? Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Usually a lot Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises How this will work “Numerical data should be kept for eternity; it’s great stuff.” - Glenn Stevens, Boston University Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Outline Motivation 1 Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Outline Motivation 1 Probability basics 2 Coin tossing Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Outline Motivation 1 Probability basics 2 Coin tossing Power calculation 3 Terminology/Concepts The Basic Calculation Clusters Covariates Details Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Outline Motivation 1 Probability basics 2 Coin tossing Power calculation 3 Terminology/Concepts The Basic Calculation Clusters Covariates Details Exercises 4 Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing A hypothesis and a kind of test Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing A hypothesis and a kind of test “Null” Hypothesis: the coin is fair 50% chance of heads, 50% chance of tails. Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing A hypothesis and a kind of test “Null” Hypothesis: the coin is fair 50% chance of heads, 50% chance of tails. Structure of the data: Toss the coin a number of times, count heads. Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing A hypothesis and a kind of test “Null” Hypothesis: the coin is fair 50% chance of heads, 50% chance of tails. Structure of the data: Toss the coin a number of times, count heads. The test: “Accept” hypothesis if within some distance of the mean under the null; “Reject” otherwise. Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing A hypothesis and a kind of test “Null” Hypothesis: the coin is fair 50% chance of heads, 50% chance of tails. Structure of the data: Toss the coin a number of times, count heads. The test: “Accept” hypothesis if within some distance of the mean under the null; “Reject” otherwise. If we only had 4 tosses of the coin, what distance cutoffs could we use? Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing A hypothesis and a kind of test “Null” Hypothesis: the coin is fair 50% chance of heads, 50% chance of tails. Structure of the data: Toss the coin a number of times, count heads. The test: “Accept” hypothesis if within some distance of the mean under the null; “Reject” otherwise. If we only had 4 tosses of the coin, what distance cutoffs could we use? Could accept (A) never, (B) when exactly the mean (2 heads), (C) when within 1 (1, 2, or 3 heads), or (D) always. Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing A hypothesis and a kind of test “Null” Hypothesis: the coin is fair 50% chance of heads, 50% chance of tails. Structure of the data: Toss the coin a number of times, count heads. The test: “Accept” hypothesis if within some distance of the mean under the null; “Reject” otherwise. If we only had 4 tosses of the coin, what distance cutoffs could we use? Could accept (A) never, (B) when exactly the mean (2 heads), (C) when within 1 (1, 2, or 3 heads), or (D) always. We don’t want to reject the null when it is true, though; How much accidental rejection would each possible cutoff give us? Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing Distribution of possible results Distribution of numbers of heads in 4 tosses of a fair coin .4 0.38 .3 0.25 0.25 Probability .2 .1 0.06 0.06 0 0 1 2 3 4 P(2)=.38; P(1...3)=.88; P(0...4)=1 Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing Not enough data. Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing Not enough data. There is no way* to create such a test with four coin tosses so that the chance of accidental rejection under the “null” hypothesis (sometimes written H 0 ) is less than 5%, a standard in social science. Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing Not enough data. There is no way* to create such a test with four coin tosses so that the chance of accidental rejection under the “null” hypothesis (sometimes written H 0 ) is less than 5%, a standard in social science. * (Except the “never reject, no matter what” rule. Not very useful.) Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing Not enough data. There is no way* to create such a test with four coin tosses so that the chance of accidental rejection under the “null” hypothesis (sometimes written H 0 ) is less than 5%, a standard in social science. * (Except the “never reject, no matter what” rule. Not very useful.) What about 20 coin tosses? Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing Distribution of possible results Distribution of numbers of heads in 20 tosses of a fair coin .2 0.18 0.16 0.16 .15 0.12 0.12 Probability .1 0.07 0.07 .05 0.04 0.04 0.01 0.01 0.000.000.000.000.00 0.000.000.000.000.00 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 P(10)=.18; P(9...11)=.5; P(8...12)=.74; P(7...13)=.88; P(6...14)=.96 Owen Ozier Sample Size and Power Calculations

Motivation Probability basics Power calculation Exercises Coin tossing The normal distribution Distribution (limiting) of any well−behaved residual error .4 Probability density .3 .2 .1 0 −4 −2 0 2 4 Deviations from mean Owen Ozier Sample Size and Power Calculations

As sample size increases more: Motivation Probability 0 .02 .04 .06 .08 Probability basics 0 1 2 3 4 5 6 Distribution of numbers of heads in 100 tosses of a fair coin 7 8 9 10 11 12 13 14 Power calculation 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Owen Ozier 31 32 33 Exercises 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 Sample Size and Power Calculations 50 Coin tossing 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100