STAT 113 Hypothesis Testing II The World According to the Null - - PowerPoint PPT Presentation

How Surprised is the Skeptic? Simulating the Skeptic’s World

STAT 113 Hypothesis Testing II

The World According to the Null Hypothesis

Colin Reimer Dawson

October 19, 2020 1 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

How Surprised is the Skeptic? Simulating the Skeptic’s World Randomization Distribution 2 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

The Lady Tasting Tea

At a 1920s party in Cambridge, UK, a lady (Dr. Muriel Bristol, a phycologist, specializing in algae) claimed she could tell whether a cup of tea had been prepared by adding milk before or after the tea was poured. A statistician, Ronald Fisher, also in attendance, proposed a blind taste test w/ 10 cups of tea, each prepared in random order.

• How much success is enough to believe her?

3 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

The Null Hypothesis

• R.A. Fisher: Formulate the “most boring” hypothesis about

the world/process/population (“nothing to see here; moving along”)

• Try to measure how surprising the data would have been if

the “boring” thing were true.

• Fisher called this boring “antihypothesis” the null hypothesis

(abbreviated as H0) 4 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

The Alternative Hypothesis

• Jerzy Neyman and Egon Pearson added the idea of a specific

alternative hypothesis to this formulation

• The “alternative” is usually the one that you started with

H0: the new drug works no better than the old one H1: the new drug works better than the old one H0: there is no relationship between bill and tip percent H1: there is some relationship between bill and tip percent

5 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Hypotheses Are About Parameters

• We know what’s true about our dataset (the sample)
• Our hypotheses propose possibilities involving the wider

context (population/process)

• Important: When formulating statistical hypotheses, they will

always be about the population/process/phenomenon

• Correct H1: A majority of all U.S. registered voters plan to

vote for Biden in November

• Incorrect H1: A majority of the registered voters in the

poll plan to vote for Biden in November

6 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Self-Check: Null and Alternative Hypotheses

For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis (H0), and (d) the alternative hypothesis (H1)

• 1. Claim: There is a positive linear association between pH and

mercury in Florida lakes. 7 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Self-Check: Null and Alternative Hypotheses

For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis (H0), and (d) the alternative hypothesis (H1)

• 1. Claim: There is a positive linear association between pH and

mercury in Florida lakes.

• Data: We measure pH and mercury levels in 50 random lakes

in Florida.

7 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Self-Check: Null and Alternative Hypotheses

For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis (H0), and (d) the alternative hypothesis (H1)

• 1. Claim: There is a positive linear association between pH and

mercury in Florida lakes.

• Data: We measure pH and mercury levels in 50 random lakes

in Florida.

• 2. Claim: Dr. Bristol can tell the difference between milk-first

and tea-first preparations better than a coin flip 7 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Self-Check: Null and Alternative Hypotheses

For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis (H0), and (d) the alternative hypothesis (H1)

• 1. Claim: There is a positive linear association between pH and

mercury in Florida lakes.

• Data: We measure pH and mercury levels in 50 random lakes

in Florida.

• 2. Claim: Dr. Bristol can tell the difference between milk-first

and tea-first preparations better than a coin flip

• Data: She tastes 10 cups in a blind taste test.

7 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Self-Check: Null and Alternative Hypotheses

For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis (H0), and (d) the alternative hypothesis (H1)

• 1. Claim: There is a positive linear association between pH and

mercury in Florida lakes.

• Data: We measure pH and mercury levels in 50 random lakes

in Florida.

• 2. Claim: Dr. Bristol can tell the difference between milk-first

and tea-first preparations better than a coin flip

• Data: She tastes 10 cups in a blind taste test.
• 3. Claim: Lab mice eat more on average if the room is light at

meal time than if it is dark at meal time. 7 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Self-Check: Null and Alternative Hypotheses

For the following research claims and datasets, identify (a) the relevant parameter(s) and the context where it applies (b) the statistic(s) that we can use to estimate the parameter(s), (c) the null hypothesis (H0), and (d) the alternative hypothesis (H1)

• 1. Claim: There is a positive linear association between pH and

mercury in Florida lakes.

• Data: We measure pH and mercury levels in 50 random lakes

in Florida.

• 2. Claim: Dr. Bristol can tell the difference between milk-first

and tea-first preparations better than a coin flip

• Data: She tastes 10 cups in a blind taste test.
• 3. Claim: Lab mice eat more on average if the room is light at

meal time than if it is dark at meal time.

• Data: 20 mice are randomly split into two groups. One group

is fed in the light, another in the dark. Their food intake in grams is measured.

7 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Outline

How Surprised is the Skeptic? Simulating the Skeptic’s World Randomization Distribution 8 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Logic of Testing H0

• Imagine two observers:

9 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Logic of Testing H0

• Imagine two observers:
• a skeptic who thinks there is nothing interesting going on

(they believe H0)

9 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Logic of Testing H0

• Imagine two observers:
• a skeptic who thinks there is nothing interesting going on

(they believe H0)

• a proponent who thinks there is something interesting there

(they believe H1)

9 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Logic of Testing H0

• Imagine two observers:
• a skeptic who thinks there is nothing interesting going on

(they believe H0)

• a proponent who thinks there is something interesting there

(they believe H1)

• Ask which values of the statistic would surprise the

proponent less than they would surprise the skeptic 9 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Logic of Testing H0

• Imagine two observers:
• a skeptic who thinks there is nothing interesting going on

(they believe H0)

• a proponent who thinks there is something interesting there

(they believe H1)

• Ask which values of the statistic would surprise the

proponent less than they would surprise the skeptic

• Of those, sort them in descending order according to how

much they favor the proponent 9 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Logic of Testing H0

• Imagine two observers:
• a skeptic who thinks there is nothing interesting going on

(they believe H0)

• a proponent who thinks there is something interesting there

(they believe H1)

• Ask which values of the statistic would surprise the

proponent less than they would surprise the skeptic

• Of those, sort them in descending order according to how

much they favor the proponent

• If the data yields a statistic which is sufficiently far up that

list, the skeptic will change their mind 9 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

• Dr. Bristol tastes 10 cups of tea and guesses the preparation
• f each. What is the statistic of interest?
• Which values should suprise a proponent less than they

surprise a skeptic?

• Which of those favor the proponent the most?

10 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

The P-value

• The skeptic assigns a weight to every possible value of the

statistic, according to how often that value should occur in the skeptic’s model of the world 11 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

The P-value

• The skeptic assigns a weight to every possible value of the

statistic, according to how often that value should occur in the skeptic’s model of the world

• These weights together must add up to 1

11 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

The P-value

• The skeptic assigns a weight to every possible value of the

statistic, according to how often that value should occur in the skeptic’s model of the world

• These weights together must add up to 1
• Then, sort the statistics according to how much they favor

the proponent’s explanation 11 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

The P-value

• The skeptic assigns a weight to every possible value of the

statistic, according to how often that value should occur in the skeptic’s model of the world

• These weights together must add up to 1
• Then, sort the statistics according to how much they favor

the proponent’s explanation

• Once we have data, start adding up weights from the top of

the list until we hit the observed statistic from our dataset 11 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

The P-value

• The skeptic assigns a weight to every possible value of the

statistic, according to how often that value should occur in the skeptic’s model of the world

• These weights together must add up to 1
• Then, sort the statistics according to how much they favor

the proponent’s explanation

• Once we have data, start adding up weights from the top of

the list until we hit the observed statistic from our dataset

• The number we get to when we stop (will be between 0 and 1)

is called the P-value 11 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Formal Definition: P-value

The probability of obtaining a result that favors H1 over H0 at least as much as what was actually observed, based on H0’s model of potential datasets 12 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Formal Definition: P-value

The probability of obtaining a result that favors H1 over H0 at least as much as what was actually observed, based on H0’s model of potential datasets

Slightly Less Formal Definition: P-value

The combined weight assigned by the skeptic to potential statistics on the list up to and including the statistic we actually got (where the potential statistics have been ordered from most to least favorable to the proponent) 12 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Outline

How Surprised is the Skeptic? Simulating the Skeptic’s World Randomization Distribution 13 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Simulating the Skeptic’s World

• Often we can simulate the world according to the skeptic (H0)

in order to figure out the weights assigned to each potential value of the statistic

• Physical simulations: Coin flips, dice, cards, etc.
• Computer simulations: R, StatKey

14 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Outline

How Surprised is the Skeptic? Simulating the Skeptic’s World Randomization Distribution 15 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Randomization Distribution

A randomization distribution is a hypothetical sampling distribution generated from a population/process which is constructed so as to be consistent with the skeptic’s (H0’s) worldview

• The randomization distribution tells us how likely any

particular value of the statistic of interest would be if the skeptic (H0) were correct 16 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Simulating a Randomization Distribution

• Suppose Dr. Bristol guessed correctly for 9 out of 10 cups of

tea

StatKey

17 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Simulating a Randomization Distribution

• Suppose Dr. Bristol guessed correctly for 9 out of 10 cups of

tea

• Because there are two choices, one of which is correct, if she

were guessing randomly, each cup can be modeled by a coin flip

StatKey

17 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Simulating a Randomization Distribution

• Suppose Dr. Bristol guessed correctly for 9 out of 10 cups of

tea

• Because there are two choices, one of which is correct, if she

were guessing randomly, each cup can be modeled by a coin flip

• We can therefore construct a virtual world in which the

taster is guessing randomly by flipping a coin 10 times

StatKey

17 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Simulating a Randomization Distribution

• Suppose Dr. Bristol guessed correctly for 9 out of 10 cups of

tea

• Because there are two choices, one of which is correct, if she

were guessing randomly, each cup can be modeled by a coin flip

• We can therefore construct a virtual world in which the

taster is guessing randomly by flipping a coin 10 times

• By repeating this simulation several thousand times, we can

see how often each value of the statistic “number of correct responses” occurs

StatKey

17 / 17

How Surprised is the Skeptic? Simulating the Skeptic’s World

Simulating a Randomization Distribution

• Suppose Dr. Bristol guessed correctly for 9 out of 10 cups of

tea

• Because there are two choices, one of which is correct, if she

were guessing randomly, each cup can be modeled by a coin flip

• We can therefore construct a virtual world in which the

taster is guessing randomly by flipping a coin 10 times

• By repeating this simulation several thousand times, we can

see how often each value of the statistic “number of correct responses” occurs

• Then, we can get a P-value by finding the proportion of

those simulations that yield between and correct guesses

StatKey

17 / 17