Which Multiple Testing Methods are Optimal? Peter H. Westfall, - - PowerPoint PPT Presentation

▶

Jan 07, 2024 275 likes •490 views

Which Multiple Testing Methods are Optimal? Peter H. Westfall, Texas Tech University Background The scientific literature has recently experienced an embarrassment of contradictory results: Ioannidis, J.P. (2005), "Contradicted

SLIDE 1

Which Multiple Testing Methods are Optimal?

Peter H. Westfall, Texas Tech University

SLIDE 2

Background

The scientific literature has recently

experienced an embarrassment of contradictory results:

Ioannidis, J.P. (2005), "Contradicted and Initially Stronger Effects in Highly

Cited Clinical Research,"J. Amer. Med. Assoc. 294, 218--228.

Bertram, L., McQueen, M. B., Mullin, K., Blacker, D., and Tanzi, R. E.

(2007), "Systematic Meta-analyses of Alzheimer Disease Genetic Association Studies: the AlzGene Database," Nature Genetics 39, 17--23.

Boffetta, P., McLaughlin, J.K., La Vecchia, C., Tarone, R.E., Lipworth, L.,

Blot, W. J., (2008), "False-Positive Results in Cancer Epidemiology: A Plea for Epistemological Modesty," J. Nat. Cancer Inst. 100, 988--995.

SLIDE 3

Goals

Compare fixed critical value methods in

terms of loss Q: Does m matter? Do data correlations matter? A: It depends on how you feel about type I versus type II errors (i.e., relative costs)

SLIDE 4

Background

“Lehmann (1957a,b) was the first to

consider multiple comparisons from a decision-theoretic viewpoint.”

– Hochberg and Tamhane (1987), Multiple Comparisons Procedures (Wiley)

SLIDE 5

Data Setup of this Talk

Data: z | θ ~Nm(θ, ρ), ρ a correlation matrix. Model: θi ~iid N(0, σ2), σ2 known.

SLIDE 6

Decision Theory

Lehmann (1957a,b) Annals
Hochberg and Tamhane (1987)
Three-decision problem: Decide either

– GT: θi > 0 – LT: θi < 0, or – NI: θi ~ 0 (or “EM”)

SLIDE 7

A Component Loss Function

LGT(θ) , LLT(θ) , LNI(θ); for example:
0.2

0.2 0.4 0.6 0.8 1 1.2

Loss

L_NI L_GT

A 1

SLIDE 8

Actual and Expected Loss

Actual loss using method “M”:
Expected Loss:
Combined Loss: (additive!?)

(M)( ,

) ( | ) ( ) ( | ) ( ) ( | ) ( )

i i i GT i i LT i i NI i

L I GT L I LT L I NI L θ θ θ θ = + + z z z z

( )

(M) (M) ,

( , )

i i i

E L

θ Ψ =

(M) (M) i

Ψ = Ψ

∑

SLIDE 9

Decision Rules

Decide

– LT if zi < −c – GT if zi > c – NI if −c ≤ zi ≤ c

If ρ = I, then c = (1 + 1/ σ2)z1-A is optimal.

⇒ For Bonferroni-like procedures to be

ptimal, A=A(m).

SLIDE 10

Does m Matter?

Theorem: If A(m) = o(1) and 1/A(m) =
(m {ln(m)}1/2), then Ψ(Bon) ~ Ψ(Optimal) .

⇒ If the loss of a single Type I error equals βm Type II errors (0<β<1), then Bonferroni is optimal and fixed significance level procedures (like FDR) are inadmissible.

Lu, Y., and Westfall, P. (2009). Is Bonferroni Admissible for Large m? American Journal of Mathematical and Management Sciences, Vol. 29 (1&2), 51-69.

SLIDE 11

From Lu, Y., and Westfall, P. (2009). Is Bonferroni Admissible for Large m?

SLIDE 12

Do Data Correlations Matter?

“Reject Hi” if |zi|>c, i=1,…,m. Let V = number of false discoveries. With higher correlations among z’s:

E(V) is unaffected
P(V>0) is lower (smaller FWER)
Var(V) is higher (potentially high # of false

discoveries)

SLIDE 13

Effect of Correlation with Additive Loss

No affect on expected value ⇒ optimal c

not affected

Affects percentiles ⇒ optimal c is affected

VaR = “Value at risk”=95th pctle of Loss (finance)

SLIDE 14

A Model for Studying Effect of Correlation

Suppose z |θ ~ Nm(θ, ρ), with ρ =λλ′ + ψ2 , λ (m x1) and ψ2 diagonal. Then ρij = λiλi. Let , where Ui ~iid U(−1,1). Then E(ρij)=0 and

2 2 1/2

/ ( )

i i i

U U s λ = +

{ }

1/2 2 1

rmsc ( ) 1 tan (1/ ).

E s s ρ

−

≡ = −

SLIDE 15

Waller-Duncan Loss

Loss

Loss(NI) Loss(GT)

LGT(θ) = −(K+1)θ, θ < 0; LGT(θ) = 0, o/w. LNI(θ) = |θ | .

SLIDE 16

90th Pctle-Minimizing Optimal c, K=100

SLIDE 17

Should Loss Be Additive?

Is the cost difference between 10 and 11

Extraterrestrial Intelligence claims the same as the cost difference between 0 and 1?

Is the cost difference between 10 and 11

shouts of “fire” in a crowded theater the same as the cost difference between 0 and 1?

SLIDE 18

‘Fire-In-The-Theater’ Loss Function

Let n1 = # Directional Errors Let n2 = # “Not Interesting” claims L1 = n1/(n1 + 1) L2 = 1/(m − n2 +1) − 1/(m +1) “Fire in the Theater” Loss = L1 + L2

SLIDE 19

Fire-In-The-Theater Loss Function Components, m=100

SLIDE 20

Expected Value-Minimizing Optimal c for Fire-In-The-Theater Loss Function

Which Multiple Testing Methods are Optimal?

Peter H. Westfall, Texas Tech University

Background

experienced an embarrassment of contradictory results:

Goals

terms of loss Q: Does m matter? Do data correlations matter? A: It depends on how you feel about type I versus type II errors (i.e., relative costs)

Background

consider multiple comparisons from a decision-theoretic viewpoint.”

– Hochberg and Tamhane (1987), Multiple Comparisons Procedures (Wiley)

Data Setup of this Talk

Data: z | θ ~Nm(θ, ρ), ρ a correlation matrix. Model: θi ~iid N(0, σ2), σ2 known.

Decision Theory

– GT: θi > 0 – LT: θi < 0, or – NI: θi ~ 0 (or “EM”)

A Component Loss Function

Actual and Expected Loss

) ( | ) ( ) ( | ) ( ) ( | ) ( )

L I GT L I LT L I NI L θ θ θ θ = + + z z z z

( )

∑

Decision Rules

– LT if zi < −c – GT if zi > c – NI if −c ≤ zi ≤ c

⇒ For Bonferroni-like procedures to be

Does m Matter?

⇒ If the loss of a single Type I error equals βm Type II errors (0<β<1), then Bonferroni is optimal and fixed significance level procedures (like FDR) are inadmissible.

Do Data Correlations Matter?

“Reject Hi” if |zi|>c, i=1,…,m. Let V = number of false discoveries. With higher correlations among z’s:

discoveries)

Effect of Correlation with Additive Loss

not affected

VaR = “Value at risk”=95th pctle of Loss (finance)

A Model for Studying Effect of Correlation

{ }

Waller-Duncan Loss

LGT(θ) = −(K+1)θ, θ < 0; LGT(θ) = 0, o/w. LNI(θ) = |θ | .

90th Pctle-Minimizing Optimal c, K=100

Should Loss Be Additive?

Extraterrestrial Intelligence claims the same as the cost difference between 0 and 1?

shouts of “fire” in a crowded theater the same as the cost difference between 0 and 1?

‘Fire-In-The-Theater’ Loss Function

Let n1 = # Directional Errors Let n2 = # “Not Interesting” claims L1 = n1/(n1 + 1) L2 = 1/(m − n2 +1) − 1/(m +1) “Fire in the Theater” Loss = L1 + L2

Fire-In-The-Theater Loss Function Components, m=100

Expected Value-Minimizing Optimal c for Fire-In-The-Theater Loss Function

Conclusions

If Type I errors are serious then:

with higher data correlation.