Cherry-picking Multiple Testing for Exploratory Research Jelle - - PowerPoint PPT Presentation

▶
cherry picking multiple testing for exploratory research
SMART_READER_LITE
LIVE PREVIEW

Cherry-picking Multiple Testing for Exploratory Research Jelle - - PowerPoint PPT Presentation

Oct 23, 2023 365 likes •802 views

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman Aldo Solari Leiden University Medical Center University of Milano-Bicocca Journ

slide-1
SLIDE 1

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Cherry-picking Multiple Testing for Exploratory Research

Jelle Goeman Aldo Solari

Leiden University Medical Center University of Milano-Bicocca

Journ´ ees de Statistique, 2012-05-24

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-2
SLIDE 2

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

A genomics data analysis result

Top 10 genes Gene p-value multiplicity-corrected p-value OCIAD2 5.5e-6 0.015 NEK3 6.7e-6 0.019 TAF5 7.1e-6 0.020 FOXD4L6 7.5e-6 0.021 ADIG 8.8e-6 0.025 ZNF19 1.3e-5 0.038 ERICH1 1.5e-5 0.044 SKP1 1.7e-5 0.050 GDF3 2.0e-5 0.059 CCDC25 2.0e-5 0.059 . . . . . . . . .

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-3
SLIDE 3

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

The empirical cycle

Confirmatory data analysis Limited number of research questions Research questions well-defined a priori Focus: strict error control Traditionally: (multiple) testing is important Exploratory data analysis Many possible research questions Research questions not well-defined a priori Focus: finding promising research avenues Traditionally: (multiple) testing not so important

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-4
SLIDE 4

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Microarray data analysis

More like exploratory than confirmatory research Probing many genes simultaneously Decision which questions are interesting taken a posteriori Findings are subject to follow up validation Still: multiple testing performed Reason: prevent unsuccessful validation experiments

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-5
SLIDE 5

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Exploratory data analysis

Mild It is not bad to select some true null hypotheses Flexible Procedures should not completely prescribe what to reject Post hoc Decide what/how much to follow up after seeing the data

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-6
SLIDE 6

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Exploratory data analysis

Mild It is not bad to select some true null hypotheses Flexible Procedures should not completely prescribe what to reject Post hoc Decide what/how much to follow up after seeing the data Multiple testing in exploratory research Should sanction mild, flexible, post hoc inference Should advise, not prescribe

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-7
SLIDE 7

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Set-up

Hypotheses H1, . . . , Hn True hypotheses T ⊆ {1, . . . , n} indices of true hypotheses Rejections R ⊆ {1, . . . , n} set of rejected hypotheses (usually random) Type I errors T ∩ R ⊆ {1, . . . , n}

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-8
SLIDE 8

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

FWER, FDR, k-FWER

User role Before seeing the data Choose error rate to be controlled FWER: : P(T ∩ R = ∅) FDR : E #(T ∩ R) #R ∨ 1

  • k-FWER

: P

  • #(R ∩ T) ≥ k
  • Procedure

Chooses R that controls the chosen error rate

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-9
SLIDE 9

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Alterative: exploratory inference

Role of the user In complete freedom the user rejects collection of hypotheses R.

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-10
SLIDE 10

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Alterative: exploratory inference

Role of the user In complete freedom the user rejects collection of hypotheses R. Role of the multiple testing procedure Inform user of the number of false rejections incurred

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-11
SLIDE 11

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Alterative: exploratory inference

Role of the user In complete freedom the user rejects collection of hypotheses R. Role of the multiple testing procedure Inform user of the number of false rejections incurred Number of false rejections = #(T ∩ R) = function of the model parameters = something we can estimate or make a confidence interval for

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-12
SLIDE 12

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Alterative: exploratory inference

Role of the user In complete freedom the user rejects collection of hypotheses R. Role of the multiple testing procedure Inform user of the number of false rejections incurred Number of false rejections = #(T ∩ R) = function of the model parameters = something we can estimate or make a confidence interval for Post hoc If we make a simultaneous CI, post hoc choice of R is allowed

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-13
SLIDE 13

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Closed Testing: ingredients

Marcus, Peritz and Gabriel (1976) Fundamental principle of FWER control Intersection hypothesis HC =

i∈C Hi, for C ⊆ {1, . . . , n}

Closure Collection of all intersection hypotheses C =

  • HC : C ⊆ {1, . . . , n}
  • Local test

Valid α-level test for every intersection hypothesis

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-14
SLIDE 14

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Closed testing (graphically)

A B C

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-15
SLIDE 15

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Closed testing (graphically)

A B C A ∩ B ∩ C A ∩ C B ∩ C A ∩ B

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-16
SLIDE 16

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Closed testing: procedure

Raw rejections Hypotheses U ⊆ C rejected by the local test Multiplicity-rejected rejections Reject H ∈ C if J ∈ U for every J ⊆ H

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-17
SLIDE 17

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Closed testing: procedure

Raw rejections Hypotheses U ⊆ C rejected by the local test Multiplicity-rejected rejections Reject H ∈ C if J ∈ U for every J ⊆ H Statement P(R ∩ T = ∅) ≥ 1 − α with R = {C ∈ C : C rejected} and T = {C ∈ C : C true}

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-18
SLIDE 18

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Closed testing: procedure

Raw rejections Hypotheses U ⊆ C rejected by the local test Multiplicity-rejected rejections Reject H ∈ C if J ∈ U for every J ⊆ H Statement P(R ∩ T = ∅) ≥ 1 − α with R = {C ∈ C : C rejected} and T = {C ∈ C : C true} Proof {R ∩ T = ∅} ⊇ {HT / ∈ U}

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-19
SLIDE 19

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Consonance

Traditionally, only rejection of elementary hypotheses is of interest A ∩ B ∩ C A B C A ∩ B A ∩ C B ∩ C The closed graph of hypotheses A, B and C

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-20
SLIDE 20

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Consonance

Traditionally, only rejection of elementary hypotheses is of interest A ∩ B ∩ C A ∩ B ∩ C A ∩ B ∩ C A B C A ∩ B A ∩ B A ∩ C A ∩ C A ∩ C B ∩ C Consonant rejections

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-21
SLIDE 21

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Consonance

Traditionally, only rejection of elementary hypotheses is of interest A ∩ B ∩ C A ∩ B ∩ C A ∩ B ∩ C A B C A ∩ B A ∩ B A ∩ C A ∩ C A ∩ C B ∩ C B ∩ C B ∩ C Non-consonant rejection of B ∩ C

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-22
SLIDE 22

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Parameter, confidence bound and coverage

Parameter τ(R) = #(T ∩ R) for a fixed set R Closed testing Let X be the collection of hypotheses rejected Confidence bound tα(R) = max(#C : C ⊆ R, HC / ∈ X}

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-23
SLIDE 23

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

In the example

A ∩ B ∩ C A ∩ B ∩ C A ∩ B ∩ C A B C A ∩ B A ∩ B A ∩ C A ∩ C A ∩ C B ∩ C B ∩ C B ∩ C tα({B, C}) = 1

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-24
SLIDE 24

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Coverage

Coverage statement P(τ(R) ≤ tα(R)) ≥ 1 − α

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-25
SLIDE 25

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Coverage

Coverage statement P(τ(R) ≤ tα(R)) ≥ 1 − α Proof {τ(R) ≤ tα(R)} ⊆ {HT / ∈ U}

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-26
SLIDE 26

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Coverage

Coverage statement P(τ(R) ≤ tα(R)) ≥ 1 − α Proof {τ(R) ≤ tα(R)} ⊆ {HT / ∈ U} Confidence set Trivial lower bound τ(R) ≥ 0 Confidence set {0, . . . , tα(R)} Confidence set for φ(R) = #R − τ(R) immediate

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-27
SLIDE 27

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Coverage

Coverage statement P(τ(R) ≤ tα(R)) ≥ 1 − α Proof {τ(R) ≤ tα(R)} ⊆ {HT / ∈ U} Confidence set Trivial lower bound τ(R) ≥ 0 Confidence set {0, . . . , tα(R)} Confidence set for φ(R) = #R − τ(R) immediate Simultaneous Simultaneous control over all R Consequence: coverage robust against post hoc selection of R

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-28
SLIDE 28

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Reject hypotheses

R confidence set for τ(R) confidence set for φ(R) {A} {0} {1} {B} {0,1} {0,1} {C} {0,1} {0,1} {A, B} {0,1} {1,2} {A, C} {0,1} {1,2} {B, C} {0,1} {1,2} {A, B, C} {0,1} {2,3}

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-29
SLIDE 29

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Mild, post hoc, flexible

Mild Allows a many or few false rejections as the researcher wants Flexible The researcher is completely free to choose the rejections Post hoc Allowed because confidence statements are simultaneous

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-30
SLIDE 30

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Mild, post hoc, flexible

Mild Allows a many or few false rejections as the researcher wants Flexible The researcher is completely free to choose the rejections Post hoc Allowed because confidence statements are simultaneous Role of the multiple testing procedure Advises rather than dictates

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-31
SLIDE 31

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Selecting covariates

Which covariates are associated with a response? Practice: typically exploratory and post hoc Example Physical data set Response: mass of male subjects Covariates: length and circumference of body parts 10 covariates 22 subjects

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-32
SLIDE 32

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Classical Forward/Backward (p-values)

covariate full model selected model (Intercept) 0.036 0.000 Forearm 0.061 0.000 Biceps 0.755 — Chest 0.420 — Neck 0.518 — Shoulder 0.905 — Waist 0.000 0.000 Height 0.033 0.005 Calf 0.303 — Thigh 0.351 0.036 Head 0.105 —

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-33
SLIDE 33

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Closed testing applied

Model Linear model, 10+1 regression coefficients Local test Test HC with an F-test of βi = 0 for i ∈ C against saturated

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-34
SLIDE 34

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Closed testing applied

Model Linear model, 10+1 regression coefficients Local test Test HC with an F-test of βi = 0 for i ∈ C against saturated Result 626 out of 1023 intersection hypotheses rejected Number of false nulls (95% conf) F/B set: at least one false null hypothesis All 10: at least two false null hypotheses

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-35
SLIDE 35

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Defining sets

Defining sets Rejected intersection hypotheses without rejected supersets Each contains at least one false null {waist} {forearm, neck, shoulder, height} {forearm, biceps, shoulder, calf} {forearm, shoulder, height, calf} {forearm, biceps, chest, neck, shoulder, thigh} {forearm, shoulder, height, thigh} {forearm, calf, thigh}

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-36
SLIDE 36

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Shortlist

Shortlist (idea suggested by Nicolai Meinshausen) Rewrite defining sets to intersection of unions At least one of these sets contains only false nulls {waist, forearm} {waist, shoulder, calf} {waist, shoulder, thigh} {waist, neck, height, calf} {waist, neck, calf, thigh} {waist, height, biceps, calf} {waist, height, calf, chest} {waist, height, biceps, thigh} {waist, height, calf, thigh}

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-37
SLIDE 37

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Shortcuts

General Procedure can be used for any local test Number of intersection hypotheses 2n: computationally hard above 20 hypotheses Concept: shortcut Smart choice of local test to save calculations Extend shortcut concept Should also yield rejected non-consonant intersections

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-38
SLIDE 38

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Shortcuts based on Simes’ inequality

Simes’ inequality If p1, . . . , pk from true hypotheses, then simultaneously p(i) > iα

k .

Use Simes as local test Reject if any p(i) ≤ iα

k

Allows shortcut with n2 rather than 2n calculations Link with Benjamini & Hochberg FDR control Same assumptions, same weak FWER control Link with Hochberg and Hommel procedures Rejects exactly the same as Hommel for FWER

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-39
SLIDE 39

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

The Rosenwald data

Rosenwald (2002) Diffuse Large B-cell Lymphoma Gene expression of 7399 probes Sample size of 240 patients Survival follow-up of median 8 years Hypothesis tests 7399 likelihood ratio tests in the Cox model

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-40
SLIDE 40

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Rejection curve (first part)

10 20 30 40 10 20 30 40 number of rejections number of rejections correct rejections (95% conf.)

  • thers

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-41
SLIDE 41

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Package “cherry” on CRAN

General closed testing Specify any local test Up to 31 hypotheses Special functions based on shortcuts Simes inequality Fisher combinations

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-42
SLIDE 42

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Conclusion

New method Between weak and strong FWER control Nothing new Just closed testing and simultaneous confidence sets Suitable for exploratory research Mild, post hoc, flexible

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari

slide-43
SLIDE 43

Exploratory data ananlysis Closed testing A Confidence Set Applications Discussion

Read more?

Goeman and Solari (2011). Multiple testing for exploratory research, with discussion Statistical Science, 26 (4), 584-597.

Cherry-picking Multiple Testing for Exploratory Research Jelle Goeman, Aldo Solari