Adaptive Monte Carlo Multiple Testing via Multi-Armed Bandits - - PowerPoint PPT Presentation

▶

Nov 20, 2023 317 likes •526 views

Adaptive Monte Carlo Multiple Testing via Multi-Armed Bandits Martin Zhang joint work with: David Tse & James Zou Stanford University Problem | Monte Carlo Multiple Hypothesis Testing SNP 1 SNP 2 SNP m Problem | Monte Carlo Multiple

SLIDE 1

Adaptive Monte Carlo Multiple Testing via Multi-Armed Bandits

Martin Zhang

joint work with:

David Tse & James Zou Stanford University

SLIDE 2

Problem | Monte Carlo Multiple Hypothesis Testing

SNP1

…

SNP2 SNPm

SLIDE 3

Problem | Monte Carlo Multiple Hypothesis Testing

P1 P2 Pm

…

SNP1

…

SNP2 SNPm

SLIDE 4

Problem | Monte Carlo Multiple Hypothesis Testing

P1 P2 Pm

…

SNP1

…

SNP2 SNPm

√

× ×

…

SLIDE 5

Problem | Monte Carlo Multiple Hypothesis Testing

Monte Carlo test

P1 P2 Pm

…

SNP1

…

SNP2 SNPm

√

× ×

…

P1 ∼ 1 n

∑

j=1

𝕁{Tnull

1,j ≥ tobs 1 }

SLIDE 6

Problem | Monte Carlo Multiple Hypothesis Testing

Monte Carlo test Benjamini Hochberg procedure

P1 P2 Pm

…

SNP1

…

SNP2 SNPm

√

× ×

…

P1 ∼ 1 n

∑

j=1

𝕁{Tnull

1,j ≥ tobs 1 }

Data-dependent # of discoveries Control FDR = 𝔽 [

false discovery discovery ]

SLIDE 7

Monte Carlo test Benjamini Hochberg procedure

P1 P2 Pm

…

SNP1

…

SNP2 SNPm

√

× ×

…

P1 ∼ 1 n

∑

j=1

𝕁{Tnull

1,j ≥ tobs 1 }

Data-dependent # of discoveries Control FDR = 𝔽 [

false discovery discovery ]

Problem | Monte Carlo Multiple Hypothesis Testing

Computational cost: nm

SLIDE 8

Monte Carlo test Benjamini Hochberg procedure

P1 P2 Pm

…

SNP1

…

SNP2 SNPm

√

× ×

…

P1 ∼ 1 n

∑

j=1

𝕁{Tnull

1,j ≥ tobs 1 }

Data-dependent # of discoveries Control FDR = 𝔽 [

false discovery discovery ]

Problem | Monte Carlo Multiple Hypothesis Testing

Computational cost: nm

hypothesis tests

m

MC samples per test

n ×

SLIDE 9

Genome-wide association studies

m = 500,000 n = 50,000,000

MC samples per test hypothesis tests

m n

Problem | Monte Carlo Multiple Hypothesis Testing

SLIDE 10

Genome-wide association studies

m = 500,000 n = 50,000,000

tal MC samples:

nm = 2.5 × 1013

T ypical computation time: ~2 months MC samples per test hypothesis tests

m n

Problem | Monte Carlo Multiple Hypothesis Testing

SLIDE 11

Genome-wide association studies

m = 500,000 n = 50,000,000

tal MC samples:

nm = 2.5 × 1013

T ypical computation time: ~2 months

Can we make it faster?

MC samples per test hypothesis tests

m n

Problem | Monte Carlo Multiple Hypothesis Testing

SLIDE 12

Results | Adaptive Monte Carlo Multiple Testing (AMT)

same discoveries with high probability; information theoretically optimal

Theorem (informal):

baseline: nm

Expected # of MC samples:

SLIDE 13

Results | Adaptive Monte Carlo Multiple Testing (AMT)

same discoveries with high probability; information theoretically optimal

Theorem (informal):

baseline: nm

Expected # of MC samples:

nm 2 months 1 hour with the same discoveries

→

GWAS example:

SLIDE 14

rank k p-value

1 2 3 4 5 6 7 8 1

Quantities to estimate:

Results | Adaptive Monte Carlo Multiple Testing (AMT)

SLIDE 15

rank k p-value

1 2 3 4 5 6 7 8 1

τ*

BH threshold τ*

Quantities to estimate:

Results | Adaptive Monte Carlo Multiple Testing (AMT)

SLIDE 16

How each p-value compares with τ*

rank k p-value

1 2 3 4 5 6 7 8 1

τ*

BH threshold τ*

Quantities to estimate:

Results | Adaptive Monte Carlo Multiple Testing (AMT)

SLIDE 17

How each p-value compares with τ*

rank k p-value

1 2 3 4 5 6 7 8 1

τ*

BH threshold τ*

Quantities to estimate:

More MC samples

Results | Adaptive Monte Carlo Multiple Testing (AMT)

SLIDE 18

How each p-value compares with τ*

rank k p-value

1 2 3 4 5 6 7 8 1

τ*

BH threshold τ*

Quantities to estimate:

More MC samples Less MC samples

Results | Adaptive Monte Carlo Multiple Testing (AMT)

SLIDE 19

How each p-value compares with τ*

rank k p-value

1 2 3 4 5 6 7 8 1

τ*

BH threshold τ*

Quantities to estimate:

More MC samples

Adaptive Estimation via Multi-Armed Bandits

Less MC samples