Familywise error rate control by interactive unmasking Boyan - - PowerPoint PPT Presentation

Familywise error rate control by interactive unmasking

• Dept. of Statistics and Data Science

Carnegie Mellon University

1

Larry Wasserman Aaditya Ramdas

Boyan Duan

2

Motivating example: tumor detection in brain image

2

Motivating example: tumor detection in brain image

Eg. for each pixel and test .

Zi ∼ N(μi,1) Hi : μi ≤ 0

Underlying true

μi

2

Motivating example: tumor detection in brain image

Eg. for each pixel and test .

Zi ∼ N(μi,1) Hi : μi ≤ 0

Underlying true

μi

Observed -values

p Pi = 1 − Φ(Zi)

2

Motivating example: tumor detection in brain image

Eg. for each pixel and test .

Zi ∼ N(μi,1) Hi : μi ≤ 0

Underlying true

μi

Observed -values

p Pi = 1 − Φ(Zi)

Task: identify non-nulls (decide whether to reject each ),

Hi

2

Motivating example: tumor detection in brain image

Eg. for each pixel and test .

Zi ∼ N(μi,1) Hi : μi ≤ 0

Underlying true

μi

Observed -values

p Pi = 1 − Φ(Zi)

Task: identify non-nulls (decide whether to reject each ),

Hi

with familywise error rate (FWER) control:

2

FWER := ℙ(#falsely rejected nulls ≥ 1) Motivating example: tumor detection in brain image

Eg. for each pixel and test .

Zi ∼ N(μi,1) Hi : μi ≤ 0

Underlying true

μi

Observed -values

p Pi = 1 − Φ(Zi)

Task: identify non-nulls (decide whether to reject each ),

Hi

with familywise error rate (FWER) control:

2

FWER := ℙ(#falsely rejected nulls ≥ 1) Motivating example: tumor detection in brain image

Eg. for each pixel and test .

Zi ∼ N(μi,1) Hi : μi ≤ 0

Underlying true

μi

Observed -values

p Pi = 1 − Φ(Zi)

≤ α

Eg.

α = 0.2

Task: identify non-nulls (decide whether to reject each ),

Hi

with familywise error rate (FWER) control:

2

FWER := ℙ(#falsely rejected nulls ≥ 1) Motivating example: tumor detection in brain image

Eg. for each pixel and test .

Zi ∼ N(μi,1) Hi : μi ≤ 0

Underlying true

μi

Observed -values

p Pi = 1 − Φ(Zi)

≤ α

Eg.

α = 0.2

taking account of side information. Eg. structure, covariates…

3

Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019)

Classical testing

Pre-fixed procedure. Eg. weighted Bonferroni: reject if .

Hi Pi/wi ≤ α/n

Classical test (single step)

3

Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019)

Classical testing

Pre-fixed procedure. Eg. weighted Bonferroni: reject if .

Hi Pi/wi ≤ α/n

Data (& side info) Classical test (single step)

3

Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019)

Classical testing

Pre-fixed procedure. Eg. weighted Bonferroni: reject if .

Hi Pi/wi ≤ α/n

• values

p

Data (& side info) Classical test (single step) Rejection set

3

Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019)

Classical testing

Pre-fixed procedure. Eg. weighted Bonferroni: reject if .

Hi Pi/wi ≤ α/n

• values

p

Data (& side info) Classical test (single step) Rejection set

3

Interactive test (multi-step)

Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019)

Classical testing

Pre-fixed procedure. Eg. weighted Bonferroni: reject if .

Hi Pi/wi ≤ α/n

• values

p

Data (& side info) Classical test (single step) Rejection set

3

Masked data (& side info)

Interactive test (multi-step)

Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019)

Classical testing

Pre-fixed procedure. Eg. weighted Bonferroni: reject if .

Hi Pi/wi ≤ α/n

• values

p

masked

• values

p

Data (& side info) Classical test (single step) Rejection set

3

Masked data (& side info)

Interactive test (multi-step)

Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019)

Classical testing

Pre-fixed procedure. Eg. weighted Bonferroni: reject if .

Hi Pi/wi ≤ α/n

• values

p

masked

• values

p

Data (& side info) Classical test (single step) Rejection set

3

Masked data (& side info)

Interactive test (multi-step)

Error control not met

Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019)

Classical testing

Pre-fixed procedure. Eg. weighted Bonferroni: reject if .

Hi Pi/wi ≤ α/n

• values

p

masked

• values

p

Data (& side info) Classical test (single step) Rejection set

3

Masked data (& side info)

Interactive test (multi-step)

Unmask data + Interact Error control not met

Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019)

Classical testing

Pre-fixed procedure. Eg. weighted Bonferroni: reject if .

Hi Pi/wi ≤ α/n

• values

p

masked

• values

p

Data (& side info) Classical test (single step) Rejection set

3

Masked data (& side info)

Interactive test (multi-step)

Unmask data + Interact Error control not met

Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019)

Classical testing

Pre-fixed procedure. Eg. weighted Bonferroni: reject if .

Hi Pi/wi ≤ α/n

• values

p

masked

• values

p

Data (& side info) Classical test (single step) Rejection set

3

Masked data (& side info)

Interactive test (multi-step)

Unmask data + Interact Error control not met

Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019)

Classical testing

Pre-fixed procedure. Eg. weighted Bonferroni: reject if .

Hi Pi/wi ≤ α/n

• values

p

masked

• values

p

Data (& side info) Classical test (single step) Rejection set

3

Masked data (& side info)

Interactive test (multi-step)

Unmask data + Interact Error control not met

Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019)

Classical testing

Pre-fixed procedure. Eg. weighted Bonferroni: reject if .

Hi Pi/wi ≤ α/n

Interactive testing

• Accommodate various structures: grid, tree …
• Be revised manually on the fly: one tumor or two?
• values

p

masked

• values

p

4

Component 1: mask -values

p

{ g(P) = min{P,1 − P} h(P) = 2 ⋅ 1{P < 0.5} − 1 P

4

Component 1: mask -values

p

g(P) h(P) P

{ g(P) = min{P,1 − P} h(P) = 2 ⋅ 1{P < 0.5} − 1 P

4

Component 1: mask -values

p

Independent for nulls

{ g(P) = min{P,1 − P} h(P) = 2 ⋅ 1{P < 0.5} − 1 P Small indicate non-nulls

g(P)

4

Component 1: mask -values

p

1 P g(P) 0.5

Independent for nulls

{ g(P) = min{P,1 − P} h(P) = 2 ⋅ 1{P < 0.5} − 1 P Small indicate non-nulls

g(P)

4

Component 1: mask -values

p

g(P) P

1 P g(P) 0.5

Independent for nulls

{ g(P) = min{P,1 − P} h(P) = 2 ⋅ 1{P < 0.5} − 1 P Small indicate non-nulls

g(P)

4

Component 1: mask -values

p

g(P) P

1 P g(P) 0.5

Independent for nulls Note: above masking works for FDR control, but not FWER control.

{ g(P) = min{P,1 − P} h(P) = 2 ⋅ 1{P < 0.5} − 1 P Small indicate non-nulls

g(P)

4

Component 1: mask -values

p

g(P) P

1 P g(P) 0.5

Independent for nulls Note: above masking works for FDR control, but not FWER control.

{ g(P) = min{P,1 − P} h(P) = 2 ⋅ 1{P < 0.5} − 1 P Small indicate non-nulls

g(P)

4

Component 1: mask -values

p

g(P) P

p* p* 1 P g(P) (Default )

p* = α/2

Independent for nulls Note: above masking works for FDR control, but not FWER control.

P Small indicate non-nulls

g(P)

4

Component 1: mask -values

p

g(P) P

p* p* 1 P g(P) { g(P; p*) = min{P,

p* 1 − p* (1 − P)}

h(P; p*) = 2 ⋅ 1{P < p*} − 1 (Default )

p* = α/2

Independent for nulls

5

Component 2: select candidate set interactively

g(P) P

g(P) h(P)

for cand. set selection for error control

5

Component 2: select candidate set interactively

g(P)

R

P

g(P) h(P)

R

for cand. set selection for error control

5

Component 2: select candidate set interactively

g(P)

R

P ̂ FWER

?

≤ α

g(P) h(P)

R

for cand. set selection for error control

5

Component 2: select candidate set interactively

g(P)

R

t + 1

progressively shrink

Rt R

t

P ̂ FWERt

?

≤ α

g(P) h(P)

Rt

for cand. set selection for error control

R0 = {1,…, n} R0 ⊇ R1 ⊇ …

5

Component 2: select candidate set interactively

g(P)

R {g(Pi)}n

i=1

t + 1

progressively shrink

Rt R

t

P ̂ FWERt

?

≤ α

g(P) h(P)

Rt

for cand. set selection for error control

R0 = {1,…, n} R0 ⊇ R1 ⊇ …

5

Component 2: select candidate set interactively

g(P)

R {g(Pi)}n

i=1

+ coordinates

t + 1

progressively shrink

Rt R

t

P ̂ FWERt

?

≤ α

g(P) h(P)

Rt

for cand. set selection for error control

R0 = {1,…, n} R0 ⊇ R1 ⊇ …

5

Component 2: select candidate set interactively

g(P)

R {g(Pi)}n

i=1

+ coordinates

t + 1

progressively shrink

Rt

(side information )

{xi}n

i=1

R

t

P ̂ FWERt

?

≤ α

g(P) h(P)

Rt

for cand. set selection for error control

R0 = {1,…, n} R0 ⊇ R1 ⊇ …

5

Component 2: select candidate set interactively

g(P)

R {g(Pi)}n

i=1

+ coordinates

+

{h(Pi)}i∉R

t + 1

t

progressively shrink

Rt

(side information )

{xi}n

i=1

R

t

P ̂ FWERt

?

≤ α

g(P) h(P)

Rt

for cand. set selection for error control

R0 = {1,…, n} R0 ⊇ R1 ⊇ …

5

Component 2: select candidate set interactively

g(P)

R {g(Pi)}n

i=1

+ coordinates

+

{h(Pi)}i∉R

t + 1

t

progressively shrink

Rt

(side information )

{xi}n

i=1

using increasing information:

R

t

P ̂ FWERt

?

≤ α

g(P) h(P)

Rt

for cand. set selection for error control

R0 = {1,…, n} R0 ⊇ R1 ⊇ …

6

Component 3: error control

FWER := ℙ(#false rejections ≥ 1) P p* g(P) h(P) Rt

6

Component 3: error control

Reject the set .

{Hi ∈ Rt : h(Pi) = 1}

FWER := ℙ(#false rejections ≥ 1) P p* g(P) h(P) Rt

6

Component 3: error control

Count

R−

t := |{Hi ∈ Rt : h(Pi) = − 1}|

Reject the set .

{Hi ∈ Rt : h(Pi) = 1}

if is null

h(Pi) ∼ Ber(p*) Hi

FWER := ℙ(#false rejections ≥ 1) P p* g(P) h(P) Rt

FWER control using binary var.: Janson & Su (2016)

6

Component 3: error control

Count

R−

t := |{Hi ∈ Rt : h(Pi) = − 1}|

Stop shrinking when .

̂

FWER t := 1 − (1 − p*)R−

t +1 ≤ α

Reject the set .

{Hi ∈ Rt : h(Pi) = 1}

if is null

h(Pi) ∼ Ber(p*) Hi

FWER := ℙ(#false rejections ≥ 1) P p* g(P) h(P) Rt

FWER control using binary var.: Janson & Su (2016)

6

Component 3: error control

Count

R−

t := |{Hi ∈ Rt : h(Pi) = − 1}|

hypotheses with 9 non-nulls ( )

10 × 10 α = 0.2

Stop shrinking when .

̂

FWER t := 1 − (1 − p*)R−

t +1 ≤ α

Reject the set .

{Hi ∈ Rt : h(Pi) = 1}

if is null

h(Pi) ∼ Ber(p*) Hi

FWER := ℙ(#false rejections ≥ 1)

FWER control using binary var.: Janson & Su (2016)

6

Component 3: error control

Count

R−

t := |{Hi ∈ Rt : h(Pi) = − 1}|

hypotheses with 9 non-nulls ( )

10 × 10 α = 0.2

Stop shrinking when .

̂

FWER t := 1 − (1 − p*)R−

t +1 ≤ α

Reject the set .

{Hi ∈ Rt : h(Pi) = 1}

if is null

h(Pi) ∼ Ber(p*) Hi

FWER := ℙ(#false rejections ≥ 1)

FWER control using binary var.: Janson & Su (2016)

6

Component 3: error control

Count

R−

t := |{Hi ∈ Rt : h(Pi) = − 1}|

hypotheses with 9 non-nulls ( )

10 × 10 α = 0.2

Stop shrinking when .

̂

FWER t := 1 − (1 − p*)R−

t +1 ≤ α

Reject the set .

{Hi ∈ Rt : h(Pi) = 1}

if is null

h(Pi) ∼ Ber(p*) Hi

FWER := ℙ(#false rejections ≥ 1)

Theorem: the i-FWER method control FWER if null -values are mutually independent, and are independently of the non-nulls.

p

FWER control using binary var.: Janson & Su (2016)

• Estimate the probability
• f being non-null.

g(P)

smoothed probability

7

Strategies to shrink

Rt

• Estimate the probability
• f being non-null.

g(P)

smoothed probability

• Change the strategy

at any step.

single center?

7

Strategies to shrink

Rt

• Estimate the probability
• f being non-null.

g(P)

smoothed probability

• Change the strategy

at any step.

single center? two centers

7

Strategies to shrink

Rt

• Estimate the probability
• f being non-null.

g(P)

smoothed probability

• Change the strategy

at any step.

• Customize the strategy to

different structures.

single center? two centers

7

Hierarchical structure

Strategies to shrink

Rt

• Estimate the probability
• f being non-null.

g(P)

smoothed probability

• Change the strategy

at any step.

• Customize the strategy to

different structures.

single center? two centers

7

Hierarchical structure

Strategies to shrink

Rt

8

• 0.0

0.2 0.4 0.6 0.8 1.0 1 2 3 4 5

alternative mean power

• i−FWER 10*10

i−FWER 30*30 Sidak 10*10 Sidak 30*30

Grid

Meinshausen (2008); Goeman and Finos (2012); Ignatiadis et al. (2016); Lei, Fithian (2018)

8

• 0.0

0.2 0.4 0.6 0.8 1.0 1 2 3 4 5

alternative mean power

• i−FWER 10*10

i−FWER 30*30 Sidak 10*10 Sidak 30*30

Grid Tree

Meinshausen (2008); Goeman and Finos (2012); Ignatiadis et al. (2016); Lei, Fithian (2018)

…

⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮

8

• 0.0

0.2 0.4 0.6 0.8 1.0 1 2 3 4 5

alternative mean power

• i−FWER 10*10

i−FWER 30*30 Sidak 10*10 Sidak 30*30

Grid Tree Real data: RNA sequence

FWER level IHW i-FWER 0.1 1552 1613 0.2 1645 1740 0.3 1708 1844

differentially expressed genes in airway smooth muscle cell lines in response to dexamethasone

Meinshausen (2008); Goeman and Finos (2012); Ignatiadis et al. (2016); Lei, Fithian (2018)

…

⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮

9

Extensions

p* p* p Tent function

Masking functions

9

Extensions

p* p* p Tent function Railway function Gap function p* p* p p* p* p

Masking functions

9

Extensions

p* p* p Tent function Railway function Gap function p* p* p p* p* p

Masking functions Interactive testing

Masking Masked data Missing bit Data independent under null Human interaction Large scale testing using ML g(P) h(P)

Familywise error rate control by interactive unmasking

• Dept. of Statistics and Data Science

Carnegie Mellon University

10

Larry Wasserman Aaditya Ramdas

Boyan Duan

Thank you!