Familywise error rate control by interactive unmasking Boyan - PowerPoint PPT Presentation

Dept. of Statistics and Data Science Carnegie Mellon University Familywise error rate control by interactive unmasking Boyan Aaditya Larry Duan Ramdas Wasserman 1

Motivating example: tumor detection in brain image 2

Motivating example: tumor detection in brain image Z i ∼ N ( μ i ,1) H i : μ i ≤ 0 Eg. for each pixel and test . Underlying true μ i 2

Motivating example: tumor detection in brain image Z i ∼ N ( μ i ,1) H i : μ i ≤ 0 Eg. for each pixel and test . p Underlying true Observed -values μ i P i = 1 − Φ ( Z i ) 2

Motivating example: tumor detection in brain image Z i ∼ N ( μ i ,1) H i : μ i ≤ 0 Eg. for each pixel and test . p Underlying true Observed -values μ i P i = 1 − Φ ( Z i ) 2

Motivating example: tumor detection in brain image Z i ∼ N ( μ i ,1) H i : μ i ≤ 0 Eg. for each pixel and test . p Underlying true Observed -values μ i P i = 1 − Φ ( Z i ) H i Task: identify non-nulls (decide whether to reject each ), 2

Motivating example: tumor detection in brain image Z i ∼ N ( μ i ,1) H i : μ i ≤ 0 Eg. for each pixel and test . p Underlying true Observed -values μ i P i = 1 − Φ ( Z i ) H i Task: identify non-nulls (decide whether to reject each ), with familywise error rate (FWER) control: FWER := ℙ (# falsely rejected nulls ≥ 1) 2

Motivating example: tumor detection in brain image Z i ∼ N ( μ i ,1) H i : μ i ≤ 0 Eg. for each pixel and test . p Underlying true Observed -values μ i P i = 1 − Φ ( Z i ) H i Task: identify non-nulls (decide whether to reject each ), with familywise error rate (FWER) control: FWER := ℙ (# falsely rejected nulls ≥ 1) α = 0.2 Eg. ≤ α 2

Motivating example: tumor detection in brain image Z i ∼ N ( μ i ,1) H i : μ i ≤ 0 Eg. for each pixel and test . p Underlying true Observed -values μ i P i = 1 − Φ ( Z i ) H i Task: identify non-nulls (decide whether to reject each ), with familywise error rate (FWER) control: FWER := ℙ (# falsely rejected nulls ≥ 1) α = 0.2 Eg. ≤ α taking account of side information. Eg. structure, covariates… 2

Classical testing H i P i / w i ≤ α / n Pre-fixed procedure. Eg. weighted Bonferroni: reject if . Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019) 3

Classical testing H i P i / w i ≤ α / n Pre-fixed procedure. Eg. weighted Bonferroni: reject if . Classical test (single step) Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019) 3

Classical testing H i P i / w i ≤ α / n Pre-fixed procedure. Eg. weighted Bonferroni: reject if . Data (& side info) p -values Classical test (single step) Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019) 3

Classical testing H i P i / w i ≤ α / n Pre-fixed procedure. Eg. weighted Bonferroni: reject if . Data (& side info) p -values Classical test (single step) Rejection set Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019) 3

Classical testing H i P i / w i ≤ α / n Pre-fixed procedure. Eg. weighted Bonferroni: reject if . Data (& side info) p -values Interactive test Classical test (single step) (multi-step) Rejection set Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019) 3

Classical testing H i P i / w i ≤ α / n Pre-fixed procedure. Eg. weighted Bonferroni: reject if . Masked data Data (& side info) (& side info) p -values Interactive test Classical test masked (single step) (multi-step) p -values Rejection set Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019) 3

Classical testing H i P i / w i ≤ α / n Pre-fixed procedure. Eg. weighted Bonferroni: reject if . Masked data Data (& side info) (& side info) p -values Interactive test Classical test masked (single step) (multi-step) p -values Rejection set Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019) 3

Classical testing H i P i / w i ≤ α / n Pre-fixed procedure. Eg. weighted Bonferroni: reject if . Masked data Data (& side info) (& side info) p -values Error control not met Interactive test Classical test masked (single step) (multi-step) p -values Rejection set Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019) 3

Classical testing H i P i / w i ≤ α / n Pre-fixed procedure. Eg. weighted Bonferroni: reject if . Masked data Data (& side info) (& side info) p -values Error control not met Unmask data Interactive test Classical test masked + (single step) (multi-step) p -values Interact Rejection set Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019) 3

Classical testing H i P i / w i ≤ α / n Pre-fixed procedure. Eg. weighted Bonferroni: reject if . Masked data Data (& side info) (& side info) p -values Error control not met Unmask data Interactive test Classical test masked + (single step) (multi-step) p -values Interact Rejection set Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019) 3

Classical testing H i P i / w i ≤ α / n Pre-fixed procedure. Eg. weighted Bonferroni: reject if . Masked data Data (& side info) (& side info) p -values Error control not met Unmask data Interactive test Classical test masked + (single step) (multi-step) p -values Interact Rejection set Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019) 3

Classical testing H i P i / w i ≤ α / n Pre-fixed procedure. Eg. weighted Bonferroni: reject if . Masked data Data (& side info) (& side info) p -values Error control not met Unmask data Interactive test Classical test masked + (single step) (multi-step) p -values Interact Interactive testing Rejection - Accommodate various structures: grid, tree … set - Be revised manually on the fly: one tumor or two? Interactive tests with FDR control: Lei, Fithian (2018); Lei, Ramdas, Fithian (2019) 3

p Component 1: mask � -values 4

p Component 1: mask � -values h ( P ) { g ( P ) = min{ P ,1 − P } P g ( P ) h ( P ) = 2 ⋅ 1{ P < 0.5} − 1 P 4

p Component 1: mask � -values { g ( P ) = min{ P ,1 − P } Independent P for nulls h ( P ) = 2 ⋅ 1{ P < 0.5} − 1 4

p Component 1: mask � -values { g ( P ) = min{ P ,1 − P } Independent P for nulls h ( P ) = 2 ⋅ 1{ P < 0.5} − 1 Small indicate non-nulls g ( P ) g ( P ) P 1 0.5 4

p Component 1: mask � -values { g ( P ) = min{ P ,1 − P } Independent P for nulls h ( P ) = 2 ⋅ 1{ P < 0.5} − 1 Small indicate non-nulls g ( P ) g ( P ) P g ( P ) P 1 0.5 4

p Component 1: mask � -values { g ( P ) = min{ P ,1 − P } Independent P for nulls h ( P ) = 2 ⋅ 1{ P < 0.5} − 1 Note: above masking works for FDR control, but not FWER control. Small indicate non-nulls g ( P ) g ( P ) P g ( P ) P 1 0.5 4

p Component 1: mask � -values { g ( P ) = min{ P ,1 − P } Independent P for nulls h ( P ) = 2 ⋅ 1{ P < 0.5} − 1 Note: above masking works for FDR control, but not FWER control. Small indicate non-nulls g ( P ) g ( P ) P g ( P ) P 1 0.5 4

p Component 1: mask � -values { g ( P ) = min{ P ,1 − P } Independent P for nulls h ( P ) = 2 ⋅ 1{ P < 0.5} − 1 Note: above masking works for FDR control, but not FWER control. Small indicate non-nulls g ( P ) g ( P ) P p * g ( P ) P p * 1 p * = α /2 (Default ) 4

p Component 1: mask � -values p * { g ( P ; p * ) = min{ P , 1 − p * (1 − P )} Independent P for nulls h ( P ; p * ) = 2 ⋅ 1{ P < p * } − 1 Small indicate non-nulls g ( P ) g ( P ) P p * g ( P ) P p * 1 p * = α /2 (Default ) 4

Component 2: select candidate set interactively g ( P ) for cand. set selection P h ( P ) for error control g ( P ) 5

Component 2: select candidate set interactively R g ( P ) for cand. set selection P h ( P ) for error control R g ( P ) 5

̂ Component 2: select candidate set interactively R g ( P ) for cand. set selection P ? h ( P ) for error control ≤ α FWER R g ( P ) 5

̂ Component 2: select candidate set interactively R t g ( P ) for cand. set selection P ? h ( P ) for error control ≤ α FWER t R 0 = {1,…, n } R R 0 ⊇ R 1 ⊇ … t R t progressively shrink R t + 1 g ( P ) 5

̂ Component 2: select candidate set interactively R t g ( P ) for cand. set selection P ? h ( P ) for error control ≤ α FWER t R 0 = {1,…, n } R R 0 ⊇ R 1 ⊇ … t R t progressively shrink R t + 1 { g ( P i )} n i =1 g ( P ) 5

̂ Component 2: select candidate set interactively R t g ( P ) for cand. set selection P ? h ( P ) for error control ≤ α FWER t R 0 = {1,…, n } R R 0 ⊇ R 1 ⊇ … t R t progressively shrink R t + 1 { g ( P i )} n i =1 + coordinates g ( P ) 5