Sparse multiple testing: can one estimate the null distribution? - - PowerPoint PPT Presentation

Sparse multiple testing: can one estimate the null distribution?

Etienne Roquain1 Joint work with A. Carpentier2, S. Delattre3, N. Verzelen4,

1LPSM, Sorbonne Université, France 2Otto-von-Guericke-Universität Magdeburg, Allemagne 3LPSM, Université de Paris, France 4INRAE, Montpellier, France

MMMS2 Luminy, 02/06/2020

Arxiv 1912.03109. "On using empirical null distributions in Benjamini-Hochberg procedure" To appear in AoS. "Estimating minimum effect with outlier selection "

ANR “Sanssouci", ANR “BASICS", GDR ISIS “TASTY"

Roquain, Etienne Sparse multiple testing 1 / 29

1

Introduction

2

Upper bound

3

Lower bound

4

Additional results

5

One-sided alternatives

Roquain, Etienne Sparse multiple testing 2 / 29

Motivation 1: null distribution unknown

M67 photography, Package photutils Original Gaussian fitting Gumbel fitting ◮ Naive null distribution fitting ◮ Impact on the risk?

Roquain, Etienne Sparse multiple testing Introduction 3 / 29

Motivation 1: null distribution unknown

M67 photography, Package photutils Original Gaussian fitting Gumbel fitting ◮ Naive null distribution fitting ◮ Impact on the risk?

Roquain, Etienne Sparse multiple testing Introduction 3 / 29

Motivation 2: null distribution wrong

Figure 4 in [Efron (2008)] ◮ Empirical null [Efron (2004,2007,2008,2009)] ◮ Impact on the risk?

Roquain, Etienne Sparse multiple testing Introduction 4 / 29

Motivation 2: null distribution wrong

Figure 4 in [Efron (2008)] ◮ Empirical null [Efron (2004,2007,2008,2009)] ◮ Impact on the risk?

Roquain, Etienne Sparse multiple testing Introduction 4 / 29

Existing work (selection)

Estimation of the null: ◮ Series of work [Efron (2004,2007,2008,2009)] ◮ Minimax rate with Fourier analysis: [Jin and Cai (2007)]; [Cai and Jin (2010)] ◮ Two group mixture model: [Efron et al. (2001)]; [Sun and Cai (2009)]; [Cai and

Sun (2009)]; [Padilla and Bickel (2012)]; [Nguyen and Matias (2014)]; [Heller and Yekutieli (2014)]; [Zablocki et al. (2017)]; [Amar et al. (2017)]; [Cai et al. (2019)]; [Rebafka et al. (2019)]

◮ Estimation in factor model: [Efron (2007a)]; [Leek and Storey (2008)]; [Friguet

et al. (2009)]; [Fan et al. (2012)]; [Fan and Han (2017)]

Impact on the risk: ◮ FDR control in symmetric, centered, one-sided case: [Barber and Candès

(2015)]; [Arias-Castro and Chen (2017)]

Lower bounds in multiple testing: ◮ [Arias-Castro and Chen (2017)]; [Rabinovich et al. (2017)]; [Castillo and R.

(2020).]

Roquain, Etienne Sparse multiple testing Introduction 5 / 29

Existing work (selection)

Estimation of the null: ◮ Series of work [Efron (2004,2007,2008,2009)] ◮ Minimax rate with Fourier analysis: [Jin and Cai (2007)]; [Cai and Jin (2010)] ◮ Two group mixture model: [Efron et al. (2001)]; [Sun and Cai (2009)]; [Cai and

Sun (2009)]; [Padilla and Bickel (2012)]; [Nguyen and Matias (2014)]; [Heller and Yekutieli (2014)]; [Zablocki et al. (2017)]; [Amar et al. (2017)]; [Cai et al. (2019)]; [Rebafka et al. (2019)]

◮ Estimation in factor model: [Efron (2007a)]; [Leek and Storey (2008)]; [Friguet

et al. (2009)]; [Fan et al. (2012)]; [Fan and Han (2017)]

Impact on the risk: ◮ FDR control in symmetric, centered, one-sided case: [Barber and Candès

(2015)]; [Arias-Castro and Chen (2017)]

Lower bounds in multiple testing: ◮ [Arias-Castro and Chen (2017)]; [Rabinovich et al. (2017)]; [Castillo and R.

(2020).]

Roquain, Etienne Sparse multiple testing Introduction 5 / 29

Existing work (selection)

Estimation of the null: ◮ Series of work [Efron (2004,2007,2008,2009)] ◮ Minimax rate with Fourier analysis: [Jin and Cai (2007)]; [Cai and Jin (2010)] ◮ Two group mixture model: [Efron et al. (2001)]; [Sun and Cai (2009)]; [Cai and

Sun (2009)]; [Padilla and Bickel (2012)]; [Nguyen and Matias (2014)]; [Heller and Yekutieli (2014)]; [Zablocki et al. (2017)]; [Amar et al. (2017)]; [Cai et al. (2019)]; [Rebafka et al. (2019)]

◮ Estimation in factor model: [Efron (2007a)]; [Leek and Storey (2008)]; [Friguet

et al. (2009)]; [Fan et al. (2012)]; [Fan and Han (2017)]

Impact on the risk: ◮ FDR control in symmetric, centered, one-sided case: [Barber and Candès

(2015)]; [Arias-Castro and Chen (2017)]

Lower bounds in multiple testing: ◮ [Arias-Castro and Chen (2017)]; [Rabinovich et al. (2017)]; [Castillo and R.

(2020).]

Roquain, Etienne Sparse multiple testing Introduction 5 / 29

Setting

Observations Y = (Yi)1≤i≤n indep , Yi ∼ Pi, parameter P = (Pi)1≤i≤n ∈ P Gaussian null assumption: Most of the Pi’s equal N(θ, σ2), for some unknown θ, σ Example: P =

• P1, N(θ, σ2), P3, N(θ, σ2), N(θ, σ2), N(θ, σ2), P7, N(θ, σ2)
• ◮ Ensures θ = θ(P) and σ = σ(P) uniquely defined

◮ Test H0,i : “Pi = N(θ(P), σ2(P))” against H1,i : “Pi = N(θ(P), σ2(P))” “item i comes from the background” “item i comes from signal”

Roquain, Etienne Sparse multiple testing Introduction 6 / 29

Setting

Observations Y = (Yi)1≤i≤n indep , Yi ∼ Pi, parameter P = (Pi)1≤i≤n ∈ P Gaussian null assumption: Most of the Pi’s equal N(θ, σ2), for some unknown θ, σ Example: P =

• P1, N(θ, σ2), P3, N(θ, σ2), N(θ, σ2), N(θ, σ2), P7, N(θ, σ2)
• ◮ Ensures θ = θ(P) and σ = σ(P) uniquely defined

◮ Test H0,i : “Pi = N(θ(P), σ2(P))” against H1,i : “Pi = N(θ(P), σ2(P))” “item i comes from the background” “item i comes from signal”

Roquain, Etienne Sparse multiple testing Introduction 6 / 29

Setting

Observations Y = (Yi)1≤i≤n indep , Yi ∼ Pi, parameter P = (Pi)1≤i≤n ∈ P Gaussian null assumption: Most of the Pi’s equal N(θ, σ2), for some unknown θ, σ Example: P =

• P1, N(θ, σ2), P3, N(θ, σ2), N(θ, σ2), N(θ, σ2), P7, N(θ, σ2)
• ◮ Ensures θ = θ(P) and σ = σ(P) uniquely defined

◮ Test H0,i : “Pi = N(θ(P), σ2(P))” against H1,i : “Pi = N(θ(P), σ2(P))” “item i comes from the background” “item i comes from signal”

Roquain, Etienne Sparse multiple testing Introduction 6 / 29

Criteria

◮ True null set H0(P) = {i : P satisfies H0,i}, n0(P) = |H0(P)| ◮ False null set H1(P) = H0(P)c, n1(P) = |H1(P)| ◮ for a procedure R(Y) ⊂ {1, . . . , n} FDP(P, R(Y)) = |R(Y) ∩ H0(P)| |R(Y)| ∨ 1 ’false discovery proportion’ EP[FDP(P, R(Y))] = FDR(P, R) ’false discovery rate’ TDP(P, R(Y)) = |R(Y) ∩ H1(P)| n1(P) ∨ 1 ’true discovery proportion’ EP[TDP(P, R(Y))] = TDR(P, R) ’true discovery rate’ ◮ Sparse multiple testing (enough background) n1(P) ≤ kn with kn ’small’

Roquain, Etienne Sparse multiple testing Introduction 7 / 29

Criteria

◮ True null set H0(P) = {i : P satisfies H0,i}, n0(P) = |H0(P)| ◮ False null set H1(P) = H0(P)c, n1(P) = |H1(P)| ◮ for a procedure R(Y) ⊂ {1, . . . , n} FDP(P, R(Y)) = |R(Y) ∩ H0(P)| |R(Y)| ∨ 1 ’false discovery proportion’ EP[FDP(P, R(Y))] = FDR(P, R) ’false discovery rate’ TDP(P, R(Y)) = |R(Y) ∩ H1(P)| n1(P) ∨ 1 ’true discovery proportion’ EP[TDP(P, R(Y))] = TDR(P, R) ’true discovery rate’ ◮ Sparse multiple testing (enough background) n1(P) ≤ kn with kn ’small’

Roquain, Etienne Sparse multiple testing Introduction 7 / 29

Criteria

◮ True null set H0(P) = {i : P satisfies H0,i}, n0(P) = |H0(P)| ◮ False null set H1(P) = H0(P)c, n1(P) = |H1(P)| ◮ for a procedure R(Y) ⊂ {1, . . . , n} FDP(P, R(Y)) = |R(Y) ∩ H0(P)| |R(Y)| ∨ 1 ’false discovery proportion’ EP[FDP(P, R(Y))] = FDR(P, R) ’false discovery rate’ TDP(P, R(Y)) = |R(Y) ∩ H1(P)| n1(P) ∨ 1 ’true discovery proportion’ EP[TDP(P, R(Y))] = TDR(P, R) ’true discovery rate’ ◮ Sparse multiple testing (enough background) n1(P) ≤ kn with kn ’small’

Roquain, Etienne Sparse multiple testing Introduction 7 / 29

Criteria

◮ True null set H0(P) = {i : P satisfies H0,i}, n0(P) = |H0(P)| ◮ False null set H1(P) = H0(P)c, n1(P) = |H1(P)| ◮ for a procedure R(Y) ⊂ {1, . . . , n} FDP(P, R(Y)) = |R(Y) ∩ H0(P)| |R(Y)| ∨ 1 ’false discovery proportion’ EP[FDP(P, R(Y))] = FDR(P, R) ’false discovery rate’ TDP(P, R(Y)) = |R(Y) ∩ H1(P)| n1(P) ∨ 1 ’true discovery proportion’ EP[TDP(P, R(Y))] = TDR(P, R) ’true discovery rate’ ◮ Sparse multiple testing (enough background) n1(P) ≤ kn with kn ’small’

Roquain, Etienne Sparse multiple testing Introduction 7 / 29

Oracle procedure BH∗

α ◮ Rescaled observation Zi = (Yi − θ(P))/σ(P) ◮ Apply the standard BH procedure to the Zi’s:

• Sorting |Z|(1) ≥ |Z|(2) ≥ · · · ≥ |Z|(n)
• Quantiles

tk = Φ

−1(αk/(2n))

• Rejection number
• k = max{k : |Z|(k) ≥ tk}
• Select the Zi’s corresponding to |Z|(1), |Z|(2), . . . , |Z|(

k).

Theorem [Benjamini and Hochberg (1995), Benjamini and Yekutieli (2001)]

∀P ∈ P, FDR(P, BH∗

α) = αn0(P)/n

≃ α under sparsity

Roquain, Etienne Sparse multiple testing Introduction 8 / 29

Oracle procedure BH∗

α ◮ Rescaled observation Zi = (Yi − θ(P))/σ(P) ◮ Apply the standard BH procedure to the Zi’s:

• Sorting |Z|(1) ≥ |Z|(2) ≥ · · · ≥ |Z|(n)
• Quantiles

tk = Φ

−1(αk/(2n))

• Rejection number
• k = max{k : |Z|(k) ≥ tk}
• Select the Zi’s corresponding to |Z|(1), |Z|(2), . . . , |Z|(

k).

Theorem [Benjamini and Hochberg (1995), Benjamini and Yekutieli (2001)]

∀P ∈ P, FDR(P, BH∗

α) = αn0(P)/n

≃ α under sparsity

Roquain, Etienne Sparse multiple testing Introduction 8 / 29

Oracle procedure BH∗

α ◮ Rescaled observation Zi = (Yi − θ(P))/σ(P) ◮ Apply the standard BH procedure to the Zi’s:

• Sorting |Z|(1) ≥ |Z|(2) ≥ · · · ≥ |Z|(n)
• Quantiles

tk = Φ

−1(αk/(2n))

• Rejection number
• k = max{k : |Z|(k) ≥ tk}
• Select the Zi’s corresponding to |Z|(1), |Z|(2), . . . , |Z|(

k).

Theorem [Benjamini and Hochberg (1995), Benjamini and Yekutieli (2001)]

∀P ∈ P, FDR(P, BH∗

α) = αn0(P)/n

≃ α under sparsity

Roquain, Etienne Sparse multiple testing Introduction 8 / 29

Oracle procedure BH∗

α

• 5

10 15 20 25 30 1 2 3 4 5

• ● ●
• ● ● ● ● ● ● ● ● ● ●
• |Z(k)|

Rejected

Roquain, Etienne Sparse multiple testing Introduction 9 / 29

Optimality under a sparsity range

Procedure R optimal: R ≈ BH∗

α both for FDP and TDP

Definition

Procedure R optimal for a sparsity kn: there exists ηn → 0, s.t. (I) lim sup

n

sup

α∈(1/n,1/2)

     sup

P∈P n1(P)≤kn

{FDR(P, R)} − α      ≤ 0 (II) lim

n

sup

α∈(1/n,1/2)

     sup

P∈P n1(P)≤kn

• PY∼P
• TDP(P, R) < TDP(P, BH⋆

α(1−ηn))

• 

    = 0 ◮ Robust criteria: alternatives arbitrary

Roquain, Etienne Sparse multiple testing Introduction 10 / 29

Optimality under a sparsity range

Procedure R optimal: R ≈ BH∗

α both for FDP and TDP

Definition

Procedure R optimal for a sparsity kn: there exists ηn → 0, s.t. (I) lim sup

n

sup

α∈(1/n,1/2)

     sup

P∈P n1(P)≤kn

{FDR(P, R)} − α      ≤ 0 (II) lim

n

sup

α∈(1/n,1/2)

     sup

P∈P n1(P)≤kn

• PY∼P
• TDP(P, R) < TDP(P, BH⋆

α(1−ηn))

• 

    = 0 ◮ Robust criteria: alternatives arbitrary

Roquain, Etienne Sparse multiple testing Introduction 10 / 29

Optimality under a sparsity range

Procedure R optimal: R ≈ BH∗

α both for FDP and TDP

Definition

Procedure R optimal for a sparsity kn: there exists ηn → 0, s.t. (I) lim sup

n

sup

α∈(1/n,1/2)

     sup

P∈P n1(P)≤kn

{FDR(P, R)} − α      ≤ 0 (II) lim

n

sup

α∈(1/n,1/2)

     sup

P∈P n1(P)≤kn

• PY∼P
• TDP(P, R) < TDP(P, BH⋆

α(1−ηn))

• 

    = 0 ◮ Robust criteria: alternatives arbitrary

Roquain, Etienne Sparse multiple testing Introduction 10 / 29

1

Introduction

2

Upper bound

3

Lower bound

4

Additional results

5

One-sided alternatives

Roquain, Etienne Sparse multiple testing Upper bound 11 / 29

Plugged BH procedure

◮ Estimation: robust minimax estimator of θ(P), σ(P):

• θ = Y(⌈n/2⌉);

σ = U(⌈n/2⌉)/Φ

−1(1/4), Ui = |Yi − Y(⌈n/2⌉)|

• f L1 max risk ≍ (kn/n) ∨ n−1/2 for sparsity kn [Huber, 1964], [Chen et al. (2018)]

◮ Plugged BH procedure BH( θ, σ)

• Rescaled observation Z ′

i = (Yi −

θ)/ σ

• Apply the standard BH procedure to the Z ′

i ’s

Roquain, Etienne Sparse multiple testing Upper bound 12 / 29

Plugged BH procedure

◮ Estimation: robust minimax estimator of θ(P), σ(P):

• θ = Y(⌈n/2⌉);

σ = U(⌈n/2⌉)/Φ

−1(1/4), Ui = |Yi − Y(⌈n/2⌉)|

• f L1 max risk ≍ (kn/n) ∨ n−1/2 for sparsity kn [Huber, 1964], [Chen et al. (2018)]

◮ Plugged BH procedure BH( θ, σ)

• Rescaled observation Z ′

i = (Yi −

θ)/ σ

• Apply the standard BH procedure to the Z ′

i ’s

Roquain, Etienne Sparse multiple testing Upper bound 12 / 29

Plugged BH procedure

• 5

10 15 20 25 30 1 2 3 4 5

• ● ●
• ● ● ● ● ● ● ● ● ● ●
• ●
• |Z(k)|

Rejected |Z'(k)| Rejected'

Roquain, Etienne Sparse multiple testing Upper bound 13 / 29

Upper bound

Heuristic: BHα( θ, σ) ≈ BH∗

α if

| θ − θ(P)| ≪ min

k

• Φ

−1(αk/n) − Φ −1(α(k + 1)/n)

• ≈ 1/
• log n

| σ − σ(P)| ≪ min

k

• (Φ

−1(αk/n) − Φ −1(α(k + 1)/n))/Φ −1(αk/n)

• ≈ 1/ log n

Suggest BHα( θ, σ) ≈ BH∗

α for kn/n ≪ 1/ log(n).

Proposition 1 [R. and Verzelen (2020)]

BHα( θ, σ) is optimal for any sparsity sequence kn ≪ n/ log(n). Proof: rescaling of p-value process; combining BH procedure and ( θ, σ) leave-one-out properties {pi( θ, σ) ≤ Tα(Y; θ, σ)} ⊂ {pi( θ(i), σ(i)) ≤ Tα(Y (i); θ(i), σ(i))}.

Roquain, Etienne Sparse multiple testing Upper bound 14 / 29

Upper bound

Heuristic: BHα( θ, σ) ≈ BH∗

α if

| θ − θ(P)| ≪ min

k

• Φ

−1(αk/n) − Φ −1(α(k + 1)/n)

• ≈ 1/
• log n

| σ − σ(P)| ≪ min

k

• (Φ

−1(αk/n) − Φ −1(α(k + 1)/n))/Φ −1(αk/n)

• ≈ 1/ log n

Suggest BHα( θ, σ) ≈ BH∗

α for kn/n ≪ 1/ log(n).

Proposition 1 [R. and Verzelen (2020)]

BHα( θ, σ) is optimal for any sparsity sequence kn ≪ n/ log(n). Proof: rescaling of p-value process; combining BH procedure and ( θ, σ) leave-one-out properties {pi( θ, σ) ≤ Tα(Y; θ, σ)} ⊂ {pi( θ(i), σ(i)) ≤ Tα(Y (i); θ(i), σ(i))}.

Roquain, Etienne Sparse multiple testing Upper bound 14 / 29

Upper bound

Heuristic: BHα( θ, σ) ≈ BH∗

α if

| θ − θ(P)| ≪ min

k

• Φ

−1(αk/n) − Φ −1(α(k + 1)/n)

• ≈ 1/
• log n

| σ − σ(P)| ≪ min

k

• (Φ

−1(αk/n) − Φ −1(α(k + 1)/n))/Φ −1(αk/n)

• ≈ 1/ log n

Suggest BHα( θ, σ) ≈ BH∗

α for kn/n ≪ 1/ log(n).

Proposition 1 [R. and Verzelen (2020)]

BHα( θ, σ) is optimal for any sparsity sequence kn ≪ n/ log(n). Proof: rescaling of p-value process; combining BH procedure and ( θ, σ) leave-one-out properties {pi( θ, σ) ≤ Tα(Y; θ, σ)} ⊂ {pi( θ(i), σ(i)) ≤ Tα(Y (i); θ(i), σ(i))}.

Roquain, Etienne Sparse multiple testing Upper bound 14 / 29

1

Introduction

2

Upper bound

3

Lower bound

4

Additional results

5

One-sided alternatives

Roquain, Etienne Sparse multiple testing Lower bound 15 / 29

Idea

Procedure BH∗

−4 −2 2 4 0.00 0.10 0.20 0.30

pi0*N(0,1) (1−pi0)*f1

◮ rejects something

−4 −2 2 4 0.00 0.10 0.20 0.30

pi0*N(0,3) (1−pi0)*f2

◮ does not reject anything

Roquain, Etienne Sparse multiple testing Lower bound 16 / 29

Idea

Any procedure R = R(Y)

−4 −2 2 4 0.00 0.10 0.20 0.30

pi0*N(0,1) (1−pi0)*f1 mixture

−4 −2 2 4 0.00 0.10 0.20 0.30

pi0*N(0,3) (1−pi0)*f2 mixture

Does not distinguish between the two!

Roquain, Etienne Sparse multiple testing Lower bound 16 / 29

Idea

Any procedure R = R(Y)

−4 −2 2 4 0.00 0.10 0.20 0.30

pi0*N(0,1) (1−pi0)*f1 mixture

−4 −2 2 4 0.00 0.10 0.20 0.30

pi0*N(0,3) (1−pi0)*f2 mixture

Does not distinguish between the two! Not able to mimic BH∗

Roquain, Etienne Sparse multiple testing Lower bound 16 / 29

Lower bound

Proposition 2 [R. and Verzelen (2020)]

For a sparsity kn ≫ n/ log(n), there exists no optimal procedure. Proof : Le Cam’s two-point reduction scheme with the above configuration. ◮ for all n ≥ c1, any α ∈ (0, 1), any k with c2

n log(2/α) log(n)

≤ k < n/2 ◮ For any multiple testing procedure R such that FDR(P, R) ≤ c3 , for any P ∈ P with n1(P) ≤ k , ◮ Then there exists some P ∈ P with n1(P) ≤ k such that we have |R(Y) ∩ H1(P)| = 0 with P-proba ≥ 2/5 |BH⋆

α/2 ∩ H1(P)| ≥ c4α−1n1/2/ log1/2 n with P-proba ≥ 4/5.

Roquain, Etienne Sparse multiple testing Lower bound 17 / 29

Lower bound

Proposition 2 [R. and Verzelen (2020)]

For a sparsity kn ≫ n/ log(n), there exists no optimal procedure. Proof : Le Cam’s two-point reduction scheme with the above configuration. ◮ for all n ≥ c1, any α ∈ (0, 1), any k with c2

n log(2/α) log(n)

≤ k < n/2 ◮ For any multiple testing procedure R such that FDR(P, R) ≤ c3 , for any P ∈ P with n1(P) ≤ k , ◮ Then there exists some P ∈ P with n1(P) ≤ k such that we have |R(Y) ∩ H1(P)| = 0 with P-proba ≥ 2/5 |BH⋆

α/2 ∩ H1(P)| ≥ c4α−1n1/2/ log1/2 n with P-proba ≥ 4/5.

Roquain, Etienne Sparse multiple testing Lower bound 17 / 29

Main result

Theorem 1 [R. and Verzelen (2020)]

(i) for a sparsity kn ≫ n/ log(n), there exists no optimal procedure (of any kind); (ii) for a sparsity kn ≪ n/ log(n), BHα( θ, σ) is optimal with ( θ, σ) above. Procedure R optimal for a sparsity kn: there exists ηn → 0, s.t. (I) lim sup

n

sup

α∈(1/n,1/2)

     sup

P∈P n1(P)≤kn

{FDR(P, R)} − α      ≤ 0 (II) lim

n

sup

α∈(1/n,1/2)

     sup

P∈P n1(P)≤kn

• PY∼P
• TDP(P, R) < TDP(P, BH⋆

α(1−ηn))

• 

    = 0

Roquain, Etienne Sparse multiple testing Lower bound 18 / 29

1

Introduction

2

Upper bound

3

Lower bound

4

Additional results

5

One-sided alternatives

Roquain, Etienne Sparse multiple testing Additional results 19 / 29

No adaptation across boundary

Remark: always possible to achieve (I) by rejecting no null Reformulation Theorem 1: (i) if kn ≫ n/ log(n), possible to achieve (I) but not with (II); (ii) if kn ≪ n/ log(n), possible to achieve optimality (both (I) and (II)). Procedure achieving (i) and (ii)? NO !

Theorem 2 [R. and Verzelen (2020)]

◮ Any procedure achieving (I) for a sparsity kn ≫ n/ log(n) will fail to achieve optimality for a sparsity kn ≪ n/ log(n). ◮ Any procedure achieving optimality for a sparsity kn ≪ n/ log(n) will fail to achieve (I) for some regime kn ≫ n/ log(n). For instance, this is the case for BHα( θ, σ).

Roquain, Etienne Sparse multiple testing Additional results 20 / 29

No adaptation across boundary

Remark: always possible to achieve (I) by rejecting no null Reformulation Theorem 1: (i) if kn ≫ n/ log(n), possible to achieve (I) but not with (II); (ii) if kn ≪ n/ log(n), possible to achieve optimality (both (I) and (II)). Procedure achieving (i) and (ii)? NO !

Theorem 2 [R. and Verzelen (2020)]

◮ Any procedure achieving (I) for a sparsity kn ≫ n/ log(n) will fail to achieve optimality for a sparsity kn ≪ n/ log(n). ◮ Any procedure achieving optimality for a sparsity kn ≪ n/ log(n) will fail to achieve (I) for some regime kn ≫ n/ log(n). For instance, this is the case for BHα( θ, σ).

Roquain, Etienne Sparse multiple testing Additional results 20 / 29

Location model

Case where σ(P) is known

◮ only estimating θ(P) ◮ the sparsity boundary becomes n/ log1/2(n)

Extension to non-Gaussian null g(· − θ)

◮ g known, symmetric, continuous and non-increasing on R+ ◮ lower-bound and upper-bound matching up to some term ◮ Subbotin case: g(x) = L−1

ζ

e−|x|ζ/ζ, ζ > 1 The sparsity boundary becomes n/(log(n))1−1/ζ.

Roquain, Etienne Sparse multiple testing Additional results 21 / 29

Location model

Case where σ(P) is known

◮ only estimating θ(P) ◮ the sparsity boundary becomes n/ log1/2(n)

Extension to non-Gaussian null g(· − θ)

◮ g known, symmetric, continuous and non-increasing on R+ ◮ lower-bound and upper-bound matching up to some term ◮ Subbotin case: g(x) = L−1

ζ

e−|x|ζ/ζ, ζ > 1 The sparsity boundary becomes n/(log(n))1−1/ζ.

Roquain, Etienne Sparse multiple testing Additional results 21 / 29

1

Introduction

2

Upper bound

3

Lower bound

4

Additional results

5

One-sided alternatives

Roquain, Etienne Sparse multiple testing One-sided alternatives 22 / 29

One-sided setting

One sided assumption: ◮ the Pi’s under the alternative are assumed N(θ, σ2) ◮ easier problem

Proposition [Carpentier, Dellatre, R., Verzelen (2020)]

Estimation of θ: ◮ Identifiable as soon as k ≤ n − 1 ◮ Minimax rate

k/n log1/2(e∨(k2/n)) for sparsity 1 ≤ k ≤ 0.9n

◮ In particular, minimax rate 1/ log1/2(n) Estimation of σ: same with log1/2 replaced by log Remark: extra log in the convergence rate, useful for mimicking the oracle!

Roquain, Etienne Sparse multiple testing One-sided alternatives 23 / 29

One-sided setting

One sided assumption: ◮ the Pi’s under the alternative are assumed N(θ, σ2) ◮ easier problem

Proposition [Carpentier, Dellatre, R., Verzelen (2020)]

Estimation of θ: ◮ Identifiable as soon as k ≤ n − 1 ◮ Minimax rate

k/n log1/2(e∨(k2/n)) for sparsity 1 ≤ k ≤ 0.9n

◮ In particular, minimax rate 1/ log1/2(n) Estimation of σ: same with log1/2 replaced by log Remark: extra log in the convergence rate, useful for mimicking the oracle!

Roquain, Etienne Sparse multiple testing One-sided alternatives 23 / 29

Upper bound in one-sided case

Plugged BHα( θ, σ) (one-sided version) with new estimators:   

• θ = Y(qn) +

σ Φ

−1 qn n

• ;
• σ =

Y(qn)−Y(q′

n)

Φ

−1(q′ n/(n−ℓ0))−Φ −1(qn/n) ,

for ℓ0 ≤ ⌊0.9n⌋, qn = ⌊n3/4⌋ and q′

n = ⌊n1/4⌋.

Theorem [Carpentier, Dellatre, R., Verzelen (2020)]

BHα( θ, σ) is optimal for any sparsity sequence kn ≤ ⌊0.9n⌋ in the following sense: ◮ FDR control at level α as before. ◮ Mimics BH∗

α in terms of TDR = E(TDP) provided that ℓ0/n ≍ n1(P)/n

◮ Optimality even without sparsity!

Roquain, Etienne Sparse multiple testing One-sided alternatives 24 / 29

Upper bound in one-sided case

Plugged BHα( θ, σ) (one-sided version) with new estimators:   

• θ = Y(qn) +

σ Φ

−1 qn n

• ;
• σ =

Y(qn)−Y(q′

n)

Φ

−1(q′ n/(n−ℓ0))−Φ −1(qn/n) ,

for ℓ0 ≤ ⌊0.9n⌋, qn = ⌊n3/4⌋ and q′

n = ⌊n1/4⌋.

Theorem [Carpentier, Dellatre, R., Verzelen (2020)]

BHα( θ, σ) is optimal for any sparsity sequence kn ≤ ⌊0.9n⌋ in the following sense: ◮ FDR control at level α as before. ◮ Mimics BH∗

α in terms of TDR = E(TDP) provided that ℓ0/n ≍ n1(P)/n

◮ Optimality even without sparsity!

Roquain, Etienne Sparse multiple testing One-sided alternatives 24 / 29

Upper bound in one-sided case

Plugged BHα( θ, σ) (one-sided version) with new estimators:   

• θ = Y(qn) +

σ Φ

−1 qn n

• ;
• σ =

Y(qn)−Y(q′

n)

Φ

−1(q′ n/(n−ℓ0))−Φ −1(qn/n) ,

for ℓ0 ≤ ⌊0.9n⌋, qn = ⌊n3/4⌋ and q′

n = ⌊n1/4⌋.

Theorem [Carpentier, Dellatre, R., Verzelen (2020)]

BHα( θ, σ) is optimal for any sparsity sequence kn ≤ ⌊0.9n⌋ in the following sense: ◮ FDR control at level α as before. ◮ Mimics BH∗

α in terms of TDR = E(TDP) provided that ℓ0/n ≍ n1(P)/n

◮ Optimality even without sparsity!

Roquain, Etienne Sparse multiple testing One-sided alternatives 24 / 29

Outlook

Take home message

◮ Challenging and useful direction of research ◮ First results on the feasibility of using empirical null in BH procedure ◮ Good news: weak sparsity kn ≪ n/ log(n) enough to mimic the oracle ◮ Bad news: it is needed

Comments

◮ Robust minimax angle, so quite ’pessimistic’ ◮ One-sided structure on the alternatives makes the problem easier

Future work

◮ More structured alternatives ◮ Less structured nulls

Roquain, Etienne Sparse multiple testing One-sided alternatives 25 / 29

Outlook

Take home message

◮ Challenging and useful direction of research ◮ First results on the feasibility of using empirical null in BH procedure ◮ Good news: weak sparsity kn ≪ n/ log(n) enough to mimic the oracle ◮ Bad news: it is needed

Comments

◮ Robust minimax angle, so quite ’pessimistic’ ◮ One-sided structure on the alternatives makes the problem easier

Future work

◮ More structured alternatives ◮ Less structured nulls

Roquain, Etienne Sparse multiple testing One-sided alternatives 25 / 29

Outlook

Take home message

◮ Challenging and useful direction of research ◮ First results on the feasibility of using empirical null in BH procedure ◮ Good news: weak sparsity kn ≪ n/ log(n) enough to mimic the oracle ◮ Bad news: it is needed

Comments

◮ Robust minimax angle, so quite ’pessimistic’ ◮ One-sided structure on the alternatives makes the problem easier

Future work

◮ More structured alternatives ◮ Less structured nulls

Roquain, Etienne Sparse multiple testing One-sided alternatives 25 / 29

Some references

◮ Arias-Castro, E. and Chen, S. (2017). Distribution-free multiple testing. Electron.

• J. Stat., 11(1):1983–2001.

◮ Barber, R. F. and Candès, E. J. (2015). Controlling the false discovery rate via

• knockoffs. Ann. Statist., 43(5):2055–2085.

◮ Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. J. Roy. Statist. Soc. Ser. B, 57(1):289–300. ◮ Carpentier, A., Delattre, S., Roquain, E., and Verzelen, N. (2018). Estimating minimum effect with outlier selection. arXiv e-prints, page arXiv:1809.08330. ◮ Efron, B. (2004). Large-scale simultaneous hypothesis testing: the choice of a null hypothesis. J. Am. Stat. Assoc., 99(465):96–104. ◮ Efron, B. (2008). Microarrays, empirical Bayes and the two-groups model. Statist. Sci., 23(1):1–22. ◮ Ghosh, D. (2012). Incorporating the empirical null hypothesis into the Benjamini-Hochberg procedure. Stat. Appl. Genet. Mol. Biol., 11(4):Art. 11, front matter+19. ◮ Huber, P . J. (1964). Robust estimation of a location parameter. The annals of mathematical statistics, pages 73–101.

Roquain, Etienne Sparse multiple testing One-sided alternatives 26 / 29

Illustration - Gaussian alternative

k = n1/2 Oracle Estimated

−5 5 10 15 0.00 0.05 0.10 0.15 0.20 FDP = 0.2 TDP = 0.67 −5 5 10 15 0.00 0.05 0.10 0.15 0.20 FDP = 0.05 TDP = 0.63

Roquain, Etienne Sparse multiple testing One-sided alternatives 27 / 29

Illustration - Gaussian alternative

k = n3/4 Oracle Estimated

−10 −5 5 10 15 0.00 0.05 0.10 0.15 0.20 FDP = 0.12 TDP = 0.91 −10 −5 5 10 15 0.00 0.05 0.10 0.15 0.20 FDP = 0.056 TDP = 0.77

Roquain, Etienne Sparse multiple testing One-sided alternatives 27 / 29

Illustration - Gaussian alternative

k = 0.4n Oracle Estimated

−10 −5 5 10 15 0.00 0.05 0.10 0.15 0.20 FDP = 0.11 TDP = 0.95 −10 −5 5 10 15 0.00 0.05 0.10 0.15 0.20 FDP = 0 TDP = 0.57

Roquain, Etienne Sparse multiple testing One-sided alternatives 27 / 29

Illustration - f1 alternative

k = n1/2 Oracle Estimated

−3 −2 −1 1 2 3 0.0 0.1 0.2 0.3 0.4 FDP = 0 TDP = 0 −3 −2 −1 1 2 3 0.0 0.1 0.2 0.3 0.4 FDP = 0 TDP = 0

Roquain, Etienne Sparse multiple testing One-sided alternatives 28 / 29

Illustration - f1 alternative

k = n3/4 Oracle Estimated

−4 −2 2 4 6 0.0 0.1 0.2 0.3 0.4 FDP = 0.17 TDP = 0.65 −4 −2 2 4 6 0.0 0.1 0.2 0.3 0.4 FDP = 0 TDP = 0.028

Roquain, Etienne Sparse multiple testing One-sided alternatives 28 / 29

Illustration - f1 alternative

k = 0.4n Oracle Estimated

−10 −5 5 10 0.0 0.1 0.2 0.3 0.4 FDP = 0.13 TDP = 1 −10 −5 5 10 0.0 0.1 0.2 0.3 0.4 FDP = 0 TDP = 0.6

Roquain, Etienne Sparse multiple testing One-sided alternatives 28 / 29

Illustration - f2 alternative

k = n1/2 Oracle Estimated

−3 −2 −1 1 2 3 0.0 0.1 0.2 0.3 0.4 FDP = 0 TDP = 0 −3 −2 −1 1 2 3 0.0 0.1 0.2 0.3 0.4 FDP = 0 TDP = 0

Roquain, Etienne Sparse multiple testing One-sided alternatives 29 / 29

Illustration - f2 alternative

k = n3/4 Oracle Estimated

−4 −2 2 4 0.00 0.05 0.10 0.15 0.20 0.25 0.30 FDP = 0 TDP = 0 −4 −2 2 4 0.00 0.05 0.10 0.15 0.20 0.25 0.30 FDP = 1 TDP = 0

Roquain, Etienne Sparse multiple testing One-sided alternatives 29 / 29

Illustration - f2 alternative

k = 0.4n Oracle Estimated

−15 −10 −5 5 10 15 0.00 0.02 0.04 0.06 0.08 0.10 FDP = 0 TDP = 0 −15 −10 −5 5 10 15 0.00 0.02 0.04 0.06 0.08 0.10 FDP = 1 TDP = 0

Roquain, Etienne Sparse multiple testing One-sided alternatives 29 / 29