FDR and Online FDR Adel Javanmard and Andrea Montanari USC and - - PowerPoint PPT Presentation

FDR and Online FDR

Adel Javanmard and Andrea Montanari

USC and Stanford

December 11, 2015

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 1 / 34

Outline

1

Large-scale Hypothesis Testing

2

Controlling FDR

3

Controlling Online FDR

4

Conclusion

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 2 / 34

Large-scale Hypothesis Testing

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 3 / 34

Assume

◮ I am the CTO of a big web company ◮ ✙ 1000 data scientists ◮ ✙ 1000 ‘brilliant ideas’ per day

◮ Users are more likely to click on the first search result ◮ Users are more likely to on top right ads ◮ Users are more engaged with page layout A

◮ How to avoid wasting company resources?

Compute ‘significance level’ from data!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 4 / 34

Assume

◮ I am the CTO of a big web company ◮ ✙ 1000 data scientists ◮ ✙ 1000 ‘brilliant ideas’ per day

◮ Users are more likely to click on the first search result ◮ Users are more likely to on top right ads ◮ Users are more engaged with page layout A

◮ How to avoid wasting company resources?

Compute ‘significance level’ from data!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 4 / 34

Assume

◮ I am the CTO of a big web company ◮ ✙ 1000 data scientists ◮ ✙ 1000 ‘brilliant ideas’ per day

◮ Users are more likely to click on the first search result ◮ Users are more likely to on top right ads ◮ Users are more engaged with page layout A

◮ How to avoid wasting company resources?

Compute ‘significance level’ from data!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 4 / 34

Assume

◮ I am the CTO of a big web company ◮ ✙ 1000 data scientists ◮ ✙ 1000 ‘brilliant ideas’ per day

◮ Users are more likely to click on the first search result ◮ Users are more likely to on top right ads ◮ Users are more engaged with page layout A

◮ How to avoid wasting company resources?

Compute ‘significance level’ from data!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 4 / 34

Assume

◮ I am the CTO of a big web company ◮ ✙ 1000 data scientists ◮ ✙ 1000 ‘brilliant ideas’ per day

◮ Users are more likely to click on the first search result ◮ Users are more likely to on top right ads ◮ Users are more engaged with page layout A

◮ How to avoid wasting company resources?

Compute ‘significance level’ from data!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 4 / 34

Assume

◮ I am the CTO of a big web company ◮ ✙ 1000 data scientists ◮ ✙ 1000 ‘brilliant ideas’ per day

◮ Users are more likely to click on the first search result ◮ Users are more likely to on top right ads ◮ Users are more engaged with page layout A

◮ How to avoid wasting company resources?

Compute ‘significance level’ from data!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 4 / 34

Example

Idea: Users click more on the first search result than on the second Null H0: Users are equaly likely to click on first and second Data:

◮ n events ◮ n1 clicks on the first result ◮ n2 ❂ n n1 clicks on the second result

Idea H0 ✮ z ✑ n1 n2 ♣n ✙ N✭0❀ 1✮

◮ If z ✢ 1, then declare it significant

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 5 / 34

Example

Idea: Users click more on the first search result than on the second Null H0: Users are equaly likely to click on first and second Data:

◮ n events ◮ n1 clicks on the first result ◮ n2 ❂ n n1 clicks on the second result

Idea H0 ✮ z ✑ n1 n2 ♣n ✙ N✭0❀ 1✮

◮ If z ✢ 1, then declare it significant

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 5 / 34

Example

Idea: Users click more on the first search result than on the second Null H0: Users are equaly likely to click on first and second Data:

◮ n events ◮ n1 clicks on the first result ◮ n2 ❂ n n1 clicks on the second result

Idea H0 ✮ z ✑ n1 n2 ♣n ✙ N✭0❀ 1✮

◮ If z ✢ 1, then declare it significant

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 5 / 34

Example

Idea: Users click more on the first search result than on the second Null H0: Users are equaly likely to click on first and second Data:

◮ n events ◮ n1 clicks on the first result ◮ n2 ❂ n n1 clicks on the second result

Idea H0 ✮ z ✑ n1 n2 ♣n ✙ N✭0❀ 1✮

◮ If z ✢ 1, then declare it significant

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 5 / 34

Formally

z ✑ n1 n2 ♣n ✙ N✭0❀ 1✮ p-value (G ✘ N✭0❀ 1✮) p ✑ P✭G ✕ z✮ ❂

❩ ✶

z

ex 2❂2 ♣ 2✙ dx

◮ Null:

p ✘ Uniform✭❬0❀ 1❪✮

(Definition)

◮ Small p:

significant

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 6 / 34

Formally

z ✑ n1 n2 ♣n ✙ N✭0❀ 1✮ p-value (G ✘ N✭0❀ 1✮) p ✑ P✭G ✕ z✮ ❂

❩ ✶

z

ex 2❂2 ♣ 2✙ dx

◮ Null:

p ✘ Uniform✭❬0❀ 1❪✮

(Definition)

◮ Small p:

significant

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 6 / 34

Company policy

Bring your idea up only if p ✔ ☛

[☛ ❂ 0✿05, Fisher’s rule of thumb]

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 7 / 34

Company policy

Bring your idea up only if p ✔ ☛

[☛ ❂ 0✿05, Fisher’s rule of thumb]

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 7 / 34

Problem

◮ M ✙ 1000 hypotheses per day ◮ M☛ ✙ 1000 ✁ 0✿05 ❂ 50 pass the test ◮ Still too much waste

New company policy (Bonferroni): Bring up your idea only if p ✔ ☛M ❂ ☛❂M

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 8 / 34

Problem

◮ M ✙ 1000 hypotheses per day ◮ M☛ ✙ 1000 ✁ 0✿05 ❂ 50 pass the test ◮ Still too much waste

New company policy (Bonferroni): Bring up your idea only if p ✔ ☛M ❂ ☛❂M

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 8 / 34

Problem

◮ M ✙ 1000 hypotheses per day ◮ M☛ ✙ 1000 ✁ 0✿05 ❂ 50 pass the test ◮ Still too much waste

New company policy (Bonferroni): Bring up your idea only if p ✔ ☛M ❂ ☛❂M

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 8 / 34

Problem with Bonferroni

Bring up your idea only if p ✔ ☛M ❂ ☛❂M

◮ More data scientists ✮ Less sensitive ◮ ☛ false positives per day ✮ Does not scale with M

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 9 / 34

Problem with Bonferroni

Bring up your idea only if p ✔ ☛M ❂ ☛❂M

◮ More data scientists ✮ Less sensitive ◮ ☛ false positives per day ✮ Does not scale with M

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 9 / 34

What do we want to achieve?

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 10 / 34

FDR (Benjamini, Hochberg, 1995)

◮ M hypotheses ◮ D ✑ Total number of discoveries (positives) ◮ FD ✑ Number of false discoveries

FDR ❂ E

♥

FD max✭D❀ 1✮

♦

Interpretation: FDR ✔ 0✿1 ✮ At most 10✪ of the discoveries is false.

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 11 / 34

FDR (Benjamini, Hochberg, 1995)

◮ M hypotheses ◮ D ✑ Total number of discoveries (positives) ◮ FD ✑ Number of false discoveries

FDR ❂ E

♥

FD max✭D❀ 1✮

♦

Interpretation: FDR ✔ 0✿1 ✮ At most 10✪ of the discoveries is false.

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 11 / 34

Controlling FDR

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 12 / 34

Setting

Null hypotheses: H0❀1❀ H0❀2❀ ✿ ✿ ✿ ❀ H0❀M p-values: p1❀ p2❀ ✿ ✿ ✿ ❀ pM Ground truth: ✒1❀ ✒2❀ ✿ ✿ ✿ ❀ ✒M ❬H0❀i ✿ ✒i ❂ 0❪ Test ouput (p ❂ ✭pi✮1✔i✔M : T1✭p✮❀ T2✭p✮❀ ✿ ✿ ✿ ❀ TM ✭p✮ ✷ ❢0❀ 1❣ ✒i ❂ 0 ✮ pi ✘ Uniform✭❬0❀ 1❪✮

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 13 / 34

Setting

Null hypotheses: H0❀1❀ H0❀2❀ ✿ ✿ ✿ ❀ H0❀M p-values: p1❀ p2❀ ✿ ✿ ✿ ❀ pM Ground truth: ✒1❀ ✒2❀ ✿ ✿ ✿ ❀ ✒M ❬H0❀i ✿ ✒i ❂ 0❪ Test ouput (p ❂ ✭pi✮1✔i✔M : T1✭p✮❀ T2✭p✮❀ ✿ ✿ ✿ ❀ TM ✭p✮ ✷ ❢0❀ 1❣ ✒i ❂ 0 ✮ pi ✘ Uniform✭❬0❀ 1❪✮

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 13 / 34

Benjamini-Hochberg procedure

◮ Order the p-values

p✭1✮ ✔ p✭2✮ ✔ ✁ ✁ ✁ ✔ p✭M✮

◮ Set threshold

I ❂ max

♥

i ✷ ❬M❪ ✿ p✭i✮ ✔ i☛ M

♦

◮ Reject at level p✭I ✮:

T❵✭p✮ ❂

✭

1 if p❵ ✔ p✭I ✮,

• therwise.

Theorem (Benjamini, Hochberg, 1995)

If the p-values are independent, and BH is used, then FDR ✔ ☛

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 14 / 34

Benjamini-Hochberg procedure

◮ Order the p-values

p✭1✮ ✔ p✭2✮ ✔ ✁ ✁ ✁ ✔ p✭M✮

◮ Set threshold

I ❂ max

♥

i ✷ ❬M❪ ✿ p✭i✮ ✔ i☛ M

♦

◮ Reject at level p✭I ✮:

T❵✭p✮ ❂

✭

1 if p❵ ✔ p✭I ✮,

• therwise.

Theorem (Benjamini, Hochberg, 1995)

If the p-values are independent, and BH is used, then FDR ✔ ☛

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 14 / 34

Interpretation

◮ M0 true nulls, M1 ❂ M M0 true non-null ◮ Reject H0❀i if pi ✔ q

FD ✙ M0q D ❂ J✭q✮ ✑ max❢i ✿ p✭i✮ ❁ q❣ FDR ✙ FDR✭q✮ ✑ M0q J✭q✮ ✔ Mq J✭q✮

• FDR✭p✭I ✮✮ ✔ Mp✭I ✮

I ✔ ☛

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 15 / 34

Interpretation

◮ M0 true nulls, M1 ❂ M M0 true non-null ◮ Reject H0❀i if pi ✔ q

FD ✙ M0q D ❂ J✭q✮ ✑ max❢i ✿ p✭i✮ ❁ q❣ FDR ✙ FDR✭q✮ ✑ M0q J✭q✮ ✔ Mq J✭q✮

• FDR✭p✭I ✮✮ ✔ Mp✭I ✮

I ✔ ☛

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 15 / 34

Interpretation

◮ M0 true nulls, M1 ❂ M M0 true non-null ◮ Reject H0❀i if pi ✔ q

FD ✙ M0q D ❂ J✭q✮ ✑ max❢i ✿ p✭i✮ ❁ q❣ FDR ✙ FDR✭q✮ ✑ M0q J✭q✮ ✔ Mq J✭q✮

• FDR✭p✭I ✮✮ ✔ Mp✭I ✮

I ✔ ☛

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 15 / 34

Controlling Online FDR

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 16 / 34

Back to our company

BH policy: Collect M p-values every day, and run BH Problems:

◮ Centralized ◮ Controls end-of-day FDR

Not end-of-year FDR ✦ Online FDR control

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 17 / 34

Back to our company

BH policy: Collect M p-values every day, and run BH Problems:

◮ Centralized ◮ Controls end-of-day FDR

Not end-of-year FDR ✦ Online FDR control

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 17 / 34

Back to our company

BH policy: Collect M p-values every day, and run BH Problems:

◮ Centralized ◮ Controls end-of-day FDR

Not end-of-year FDR ✦ Online FDR control

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 17 / 34

Setting

Null hypotheses: H0❀1❀ H0❀2❀ ✿ ✿ ✿ ❀ H0❀M Sequence of p-values: one at each time p1❀ p2❀ p3❀ ✿ ✿ ✿ Ground truth: ✒1❀ ✒2❀ ✒3❀ ✿ ✿ ✿ ❬H0❀i ✿ ✒i ❂ 0❪ Test ouput (pt

1 ❂ ✭p1❀ ✿ ✿ ✿ ❀ pt✮:

T1✭p1

1✮❀ T2✭p2 1✮❀ T3✭p3 1✮❀ ✁ ✁ ✁ ✷ ❢0❀ 1❣

[Foster, Stine, 2007]

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 18 / 34

Setting

Null hypotheses: H0❀1❀ H0❀2❀ ✿ ✿ ✿ ❀ H0❀M Sequence of p-values: one at each time p1❀ p2❀ p3❀ ✿ ✿ ✿ Ground truth: ✒1❀ ✒2❀ ✒3❀ ✿ ✿ ✿ ❬H0❀i ✿ ✒i ❂ 0❪ Test ouput (pt

1 ❂ ✭p1❀ ✿ ✿ ✿ ❀ pt✮:

T1✭p1✮❀ T2✭p2❀ T1✮❀ T3✭p3❀ T1❀ T2✮❀ ✁ ✁ ✁ ✷ ❢0❀ 1❣ [Foster, Stine, 2007]

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 19 / 34

What do we want to control?

◮ FD✭n✮ ✑False discoveries up to time n ◮ D✭n✮ ✑ Total number of discoveries up to time n

FDR✭n✮ ✑ E

♥

FD✭n✮ max✭D✭n✮❀ 1✮

♦ Want FDR✭n✮ ✔ ☛ for all n, ✒

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 20 / 34

What do we want to control?

◮ FD✭n✮ ✑False discoveries up to time n ◮ D✭n✮ ✑ Total number of discoveries up to time n

FDR✭n✮ ✑ E

♥

FD✭n✮ max✭D✭n✮❀ 1✮

♦ Want FDR✭n✮ ✔ ☛ for all n, ✒

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 20 / 34

Trivial approach (Bonferroni)

◮ Choose ☞i ✷ ❬0❀ 1❪, P✶ i❂1 ☞i ✔ ☛ ◮ Set

Ti ❂

✭

1 if pi ✔ ☞i,

• therwise.

Indeed FDR✭n✮ ✔ E❢FD✭n✮❣ ✔

❳

i✿ ✒i❂0

P✭pi ✔ ☞i✮ ❂

❳

i✿ ✒i❂0

☞i ✔ ☛ Very conservative!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 21 / 34

Trivial approach (Bonferroni)

◮ Choose ☞i ✷ ❬0❀ 1❪, P✶ i❂1 ☞i ✔ ☛ ◮ Set

Ti ❂

✭

1 if pi ✔ ☞i,

• therwise.

Indeed FDR✭n✮ ✔ E❢FD✭n✮❣ ✔

❳

i✿ ✒i❂0

P✭pi ✔ ☞i✮ ❂

❳

i✿ ✒i❂0

☞i ✔ ☛ Very conservative!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 21 / 34

Trivial approach (Bonferroni)

◮ Choose ☞i ✷ ❬0❀ 1❪, P✶ i❂1 ☞i ✔ ☛ ◮ Set

Ti ❂

✭

1 if pi ✔ ☞i,

• therwise.

Indeed FDR✭n✮ ✔ E❢FD✭n✮❣ ✔

❳

i✿ ✒i❂0

P✭pi ✔ ☞i✮ ❂

❳

i✿ ✒i❂0

☞i ✔ ☛ Very conservative!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 21 / 34

A simple rule

LORD

(Levels based On Recent Discovery)

◮ Choose ☞i ✷ ❬0❀ 1❪, P✶ i❂1 ☞i ✔ ☛ ◮ ✜i ✑ Time of the last discovery before i ◮ Set

Ti ❂

✭

1 if pi ✔ ☞i✜i,

• therwise.

Each discovery resets everything.

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 22 / 34

A simple rule

LORD

(Levels based On Recent Discovery)

◮ Choose ☞i ✷ ❬0❀ 1❪, P✶ i❂1 ☞i ✔ ☛ ◮ ✜i ✑ Time of the last discovery before i ◮ Set

Ti ❂

✭

1 if pi ✔ ☞i✜i,

• therwise.

Each discovery resets everything.

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 22 / 34

A theorem

Theorem (Javanmard, Montanari, 2015)

If the null p-values are indepenent, then LORD satifies sup

✒

sup

n FDR✭n✮ ✔ ☛ ✿

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 23 / 34

Remarks

◮ Foster, Stine 2007:

◮ Introduced model ◮ Introduced alpha investing rules ◮ Proved they control mFDR (see next)

◮ Last theorem applies to generalized alpha investing ◮ LORD uses very little information on the past!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 24 / 34

Remarks

◮ Foster, Stine 2007:

◮ Introduced model ◮ Introduced alpha investing rules ◮ Proved they control mFDR (see next)

◮ Last theorem applies to generalized alpha investing ◮ LORD uses very little information on the past!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 24 / 34

Remarks

◮ Foster, Stine 2007:

◮ Introduced model ◮ Introduced alpha investing rules ◮ Proved they control mFDR (see next)

◮ Last theorem applies to generalized alpha investing ◮ LORD uses very little information on the past!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 24 / 34

FDRvs mFDR

mFDR✑✭n✮ ❂ E✒❢FD✭n✮❣ E✒❢D✭n✮❣ ✰ ✑ mFDR control ✻✮ FDR control

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 25 / 34

Example

2 2.5 3 3.5 4 0.1 0.2 0.3 0.4 0.5 0.6 0.7 FDR mFDR

t

Data

◮ Z1❀ ✿ ✿ ✿ ❀ Zn0 ✘iid N✭0❀ 1✮, ✭Zn0✰1❀ ✿ ✿ ✿ ❀ Zn✮ ✘ N✭✒✄ 1❀ ✚11T ✰ ✚I✮ ◮ n ❂ 3000, n0 ❂ 2700, ✒✄ ❂ 2, ✚ ❂ 0✿9

Rule Ti ❂

✭

1 if ❥Zi❥ ✕ t,

• therwise.

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 26 / 34

Statistical power?

Two-groups model ✒i ✘iid Bernoulli✭✙✮ ❀ P✒i✭pi ✔ x✮ ❂

✭

F✭x✮ ❂ x if ✒i ❂ 0, G✭x✮

• therwise.

‘Discoveries should keep coming’

◮ A good rule should have D✭n✮ ❂ ✂✭n✮.

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 27 / 34

Two experiments

500 1000 1500 2000 50 100 150 200 250 300 350 400 450 500 π = 0.01 π = 0.02 π = 0.04 π = 0.08 π = 0.16 π = 0.32 500 1000 1500 2000 50 100 150 200 250 300 350 400 450 500 π = 0.01 π = 0.02 π = 0.04 π = 0.08 π = 0.16 π = 0.32

n n

◮ Left: ✒i ✘iid ✭1 ✙✮✍0 ✰ ✙N✭0❀ ✛2✮, Zi ✘ N✭0❀ ✒i✮ ◮ Right: ✒i re-ordered, decreasing ❥✒i❥

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 28 / 34

A theorem

Theorem (Javanmard, Montanari, 2015)

Assume the two-groups model, and use of LORD. Then, almost surely lim

n✦✶

1 n D✭n✮ ✕ ❆✭G❀ ☞✮ ❀ ❆✭G❀ ☞✮ ✑

✥ ✶ ❳

k❂1

ePk

❵❂1 G✭☞❵✮

✦1

✿

◮ ❆✭G❀ ☞✮ ❃ 0 strictly if G✭☞❵✮ ❃ ✭1 ✰ ✧✮❂❵ for all ❵ large enough. ◮ Sufficient G✭x✮ ✙ G0x 1✰✍ as x ✦ 0.

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 29 / 34

Comparison under the Gaussian two-groups model

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.01 0.02 0.03 0.04 0.05 0.06 BH LOND LORD Bonferroni alpha−Investing

✙ FDR

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2 LOND LORD Bonferroni alpha−Investing

✙ Relative power to BH

TD✭n✮ ❂ True discoveries RelativePower✭n✮ ✑ E

♥

TD✭n✮ max✭TDBH✭n✮❀ 1✮

♦

✿

◮ n ❂ 1000, ✛2 ❂ 2 log n

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 30 / 34

Conclusion

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 31 / 34

What if I am not CTO of a big-data company?

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 32 / 34

Take the “company” as a metaphor for science

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 33 / 34

Conclusion

◮ FDR control is fundamental for reasoning about data ◮ Online FDR is likely more realistic

Thanks!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 34 / 34

Conclusion

◮ FDR control is fundamental for reasoning about data ◮ Online FDR is likely more realistic

Thanks!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 34 / 34

Conclusion

◮ FDR control is fundamental for reasoning about data ◮ Online FDR is likely more realistic

Thanks!

Andrea Montanari (Stanford) FDR and Online FDR December 11, 2015 34 / 34