On Combining Significances. Trivial examples. S.I. Bityukov and N.V. - - PDF document

PhyStat 2011 CERN, Geneva, January 18, 2011

On Combining Significances. Trivial examples.

S.I. Bityukov and N.V. Krasnikov Institute for nuclear research RAS, Moscow, Russia “Suppose one experiment sees a 3-sigma effect and another sees a 4-sigma effect. What is combined sig- nificance?” (R. Cousins (2008). His conclusion - the question is not well-posed.) In this talk we discuss the problem of significances combining.

Plan

• Introduction
• Combining methods
• Examples
• Weighted combination
• The case of small statistics
• Systematics
• Conclusion

1

PhyStat 2011 CERN, Geneva, January 18, 2011

Introduction

Significance S is related with the probability by the relation (one-sided tail probability) S = Φ−1(1 − p) = −Φ−1(p), Φ(S) = 1 √ 2π

S

−∞ e

−t2 2 dt =

1 + erf( S

√ 2)

2 , S = √ 2erf−1(1 − 2p). For example, S = 5 corresponds to a p−value of 2.9 · 10−7. S = 4 = ⇒ p = 3.2 · 10−5. S = 3 = ⇒ p = 0.0013. S = 2 = ⇒ p = 0.023. S = 1 = ⇒ p = 0.16. So the problem – how to combine a set of a p−values

• r a set of of S−significances ?

2

PhyStat 2011 CERN, Geneva, January 18, 2011

Combining methods I

There are several methods for significances combin- ing:

• 1. Fisher’s (R.A. Fisher, 1932) method based on the

choice of P =

N

• i=1 pi. A simple way to combine pi is

the use of relation −2

• i ln pi = χ2

2N,p, where χ2 ν,p denotes the upper p

point of the probability integral of a central chi- squared of ν degrees of freedom. For two p1 and p2 combining (for example, F. James (2006)) p(p1, p2) = p1p2(1 − ln(p1p2)). Plus a lot of generalizations: I.J. Good (1955),

• H. Lancaster (1961) et al.
• 2. Tippett’s (L. Tippett, 1931) method using the small-

est pi p = 1 − (1 − (min pi))N ≈ N · min pi. Plus generalizations: B. Wilkinson (1991), . . . .

3

PhyStat 2011 CERN, Geneva, January 18, 2011

Combining methods II

• 3. Stouffer’s (S. Stouffer et al., 1949) method adding

the inverse normal of the pi’s

Φ−1(pi) =

√ NΦ−1(p), equivalently S =

Si

√ N . Plus generalizations:

• F. Mosteller and R. Bush

(1954), T. Liptak (1958), . . . , S. Bityukov et al. (2006). Here we shall compare the Fisher and Stouffer ap- proaches. Our conclusion: For high energy physics with Pois- son distribution in many cases the Stouffer approach is more natural.

4

PhyStat 2011 CERN, Geneva, January 18, 2011

Example I

Suppose the CMS experiment for some signature measures in july (2010) 10300 events and in august it detects 9700 events with the theoretical expectation λjuly = λaugust = 10000 in Poisson distribution Pois(n, λ) = λn n! e−λ. For λ ≫ 1, nobs ≫ 1 Poisson distribution is approx- imated by normal distribution with mean µ = λ and variance σ2 = λ and we find that Sjuly = |10300 − 10000| √ 10000 = 3, Saugust = |9700 − 10000| √ 10000 = 3. If we analyze data for july plus august we find (using the fact that the sum of two Poisson processes with λ1 and λ2 is a Poisson process with λ = λ1 + λ2) that Sjuly+august = |nobs,july + nobs,august − λjuly − λaugust|

λjuly + λaugust

= 0 in perfect agreement with a theory.

5

PhyStat 2011 CERN, Geneva, January 18, 2011

Example I

If we combine significances using formula p(p1, p2) = p1 p2(1 − ln(p1 p2)), we find that (remember that S = 3 = ⇒ pi = 0.0013, i = 1, 2) p = 0.000026; SFisher

july+august = 4.05

Example II

For other example with njuly = 10300, naugust = 10400 λjuly = λaugust = 10000 we find that Sjuly = 3, Saugust = 4, and Sjuly+august = 3 + 4 √ 2 ≈ 4.95. Fisher’s method gives: pjuly+august = 0.00000077; SFisher

july+august = 4.80 6

PhyStat 2011 CERN, Geneva, January 18, 2011

Weighted combination

For Poisson distribution with nobs ≫ 1 and λ ≫ 1 (when we can approximate this distribution by Gaus- sian) we can define significance as S1 = nobs1 − n01 σ1 , S2 = nobs2 − n02 σ2 . Here Si > 0 corresponds for excess of events, Si < 0 corresponds for shortage of events and σi = √λi, i = 1, 2. Usualy people use S = |nobs − n0 σ |. The rule for significances combining is S(S1, S2) = S1 σ1 + S2 σ2

• σ2

1 + σ2 2

. For general case S(S1, S2, . . . , Sn) = S1 σ1 + . . . + Sn σ2

• σ2

1 + . . . + σ2 2

.

7

PhyStat 2011 CERN, Geneva, January 18, 2011

The case of small statistics

Consider now the case of small statistics. Namely, as an example, consider CMS experiment with nobs,july = nobs,august = 1 and λ = λaugust = λ ≪ 1. Then the proba- bility to observe n ≥ 1 events is determined by Pois(n ≥ 1, λ) =

∞

• i=1 Pois(i, λ) ≈ λ.

Correspondingly, Pois(n ≥ 2, λ) =

∞

• i=2 Pois(i, λ) ≈ λ2

2 and we find that Pjuly = λjuly, Paugust = λaugust, Pjuly+august =

(λjuly+λaugust)2 2

=

(Pjuly+Paugust)2 2

.

• A. For Pjuly = Paugust = 0.023 (2σ : one − side)

Pjuly+august = 0.001 (3.07σ). Compare with Fisher’s formula P Fisher

july+august = Pjuly · Paugust(1 − ln(Pjuly · Paugust)).

Fisher’s method gives the value P Fisher

july+august = 0.0044 (2.62σ). 8

PhyStat 2011 CERN, Geneva, January 18, 2011

The case of small statistics

• B. For the case Sjuly = 3σ, Saugust = 4σ, njuly = 1 and

naugust = 1 Pjuly+august = 0.95 · 10−6 (4.76σ) P Fisher

july+august = 0.77 · 10−6 (4.81σ)

• C. For the case Sjuly = 3σ, Saugust = 3σ, njuly = 1 and

naugust = 1 Pjuly+august = 0.36 · 10−5 (4.49σ) P Fisher

july+august = 0.26 · 10−4 (4.05σ)

In general for nobs ≫ λ

∞

• n=n1 P(n, λ1) ≈ λn1

1

n1!e−λ1;

∞

• n=n2 P(n, λ2) ≈ λn2

2

n2!e−λ2 and

∞

• n=n1+n2

P(n, λ1 + λ2) ≈ (λ1 + λ2)n1+n2 n1 + n2! e−(λ1+λ2).

9

PhyStat 2011 CERN, Geneva, January 18, 2011

Systematics

Let us consider the influence of systematic effects related with nonexact knowledge of parameter λ in Poisson formula. Suppose nobs,july = nobs,august = 600, λjuly = λaugust = 300, and ǫ = 1 3 (uncertainty in the parameter λ determina- tion). For such case the significance is determined by ap- proximate formula S = nobs − λ

• λ + (ǫλ)2.

According to this formula Sjuly = Saugust = 300

• 300 + (100)2 ≈ 3.

For july+august Sjuly+august = 600

• 600 + (2 · 100)2 ≈ 3.

So we find that july+august combining does not help to increase significance, since the systematic er- ror ∼ (ǫλ)2 dominates.

10

PhyStat 2011 CERN, Geneva, January 18, 2011

Conclusions

To conclude we think that in many cases for Poisson statistics the most natural rules of significances com- bining is the use of the fact that the sum of Poisson processes is the Poisson process.

                  

Pois(n1, λ1) Pois(n2, λ2) . . . = ⇒ Pois(

ni, λi)

Pois(nn, λn) In fact it is natural generalization of the original Stouffer method.

11

PhyStat 2011 CERN, Geneva, January 18, 2011

References [1] R.D. Cousins, Annotated Bibliography of Some Papers on Combining Significances or p -values, arXiv:0705.2209 [physics.data-an]. [2] R.A. Fisher, Statistical Methods for Research Workers, Hafner, Darien, Connecticut, 14th edition, 1970. The method

• f combining significances seems to have appeared in the 4th

edition of 1932. [3] F. James, Statistical Methods in Experimental Physics, 2nd edition, World Scientific, 2006. [4] I.J. Good, On the weighted combination of significance tests, Journal of the Royal Statistical Society. Series B (Methodological) 17(2):264-265. [5] H. Lancaster, The combination of probabilities: an applica- tion of orthinormal functions. Australian Journal of Statis- tics, 3, 20-33, 1961. [6] L. Tippett, The Methods of Statistics, Williams and Nor- gate, Ltd., London, 1st edition. Sec. 3.5, 53-6, 1931, as cited by Birnbaum and by Westberg. [7] B. Wilkinson, A statistical consideration in psychological research, Psychological Bulletin, 48, 156-158, 1951. [8] S. Stouffer, E. Suchman, L. DeVinnery, S. Star, and R.W. Jr, The American Soldier, volume I: Adjustment during Army

• Life. Princeton University Press, 1949.

[9] F. Mosteller and R. Bush, Selected quantitative techniques, in ed. G. Lindzey, Handbookof Social Psychology vol. I, 289-334, Addison-Wesley, Cambridge, Mass., 1954.

12

PhyStat 2011 CERN, Geneva, January 18, 2011

[10] T. Liptak, On the combination of independent tests, Maguar Tud. Akad. Mat. Kutato Int. Kozl., 3, 171-197, 1958. [11] S.Bityukov, N.Krasnikov and A.Nikitenko, On the combin- ing significances, arXiv:physics/0612178(2006); S.Bityukov, N.Krasnikov, A.Nikitenko and V.Smirnova, Two approaches to combining significances, PoS ACAT08:118(2008).

13