Practical Statistics for Particle Physics Kyle Cranmer, New York - - PowerPoint PPT Presentation

SLIDE 1

Kyle Cranmer (NYU) CERN Summer School, July 2013 Center for Cosmology and Particle Physics

Kyle Cranmer,

New York University

Practical Statistics for Particle Physics

1

SLIDE 2

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Statistics plays a vital role in science, it is the way that we:

• quantify our knowledge and uncertainty
• communicate results of experiments

Big questions:

• how do we make discoveries, measure or exclude theoretical parameters, ...
• how do we get the most out of our data
• how do we incorporate uncertainties
• how do we make decisions

Statistics is a very big field, and it is not possible to cover everything in 4 hours. In these talks I will try to:

• explain some fundamental ideas & prove a few things
• enrich what you already know
• expose you to some new ideas

I will try to go slowly, because if you are not following the logic, then it is not very interesting.

• Please feel free to ask questions and interrupt at any time

2

Introduction

SLIDE 3

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Further Reading

By physicists, for physicists

• G. Cowan, Statistical Data Analysis, Clarendon Press, Oxford, 1998.

R.J.Barlow, A Guide to the Use of Statistical Methods in the Physical Sciences, John Wiley, 1989;

• F. James, Statistical Methods in Experimental Physics, 2nd ed., World Scientific, 2006;
• W.T. Eadie et al., North-Holland, 1971 (1st ed., hard to find);

S.Brandt, Statistical and Computational Methods in Data Analysis, Springer, New York, 1998. L.Lyons, Statistics for Nuclear and Particle Physics, CUP, 1986. My favorite statistics book by a statistician:

3

SLIDE 4

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Other lectures

Fred James’s lectures Glen Cowan’s lectures Louis Lyons Bob Cousins gave a CMS lecture, may give it more publicly Gary Feldman “Journeys of an Accidental Statistician” The PhyStat conference series at PhyStat.org:

4

http://www.desy.de/~acatrain/ http://www.pp.rhul.ac.uk/~cowan/stat_cern.html http://preprints.cern.ch/cgi-bin/setlink?base=AT&categ=Academic_Training&id=AT00000799 http://indico.cern.ch/conferenceDisplay.py?confId=a063350 http://www.hepl.harvard.edu/~feldman/Journeys.pdf

SLIDE 5

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Lecture notes

5

Contents

1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2 Conceptual building blocks for modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2.1 Probability densities and the likelihood function . . . . . . . . . . . . . . . . . . . . . . 3 2.2 Auxiliary measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.3 Frequentist and Bayesian reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.4 Consistent Bayesian and Frequentist modeling of constraint terms . . . . . . . . . . . . 7 3 Physics questions formulated in statistical language . . . . . . . . . . . . . . . . . . . . . 8 3.1 Measurement as parameter estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2 Discovery as hypothesis tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.3 Excluded and allowed regions as confidence intervals . . . . . . . . . . . . . . . . . . . 11 4 Modeling and the Scientific Narrative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 4.1 Simulation Narrative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 4.2 Data-Driven Narrative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 4.3 Effective Model Narrative . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.4 The Matrix Element Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 4.5 Event-by-event resolution, conditional modeling, and Punzi factors . . . . . . . . . . . . 28 5 Frequentist Statistical Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 5.1 The test statistics and estimators of µ and θ . . . . . . . . . . . . . . . . . . . . . . . . 29 5.2 The distribution of the test statistic and p-values . . . . . . . . . . . . . . . . . . . . . . 31 5.3 Expected sensitivity and bands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 5.4 Ensemble of pseudo-experiments generated with “Toy” Monte Carlo . . . . . . . . . . . 33 5.5 Asymptotic Formulas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 5.6 Importance Sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 5.7 Look-elsewhere effect, trials factor, Bonferoni . . . . . . . . . . . . . . . . . . . . . . . 37 5.8 One-sided intervals, CLs, power-constraints, and Negatively Biased Relevant Subsets . . 37 6 Bayesian Procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 6.1 Hybrid Bayesian-Frequentist methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 6.2 Markov Chain Monte Carlo and the Metropolis-Hastings Algorithm . . . . . . . . . . . 40 6.3 Jeffreys’s and Reference Prior . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 6.4 Likelihood Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 7 Unfolding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

Practical Statistics for the LHC

Kyle Cranmer Center for Cosmology and Particle Physics, Physics Department, New York University, USA Abstract This document is a pedagogical introduction to statistics for particle physics. Emphasis is placed on the terminology, concepts, and methods being used at the Large Hadron Collider. The document addresses both the statistical tests applied to a model of the data and the modeling itself . I expect to release updated versions of this document in the future.

SLIDE 6

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Outline

Lecture 1: Preliminaries

• Probability Density Function vs. Likelihood
• Monte Carlo
• Point estimates and maximum likelihood estimators

Lecture 2: Building a probability model

• A generic template for high energy physics
• Examples of different “narratives”

Lecture 3: Hypothesis testing

• The Neyman-Pearson lemma and the likelihood ratio
• Composite models and the profile likelihood ratio
• Review of ingredients for a hypothesis test

Lecture 4: Limits & Confidence Intervals

• The meaning of confidence intervals as inverted hypothesis tests
• Asymptotic properties of likelihood ratios
• Bayesian approach

6

SLIDE 7

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Lecture 1

7

SLIDE 8

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Terms

The next 3 lectures will rely on a clear understanding of these terms:

• Random variables / “observables” x
• Probability mass and probility density function (pdf) p(x)
• Parametrized Family of pdfs / “model” p(x|α)
• Parameter α
• Likelihood L(α)
• Estimate (of a parameter) α(x)

8

^

SLIDE 9

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Random variable / observable

“Observables” are quantities that we observe or measure directly

• They are random variables under repeated observation

Discrete observables:

• number of particles seen in a detector in some time interval
• particle type (electron, muon, ...) or charge (+,-,0)

Continuous observables:

• energy or momentum measured in a detector
• invariant mass formed from multiple particles

9

SLIDE 10

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Probability Mass Functions

When dealing with discrete random variables, define a Probability Mass Function as probability for ith possibility

Defined as limit of long term frequency

• probability of rolling a 3 := limit #trials→∞ (# rolls with 3 / # trials)
• you don’t need an infinite sample for definition to be useful

And it is normalized

10

P(xi) = pi X

i

P(xi) = 1

SLIDE 11

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Probability Density Functions

When dealing with continuous random variables, need to introduce the notion of a Probability Density Function Note, is NOT a probability PDFs are always normalized to unity:

11

P(x ∈ [x, x + dx]) = f(x)dx ∞

−∞

f(x)dx = 1 f(x)

x

• 3
• 2
• 1

1 2 3 f(x) 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

SLIDE 12

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Probability Density Functions

When dealing with continuous random variables, need to introduce the notion of a Probability Density Function Note, is NOT a probability PDFs are always normalized to unity:

11

P(x ∈ [x, x + dx]) = f(x)dx ∞

−∞

f(x)dx = 1 f(x)

x

• 3
• 2
• 1

1 2 3 f(x) 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

SLIDE 13

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Cumulative Density Functions

Often useful to use a cumulative distribution:

• in 1-dimension:

12

x

• 3
• 2
• 1

1 2 3 F(x) 0.2 0.4 0.6 0.8 1 x

• 3
• 2
• 1

1 2 3 f(x) 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

x

⇤

f(x⇥)dx⇥ = F(x)

SLIDE 14

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Cumulative Density Functions

Often useful to use a cumulative distribution:

• in 1-dimension:

12

f(x) = ∂F(x) ∂x

x

• 3
• 2
• 1

1 2 3 F(x) 0.2 0.4 0.6 0.8 1 x

• 3
• 2
• 1

1 2 3 f(x) 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

x

⇤

f(x⇥)dx⇥ = F(x)

• alternatively, define density

as partial of cumulative:

SLIDE 15

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Cumulative Density Functions

Often useful to use a cumulative distribution:

• in 1-dimension:

12

f(x) = ∂F(x) ∂x

x

• 3
• 2
• 1

1 2 3 F(x) 0.2 0.4 0.6 0.8 1 x

• 3
• 2
• 1

1 2 3 f(x) 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

x

⇤

f(x⇥)dx⇥ = F(x)

• alternatively, define density

as partial of cumulative:

• same relationship as total and

difgerential cross section:

f(E) = 1 σ ∂σ ∂E

SLIDE 16

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Cumulative Density Functions

Often useful to use a cumulative distribution:

• in 1-dimension:

12

f(x) = ∂F(x) ∂x

x

• 3
• 2
• 1

1 2 3 F(x) 0.2 0.4 0.6 0.8 1 x

• 3
• 2
• 1

1 2 3 f(x) 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

x

⇤

f(x⇥)dx⇥ = F(x)

• alternatively, define density

as partial of cumulative:

• same relationship as total and

difgerential cross section:

f(E) = 1 σ ∂σ ∂E

SLIDE 17

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Histogram {xi}→ f(x)

GIven a set of observations {xi} we can approximate the pdf with a histogram. Think of a pdf as a histogram with:

13

infinite data sample, zero bin width, normalized to unit area.

[G. Cowan]

SLIDE 18

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Accept/Reject Monte Carlo f(x) → {xi}

14

Mote Carlo techniques produce samples {xi} from f(x)

SLIDE 19

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Parametrized families / models

Often we are interested in a parametried family of pdfs

• We will write these as: said “f of x given α”
• where α are the parameters of the “model” (written in greek characters)

A discrete example:

• The Poisson distribution is a probability mass function for n, the

number of events one observes, when one expects μ events A continuous example

• The Gaussian distribution is a probability density function for a

continuous variable x characterized by a mean μ and standard deviation σ

15

f(x|α)

Pois(n|µ) = µn e−µ n!

G(x|µ, σ) = 1 √ 2πσ e− (x−µ)2

2σ2

SLIDE 20

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

The Likelihood Function

Consider the Poisson distribution describes a discrete event count n for a real-valued mean µ. The likelihood of µ given n is the same equation evaluated as a function of µ

• Now it’s a continuous function
• But it is not a pdf!

Common to plot the -ln L (or -2 ln L)

• helps avoid thinking of it as a PDF
• connection to χ2 distribution

16

Figure from R. Cousins,

• Am. J. Phys. 63 398 (1995)

L(µ) = Pois(n|µ) Pois(n|µ) = µn e−µ n!

SLIDE 21

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Repeated observations

In particle physics we are usually able to perform repeated

• bservations of x that are independent and identically distributed
• These repeated observations are written {xi}
• and the likelihood in that case is
• and the log-likelihood is

17

L(α) = Y

i

f(xi|α) log L(α) = X

i

log f(xi|α)

SLIDE 22

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Estimators

Given some model and a set of observations {xi} often one wants to estimate the true value of α (assuming the model is true). An estimator is function of the data written

• Since the data are random, so is the resulting estimate
• often it is just written , where the x-dependence is implicit
• one can compute expectation of the estimator

Properties of estimators:

• bias (unbiased means bias=0)
• variance
• asymptotic bias limit of bias with infinite observations

18

f(x|α)

ˆ α(x1, . . . xn)

ˆ α

E[ˆ α(x)|α] − α

E[(ˆ α(x) − α)2|α] = Z (ˆ α(x) − α)2f(x|α)dx E[ˆ α(x)|α] = Z ˆ α(x)f(x|α)dx

SLIDE 23

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Maximum likelihood estimators

There are many different possible estimators, but the most well- known and well-studied is the maximum likelihood estimator (MLE)

19

Figure from R. Cousins,

• Am. J. Phys. 63 398 (1995)

ˆ α(x) = argmaxαL(α) = argmaxαf(x|α)

This is just the value of α that maximizes the likelihood

Example: the Poisson distribution

Maximizing L(μ) is the same as minimizing -ln L(μ)

Pois(n|µ) = µn e−µ n!

⇒ ˆ µ = n

− d dµ ln L(µ)

• ˆ

µ = 0 = d

dµ @µ − n ln µ + ln n! |{z}

const

1 A = 1 − n µ

In this case, the MLE is unbiased b/c E[n]=μ

SLIDE 24

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

A second example

Consider a set of observations {xi} and we want to estimate the mean of a Gaussian with known σ which gives (an unbiased estimator) . However, the MLE is biased It can be shown that is unbiased Thus, the MLE is asymptotially unbiased .

20

G(x|µ, σ) = 1 √ 2πσ e− (x−µ)2

2σ2

− d dµ ln L(µ)

• ˆ

µ = 0 = d

dµ @X

i

(xi − µ)2 2σ2 + ln √ 2πσ | {z }

const

1 A = X

i

(xi − µ) σ2

⇒ ˆ µ = 1 N X

i

xi

ˆ σ2 = 1 N X

i

(xi − µ)2

ˆ σ2 = 1 N − 1 X

i

(xi − µ)2

SLIDE 25

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Define covariance cov[x,y] (also use matrix notation Vxy) as Correlation coefficient (dimensionless) defined as If x, y, independent, i.e., , then → x and y, ‘uncorrelated’ N.B. converse not always true.

Covariance & Correlation

[G. Cowan]

SLIDE 26

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Correlation Coefficient examples

[G. Cowan]

SLIDE 27

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Correlation Coefficient examples

http://en.wikipedia.org/wiki/Correlation_and_dependence

SLIDE 28

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Mutual Information

A more general notion of ‘correlation’ comes from Mutual Information:

• it is symmetric: I(X;Y) = I(Y;X)
• if and only if X,Y totally independent: I(X;Y)=0
• possible for X,Y to be uncorrelated, but not independent

24

X Y

Mutual Information doesn’t seem to be used much within HEP, but it seems quite useful

SLIDE 29

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Cramér-Rao Bound

The minimum variance bound on an estimator is given by the Cramér-Rao inequality:

• simple univariate case:
• For an unbiased estimator the Cramér-Rao bound states
• where I(θ) is the Fisher information
• General form for multiple parameters:

Maximum Likelihood Estimators asymptotically reach this bound

25

var(ˆ θ) = E[(θ − ˆ θ)2] var(ˆ θ) ≥ 1 I(θ)

cov[ˆ θ|θ]ij ≥ I−1

ij (θ)

SLIDE 30

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Change of variables

What happens with x→ cos(x)

26

SLIDE 31

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Change of variables

If f(x) is the pdf for x and y(x) is a change of variables, then the pdf g(y) must satisfy We can rewrite the integral on the right therefore, the two pdfs are related by a Jacobian factor

27

Z y(xb)

y(xa)

g(y)dy = Z xb

xa

g(y(x))

• dy

dx

• dx

f(x) = g(y)

• dy

dx

• P(xa < x < xb) ≡

Z xb

xa

f(x)dx = Z y(xb)

y(xa)

g(y)dy ≡ P(y(xa) < y < y(xb))

SLIDE 32

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

An example

28

f(x) = g(y)

• dy

dx

• y(x) = cos(x)

g(y) = 1 2π 1 | sin(x)| = 1 2π 1 p 1 − y2

f(x) = 1 2π

SLIDE 33

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Summary

29

Change of variable x, change of parameter θ

• For pdf p(x|θ) and change of variable from x to y(x):

p(y(x)|θ) = p(x|θ) / |dy/dx|. Jacobian modifies probability density, guaranties that P( y(x1)< y < y(x2) ) = P(x1 < x < x2 ), i.e., that Probabilities are invariant under change of variable x. – Mode of probability density is not invariant (so, e.g., – Mode of probability density is not invariant (so, e.g., criterion of maximum probability density is ill-defined). – Likelihood ratio is invariant under change of variable x. (Jacobian in denominator cancels that in numerator).

• For likelihood L(θ) and reparametrization from θ to u(θ):

L(θ) = L(u(θ)) (!). – Likelihood L (θ) is invariant under reparametrization of parameter θ (reinforcing fact that L L is not a pdf in θ).

Bob Cousins, CMS, 2008

SLIDE 34

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Probability Integral Transform

Consider a specific change of variables related to the cumulative for some arbitrary f(x) Using our general change of variables formula: We find for this case the Jacobian factor is Thus

30

y(x) = Z x

1

f(x0)dx0 f(x) = g(y)

• dy

dx

• dy

dx

• = f(x)

g(y) = 1

SLIDE 35

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

A more efficient Monte Carlo technique

No inefficiency Requires inverse of cumulative F-1(y) Recall

31

x

• 3
• 2
• 1

1 2 3 F(x) 0.2 0.4 0.6 0.8 1 x

• 3
• 2
• 1

1 2 3 f(x) 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4

1) y=rand() 2) x=F-1(y)

f(x) = ∂F(x) ∂x

y=

SLIDE 36

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Summary

32

Probability Integral Transform

“…seems likely to be one of the most fruitful conceptions introduced into statistical theory in the last few years” − Egon Pearson (1938) Given continuous x ∈ (a,b), and its pdf p(x), let y(x) = !a

x p(x′) dx′ .

Then y ∈ (0,1) and p(y) = 1 (uniform) for all y. (!) So there always exists a metric in which the pdf is uniform. Many issues become more clear (or trivial) after this transformation*. (If x is discrete, some complications.) The specification of a Bayesian prior pdf p(µ) for parameter µ is equivalent to the choice of the metric f(µ) in which the pdf is uniform. This is a deep issue, not always recognized as such by users of flat prior pdf’s in HEP!

*And the inverse transformation provides for efficient M.C. generation of p(x) starting from RAN().

Bob Cousins, CMS, 2008 16

SLIDE 37

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Bayes’ Theorem

Bayes’ theorem relates the conditional and marginal probabilities of events A & B

! ▪! P(A) is the prior probability or marginal probability of

• A. It is "prior" in the sense that it does not take into

account any information about B. ! ▪! P(A|B ) is the conditional probability of A, given B. It is also called the posterior probability because it is derived from or depends upon the specified value of B. ! ▪! P(B |A) is the conditional probability of B given A. ! ▪! P(B ) is the prior or marginal probability of B, and acts as a normalizing constant.

33

P(A|B) = P(B|A)P(A) P(B)

π(θ|x) = f(x|θ)π(θ) N ∝ L(θ)π(θ)

SLIDE 38

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

... in pictures (from Bob Cousins)

34

P, Conditional P, and Derivation of Bayes’ Theorem in Pictures

A B

Whole space

P(A) = P(B) = P(B|A) = P(A|B) = P(B) × P(A|B) = × = P(A ∩ B) = P(A) × P(B|A) = × = = P(A ∩ B) = P(A ∩ B) ! P(B|A) = P(A|B) × P(B) / P(A)

Bob Cousins, CMS, 2008

SLIDE 39

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

... in pictures (from Bob Cousins)

34

P, Conditional P, and Derivation of Bayes’ Theorem in Pictures

A B

Whole space

P(A) = P(B) = P(B|A) = P(A|B) = P(B) × P(A|B) = × = P(A ∩ B) = P(A) × P(B|A) = × = = P(A ∩ B) = P(A ∩ B) ! P(B|A) = P(A|B) × P(B) / P(A)

Bob Cousins, CMS, 2008

Don’t forget about “Whole space” . I will drop it from the notation typically, but occasionally it is important.

Ω

SLIDE 40

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Louis’s Example

35

P (Data;Theory) P (Theory;Data)

!

Theory = male or female Data = pregnant or not pregnant P (pregnant ; female) ~ 3% but P (female ; pregnant) >>>3%

SLIDE 41

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Axioms of Probability

These Axioms are a mathematical starting point for probability and statistics

• 1. probability for every element, E, is non-

negative

• 2. probability for the entire space of

possibilities is 1

• 3. if elements Ei are disjoint, probability is

additive Consequences:

36

Kolmogorov axioms (1933)

SLIDE 42

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Different definitions of Probability

37

http://plato.stanford.edu/archives/sum2003/entries/probability-interpret/#3.1

|⇤ | ⇥⌅|2 = 1 2 Frequentist

• defined as limit of long term frequency
• probability of rolling a 3 := limit of (# rolls with 3 / # trials)
• you don’t need an infinite sample for definition to be useful
• sometimes ensemble doesn’t exist
• eg. P(Higgs mass = 120 GeV), P(it will snow tomorrow)
• Intuitive if you are familiar with Monte Carlo methods
• compatible with orthodox interpretation of probability in Quantum
• Mechanics. Probability to measure spin projected on x-axis if spin of beam

is polarized along +z Subjective Bayesian

• Probability is a degree of belief (personal, subjective)
• can be made quantitative based on betting odds
• most people’s subjective probabilities are not coherent and do not obey

laws of probability

SLIDE 43

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Joke

38

“Bayesians address the question everyone is interested in, by using assumptions no-one believes” “Frequentists use impeccable logic to deal with an issue of no interest to anyone”

• L. Lyons

SLIDE 44

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Lecture 2

39

SLIDE 45

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Modeling: The Scientific Narrative

40

SLIDE 46

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Before one can discuss statistical tests, one must have a “model” for the data.

• by “model”, I mean the full structure of P(data | parameters)
• holding parameters fixed gives a PDF for data
• provides ability to generate pseudo-data (via Monte Carlo)
• holding data fixed gives a likelihood function for parameters
• note, likelihood function is not as general as the full model because it

doesn’t allow you to generate pseudo-data

Both Bayesian and Frequentist methods start with the model

• it’s the objective part that everyone can agree on
• it’s the place where our physics knowledge, understanding, and

intuiting comes in

• building a better model is the best way to improve your statistical

procedure

41

Building a model of the data

SLIDE 47

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Visualizing probability models

42

G(x|µ, σ)

(µ, σ)

I will represent PDFs graphically as below (directed acyclic graph)

• eg. a Gaussian is parametrized by
• every node is a real-valued function of the nodes below

G x µ σ

SLIDE 48

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

RooFit: A data modeling toolkit

43

Wouter Verkerke, UCSB

– Composition (‘plug & play’) – Convolution g(x;m,s) m(y;a0,a1)

=

⊗

=

g(x,y;a0,a1,s)

Possible in any PDF No explicit support in PDF code needed Wouter Verkerke,

– Addition – Multiplication

+ =

*

=

RooAddPdf

sum

RooGaussian

gauss1

RooGaussian

gauss2

RooArgusBG

argus

RooRealVar

g1frac

RooRealVar

g2frac

RooRealVar

x

RooRealVar

sigma

RooRealVar

mean1

RooRealVar

mean2

RooRealVar

argpar

RooRealVar

cutoff

RooFit is a major tool developed at BaBar for data modeling. RooStats provides higher-level statistical tools based on these PDFs.

SLIDE 49

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

The Scientific Narrative

The model can be seen as a quantitative summary of the analysis

• If you were asked to justify your modeling, you would tell a

story about why you know what you know

• based on previous results and studies performed along the way
• the quality of the result is largely tied to how convincing this

story is and how tightly it is connected to model I will describe a few “narrative styles”

• The “Monte Carlo Simulation” narrative
• The “Data Driven” narrative
• The “Effective Modeling” narrative

Real-life analyses often use a mixture of these

44

SLIDE 50

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

The Monte Carlo Simulation narrative

45

Let’s start with “the Monte Carlo simulation narrative”, which is probably the most familiar

Matrix Element Transfer Functions Phase-space Integral

L(x|H0) = W W H µ+ µ−

• P =

SLIDE 51

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013 46

The simulation narrative

P = |f|i⇥|2 f|f⇥i|i⇥ P → Lσ dσ → |M|2dΩ

The language of the Standard Model is Quantum Field Theory Phase space Ω defines initial measure, sampled via Monte Carlo

1)

SLIDE 52

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013 46

The simulation narrative

P = |f|i⇥|2 f|f⇥i|i⇥ P → Lσ dσ → |M|2dΩ

The language of the Standard Model is Quantum Field Theory Phase space Ω defines initial measure, sampled via Monte Carlo

1)

SLIDE 53

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013 46

The simulation narrative

P = |f|i⇥|2 f|f⇥i|i⇥ P → Lσ dσ → |M|2dΩ

The language of the Standard Model is Quantum Field Theory Phase space Ω defines initial measure, sampled via Monte Carlo

1)

LSM = 1 4Wµν · Wµν − 1 4BµνBµν − 1 4Ga

µνGµν a

• kinetic energies and self-interactions of the gauge bosons

+ ¯ Lγµ(i∂µ − 1 2gτ · Wµ − 1 2g′Y Bµ)L + ¯ Rγµ(i∂µ − 1 2g′Y Bµ)R

• kinetic energies and electroweak interactions of fermions

+ 1 2

• (i∂µ − 1

2gτ · Wµ − 1 2g′Y Bµ) φ

• 2 − V (φ)
• W ±,Z,γ,and Higgs masses and couplings

+ g′′(¯ qγµTaq) Ga

µ

• interactions between quarks and gluons

+ (G1 ¯ LφR + G2 ¯ LφcR + h.c.)

• fermion masses and couplings to Higgs

¯ RφcL

W, Z H

SLIDE 54

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013 47

a) Perturbation theory used to systematically approximate the theory. b) splitting functions, Sudokov form factors, and hadronization models c) all sampled via accept/reject Monte Carlo P(particles | partons)

2)

The simulation narrative

SLIDE 55

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013 47

• hard scattering
• (QED) initial/final

state radiation

• partonic decays, e.g.

t → bW

• parton shower

evolution

• nonperturbative

gluon splitting

• colour singlets
• colourless clusters
• cluster fission
• cluster → hadrons
• hadronic decays

a) Perturbation theory used to systematically approximate the theory. b) splitting functions, Sudokov form factors, and hadronization models c) all sampled via accept/reject Monte Carlo P(particles | partons)

2)

The simulation narrative

SLIDE 56

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013 48

Next, the interaction of outgoing particles with the detector is simulated.

Detailed simulations of particle interactions with matter. Accept/reject style Monte Carlo integration of very complicated function P(detector readout | initial particles)

3)

The simulation narrative

SLIDE 57

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013 49

From the simulated response of the detector, we run reconstruction

algorithms on the simulated data as if it were from real data. This allows us to look at distribution of any observable that we can measure in data.

P( observable | detector readout)

4)

The simulation narrative

[GeV]

miss T

E 50 100 150 200 250 Events / 5 GeV

• 3

10

• 2

10

• 1

10 1 10

2

10

3

10

4

10

5

10

6

10

data Z+jets top Diboson W+jets Multijet =400 GeV)

H

Signal (m

ATLAS

=400 GeV)

H

(m ν ν ee → H

• 1

L dt = 35 pb

∫

= 7 TeV s

e+ e- mu- mu+

SLIDE 58

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013 49

From the simulated response of the detector, we run reconstruction

algorithms on the simulated data as if it were from real data. This allows us to look at distribution of any observable that we can measure in data.

P( observable | detector readout)

4)

The simulation narrative

[GeV]

miss T

E 50 100 150 200 250 Events / 5 GeV

• 3

10

• 2

10

• 1

10 1 10

2

10

3

10

4

10

5

10

6

10

data Z+jets top Diboson W+jets Multijet =400 GeV)

H

Signal (m

ATLAS

=400 GeV)

H

(m ν ν ee → H

• 1

L dt = 35 pb

∫

= 7 TeV s

e+ e- mu- mu+

SLIDE 59

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

The Effective Model Narrative

In contrast, one can describe a distribution with some parametric function

• “we fit background to a polynomial”, exponential, ...
• While this is convenient and the fit may be good, the narrative is weak

50

)]

2

[pb/(GeV/c

jj

/ dm σ d

• 6

10

• 5

10

• 4

10

• 3

10

• 2

10

• 1

10 1 10

2

10

3

10

4

10

)

• 1

CDF Run II Data (1.13 fb Fit = 1)

s

Excited quark ( f = f’ = f

2

300 GeV/c

2

500 GeV/c

2

700 GeV/c

2

900 GeV/c

2

1100 GeV/c

(a)

]

2

[GeV/c

jj

m

200 400 600 800 1000 1200 1400

(Data - Fit) / Fit

• 0.4
• 0.2

0.2 0.4 0.6 0.8

(b)

200 300 400 500 600 700

• 0.04
• 0.02

0.02 0.04

. . . PHYSICAL REVIEW D 79, 112002 (2009)

d dmjj ¼ p0ð1 xÞp1=xp2þp3lnðxÞ; x ¼ mjj= ffiffi ffi s p ;

500 1000 1500

Events

1 10

2

10

3

10

4

10

Data Fit

(500)

q*

(800)

q*

(1200)

q*

[GeV]

jj

Reconstructed m 500 1000 1500 B (D - B) /

• 2

2

ATLAS

= 7 TeV s

• 1

= 315 nb dt L

∫

SLIDE 60

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

The Effective Model Narrative

In contrast, one can describe a distribution with some parametric function

• “we fit background to a polynomial”, exponential, ...
• While this is convenient and the fit may be good, the narrative is weak

51

Events / 2 GeV

2000 4000 6000 8000 10000

ATLAS Preliminary γ γ → H

• 1

Ldt = 4.8 fb

∫

= 7 TeV, s

• 1

Ldt = 20.7 fb

∫

= 8 TeV, s Selected diphoton sample Data 2011+2012 =126.8 GeV)

H

Sig+Bkg Fit (m Bkg (4th order polynomial)

[GeV]

γ γ

m

100 110 120 130 140 150 160

Events - Fitted bkg

• 200
• 100

100 200 300 400 500

SLIDE 61

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

P(non|s) =

• db Pois(non|s + b) π(b),

Let’s consider a simplified problem that has been studied quite a bit to gain some insight into our more realistic and difficult problems

• number counting with background uncertainty
• in our main measurement we observe non with s+b expected
• and the background has some uncertainty
• but what is “background uncertainty”? Where did it come from?
• maybe we would say background is known to 10% or that it has some pdf
• then we often do a smearing of the background:
• Where does come from?
• did you realize that this is a Bayesian procedure that depends on some prior

assumption about what b is?

What do we mean by uncertainty?

52

π(b)

Pois(non|s + b)

π(b)

SLIDE 62

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

The Data-driven narrative

Regions in the data with negligible signal expected are used as control samples

• simulated events are used to estimate

extrapolation coefficients

• extrapolation coefficients may have

theoretical and experimental uncertainties

53 ]

2

[GeV/c

ll

m

20 40 60 80 100 120 140 160 180 200

events / bin

• 1

10 1 10

2

10

3

10

4

10 CMS Preliminary

=160 GeV

H

Signal, m W+Jets, tW di-boson t t Drell-Yan

Channel

• e

+

e

S.R.

WW → H

WW Top +jets W

) WW C.R.(

WW

Top

+jets W

C.R.(Top)

Top

+jets) W C.R.(

+jets W

W W C . R .

N

W W S . R .

N =

W W

α

Top C.R.

N

Top S.R.

N =

Top

α

+jets W C.R.

N

+jets W S.R.

N =

+jets W

α

Top C.R.(Top)

N

Top ) WW C.R.(

N =

Top

β

+jets W +jets) W C.R.(

N

+jets W ) WW C.R.(

N =

+jets W

β

Figure 10: Flow chart describing the four data samples used in the H → WW (∗) → νν analysis. S.R and C.R. stand for signal and control regions, respectively.

SLIDE 63

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

The Data-driven narrative

Regions in the data with negligible signal expected are used as control samples

• simulated events are used to estimate

extrapolation coefficients

• extrapolation coefficients may have

theoretical and experimental uncertainties

53 ]

2

[GeV/c

ll

m

20 40 60 80 100 120 140 160 180 200

events / bin

• 1

10 1 10

2

10

3

10

4

10 CMS Preliminary

=160 GeV

H

Signal, m W+Jets, tW di-boson t t Drell-Yan

Channel

• e

+

e

S.R.

WW → H

WW Top +jets W

) WW C.R.(

WW

Top

+jets W

C.R.(Top)

Top

+jets) W C.R.(

+jets W

W W C . R .

N

W W S . R .

N =

W W

α

Top C.R.

N

Top S.R.

N =

Top

α

+jets W C.R.

N

+jets W S.R.

N =

+jets W

α

Top C.R.(Top)

N

Top ) WW C.R.(

N =

Top

β

+jets W +jets) W C.R.(

N

+jets W ) WW C.R.(

N =

+jets W

β

Figure 10: Flow chart describing the four data samples used in the H → WW (∗) → νν analysis. S.R and C.R. stand for signal and control regions, respectively.

C.R. S.R. αWW

Notation for next slides:

# in S.R. → non # in C.R. → noff α WW → τ

SLIDE 64

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

P(non|s) =

• db Pois(non|s + b) π(b),

The “on/off” problem

Now let’s say that the background was estimated from some control region or sideband measurement.

• We can treat these two measurements simultaneously:
• main measurement: observe non with s+b expected
• sideband measurement: observe noff with expected
• In this approach “background uncertainty” is a statistical error
• justification and accounting of background uncertainty is much more clear

How does this relate to the smearing approach?

• while is based on data, it still depends on some original prior

54

τb

P(non, noff|s, b)

⌅ ⇤⇥ ⇧

joint model

= Pois(non|s + b)

⌅ ⇤⇥ ⇧

main measurement

Pois(noff|τb)

⌅ ⇤⇥ ⇧

sideband

.

π(b) = P(b|noff) = P(noff|b)η(b)

dbP(noff|b)η(b).

π(b)

η(b)

SLIDE 65

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

A General Purpose Statistical Model

55

SLIDE 66

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Marked Poisson Process

Channel: a subset of the data defined by some selection requirements.

• eg. all events with 4 electrons with energy > 10 GeV
• n: number of events observed in the channel
• ν: number of events expected in the channel

Discriminating variable: a property of those events that can be measured and which helps discriminate the signal from background

• eg. the invariant mass of two particles
• f(x): the p.d.f. of the discriminating variable x

Marked Poisson Process:

56

f(D|ν) = Pois(n|ν)

n

Y

e=1

f(xe)

D = {x1, . . . , xn}

SLIDE 67

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Mixture model

Sample: a sample of simulated events corresponding to particular type interaction that populates the channel.

• statisticians call this a mixture model

57

νtot = X

s∈samples

νs f(x) = 1 νtot X

s∈samples

νsfs(x) ,

[GeV]

miss T

E 50 100 150 200 250 Events / 5 GeV

• 3

10

• 2

10

• 1

10 1 10

2

10

3

10

4

10

5

10

6

10

data Z+jets top Diboson W+jets Multijet =400 GeV)

H

Signal (m

ATLAS

=400 GeV)

H

(m ν ν ee → H

• 1

L dt = 35 pb

∫

= 7 TeV s

RooRealSumPdf h2mu2nu_200_model_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_Signal_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_tt_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_WW_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_WZ_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_ZZ_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_W_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_Z_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_MultiJet_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooRealVar binWidth_obs_h2mu2nu_200_0_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_1_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_2_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_3_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_4_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_5_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_6_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_7_zz2l2nu

SLIDE 68

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Parametrizing the model

Parameters of interest (µ): parameters of the theory that modify the rates and shapes of the distributions, eg.

• the mass of a hypothesized particle
• the “signal strength” μ=0 no signal, μ=1 predicted signal rate

Nuisance parameters (θ or αp): associated to uncertainty in:

• response of the detector (calibration)
• phenomenological model of interaction in non-perturbative regime

Lead to a parametrized model:

58

ν → ν(α), f(x) → f(x|α)

α = (µ, θ)

f(D|α) = Pois(n|ν(α))

n

Y

e=1

f(xe|α)

SLIDE 69

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Z+jets top Diboson ... syst 1 syst 2 ...

59

Tabulate effect of individual variations of sources of systematic uncertainty

• typically one at a time evaluated at nominal and “± 1 σ”
• use some form of interpolation to parametrize pth variation in terms of

nuisance parameter αp

Incorporating Systematic Effects

[GeV]

miss T

E 50 100 150 200 250 Events / 5 GeV

• 3

10

• 2

10

• 1

10 1 10

2

10

3

10

4

10

5

10

6

10

data Z+jets top Diboson W+jets Multijet =400 GeV)

H

Signal (m

ATLAS

=400 GeV)

H

(m ν ν ee → H

• 1

L dt = 35 pb

∫

= 7 TeV s

f(D|α) = Pois(n|ν(α))

n

Y

e=1

f(xe|α)

SLIDE 70

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013 59

• bs_H_4e_channel_200

170 180 190 200 210 220 230 alpha_XS_BKG_ZZ_4l_zz4l 0.05 0.1 0.15 0.2 0.25

f(x) x

Tabulate effect of individual variations of sources of systematic uncertainty

• typically one at a time evaluated at nominal and “± 1 σ”
• use some form of interpolation to parametrize pth variation in terms of

nuisance parameter αp

Incorporating Systematic Effects

[GeV]

miss T

E 50 100 150 200 250 Events / 5 GeV

• 3

10

• 2

10

• 1

10 1 10

2

10

3

10

4

10

5

10

6

10

data Z+jets top Diboson W+jets Multijet =400 GeV)

H

Signal (m

ATLAS

=400 GeV)

H

(m ν ν ee → H

• 1

L dt = 35 pb

∫

= 7 TeV s

f(D|α) = Pois(n|ν(α))

n

Y

e=1

f(xe|α)

SLIDE 71

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013 59

Tabulate effect of individual variations of sources of systematic uncertainty

• typically one at a time evaluated at nominal and “± 1 σ”
• use some form of interpolation to parametrize pth variation in terms of

nuisance parameter αp

Incorporating Systematic Effects

[GeV]

miss T

E 50 100 150 200 250 Events / 5 GeV

• 3

10

• 2

10

• 1

10 1 10

2

10

3

10

4

10

5

10

6

10

data Z+jets top Diboson W+jets Multijet =400 GeV)

H

Signal (m

ATLAS

=400 GeV)

H

(m ν ν ee → H

• 1

L dt = 35 pb

∫

= 7 TeV s

f(D|α) = Pois(n|ν(α))

n

Y

e=1

f(xe|α)

SLIDE 72

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Visualizing the model for one channel

60

• bs_H_4e_channel_200

170 180 190 200 210 220 230 alpha_XS_BKG_ZZ_4l_zz4l 0.05 0.1 0.15 0.2 0.25

f(x) x

[GeV]

miss T

E 50 100 150 200 250 Events / 5 GeV

• 3

10

• 2

10

• 1

10 1 10

2

10

3

10

4

10

5

10

6

10

data Z+jets top Diboson W+jets Multijet =400 GeV)

H

Signal (m

ATLAS

=400 GeV)

H

(m ν ν ee → H

• 1

L dt = 35 pb

∫

= 7 TeV s

RooRealSumPdf h2mu2nu_200_model_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_Signal_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_tt_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_WW_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_WZ_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_ZZ_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_W_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_Z_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_MultiJet_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooRealVar binWidth_obs_h2mu2nu_200_0_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_1_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_2_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_3_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_4_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_5_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_6_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_7_zz2l2nu PiecewiseInterpolation Signal_h2mu2nu_200_Hist_alpha_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alphanominal_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooRealVar

• bs_h2mu2nu_200

RooHistFunc Signal_h2mu2nu_200_Hist_alpha_0low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_1low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_2low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_3low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_4low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_5low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_6low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_7low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_8low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_0high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_1high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_2high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_3high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_4high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_5high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_6high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_7high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_8high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooRealVar M_SCALE RooRealVar E_RES RooRealVar JER RooRealVar B_EFF RooRealVar JES RooRealVar M_RES_ID RooRealVar M_RES_MS RooRealVar E_SCALE RooProduct Signal_h2mu2nu_200_overallNorm_x_sigma_epsilon_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooStats::HistFactory::LinInterpVar Signal_h2mu2nu_200_epsilon_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooRealVar alpha_ShapeError_zz2l2nu RooRealVar alpha_StatsError_zz2l2nu RooRealVar E_EFF RooRealVar M_EFF RooRealVar MJET2E RooRealVar MJET2MU RooRealVar XS_GG RooRealVar XS_TOP RooRealVar XS_W RooRealVar XS_WW RooRealVar XS_WZ RooRealVar XS_Z RooRealVar XS_ZZ RooRealVar mu RooGaussian JES_gaus RooRealVar JES_sigma RooRealVar JES_mean

x

αp

f

fi(x) → fi(x|α)

νi → νi(α),

αp

SLIDE 73

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Visualizing the model for one channel

61

RooProdPdf model_h2mu2nu_200_zz2l2nu_edit RooRealSumPdf h2mu2nu_200_model_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_Signal_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct Signal_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit PiecewiseInterpolation Signal_h2mu2nu_200_Hist_alpha_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alphanominal_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooRealVar

• bs_h2mu2nu_200

RooHistFunc Signal_h2mu2nu_200_Hist_alpha_0low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_1low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_2low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_3low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_4low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_5low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_6low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_7low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_8low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_0high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_1high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_2high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_3high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_4high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_5high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_6high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_7high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Signal_h2mu2nu_200_Hist_alpha_8high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooRealVar M_SCALE RooRealVar E_RES RooRealVar JER RooRealVar B_EFF RooRealVar JES RooRealVar M_RES_ID RooRealVar M_RES_MS RooRealVar E_SCALE RooProduct Signal_h2mu2nu_200_overallNorm_x_sigma_epsilon_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooStats::HistFactory::LinInterpVar Signal_h2mu2nu_200_epsilon_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooRealVar alpha_ShapeError_zz2l2nu RooRealVar alpha_StatsError_zz2l2nu RooRealVar E_EFF RooRealVar M_EFF RooRealVar MJET2E RooRealVar MJET2MU RooRealVar XS_GG RooRealVar XS_TOP RooRealVar XS_W RooRealVar XS_WW RooRealVar XS_WZ RooRealVar XS_Z RooRealVar XS_ZZ RooRealVar mu RooAddition lumi RooRealVar lumi_mean_val RooProduct lumi_sigmaTimesDelta RooRealVar lumi_sigma_val RooRealVar LUMI RooProduct L_x_tt_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct tt_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit PiecewiseInterpolation tt_h2mu2nu_200_Hist_alpha_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alphanominal_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_0low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_1low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_2low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_3low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_4low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_5low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_6low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_7low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_8low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_0high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_1high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_2high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_3high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_4high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_5high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_6high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_7high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc tt_h2mu2nu_200_Hist_alpha_8high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooStats::HistFactory::LinInterpVar tt_h2mu2nu_200_epsilon_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooConstVar 1 RooProduct L_x_WW_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct WW_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit PiecewiseInterpolation WW_h2mu2nu_200_Hist_alpha_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alphanominal_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_0low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_1low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_2low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_3low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_4low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_5low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_6low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_7low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_8low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_0high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_1high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_2high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_3high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_4high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_5high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_6high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_7high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WW_h2mu2nu_200_Hist_alpha_8high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooStats::HistFactory::LinInterpVar WW_h2mu2nu_200_epsilon_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_WZ_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct WZ_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit PiecewiseInterpolation WZ_h2mu2nu_200_Hist_alpha_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alphanominal_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_0low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_1low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_2low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_3low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_4low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_5low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_6low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_7low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_8low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_0high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_1high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_2high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_3high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_4high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_5high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_6high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_7high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc WZ_h2mu2nu_200_Hist_alpha_8high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooStats::HistFactory::LinInterpVar WZ_h2mu2nu_200_epsilon_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_ZZ_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct ZZ_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit PiecewiseInterpolation ZZ_h2mu2nu_200_Hist_alpha_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alphanominal_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_0low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_1low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_2low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_3low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_4low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_5low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_6low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_7low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_8low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_0high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_1high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_2high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_3high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_4high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_5high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_6high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_7high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc ZZ_h2mu2nu_200_Hist_alpha_8high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooStats::HistFactory::LinInterpVar ZZ_h2mu2nu_200_epsilon_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_W_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct W_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit PiecewiseInterpolation W_h2mu2nu_200_Hist_alpha_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alphanominal_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_0low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_1low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_2low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_3low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_4low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_5low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_6low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_7low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_8low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_9low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_0high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_1high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_2high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_3high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_4high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_5high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_6high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_7high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_8high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc W_h2mu2nu_200_Hist_alpha_9high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooStats::HistFactory::LinInterpVar W_h2mu2nu_200_epsilon_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_Z_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct Z_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit PiecewiseInterpolation Z_h2mu2nu_200_Hist_alpha_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alphanominal_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_0low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_1low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_2low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_3low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_4low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_5low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_6low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_7low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_8low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_9low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_0high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_1high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_2high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_3high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_4high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_5high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_6high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_7high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_8high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc Z_h2mu2nu_200_Hist_alpha_9high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooStats::HistFactory::LinInterpVar Z_h2mu2nu_200_epsilon_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct L_x_MultiJet_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooProduct MultiJet_h2mu2nu_200_overallSyst_x_HistSyst_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit PiecewiseInterpolation MultiJet_h2mu2nu_200_Hist_alpha_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc MultiJet_h2mu2nu_200_Hist_alphanominal_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc MultiJet_h2mu2nu_200_Hist_alpha_0low_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooHistFunc MultiJet_h2mu2nu_200_Hist_alpha_0high_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooStats::HistFactory::LinInterpVar MultiJet_h2mu2nu_200_epsilon_zz2l2nu_model_h2mu2nu_200_zz2l2nu_edit RooRealVar binWidth_obs_h2mu2nu_200_0_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_1_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_2_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_3_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_4_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_5_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_6_zz2l2nu RooRealVar binWidth_obs_h2mu2nu_200_7_zz2l2nu RooGaussian alpha_ShapeErrorConstraint_zz2l2nu RooRealVar nom_alpha_ShapeError_zz2l2nu RooGaussian alpha_StatsErrorConstraint_zz2l2nu RooRealVar nom_alpha_StatsError_zz2l2nu RooGaussian LUMI_gaus RooRealVar LUMI_mean RooRealVar LUMI_sigma RooGaussian E_EFF_gaus RooRealVar E_EFF_sigma RooRealVar E_EFF_mean RooGaussian M_EFF_gaus RooRealVar M_EFF_sigma RooRealVar M_EFF_mean RooGaussian MJET2E_gaus RooRealVar MJET2E_sigma RooRealVar MJET2E_mean RooGaussian MJET2MU_gaus RooRealVar MJET2MU_sigma RooRealVar MJET2MU_mean RooGaussian XS_GG_gaus RooRealVar XS_GG_sigma RooRealVar XS_GG_mean RooGaussian XS_TOP_gaus RooRealVar XS_TOP_sigma RooRealVar XS_TOP_mean RooGaussian XS_W_gaus RooRealVar XS_W_sigma RooRealVar XS_W_mean RooGaussian XS_WW_gaus RooRealVar XS_WW_sigma RooRealVar XS_WW_mean RooGaussian XS_WZ_gaus RooRealVar XS_WZ_sigma RooRealVar XS_WZ_mean RooGaussian XS_Z_gaus RooRealVar XS_Z_sigma RooRealVar XS_Z_mean RooGaussian XS_ZZ_gaus RooRealVar XS_ZZ_sigma RooRealVar XS_ZZ_mean RooGaussian M_SCALE_gaus RooRealVar M_SCALE_sigma RooRealVar M_SCALE_mean RooGaussian E_RES_gaus RooRealVar E_RES_sigma RooRealVar E_RES_mean RooGaussian JER_gaus RooRealVar JER_sigma RooRealVar JER_mean RooGaussian B_EFF_gaus RooRealVar B_EFF_sigma RooRealVar B_EFF_mean RooGaussian JES_gaus RooRealVar JES_sigma RooRealVar JES_mean RooGaussian M_RES_ID_gaus RooRealVar M_RES_ID_sigma RooRealVar M_RES_ID_mean RooGaussian M_RES_MS_gaus RooRealVar M_RES_MS_sigma RooRealVar M_RES_MS_mean RooGaussian E_SCALE_gaus RooRealVar E_SCALE_sigma RooRealVar E_SCALE_mean

[GeV]

miss T

E 50 100 150 200 250 Events / 5 GeV

• 3

10

• 2

10

• 1

10 1 10

2

10

3

10

4

10

5

10

6

10

data Z+jets top Diboson W+jets Multijet =400 GeV)

H

Signal (m

ATLAS

=400 GeV)

H

(m ν ν ee → H

• 1

L dt = 35 pb

∫

= 7 TeV s

After parametrizing each component of the mixture model, the pdf for a single channel might look like this

SLIDE 74

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Simultaneous multi-channel model

Simultaneous Multi-Channel Model: Several disjoint regions of the data are modeled simultaneously. Identification of common parameters across many channels requires coordination between groups such that meaning of the parameters are really the same. where Control Regions: Some channels are not populated by signal processes, but are used to constrain the nuisance parameters

• attempt to describe systematics in a statistical language
• Prototypical Example: “on/off” problem with unknown

62

Dsim = {D1, . . . , Dcmax}

fsim(Dsim|α) = Y

c∈channels

" Pois(nc|νc(α))

nc

Y

e=1

fc(xce|α) #

νb

f(n, m|µ, νb) = Pois(n|µ + νb) | {z }

signal region

· Pois(m|τνb) | {z }

control region

SLIDE 75

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Constraint terms

Often detailed statistical model for auxiliary measurements that measure certain nuisance parameters are not available.

• one typically has MLE for αp, denoted ap and standard error

Constraint Terms: are idealized pdfs for the MLE.

• common choices are Gaussian, Poisson, and log-normal
• New: careful to write constraint term a frequentist way
• Previously: with uniform η

Simultaneous Multi-Channel Model with constraints: where

63

fp(ap|αp) for p ∈ S

for p ∈ S

Dsim = {D1, . . . , Dcmax} G = {ap}

, ftot(Dsim, G|α) = Y

c∈channels

" Pois(nc|νc(α))

nc

Y

e=1

fc(xce|α) # · Y

p∈S

fp(ap|αp)

π(αp|ap) = fp(ap|αp)η(αp)

SLIDE 76

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Conceptual building blocks

64

Experiment Ensemble Channel c ∈ channels fc (x | α) Event e ∈ events {1…nc} Observable(s) xec Sample s ∈ samples Distribution fsc (x | α) Expected Number of Events νs Constraint Term fp(ap | αp ) p ∈ parameters with constraints global observable a Parameter α, θ, μ Shape Variation fscp(x | αp = X ) A B C

Legend: A "has many" Bs. B "has a" C. Dashed is optional.

We will use the following mnemonic index conventions:

• e ∈ events
• b ∈ bins
• c ∈ channels
• s ∈ samples
• p ∈ parameters

SLIDE 77

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Combined ATLAS Higgs Search

State of the art: At the time of the discovery, the combined Higgs search included 100 disjoint channels and >500 nuisance parameters

• Models for individual channels come from about 11 sub-groups performing

dedicated searches for specific Higgs decay modes

• In addition low-level performance groups provide tools for evaluating

systematic effects and corresponding constraint terms

65

Higgs Decay Subsequent Additional Sub-Channels mH L [fb−1] Decay Range H → γγ – 9 sub-channels (pTt⊗ηγ ⊗conversion) 110-150 4.9 H → ZZ ℓℓℓ′ℓ′ {4e,2e2µ,2µ2e,4µ} 110-600 4.8 ℓℓν ν {ee,µµ} ⊗ {low pile-up, high pile-up} 200-280-600 4.7 ℓℓq q {b-tagged, untagged} 200-300-600 4.7 H → WW ℓνℓν {ee,eµ,µµ} ⊗ {0-jet, 1-jet, VBF} 110-300-600 4.7 ℓνqq′ {e,µ} ⊗ {0-jet, 1-jet} 300-600 4.7 H → τ+τ− ℓℓ4ν {eµ}⊗{0-jet} ⊕ {1-jet, VBF, VH} 110-150 4.7 ℓτhad3ν {e,µ} ⊗ {0-jet} ⊗ {Emiss

T

≷ 20 GeV} 110-150 4.7 ⊕ {e,µ} ⊗ {1-jet, VBF} τhadτhad2ν {1-jet} 110-150 4.7 VH → bb Z → νν Emiss

T

∈ {120−160,160−200,≥ 200 GeV} 110-130 4.6 W → ℓν pW

T ∈ {< 50,50−100,100−200,≥ 200 GeV}

110-130 4.7 Z → ℓℓ pZ

T ∈ {< 50,50−100,100−200,≥ 200 GeV}

110-130 4.7

SLIDE 78

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

Visualizing the combined model

66

State of the art: At the time of the discovery, the combined Higgs search included 100 disjoint channels and >500 nuisance parameters RooFit / RooStats: is the modeling language (C++) which provides technologies for collaborative modeling

• provides technology to publish likelihood functions digitally
• and more, it’s the full model so we can also generate pseudo-data

ftot(Dsim, G|α) = Y

c∈channels

" Pois(nc|νc(α))

nc

Y

e=1

fc(xce|α) # · Y

p∈S

fp(ap|αp)

SLIDE 79

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CLASHEP, Peru, March 2013

Evolution of Model Complexity

67

[GeV]

γ γ

m 100 110 120 130 140 150 160 Events / GeV 100 200 300 400 500 600 700 800

Data 2011 Total background =125 GeV, 1 x SM H m

ATLAS

• 1

Ldt = 4.9 fb

∫

= 7 TeV, s

γ γ → H

(a)

[GeV]

llll

m 100 200 300 400 500 600 Events / 10 GeV 2 4 6 8 10 12 14

Data 2011 Total background =125 GeV, 1 x SM H m

ATLAS

• 1

Ldt = 4.8 fb

∫

= 7 TeV, s

4l →

(*)

ZZ → H

(b)

[GeV]

llll

m 100 120 140 160 180 200 220 240 Events / 5 GeV 2 4 6 8 10 12

Data 2011 Total background =125 GeV, 1 x SM H m

ATLAS

• 1

Ldt = 4.8 fb

∫

= 7 TeV, s

4l →

(*)

ZZ → H

(c)

[GeV]

T

m 200 300 400 500 600 700 800 900 Events / 50 GeV 10 20 30 40 50 60 70 80 90

Data 2011 Total background =350 GeV, 1 x SM H m

ATLAS

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

ν ν ll → ZZ → H

(d)

[GeV]

llbb

m 200 300 400 500 600 700 800 Events / 18 GeV 1 2 3 4 5 6 7 8

Data 2011 Total background =350 GeV, 1 x SM H m

ATLAS

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

llbb → ZZ → H

(e)

[GeV]

lljj

m 200 300 400 500 600 700 800 Events / 20 GeV 20 40 60 80 100 120 140 160 180 200

Data 2011 Total background =350 GeV, 1 x SM H m

ATLAS

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

llqq → ZZ → H

(f)

[GeV]

T

m 60 80 100 120 140 160 180 200 220 240 Events / 10 GeV 10 20 30 40 50 60 70 80 90 100

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

ATLAS

Data 2011 =125 GeV, 1 x SM H m Total background

+0j ν l ν l → WW → H

(g)

[GeV]

T

m 60 80 100 120 140 160 180 200 220 240 Events / 10 GeV 5 10 15 20 25 30

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

ATLAS

Data 2011 =125 GeV, 1xSM H m Total background +1j ν l ν l → WW → H

(h)

[GeV]

T

m 50 100 150 200 250 300 350 400 450 Events / 10 GeV 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

ATLAS

Data 2011 =125 GeV, 1 x SM H m Total background

+2j ν l ν l → WW → H Not final selection

(i)

[GeV]

eff

m 40 60 80 100 120 140 160 180 200 Events / 15 GeV 500 1000 1500 2000 2500 3000 3500 4000

Data 2011 Total background =125 GeV, 10 x SM H m

ATLAS

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

+ 0j

lep

τ

lep

τ → H

(a)

[GeV]

τ τ

m 50 100 150 200 250 300 Events / 20 GeV 50 100 150 200 250

Data 2011 Total background =125 GeV, 10 x SM H m

ATLAS

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

+ 1j

lep

τ

lep

τ → H

(b)

[GeV]

τ τ

m 50 100 150 200 250 Events / 40 GeV 20 40 60 80 100 120 140 160 180

Data 2011 Total background =125 GeV, 10 x SM H m

ATLAS

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

+ 2j

lep

τ

lep

τ → H

(c)

[GeV]

MMC

m 50 100 150 200 250 300 350 400 Events / 10 GeV 200 400 600 800 1000 1200 1400 1600 1800

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

ATLAS

Data 2011 =125 GeV, 10xSM H m Total background

+ 0/1j

had

τ

lep

τ → H

(d)

[GeV]

MMC

m 50 100 150 200 250 300 350 400 Events / 20 GeV 5 10 15 20 25 30 35 40

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

ATLAS

Data 2011 =125 GeV, 10 x SM H m Total background

+ 2j

had

τ

lep

τ → H

(e)

[GeV]

τ τ

m 60 80 100 120 140 160 180 Events / 12 GeV 20 40 60 80 100 120

Data 2011 Total background =125 GeV, 10 x SM H m

ATLAS

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s had

τ

had

τ → H

(f)

[GeV]

b b

m 80-150 80-150 80-150 80-150 Events / 10 GeV 5 10 15 20 25 30 35 40 45 50

Data 2011 Total background =125 GeV, 5 x SM H m

ATLAS

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

b llb → ZH

(g)

[GeV]

b b

m 80-150 80-150 80-150 80-150 Events / 10 GeV 20 40 60 80 100 120 140 160 180

Data 2011 Total background =125 GeV, 5 x SM H m

ATLAS

• 1

Ldt = 4.7 fb

∫

= 7 TeV, s

b b ν l → WH

(h)

[GeV]

b b

m 80-150 80-150 80-150 Events / 10 GeV 5 10 15 20 25 30 35 40

Data 2011 Total background =125 GeV, 5 x SM H m

ATLAS

• 1

Ldt = 4.6 fb

∫

= 7 TeV, s

b b ν ν → ZH

(i)

25 50 75 100 Jan-09 Jan-10 Jan-11 Jan-12 Jan-13

Number of Datasets Combined

5000 10000 15000 20000 Jan-09 Jan-10 Jan-11 Jan-12 Jan-13

Number of Model Components

150 300 450 600 Jan-09 Jan-10 Jan-11 Jan-12 Jan-13

Number of Parameters in Likelihood

SLIDE 80

Kyle Cranmer (NYU)

Center for Cosmology and Particle Physics

CERN Summer School, July 2013

HistFactory

32 page documentation of HistFactory tool + manual

• currently a “living document”

68

http://cds.cern.ch/record/1456844