Guiding New Physics Searches with Unsupervised Learning [DS, - - PowerPoint PPT Presentation

▶
guiding new physics searches with unsupervised learning
SMART_READER_LITE
LIVE PREVIEW

Guiding New Physics Searches with Unsupervised Learning [DS, - - PowerPoint PPT Presentation

Dec 30, 2023 326 likes •716 views

IML Working Group, CERN 2018-10-12 Guiding New Physics Searches with Unsupervised Learning [DS, Jacques - 1807.06038] Andrea De Simone andrea.desimone@sissa.it > New Physics ? Searches for New Physics

slide-1
SLIDE 1

Andrea De Simone

Guiding New Physics 
 Searches with
 Unsupervised Learning

IML Working Group, CERN 2018-10-12

andrea.desimone@sissa.it

[DS, Jacques - 1807.06038]

slide-2
SLIDE 2

> New Physics ?

  • A. De Simone
  • 1. New Physics (NP) is not accessible by LHC 


new particles are too light/heavy 


  • r interacting too weakly 





  • 2. We have not explored all the possibilities


new physics may be buried under large bkg 


  • r hiding behind unusual signatures

MAYBE:

2

Searches for New Physics Beyond the Standard Model have been negative so far…

slide-3
SLIDE 3

> New Physics ?

  • A. De Simone

“Don’t want to miss a thing” (in data)
 closer look at current data

get ready for upcoming data from next run
 
 


Model-independent search
 searches for specific models may be: 


  • time-consuming

  • insensitive to unexpected/unknown processes

3

slide-4
SLIDE 4

> New Statistical Test

  • A. De Simone

4

  • 1. model-independent: 


no assumption about underlying physical model to intepret data

more general


  • 2. non-parametric:


compare two samples as a whole (not just their means, etc.)
 fewer assumptions, no max likelihood estim.

  • 3. un-binned:


high-dim feature space partitioned without rectangular bins

retain full multi-dim info of data

Want a statistical test for NP which is:

slide-5
SLIDE 5

> Outline

  • A. De Simone
  • 1. Statistical test of dataset compatibility

  • Nearest-Neighbors Two-Sample Test

  • Identify Discrepancies

  • Include Uncertainties




  • 2. Applications to High-Energy Physics

5

slide-6
SLIDE 6

> Outline

  • A. De Simone

6

  • 1. Statistical test of dataset compatibility

  • Nearest-Neighbors Two-Sample Test

  • Identify Discrepancies

  • Include Uncertainties




  • 2. Applications to High-Energy Physics
slide-7
SLIDE 7
  • A. De Simone

> Two-sample Test

7

[a.k.a. “homogeneity test”]

Two sets:

probability distributions pB,pT unknown Benchmark: Trial:

B = {x0

1, . . . , x0 NB} iid

∼ pB T = {x1, . . . , xNT }

iid

∼ pT xi, x0

i ∈ RD

e.g.: simulated SM bkg real measured data

slide-8
SLIDE 8

easy…

  • A. De Simone

Are B,T drawn from the same prob. distribution?

> Two-sample Test

8

Two sets:

probability distributions pB,pT unknown Benchmark: Trial:

T = {x1, . . . , xNT }

iid

∼ pT xi, x0

i ∈ RD

B = {x0

1, . . . , x0 NB} iid

∼ pB

slide-9
SLIDE 9
  • A. De Simone

> Two-sample Test

9

Are B,T drawn from the same prob. distribution?

… hard!

Two sets:

probability distributions pB,pT unknown Benchmark: Trial:

T = {x1, . . . , xNT }

iid

∼ pT xi, x0

i ∈ RD

B = {x0

1, . . . , x0 NB} iid

∼ pB

slide-10
SLIDE 10

> Two-sample Test

  • A. De Simone

10

RECIPE:

  • 1. Density Estimator

reconstruct PDFs from samples 


  • 2. Test Statistic (TS)


measure “distance” between PDFs
 


  • 3. TS distribution

associate probabilities to TS 
 under null hypothesis H0: pB = pT
 


  • 4. p -value


accept/reject H0

slide-11
SLIDE 11
  • A. De Simone

T Divide the space in squared bins?

11

✓ easy ✓ can use simple statistics (e.g. )
 ✘ hard/slow/impossible in high-D

χ2 B

Nearest Neighbors!

[Schilling - 1986][Henze - 1988] [Wang et al. - 2005,2006] [Dasu et al. - 2006][Perez-Cruz - 2008] [Sugiyama et al. - 2011][Kremer et al, 2015]

Need un-binned 
 multivariate approach

> 1. Density Estimator

Find PDFs estimators:
 e.g. based on densities of points: ˆ pB(x), ˆ pT (x) ˆ pB,T (x) = ρB,T (x) NB,T

slide-12
SLIDE 12
  • A. De Simone

T B

xj 12 xj

  • Fix integer K.

  • Choose query point xj in T and 


draw it in B.


> 1. Density Estimator

slide-13
SLIDE 13
  • A. De Simone

T

rj,B xj 13 xj

B

  • Fix integer K.

  • Choose query point xj in T and 


draw it in B.


  • Find the distance rj,B of the 


Kth-NN of xj in B.


> 1. Density Estimator

slide-14
SLIDE 14
  • A. De Simone

T

xj rj,T 14 xj rj,B

B

  • Fix integer K.

  • Choose query point xj in T and 


draw it in B.


  • Find the distance rj,B of the 


Kth-NN of xj in B.


  • Find the distance rj,T of the 


Kth-NN of xj in T.


> 1. Density Estimator

slide-15
SLIDE 15
  • A. De Simone

T

  • Fix integer K.

  • Choose query point xj in T and 


draw it in B.


  • Find the distance rj,B of the 


Kth-NN of xj in B.


  • Find the distance rj,T of the 


Kth-NN of xj in T.


  • Estimate PDFs:

xj 15 xj rj,T rj,B

B ˆ pB(xj) = K NB 1 ωDrD

j,B

ˆ pT (xj) = K NT − 1 1 ωDrD

j,T

> 1. Density Estimator

slide-16
SLIDE 16
  • A. De Simone

> 2. Test Statistic

  • Measure of the “distance” between 2 PDFs

  • Define Test Statistic:


(detect under-/over-densities)

  • Related to Kullback-Leibler divergence as:



 


  • From NN-estimated PDFs:





  • Theorem: this estimator converges to DKL(pB ||pT),


in large sample limit

16

[Wang et al. - 2005,2006]

DKL(p||q) ≡ Z

RD p(x) log p(x)

q(x)dx

TS(B, T ) = 1 NT

NT

X

j=1

log ˆ pT (xj) ˆ pB(xj) TS(B, T ) = ˆ DKL(ˆ pT ||ˆ pB) TSobs = D NT

NT

X

j=1

log rj,B rj,T + log NB NT − 1

slide-17
SLIDE 17

> 3. Test Statistic Distribution

  • A. De Simone

17

Permutation test!

How is TS distributed? Assume pB=pT. Union set: T e T e B B

Random reshuffle Compute the test 
 statistic TSn on:

( ˜ B, ˜ T ) U = T ∪ B Repeat many times. f(TS|H0) ← {TSn}

Distribution of TS under H0:

[asymptotically normal with zero mean]

U

slide-18
SLIDE 18

> 4. p-value

  • A. De Simone

18

TS → TS0 ≡ TS − ˆ µ ˆ σ f 0(TS0|H0) = ˆ σf(ˆ µ + ˆ σTS0|H0) ˆ µ, ˆ σ :

  • mean,variance of TS distribution





  • Standardize the TS:





  • TS’ distributed according to 





  • Two-sided p-value:



 
 


  • Equivalent significance:

f(TS|H0) p = 2 Z +1

|TS0

  • bs|

f 0(TS0|H0)dTS0

Z ≡ Φ−1(1 − p/2)

slide-19
SLIDE 19

> 2D Gaussian Example

  • A. De Simone

19

exact KL 
 divergence

µB = ✓1.0 1.0 ◆ µT = ✓1.2 1.2 ◆

ΣB = ΣT = ✓1 1 ◆

pB = N(µB, ΣB) pT = N(µT , ΣT )

K = 5, Nperm = 1000

µB = ✓1.0 1.0 ◆ µT = ✓1.15 1.15 ◆

more data, more power

slide-20
SLIDE 20

INPUT: OUTPUT:


  • A. De Simone

p -value of the null hypothesis H0: pB = pT

[check compatibility between 2 samples] pB,pT unknown

20

Trial sample: Benchmark sample: K:

Nperm: T = {x1, . . . , xNT }

iid

∼ pT B = {x0

1, . . . , x0 NB} iid

∼ pB xi, x0

i ∈ RD

number of nearest neighbors number of permutations

> NN2ST: Summary

slide-21
SLIDE 21

K

  • N

N d e n s i t y r a t i

  • e

s t i m a t i

  • n

Test Statistic

p e r m u t a t i

  • n

t e s t

p value TS distribution

  • |TSobs|

TSobs

Benchmark sample Trial sample

|TSobs|

  • A. De Simone

21

> NN2ST: Summary

github.com/de-simone/NN2ST Python code:

slide-22
SLIDE 22

> Outline

  • A. De Simone
  • 1. Statistical test of dataset compatibility

  • Nearest-Neighbors Two-Sample Test

  • Identify Discrepancies

  • Include Uncertainties




  • 2. Applications to High-Energy Physics

22

slide-23
SLIDE 23

> Where are the discrepancies?

  • A. De Simone

23

  • 1. “Score” field over T: with:



 
 
 
 
 
 


  • 2. Identify points where 


They contribute the most to large TSobs

high-discrepancy (anomalous) regions


  • 3. Apply a clustering algorithm to group them

Bonus: Characterize regions with significant discrepancies

Z(x) > c Z(xj) ≡ u(xj) − ¯ u σu

Z x

u(xj) ≡ log rj,B rj,T

TSobs = D ¯ u + const

slide-24
SLIDE 24

> Outline

  • A. De Simone
  • 1. Statistical test of dataset compatibility

  • Nearest-Neighbors Two-Sample Test

  • Identify Discrepancies

  • Include Uncertainties




  • 2. Applications to High-Energy Physics

24

slide-25
SLIDE 25

> Sample Uncertainties

  • A. De Simone

25

  • 1. Model feature uncertainties



 [e.g. zero-mean gaussians]


  • 2. New samples by adding random noise


sampled from FB,T :
 
 
 


  • 3. Compute TS on new samples





  • 4. Repeat many times to reconstruct f(U)

How to include sample uncertainties?

T B FB(x), FT (x) Tu = {xi + ∆xi}NT

i=1

Bu = {x0

i + ∆x0 i}NB i=1

TSu ≡ TS(Bu, Tu) = TSobs + U

slide-26
SLIDE 26

> Sample Uncertainties

  • A. De Simone

26

  • f(TSu) is a convolution:


f(TSu) more spread than f(TS)
 


  • p-value computed from f(TSu) 





  • weaker significance, 


power degradation

How to include sample uncertainties?

f(TSu|H0) = f(TS|H0) ∗ f(U)

TSobs

slide-27
SLIDE 27

> 2D Gaussian with Uncertainties

  • A. De Simone

27

i = ✏xi gaussian uncorrelated errors (diagonal covariance) with fixed relative uncertainty

ΣB = ΣT = ✓ 1 1 ◆

pB = N(µB, ΣB) pT = N(µT , ΣT )

µB = ✓1.0 1.0 ◆ µT = ✓1.15 1.15 ◆

B,T gaussian samples: for each feature component i

slide-28
SLIDE 28

> NN2ST: Summary

  • A. De Simone

28

✓ general, model-independent
 ✓ fast, no optimization 


[ NB,T =20k, K=5, Nperm =1k, D=2: t ~ 2 mins NB,T =20k, K=5, Nperm =1k, D=8: t ~ 50 mins ]

✓ sensitive to unspecified signals
 ✓ useful when no variable can separate sig/bkg ✓ helps finding signal regions, optimal cuts, …
 ✓ flexible to incorporate uncertainties





✘ need to run for each sample pair
 ✘ permutation test is bottleneck

slide-29
SLIDE 29

> Outline

  • A. De Simone
  • 1. Statistical test of dataset compatibility

  • Nearest-Neighbors Two-Sample Test

  • Identify Discrepancies

  • Include Uncertainties




  • 2. Applications to High-Energy Physics

29

slide-30
SLIDE 30

> Our Method

  • A. De Simone

Reject 
 null hypothesis? NN2ST

30

Bkg Simulation Data

yes

hint of new physics! select regions
 to explore

(Trial) (Benchmark) no

No signal in data

slide-31
SLIDE 31
  • A. De Simone

31 mDM = 100 GeV mZ' = 1.2, 2, 3 TeV gDM = 1, gq = 0.1

DM + Z’ vector mediator

Z’ proton proton jet DM DM

> DM search @ LHC

√s = 13 TeV

Z → ν¯ ν + (1, 2) j

  • “proof-of-principle” study

  • bkg: (𝜏bkg=202.6 pb)


sub-leading bkgs not included


  • no full detector effects 


(generic Delphes profile)

Benchmark: BKG1
 Trials: BKG2 + SIG
 K = 5 
 Nperm = 3000

8 features:

  • n. of jets
  • of 2 leading jets
  • Emiss

T

, HT pT , η ∆φEmiss

T

,j1

slide-32
SLIDE 32
  • A. De Simone

32

B: BKG1 (20k events)
 T1: BKG2 (20k events) + SIG1 (2010 events) T2: BKG2 (20k events) + SIG2 (375 events) T3: BKG2 (20k events) + SIG3 (59 events)

Sample MZ’ 𝜏signal Zno uncert. Z10% rel uncert. T1 1.2 TeV 20.4 pb 40 𝜏

26 𝜏

T2 2 TeV 3.8 pb 13 𝜏 12 𝜏 T3 3 TeV 0.6 pb 2.7 𝜏 2.5 𝜏

  • systematics: expect further degradation of results
  • the method has value, it is worth exploring

Nsig = NB × σsignal σbkg

> DM search @ LHC

still not real-world

slide-33
SLIDE 33

Nsig = NB × σsignal σbkg

  • A. De Simone

33

NB = 20 000

NT = NB + Nsig more data, more power stronger signal easier to discover

> DM search @ LHC

slide-34
SLIDE 34

> Outlook

  • A. De Simone

34

  • adaptive choice of K

  • identifying discrepant regions in realistic situations


(with Z-score method)
 


  • validation tool for bkg: 


compatibility between MC sims. and data in control regions

  • scalability

  • … your suggestions?

Directions for future work:

slide-35
SLIDE 35
  • 1. New Statistical Test for BSM Physics

  • assess degree of compatibility between 2 samples

  • rooted on nearest neighbors, solid math foundations








  • 2. NN2ST as a discovery tool

  • powerful and model-independent

  • lots of applications




  • 3. NN2ST to guide searches 

  • identify regions of discrepancies

> Take-Home Messages

  • A. De Simone

35

slide-36
SLIDE 36

BACK UP

slide-37
SLIDE 37

> Model Selection

  • A. De Simone

how to choose K?

37

Define the mean-square error: Estimate loss:
 
 Select optimal K minimizing the loss.
 Alternatively: Point-Adaptive k-NN (PAk)

Model Selection!

True: Estimated:

L(r, ˆ r) = 1 2 Z [ˆ r(x0) − r(x0)]2 pB(x0)dx0 = 1 2 Z ˆ r(x0)2pB(x0)dx0 − Z ˆ r(x)pT (x)dx + 1 2 Z r(x0)2pB(x0)dx0

r(x) = pT (x) pB(x) ˆ r(x) = ˆ pT (x) ˆ pB(x) ˆ L(r, ˆ r) = 1 2NB X

x0∈B

ˆ r(x0)2 − 1 NT X

x∈T

ˆ r(x)

[1802.10549]