Dualities in and from Machine Learning Sven Krippendorf Deep - - PowerPoint PPT Presentation

Dualities in and from Machine Learning

Sven Krippendorf
 Deep Learning and Physics 2019
 Yukawa Institute for Theoretical Physics, Kyoto October 31st 2019

• 1

Current ML applications
 in high energy

2

Spend 2 more slides on

Improving sensitivity

• ML-techniques heavily used in experimental bounds.
• Brief example: Improving sensitivity for ultra-light axion-

like particles, compared to previous bounds.

• ML algorithms good at classification. Detecting particles

is a classification problem. Our classifiers:
 
 
 


• Training: Simulate data with and without axions for

appropriate X-ray sources

• Bounds: Compare fake & real data performance
• Algorithms (sklearn): decision trees, boosted decision

trees, random forests, Gaussian Naive Bayes, Gaussian Process classifier, SVM, …

Day, SK 1907.07642

Classifier

0 (no axions) 1 (axions) Spectrum

Previous bounds:
 NGC1275: 1605.01043, Other sources: 1704.05256, Athena bounds: 1707.00176 with: Conlon, Day, Jennings; Berg, Muia, Powell, Rummel

AGN

Cluster

B

axion
 conversion

γ X-ray telescope

galactic absorption

|γ(E)i ! α|γ(E)i + β|a(E)i

3

Constraining ALPs

• Photon-axion interconversion in background magnetic fields:

• One interesting parameter region can be obtained for

photons from sources in and behind galaxy cluster magnetic fields.


4

ℒ ⊃ − gaγγ 4 aF ˜ F = gaγγE ⋅ B

0.010 0.100 1 10 100 ω [keV] 0.05 0.10 0.15 0.20 0.25 Pγ → a

sweet spot

too rapid oscillations suppressed conversion

no oscillations

NGC1275: 1605.01043 Other sources: 1704.05256 Athena bounds: 1707.00176 with: Conlon, Day, Jennings, Rummel; Berg, Muia, Powell

Pγ→a = 1 2 Θ2 1 + Θ2 sin2 ⇣ ∆ p 1 + Θ2 ⌘

Θ = 0.28 ✓ B⊥ 1 µG ◆ ⇣ ω 1 keV ⌘ ✓10−3cm−3 ne ◆ ✓1011GeV M ◆

∆ = 0.54 ⇣ ne 10−3cm−3 ⌘ ✓ L 10kpc ◆ ✓1keV ω ◆

Improving sensitivity

• Data: Chandra X-ray observations of bright point sources

(AGN, Quasar) in or behind galaxy clusters

• Bounds for ALPs with m<10-12 eV due to absence of

characteristic spectral modulations caused by interconversion between photons and axions in cluster background magnetic field

Picture: spectral distortui Picture: bounds overview Table: our results

Axion Coupling |GAγγ | (GeV-1) Axion Mass mA (eV) 10-18 10-16 10-14 10-12 10-10 10-8 10-6 10-30 10-25 10-20 10-15 10-10 10-5 100

LSW (OSQAR) Helioscopes (CAST) Haloscopes (ADMX) Telescopes

x 10-12 GeV-1 AB C DTC GaussianNB QDA RFC Previous A1367 (resid.) 1.9

• 2.4

A1367 (up-resid.) 2.0

• 1.9
• 2.4

A1795 Quasar (resid.)

• 1.7
• 1.4

>10.0 A1795 Quasar (up-resid.)

• >10.0

A1795 Sy1 (resid.) 1.0 0.8 1.2 1.1 0.7 1.5 A1795 Sy1 (up-resid.) 1.1 1.1 1.1 1.0 0.8 1.5

5

ML for the string landscape?

Many talks remove slides

many talks: Halverson, Ruehle, Shiu

6

Other avenues?

7

– Gary Shiu

“Don’t ask what ML can do for you, ask what you can do for ML.”

8

Physics ⋂ ML

9

Dualities

Betzler, SK: 191x.xxxxx

10

The problem

• Obtain correlators at high(er) accuracy:


• In physics (incl. condensed matter, holography, string/field

theory), we often use clever data representations to evaluate correlators.

• Multiple representations can be useful ➔ dualities
• Dual representations are very good representations to

evaluate certain correlation functions (mapping strongly coupled data products to weakly coupled data products).

• Examples …

⟨f(ϕi)⟩

think of this as properties of your data

11

How are dualities useful in practice?


aka connecting physics questions to data questions

12

Examples: dual representations

• Discrete Fourier transformation:


• Is there a signal under the noise?

pk =

n

∑

j=1

xj e−2πijk/n

xk = 1 n

n

∑

j=1

pj e2πijk/n

13

Dual Original

• 2D Ising model: high - low temperature self-duality

H = − J∑

⟨i,j⟩

sisj

Z = ∑ e−βH(s)

˜ β = − 1 2 log tanh β

β = 1 kBT

H = − J∑

⟨i,j⟩

σiσj

Z = ∑ e− ˜

βH(σ)

Tcritical

Ordered rep. ↔ Disordered rep.

Position space? Momentum space?

Krammers, Wannier 1941; Onsager 1943; review: Savit 1980

14

Examples: dual representations

Which data problem?

• Some correlation function which is easier evaluated on

dual variables.


• Can we classify the temperature for low-temperature

configurations? Which temperature is a sample drawn from (at low temperatures)?

⟨σiσj⟩, ⟨E(σ)⟩, ⟨M(σ)⟩

They look rather similar. How about in the dual rep.?

Replace Images

15

Data question on Ising

• But at the dual temperatures, our data takes a different

shape:
 
 
 
 
 
 


• It is easier to classify temperature of a low-temperature

configuration in the dual representation …

• How come?

P(s) = eE/T Z , P(σ) = e ˜

E/ ˜ T

Z ΔT ≪ Δ ˜ T ⟨ΔE⟩ ≪ ⟨Δ ˜ E⟩

16

Duality

Data question on Ising

• Let’s look at the overlap of energy distributions in finite

size samples
 
 
 
 
 
 
 
 
 Let’s check for performance.

17

Original variables Dual variables

Ising: simple network

• Let’s confirm this

at simple networks

• Side remark: way
• utperforming

standard sklearn classifiers

18

Change figures

Example: dual representations

• 1D Ising with multiple spin interactions:


normal: (here: )


dual: where

• Two data questions: 1) energy of spin configuration


2) metastable configuration

H(s) = − J

N−n+1

∑

k=1 n−1

∏

l=0

sk+l − B

N

∑

k=1

sk

B = 0

H(σ) = − J

N−n+1

∑

k=1

σk σk =

n−1

∏

l=0

sk+l

• cf. 1605.05199

+ + + +

s σ

N = 10 n = 3

Ghost spins (fixed value)

19

Example: dual representations

• Two data questions: 1) energy of spin configuration


2) metastable configuration


• cf. 1605.05199

+ + + +

s σ

N = 10 n = 3

• - + - - - + - - +

+ + + - + + + + - +

configuration Energy metastable stable

Add evaluation of metastability on dual and normal variables

20

• Performance on

metastability classification (single hidden layer)

• Deeper networks or CNNs

perform better to a certain degree but at large N or n show the same feature.

Example: dual representations

n = 4 n = 5 n = 8 n = 9 n = 12 6 · 102 0.9113 0.8688 0.8788 0.8813 0.8803 3 · 103 − 0.9243 0.9215 0.9223 0.9295 9.5 · 103 − − 0.9424 0.9475 0.9739 n = 4 n = 5 n = 8 n = 9 n = 12 6 · 102 0.9911 0.9783 0.9819 0.9855 0.9909 3 · 103 − 0.9958 0.9977 0.9994 1.0000 9.5 · 103 − − 1.0000 1.0000 1.0000

600 training
 samples

(b) 3000 training samples.

3000 training
 samples Normal variables Dual variables

21

Upshot: dual representations simplify

answering certain data questions
 (i.e. simple networks sufficient)

22

Why interesting for ML?

23

• Finding good representations which allow to answer the data question

is hard (if not impossible).
 
 
 
 
 
 
 
 


• In this talk we use physics examples. A generalisation to other data

products/questions would be interesting. Here, the data question on the dual network can be addressed with very simple networks.

Why interesting for ML?

Normal Frame Dual Frame Data Question ✘ ✔

Input Data:
 Normal Frame

Neural
 Network

Output: Answer
 to Data Question

Neural
 Network

Output

Neural
 Network

Dual Representation

24

DFT: simple network

• Supervised learning task (binary classification):


• Network


{((xR, xI), y)} {((pR, pI), y)} N discrete values y = 0 y = 1 noise + signal noise

Layer Shape Parameters Conv1D (2000,2) 4 Activation (2000,2)

• Dense

1 4001 Activation 1

• For this network

classification works in momentum space,
 but not in position space.

25

Utilising dual representation

• Goal: improve performance on position space.
• Deeper network? Can do the job in principle


[DFT can be implemented with a single dense layer]

• However finding it dynamically is `impossible’ with standard
• ptimisers, initialisations, and regularisers.


Layer Shape Parameters Dense (2000,2) 16000000 Conv1D (2000,2) 4 Activation (2000,2)

• Dense

1 4001 Activation 1

• DFT

Random starting point

26

Can we improve the situation by favouring dual representations?

27

How to utilise dualities?

• Learn dual transformations when explicitly known (trivial)

and use them as intermediate step in architecture.

• Enforce dual representations via feature separation.
• When not explicitly known, we can match features of

distributions (example 2D Ising high-low-temperature duality)

• Re-obtaining “dualities” (1D Ising) by demanding good

performance on medium-hard correlation in intermediate layer where no loss of information is present (beyond known dualities).

28

DFT from modified loss

• Adapt loss function to achieve feature separation, i.e. separating

the two classes of data (inspired by triplet loss)
 [towards generating dual representations dynamically]
 
 
 
 
 
 


• Note: different data question (signal injected in position and

momentum space) can lead to multiple minima in the loss landscape, i.e. using momentum space and position space.

29

1503.03832

Loss = |ynoise|2 − |ysignal|2 + α

Loss = max {0, β − ∑ |w|2 }

DFT Assist finding DFT
 with modified loss Separation Decorrelation via weight regularisation

+∑

i≠j

max{0,(wi ⋅ wj)}

DFT from feature separation

• The final result is a simple network with few parameters

utilising `dual’ representation. (First layer in
 deeper network)

Fourier Network Rep. Input sample:

30

Feature separation in Ising

• Which neural network architecture? Several architectures,

so far most promising: U-Net (1505.04597)

s-configuration σ-configuration

MaxPooling

Conv.

UpSampling

UNet Pix2Pix

31

Matching dual energy distribution

Enforcing dual representation

• Observation: deep networks do not

find these good representation by themselves

• Which ingredient is missing on the

neural network side?

• Work around when task which is

accessible in both frames

• Step 1: Find better representation
• Step 2: Use this representation on

previously inaccessible task
 
 
 


DFT Assist finding DFT
 with modified loss

Original Different rep. Original Simple Task

Autoencoder fixed

Sophisticated Task

Add details on network

32

Example: 1D Ising model

• Strategy from previous page shows improvement on

performance:


n = 4 n = 5 n = 8 n = 9 n = 12 6 · 102 0.9113 0.8688 0.8788 0.8813 0.8803 3 · 103 − 0.9243 0.9215 0.9223 0.9295 9.5 · 103 − − 0.9424 0.9475 0.9739 n = 4 n = 5 n = 8 n = 9 n = 12 6 · 102 0.9911 0.9783 0.9819 0.9855 0.9909 3 · 103 − 0.9958 0.9977 0.9994 1.0000 9.5 · 103 − − 1.0000 1.0000 1.0000

Now with “dual” representation: test accuracy over 99%

33

Intermediate representation

• Visualise dependency of intermediate variables

and input variables :


• Resembles structure from actual dual variables (results

depend on initialisation), similar but not identical islands
 
 
 
 


fi(s) s

Mij = ⟨(fi(s1, …, sj, …sN) − fi(s1, …, − sj, …sN))

2

⟩

1 N ∑N k=1 ⟨(fi(s1, …, sk, …sN) − fi(s1, …, − sk, …sN)) 2⟩

Actual Dual variables Intermediate variables

34

Recap

35

Perspective

• Large test arsenal of physics dualities from CM, holography, field/string

theory.

• Dualities are our best tools to describe strongly coupled systems.
• Allow us to get EFT-operators at higher accuracy than normally allowed

(theory: large number of diagrams, experiment: large amount of data). Think about Yukawa couplings in heterotic standard embedding

• Dual data representations can be obtained via ML. We can design the

dual representation to have particular properties, e.g. display certain quantities straight-forwardly. Can we find new useful physics dualities in this way, i.e. new ways to describe dynamical systems? Which correlators should we use to obtain dual representations?

• Twist to DL procedures? Intermediate layer with no information loss and

representation suitable for “medium-hard” task. This pre-trained structure should be useful for more sophisticated tasks in these variables.

36

“Therefore we can see that the dualities we have been dealing with for antisymmetric tensors are only particular cases of Fourier transforms and finding the dual action reduces to finding Fourier transforms. “

Dualities and Fourier transformation

• Review on dualities and global symmetries (Quevedo: hep-

th/9706210), several dualities can be seen as Fourier transformation:
 
 


• For instance duality between massive antisymmetric

tensors.
 
 
 


S = ∫ dDx (F(∂Hh) + G(Hh)) ˜ S = ∫ dDx ( ˜ F( ˜ Bd−h) + ˜ G(∂ ˜ Bd−h)) Z = ∫ 𝒠 ˜ Bd−h𝒠Hh e ∫ dDx(H⋅d ˜

Bd−h+G(Hh)+ ˜ F( ˜ Bd−h))

˜ Bd−h Hh e ˜

F ↔ eF

FT

37

Conclusions

• Dualities tightly connected to data representations.
• Finding good data representations is at the heart of ML and
• ften `good' representations are not found.
• `Good’ representations can be enforced when known to exist.
• This enables classification for previously `inaccessible’ tasks.


(1D Ising metastable configurations)

• Interesting representations can be found from feature separation

(e.g. DFT)

• Interesting aspect: reduced complexity of networks
• Dualities in Physics motivate multiple minima in a different

landscape, those of the cost functions of neural networks.

• Lots of different kinds of dualities in Physics as playground for

efficient neural network architectures.

38

Thank you!

39

40

Neural Networks

• Layout of neural nets:


• Cost function depending on parameters of network

• Parameters of network updated using gradient descent

. . . . . . Input Output Hidden layers Weights

yi = activation (wijxj + bi)

xi

yi

Input: Output:

{θA} = {wij, bk}

Parameters of network:

cost(θA) = X

training set

|ydesired − ypredicted(θA)|

θA ! θA η rAcost(θA)

41

Constraining ALPs

• Photon-axion interconversion in background magnetic fields:

• One interesting parameter region can be obtained for

photons from sources in and behind galaxy cluster magnetic fields.


42

ℒ ⊃ − gaγγ 4 aF ˜ F = gaγγE ⋅ B

0.010 0.100 1 10 100 ω [keV] 0.05 0.10 0.15 0.20 0.25 Pγ → a

sweet spot

too rapid oscillations suppressed conversion

no oscillations

NGC1275: 1605.01043 Other sources: 1704.05256 Athena bounds: 1707.00176 with: Conlon, Day, Jennings, Rummel; Berg, Muia, Powell

Pγ→a = 1 2 Θ2 1 + Θ2 sin2 ⇣ ∆ p 1 + Θ2 ⌘

Θ = 0.28 ✓ B⊥ 1 µG ◆ ⇣ ω 1 keV ⌘ ✓10−3cm−3 ne ◆ ✓1011GeV M ◆

∆ = 0.54 ⇣ ne 10−3cm−3 ⌘ ✓ L 10kpc ◆ ✓1keV ω ◆

Standard bounds

• Spectral distortions (but unknown where & strength as

dependent on magnetic field)

• Poisson noise vs. signal
• Essentially: fluctuations larger than noise
• Human-made: Fourier bounds no real improvement

43

Conlon, Rummel 1808.05916

10-18 10-16 10-14 10-12 10-10 10-8 10-6 10-30 10-25 10-20 10-15 10-10 10-5 100 Axion Coupling |GAγγ | (GeV-1) Axion Mass mA (eV) 10-18 10-16 10-14 10-12 10-10 10-8 10-6 10-30 10-25 10-20 10-15 10-10 10-5 100

LSW (OSQAR) Helioscopes (CAST) Haloscopes (ADMX) Telescopes

VMB (PVLAS)

SN 1987A

Axion Coupling |GAγγ | (GeV-1) Axion Mass mA (eV) 10-18 10-16 10-14 10-12 10-10 10-8 10-6 10-30 10-25 10-20 10-15 10-10 10-5 100

LSW (OSQAR) Helioscopes (CAST) Haloscopes (ADMX) Telescopes

VMB (PVLAS)

SN 1987A

NGC1275: 1605.01043 Other sources: 1704.05256 Athena bounds: 1707.00176 with: Conlon, Day, Jennings; Berg, Muia, Powell, Rummel