Machine learning for energy landscapes Tristan Bereau Van t Hoff - - PowerPoint PPT Presentation

Machine learning for energy landscapes

Tristan Bereau Van ’t Hoff Institute for Molecular Sciences & Informatics Institute University of Amsterdam

Machine learning for energy landscapes

Tristan Bereau Van ’t Hoff Institute for Molecular Sciences & Informatics Institute University of Amsterdam

• Introduction to kernel-based ML
• Incorporation of physical symmetries, conservation laws

Today

Machine learning for energy landscapes

Tristan Bereau Van ’t Hoff Institute for Molecular Sciences & Informatics Institute University of Amsterdam

• Introduction to kernel-based ML
• Incorporation of physical symmetries, conservation laws

Today Supervised ML in chemistry and materials science Thursday

Mohamad Moosavi Kevin Jablonka

Molecular dynamics

2

Molecular dynamics

2

F = ma

Molecular dynamics

2

F = ma

Specify interparticle forces: “force field”

Molecular dynamics

2

F = ma

Specify interparticle forces: “force field” Numerically integrate particle positions

Molecular dynamics

2

F = ma

Specify interparticle forces: “force field” Numerically integrate particle positions

Wikipedia

Molecular dynamics

2

fs ps ns s

µ

ms s

F = ma

Specify interparticle forces: “force field” Numerically integrate particle positions

Wikipedia

Molecular dynamics

2

fs ps ns s

µ

ms s integration time step

F = ma

Specify interparticle forces: “force field” Numerically integrate particle positions

Wikipedia

Molecular dynamics

2

fs ps ns s

µ

ms s integration time step Timescales of interest

F = ma

Specify interparticle forces: “force field” Numerically integrate particle positions

Wikipedia

Emergent complexity

Molecular dynamics

2

fs ps ns s

µ

ms s integration time step Timescales of interest

F = ma

Specify interparticle forces: “force field” Numerically integrate particle positions

Wikipedia

Data as the 4th pillar of science

3

• A. Agrawal and A. Choudhary APL Mater. 4 053208 (2016); https://www.bigmax.mpg.de

Data science

4

Databases Machine learning Hardware

Data science

4

Databases Machine learning Hardware

DeepMind Google

Links to machine learning

5

Potential energy surface

Can we build a more accurate PES? Can we easily build an accurate PES? Can we make the numerical integration faster and/or more efficient? …

Interpolation of a high-dimensional function

Teaser: let’s fit data points

6

Good Not good

Teaser: let’s fit data points

6

Good Not good

Teaser: let’s fit data points

6

Good Not good I want to be here

Teaser: let’s fit data points

6

Good Not good I want to be here

ruthlessly stolen from A. von Lilienfeld

Multivariate function approximation

7

Sparse data infer smooth function

Multivariate function approximation

7

Sparse data infer smooth function “inverse problems”

Multivariate function approximation

7

Sparse data infer smooth function

Rupp, International Journal of Quantum Chemistry 115 (2015)

property

Ê Ê Ê Ê Ê Ê

molecular structure

property descriptor sparse data ground truth interpolation

“inverse problems”

Multivariate function approximation

7

Sparse data infer smooth function

Rupp, International Journal of Quantum Chemistry 115 (2015)

property

Ê Ê Ê Ê Ê Ê

molecular structure

property descriptor sparse data ground truth interpolation

“inverse problems” Regression: prediction of f : ℝd → ℝ based on noisy data points D = (X, y) = {xn, yn}N

n=1

yn = f(xn) + ε

What is f ? x y

Kernel methods are vintage

8

Kernel Deep learning

• needs a representation
• linear algebra
• can be efficient with small

data

{ {

• learns the representation
• complex mathematical

structure

• data hungry

Learning from experience

9

• [7] Visualization of high-dimensional space

(genera (snake seems like overfitting, fitting to well, cf. Lecture 2)

• Inductive (based on examples)

source: xkcd

Extrapolation in machine learning

10

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 r [æ] 5 10 15 20 25 30 35 ULJ(r) [≤]

ULJ(r) Training points Prediction

ULJ(r) = 4✏ ⇣ r ⌘12 − ⇣ r ⌘6

Extrapolation in machine learning

10

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 r [æ] 5 10 15 20 25 30 35 ULJ(r) [≤]

ULJ(r) Training points Prediction

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 r [æ] 5 10 15 20 25 30 35 ULJ(r) [≤]

ULJ(r) Training points Prediction

ULJ(r) = 4✏ ⇣ r ⌘12 − ⇣ r ⌘6

Bayesian inference

11

Prediction { Prior beliefs Sampled data

Bayesian inference

11

Prediction { Prior beliefs Sampled data

f f*

training values test values

y

• bservations

Bayesian inference

11

Prediction { Prior beliefs Sampled data

f f*

training values test values

y

• bservations

Bayes’ formula

p(f, f*|y) = p(y|f) p(f, f*) p(y)

likelihood prior posterior normalization

Bayesian inference

12

Bayes’ formula

p(f, f*|y) = p(y|f) p(f, f*) p(y)

likelihood prior posterior normalization

• +

+ +

• Rasmussen, Advanced lectures on machine learning. Springer, 63-71 (2004)

Gaussian processes

13

f ∼ GP(m, k)

Rasmussen, Advanced lectures on machine learning. Springer, 63-71 (2004)

Gaussian processes

13

f ∼ GP(m, k)

random variable: value of the stochastic function at x mean covariance

f(x)

Σij = k(xi, xj)

µi = m(xi)

Rasmussen, Advanced lectures on machine learning. Springer, 63-71 (2004)

Gaussian processes

13

f ∼ GP(m, k)

random variable: value of the stochastic function at x mean covariance

f(x)

Σij = k(xi, xj)

µi = m(xi)

Rasmussen, Advanced lectures on machine learning. Springer, 63-71 (2004)

Kα = p

kernel target property

Linear-ridge vs kernel-ridge regression

14

Linear-ridge regression kernel-ridge regression (ML)

Ax = b

Kα = p

Linear-ridge vs kernel-ridge regression

14

Linear-ridge regression kernel-ridge regression (ML)

Ax = b

Kα = p

N m m N

Linear-ridge vs kernel-ridge regression

14

Linear-ridge regression kernel-ridge regression (ML)

Ax = b

Kα = p

N m m N

in general: m ⌧ N

Linear-ridge vs kernel-ridge regression

14

Linear-ridge regression kernel-ridge regression (ML)

Ax = b

Kα = p

N m m N

in general: m ⌧ N

N N N N

Linear-ridge vs kernel-ridge regression

14

Linear-ridge regression kernel-ridge regression (ML)

Ax = b

Kα = p

N m m N

in general: m ⌧ N

N N N N

Kij =Kij(xi, xj) =Kij(|xi − xj|) = exp ✓ −|xi − xj| σ ◆

Linear-ridge vs kernel-ridge regression

14

Linear-ridge regression kernel-ridge regression (ML)

Ax = b

Kα = p

N m m N

in general: m ⌧ N

N N N N

Kij =Kij(xi, xj) =Kij(|xi − xj|) = exp ✓ −|xi − xj| σ ◆

error # training points

k e r n e l linear

Linear-ridge vs kernel-ridge regression

14

Linear-ridge regression kernel-ridge regression (ML)

Ax = b

Kα = p

N m m N

in general: m ⌧ N

N N N N

Kij =Kij(xi, xj) =Kij(|xi − xj|) = exp ✓ −|xi − xj| σ ◆

error # training points

k e r n e l linear

• +

+ +

• measures similarity

between data points

Kernel machine learning 101

15

ULJ(r) Training points Prediction

ULJ(r) = 4✏ ⇣ r ⌘12 − ⇣ r ⌘6

K(r, r′ ) = exp (− (r − r′ )2 2σ2 ) 1) Define representation and kernel

Kernel machine learning 101

15

ULJ(r) Training points Prediction

ULJ(r) = 4✏ ⇣ r ⌘12 − ⇣ r ⌘6

K(r, r′ ) = exp (− (r − r′ )2 2σ2 ) 1) Define representation and kernel

Kα = U α = (K + λ𝕁)−1U

2) Train your model:

(K + λI) α = U

Regularization: “hyperparameter” scales noise level Inverse is ill defined Optimize weight coefficients on training set

Kernel machine learning 101

15

ULJ(r) Training points Prediction

ULJ(r) = 4✏ ⇣ r ⌘12 − ⇣ r ⌘6

K(r, r′ ) = exp (− (r − r′ )2 2σ2 ) 1) Define representation and kernel

Training points Query sample Predicted energy

U(r) = ∑

i

αiK(r*

i , r)

3) Make a prediction:

Kα = U α = (K + λ𝕁)−1U

2) Train your model:

(K + λI) α = U

Regularization: “hyperparameter” scales noise level Inverse is ill defined Optimize weight coefficients on training set

Extrapolation in machine learning

16

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 r [æ] 5 10 15 20 25 30 35 ULJ(r) [≤]

ULJ(r) Training points Prediction

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 r [æ] 5 10 15 20 25 30 35 ULJ(r) [≤]

ULJ(r) Training points Prediction

ULJ(r) = 4✏ ⇣ r ⌘12 − ⇣ r ⌘6

K(r, r′ ) = exp (− (r − r′ )2 2σ2 )

U(r) = ∑

i

αiK(r*

i , r)

Define kernel

Training points Query sample Predicted energy

α = (K + λ𝕁)−1U

Make a prediction: Train your model:

Extrapolation in machine learning

16

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 r [æ] 5 10 15 20 25 30 35 ULJ(r) [≤]

ULJ(r) Training points Prediction

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 r [æ] 5 10 15 20 25 30 35 ULJ(r) [≤]

ULJ(r) Training points Prediction

ULJ(r) = 4✏ ⇣ r ⌘12 − ⇣ r ⌘6

K(r, r′ ) = exp (− (r − r′ )2 2σ2 )

U(r) = ∑

i

αiK(r*

i , r)

Define kernel

Training points Query sample Predicted energy

α = (K + λ𝕁)−1U

Make a prediction: Train your model: Conformational space missing from training

Extrapolation in machine learning

16

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 r [æ] 5 10 15 20 25 30 35 ULJ(r) [≤]

ULJ(r) Training points Prediction

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 r [æ] 5 10 15 20 25 30 35 ULJ(r) [≤]

ULJ(r) Training points Prediction

Uniform prior

ULJ(r) = 4✏ ⇣ r ⌘12 − ⇣ r ⌘6

K(r, r′ ) = exp (− (r − r′ )2 2σ2 )

U(r) = ∑

i

αiK(r*

i , r)

Define kernel

Training points Query sample Predicted energy

α = (K + λ𝕁)−1U

Make a prediction: Train your model: Conformational space missing from training

0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0 r [æ] 5 10 15 20 25 30 35 ULJ(r) [≤]

ULJ(r) Training points Prediction

Conformational space missing from training

Linking conformational and interpolation spaces

17

• [7] Visualization of high-dimensional space

(genera (snake seems like overfitting, fitting to well, cf. Lecture 2)

• ML training set size is limited

(kernels!) Use physics to reduce the interpolation space

Symmetries and conservation laws

Mechanics 101: Principle of least action

19

𝒯[x(t)] = ∫

t2 t1

dt L[x(t), · x(t), t]

action microtrajectory Lagrangian L = T − V kinetic energy potential energy

Mechanics 101: Principle of least action

19

𝒯[x(t)] = ∫

t2 t1

dt L[x(t), · x(t), t] Hamilton’s principle: system minimizes action (variational principle)

· 𝒯[x*(t)] = 0

action microtrajectory Lagrangian L = T − V kinetic energy potential energy

Mechanics 101: Principle of least action

19

𝒯[x(t)] = ∫

t2 t1

dt L[x(t), · x(t), t] Hamilton’s principle: system minimizes action (variational principle)

· 𝒯[x*(t)] = 0

action microtrajectory Lagrangian

stationarity under small perturbations leads to Euler-Lagrange equations

δ𝒯 = ∫

t2 t1

dt L(x* + ε, · x* + · ε, t) − L(x*, · x*, t) = ∫

t2 t1

dt (ε ∂L ∂x + · ε ∂L ∂x ) = ∫

t2 t1

dt (ε ∂L ∂x − ε d dt ∂L ∂· x ) = 0

L = T − V kinetic energy potential energy

Mechanics 101: Principle of least action

19

𝒯[x(t)] = ∫

t2 t1

dt L[x(t), · x(t), t] Hamilton’s principle: system minimizes action (variational principle)

· 𝒯[x*(t)] = 0

action microtrajectory Lagrangian

stationarity under small perturbations leads to Euler-Lagrange equations

δ𝒯 = ∫

t2 t1

dt L(x* + ε, · x* + · ε, t) − L(x*, · x*, t) = ∫

t2 t1

dt (ε ∂L ∂x + · ε ∂L ∂x ) = ∫

t2 t1

dt (ε ∂L ∂x − ε d dt ∂L ∂· x ) = 0

integration by parts &

ε(t1) = ε(t2) = 0

L = T − V kinetic energy potential energy

From symmetries, to invariants, to conserved quantities

20

𝒯[x(t), y(t), z(t)] = ∫ dt m 2 (· x2 + · y2 + · z2) − mgz

From symmetries, to invariants, to conserved quantities

20

𝒯[x(t), y(t), z(t)] = ∫ dt m 2 (· x2 + · y2 + · z2) − mgz Introduce constant translations along and :

x y

𝒯[x(t) + x0, y(t) + y0, z(t)] = ∫ dt m 2 (· x2 + · y2 + · z2) − mgz = 𝒯[x(t), y(t), z(t)]

From symmetries, to invariants, to conserved quantities

20

𝒯[x(t), y(t), z(t)] = ∫ dt m 2 (· x2 + · y2 + · z2) − mgz Introduce constant translations along and :

x y

𝒯[x(t) + x0, y(t) + y0, z(t)] = ∫ dt m 2 (· x2 + · y2 + · z2) − mgz = 𝒯[x(t), y(t), z(t)] (Translational) symmetry leaves the action invariant. It leaves the Euler-Lagrange equation unchanged: ∂L ∂x − d dt ∂L ∂· x = 0

From symmetries, to invariants, to conserved quantities

20

𝒯[x(t), y(t), z(t)] = ∫ dt m 2 (· x2 + · y2 + · z2) − mgz Introduce constant translations along and :

x y

𝒯[x(t) + x0, y(t) + y0, z(t)] = ∫ dt m 2 (· x2 + · y2 + · z2) − mgz = 𝒯[x(t), y(t), z(t)] (Translational) symmetry leaves the action invariant. It leaves the Euler-Lagrange equation unchanged: ∂L ∂x − d dt ∂L ∂· x = 0 ∂L ∂x = 0 ∂L ∂· x = m · x = const . Translational invariance implies linear momentum conversation

From symmetries to conserved quantities (cont’d)

21

𝒯[r(t)] = ∫ dt m 2 · r2 − V(r) Rotational symmetry Apply transformation r → r′ where r′ (t) = Rr(t) = r(t) + α × r(t) One can show that 𝒯[r(t) + α × r(t)] = 𝒯[r(t)] Conservation of angular momentum Time translation Apply transformation r → r′ where r′ (t + ϵ) = r(t) One can show that 𝒯[r′ (t + ϵ)] = 𝒯[r(t)] Conservation of energy

(up to a boundary term) Bañados & Reyes, arXiv:1601.03616v3

Noether’s theorem

22

3 examples:

• Translational symmetry: Linear momentum conservation
• Rotational symmetry: Angular momentum conservation
• Time translation: Energy conservation

To every differentiable symmetry generated by local actions there corresponds a conserved quantity

Noether’s theorem

22

3 examples:

• Translational symmetry: Linear momentum conservation
• Rotational symmetry: Angular momentum conservation
• Time translation: Energy conservation

To every differentiable symmetry generated by local actions there corresponds a conserved quantity

Ceriotti, JCP 150 (2019)

2 ways of encoding symmetries:

• Representation
• ML model f

Encoding symmetries in the representation

Translational and rotational symmetries

25

Behler-Parrinello Coulomb matrix

i

X

all ji

eRijRs2fcRij:

G2

i 21 X all j;ki

1 cosijk eR2

Behler & Parrinello, Phys Rev Lett 98 (2007) Cartesian coordinates Symmetry functions Distances Angles

Rupp, Tkatchenko, Müller, von Lilienfeld, Phys Rev Lett, 108 (2012)

• =

∀ = | − | ∀ ≠ ⎧ ⎨ ⎪ ⎪ ⎩ ⎪ ⎪ C Z i j ZZ i j R R 1 2

ij i i j i j 2.4

Distances

Representation: the Coulomb matrix

26

Hansen et al., J Chem Theory Comput, 9 (2013)

Symmetries of the representation should emulate symmetries of the system

• 1. Translation
• 2. Rotations
• 3. Mirror reflection

~ Coulomb’s law E = qiqj |ri − rj|

Representation: the Coulomb matrix

26

Hansen et al., J Chem Theory Comput, 9 (2013)

Symmetries of the representation should emulate symmetries of the system

• 1. Translation
• 2. Rotations
• 3. Mirror reflection

Problems:

• 1. Dimensionality from # atoms

2. Ordering of the atoms ~ Coulomb’s law E = qiqj |ri − rj|

Optimizing the representation links to the physics

27

Huang and von Lilienfeld, J Chem Phys 145 (2016)

Learning a Gaussian function Learning atomization energies

Optimizing the representation links to the physics

27

Huang and von Lilienfeld, J Chem Phys 145 (2016)

Learning a Gaussian function Learning atomization energies Non-unique representation!

Optimizing the representation links to the physics

27

Huang and von Lilienfeld, J Chem Phys 145 (2016)

Learning a Gaussian function Learning atomization energies Non-unique representation!

ULJ(r) = 4✏ ⇣ r ⌘12 − ⇣ r ⌘6

Optimizing the representation links to the physics

27

Huang and von Lilienfeld, J Chem Phys 145 (2016)

Learning a Gaussian function Learning atomization energies Non-unique representation!

ULJ(r) = 4✏ ⇣ r ⌘12 − ⇣ r ⌘6

Empirically test for relevant physics

Encoding symmetries in the ML model

Encoding symmetries in ML models using group theory

29

Risi Kondor, Group theoretical methods in machine learning, PhD thesis (2008)

Action of group on input sample

G

x ↦ Tg(x)

Encoding symmetries in ML models using group theory

29

Risi Kondor, Group theoretical methods in machine learning, PhD thesis (2008)

Action of group on input sample

G

x ↦ Tg(x)

Can we find a kernel that is invariant to this group action?

f(Tg(x)) = f(x)∀g ∈ G k(x, x′ ) = k(Tg(x), Tg′ (x′ ))

Encoding symmetries in ML models using group theory

29

Risi Kondor, Group theoretical methods in machine learning, PhD thesis (2008)

Action of group on input sample

G

x ↦ Tg(x)

Can we find a kernel that is invariant to this group action?

f(Tg(x)) = f(x)∀g ∈ G k(x, x′ ) = k(Tg(x), Tg′ (x′ ))

To ensure invariance, symmetrize the kernel

kG(x, x′ ) = 1 |G| ∑

g∈G

k(x, Tg(x′ ))

Example of symmetrized kernel

30

Vanilla/naïve kernel

k(ρ, ρ′ ) = ∫ drρ(r)ρ(r′ )

Example of symmetrized kernel

30

Vanilla/naïve kernel

k(ρ, ρ′ ) = ∫ drρ(r)ρ(r′ )

SOAP kernel/representation*

Bartók, Kondor, Csányi, Phys Rev B 87 (2013) *Smooth Overlap of Atomic Positions: is a distance metric between two samples

k(ρ, ρ0) = Z

• S(ρ, ˆ

Rρ0)

• n

d ˆ R = = Z d ˆ R

• Z

ρ(r)ρ0( ˆ Rr)dr

• n

̂ R

Invariant vs. covariant properties

31

Glielmo, Sollich, De Vita, Phys Rev B 95 (2017)

Tensorial property (e.g., dipole moment, force) rotates with the sample

Invariant vs. covariant properties

31

Glielmo, Sollich, De Vita, Phys Rev B 95 (2017)

Tensorial property (e.g., dipole moment, force) rotates with the sample

Invariant vs. covariant properties

31

Glielmo, Sollich, De Vita, Phys Rev B 95 (2017)

Tensorial property (e.g., dipole moment, force) rotates with the sample

Invariant vs. covariant properties

31

Glielmo, Sollich, De Vita, Phys Rev B 95 (2017)

Tensorial property (e.g., dipole moment, force) rotates with the sample “Build kernel so as to encode the rotational properties of the target property”

Covariant kernels

32

Encode rotational properties of the target property in the kernel

Glielmo, Sollich, De Vita, Phys Rev B 95 (2017)

Covariant kernels

32

Encode rotational properties of the target property in the kernel

ˆ f(Sρ | D) = Sˆ f(ρ | D).

Descriptor Training data Transformation (rotation/inversion) Force prediction

Glielmo, Sollich, De Vita, Phys Rev B 95 (2017)

Covariant kernels

32

Encode rotational properties of the target property in the kernel

ˆ f(Sρ | D) = Sˆ f(ρ | D).

Descriptor Training data Transformation (rotation/inversion) Force prediction “Transform the configuration, and the prediction transforms with it”

Glielmo, Sollich, De Vita, Phys Rev B 95 (2017)

Covariant kernels

33

K(Sρ, S0ρ0) = SK(ρ, ρ0)S0T.

Configurations Transformations (rotation/inversion) Kernel

Glielmo, Sollich, De Vita, Phys Rev B 95 (2017)

K(ρ,ρ0) =

Z

dRRkb(ρ,Rρ0)

Kµ(ρ, ρ0) = 1 L X

ij

φ(ri, rj)ri ⌦ r0T

j ,

⇣ ⌘

ri rj

Conclusions

34

Take advantage of symmetries Noether: symmetry leads to conservation law

• [7] Visualization of high-dimensional space

(genera (snake seems like overfitting, fitting to well, cf. Lecture 2)

• 0.2

Extrapolation in ML models of energy landscapes Can lead to catastrophic physics

K(Sρ, S0ρ0) = SK(ρ, ρ0)S0T.

Build symmetries in ML model Work with subset of kernels that a priori satisfy conservation law