Deep Reinforcement Learning Olexandr Isayev, Ph.D. University of - - PowerPoint PPT Presentation

Olexandr Isayev, Ph.D.

University of North Carolina at Chapel Hill

• lexandr@unc.edu

http://olexandrisayev.com

De Novo molecular design with Deep Reinforcement Learning

@olexandr

About me

Ph.D. in Chemistry (computational) Minor in CS/ML Worked in Federal research lab on HPC & GPU computing to solve chemical problems Now I am faculty at the University of North Carolina, Chapel Hill We use ML & AI to solve challenging problems in chemistry http://olexandrisayev.com Twitter: @olexandr

• lexandr@unc.edu

A public-private partnership that supports the discovery

• f new medicines through
• pen access research

www.thesgc.org

The Long and Winding Road to Drug Discovery

Data Science approaches useful across the pipeline, but very different techniques aim for success, but if not: fail early, fail cheap

internal rate of return (IRR) Source: Endpoints News https://endpts.com

Drowning in Data

…but starving for Knowledge

7

The growing appreciation of molecular modeling and informatics

“Behold the rise of the machines”

Summary of recent AI-based studies

• n chemical library design

Molecular representations Generative models Method of biasing generated compounds

• Fingerprints
• SMILES
• Graphs
• Autoencoders
• Generative

adversarial models (GANs)

• Recurrent neural

networks (RNNs)

• Convolutional

neural networks (CNNs)

• None
• Latent space
• ptimization
• Fine-tuning on small

subset of molecules with the desired property

• Reinforcement

Learning

De Novo molecular design with Deep Reinforcement Learning

Molecules Patent pending Predictive Deep Network Generative Deep Network Tm; LogP; pIC50; etc

General Approach Application to Molecular design

arXiv:1711.10907

~106 – 109 molecules VIRTUAL SCREENING CHEMICAL STRUCTURES CHEMICAL DESCRIPTORS PROPERTY/ ACTIVITY PREDICTIVE QSAR MODELS INACTIVES (confirmed inactives) QSAR MAGIC HITS (confirmed actives) CHEMICAL DATABASE

Drug discovery pipeline

Design of the ReLeaSE* method

Challenges:

• Generate chemically

feasible SMILES

• Develop SMILES-

based QSAR model

• Employ Predictive ML

model to bias library generation

*Popova, Mariya, Olexandr Isayev, and Alexander Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Language of SMILEs

Generative model

Did the training converge ?

NO YES

<START> c <START>c1ccc(O)cc1<END> c 1 1 c c c c ) + loss c ( ( F + loss O ) ) c c c c 1 1 <END>

Softmax loss

1.5M molecules from ChEMBL

c1ccc(O)cc1

FC(F)COc1ccc2c(Nc3ccc(Cl)c(Cl)c3)ncnc2c1

Generative model Predictive model

Reinforcement learning for chemical design

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

FC(F)COc1ccc2c(Nc3ccc(Cl)c(Cl)c3)ncnc2c1

Generative model Predictive model

Reinforcement learning for chemical design

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Generative model Predictive model

INACTIVE!

Reinforcement learning for chemical design

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Generative model Predictive model

INACTIVE!

Reinforcement learning for chemical design

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Generative model Predictive model

INACTIVE!

Reinforcement learning for chemical design

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Generative model Predictive model

Reinforcement learning for chemical design

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Fc1ccc2c(Nc3ccc(F)c(F)c3)ncnc2c1

Generative model Predictive model

Reinforcement learning for chemical design

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Generative model Predictive model

Fc1ccc2c(Nc3ccc(F)c(F)c3)ncnc2c1

Reinforcement learning for chemical design

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Generative model Predictive model

ACTIVE!

Reinforcement learning for chemical design

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Generative model Predictive model

ACTIVE!

Reinforcement learning for chemical design

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Generative model Predictive model

ACTIVE!

Reinforcement learning for chemical design

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Generative model Predictive model

Reinforcement learning for chemical design

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Technical details

• Models were trained on Nvidia Titan X and Titan V

GPUs

• Training generative model on ChEMBL took ~ 25 days
• Training predictive models took ~ 2 hours
• Biasing generative model with reinforcement learning

for one property ~ 1 day

• Generative model produces 1000s compounds per

minute

Optimized Baseline Partition coefficient (logP)

Results: Biasing target properties in the designed libraries

JAK2 Inhibition (pIC50)

4 8 6 10

• 2

12 2

*

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

CAS 236-084-2 (buffer reagent) ZINC37859566 New molecule SIMILAR SCAFFOLDS NEW CHEMOTYPE

JAK2 (Kinase) inhibition

Train data distribution Maximized property distribution Minimized property distribution arXiv:1711.10907

Results: analysis of similarity

Distribution of Tanimoto similarity to the nearest neighbor in training dataset for compounds predicted to be active for EGFR by consensus of QSAR models:

1.0 0.9 0.8 0.7 0.6 0.5 Tanimoto similarity

Similarity= 0.57 Similarity= 0.69 Similarity = 0.86

Results: Synthetic accessibility score* of the designed libraries

*Ertl, Peter, and Ansgar Schuffenhauer. "Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions." Journal of cheminformatics 1.1 (2009): 8.

Target predictions for generated compounds using SEA*

*Keiser MJ, Roth BL, Armbruster BN, Ernsberger P, Irwin JJ, Shoichet BK. Relating protein pharmacology by ligand chemistry. Nat Biotech 25 (2), 197-206 (2007).

Target predictions for generated compounds using SEA*

*Keiser MJ, Roth BL, Armbruster BN, Ernsberger P, Irwin JJ, Shoichet BK. Relating protein pharmacology by ligand chemistry. Nat Biotech 25 (2), 197-206 (2007).

Model visualization for JAK2 (projection using t-SNE)

ZINC19982368 pIC50 = 8.64 ZINC66347860 pIC50 = 3.31 pIC50 = 10.37 pIC50 = 0.63 ZINC2876515 pIC50 = 8.39

10 9 4 7 6 5 8 3 2 1

ZINC3549031 pIC50 = 3.76 ZINC469992 pIC50 = 8.23

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Examples of Stack-RNN cells with interpretable gate activations

• M. Popova, O. Isayev, A. Tropsha. "Deep reinforcement learning for de-novo drug design." arXiv preprint arXiv:1711.10907 (2017).

Summary

• AI methods coupled with SMILES representation afford biased

libraries generation

• The system naturally embeds reinforcement to produce novel

structure with the desired property

• The system can be tuned to bias libraries towards specific property

ranges

• Next phase is experimental validation of hits by UNC SGC team