James Phillips Beckman Institute, University of Illinois - - PowerPoint PPT Presentation

▶
james phillips
SMART_READER_LITE
LIVE PREVIEW

James Phillips Beckman Institute, University of Illinois - - PowerPoint PPT Presentation

Dec 11, 2022 144 likes •571 views

Attacking HIV with Petascale Molecular Dynamics Simulations on Titan and Blue Waters James Phillips Beckman Institute, University of Illinois http://www.ks.uiuc.edu/Research/namd/ Biomedical Technology Research Center for Macromolecular

slide-1
SLIDE 1

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Attacking HIV with Petascale Molecular Dynamics Simulations on Titan and Blue Waters

James Phillips

Beckman Institute, University of Illinois http://www.ks.uiuc.edu/Research/namd/

slide-2
SLIDE 2

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Developers of the widely used computational biology software VMD and NAMD

250,000 registered VMD users 80,000 registered NAMD users 600 publications (since 1972)

  • ver 54,000 citations

5 faculty members 8 developers 1 systems administrator 17 postdocs 46 graduate students 3 administrative staff

research projects include: virus capsids, ribosome, photosynthesis, protein folding, membrane reshaping, animal magnetoreception

Tajkorshid, Luthey-Schulten, Stone, Schulten, Phillips, Kale, Mallon

NIH Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics

Achievements Built on People

Renewed 2012-2017 with 10.0 score (NIH)

slide-3
SLIDE 3

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Related talks (to stream)

  • S6623 - Advances in NAMD GPU Performance
  • Antti-Pekka Hynninen, Oak Ridge National Laboratory
  • S6253 - VMD: Petascale Molecular Visualization

and Analysis with Remote Video Streaming

  • John Stone, University of Illinois at Urbana-Champaign

3

slide-4
SLIDE 4

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Need for petascale: Simulation follows structural discovery

1990 1994 1998 2002 2006 2010 104 105 106 107 108 2014 Lysozyme ApoA1 ATP Synthase STMV Ribosome HIV capsid Number of atoms 1986

slide-5
SLIDE 5

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Structural Route to the HIV-1 Capsid

Zhao et al. , Nature 497: 643-646 (2013)

High res. EM of hexameric tubules, tomography of capsids, all-atom model of capsid by MDFF

Pornillos et al. , Cell 2009, Nature 2011

crystal structures of separated hexamer and pentamer

Ganser et al. Science, 1999

1st TEM (1999) 1st tomography (2003)

Briggs et al. EMBO J, 2003 Briggs et al. Structure, 2006

cryo-ET (2006)

Byeon et al., Cell 2009 Li et al., Nature, 2000

hexameric tubules

slide-6
SLIDE 6

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Host Cell

Capsid uncoating Integration into the host’s chromatin

Virion

Nuclear Import

Binding Fusion Budding

Capsid shepherds HIV RNA inside the cell

slide-7
SLIDE 7

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016 Fusion/Entry inhibitors Protease inhibitors Reverse Transcription (RNA to DNA) inhibitors Integrase inhibitors Currently no drug targets capsid uncoating or nuclear import !

How is HIV treated today?

Host Cell

slide-8
SLIDE 8

HIV capsid contains 186 hexamers, 12 pentamers, 1300+ proteins, 4.2 million atoms

slide-9
SLIDE 9

Malleability of HIV-1 CA

Hexamer of hexamers bite angles along chiral axis Native capsid bite angle distribution pentamers

hexamers

1300 proteins in different conformations

  • G. Zhao, et al. Nature 497 (2013)
slide-10
SLIDE 10

One-Microsecond Simulation Includes 64 Million Atoms

Key person: Juan Perilla (UIUC)

slide-11
SLIDE 11

Capsid acts as an osmotic regulator

Results from 64 M atom, 1 µs molecular dynamics simulation! Chloride ions permeate through the hexameric center

slide-12
SLIDE 12

A204 E213 K203 I201

Nature 497, 643-646

A204C mutant in vitro

Peijun Zhang - U. Pittsburgh

Curvature is regulated by the trimer interface at the atomic level

  • G. Zhao, et al. Nature 497 (2013)

HIV-CA wild-type in vitro

slide-13
SLIDE 13

HIV uncoating relies on cell factor Cyclophilin A

  • F. Diaz-Griffero, Viruses (2011)

Binding

  • f cypA

Infection

  • f nucleus

Binding

  • f E2

Premature uncoating No infection

cypA binding pattern prevents TRIM binding, but leaves nuclear pore interactions intact TRIM lattice Cell Nucleus Outside

  • f Cell
slide-14
SLIDE 14

Simulation reveals how CypA stabilizes capsid

interaction confirmed by NMR

  • nly polarizable force field yields

stable bridge interaction

slide-15
SLIDE 15

A204 E213 K203 I201

Curvature regulated by trimeric interface. Nature, 497 643-646 CypA bridges adjacent capsid subunits Nature Communications, 7 10714

Chemical Detail is Essential to Capsid Function

Ions permeate through the capsid. In preparation. Pfizer PF74 inhibits HIV-1 infection at an early step during infection. Journal of Physical Chemistry Letters, accepted

slide-16
SLIDE 16

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Peijun Zhang Angela M. Gronenborn Department of Structural Biology Center for HIV Protein Interactions University of Pittsburgh School of Medicine Christopher Aiken Department of Pathology and Immunology Vanderbilt University School of Medicine Juan R. Perilla Klaus Schulten Theoretical and Computational Biophysics Group

HIV Acknowledgments

Laxmikant Kale Parallel Programming Lab

  • Dept. of Computer Science

University of Illinois at Urbana-Champaign

slide-17
SLIDE 17

MDFF produces optimal overlap with EM data

“Dock” array models into EM density

Beyond Viruses: Modeling the Bacterial Brain

Cassidy, C. K. et al. "CryoEM and computer simulations reveal a novel kinase conformational switch in bacterial chemotaxis signaling." eLife 4 (2015): e08419. Chemosensory Array Core-signaling Unit (PDB: 3JA6)

slide-18
SLIDE 18

IF apo IF bound in

  • ut

in

  • ut

OF apo

  • ut

in

  • ut

in OF bound

apo +substrate

Law,%et#al.,%Biochemistry#46,%12190%(2007).%

Beyond Large: Complete Description of Transport Cycle via Bias-Exchange Umbrella Sampling (Moradi, Tajkhorshid)

  • M. Moradi, G. Enkavi, and E. Tajkhorshid (2015) Nature Communications, 6:8393.
slide-19
SLIDE 19

Beyond Illinois: Influenza Virus Coat, Amaro Lab, UCSD

http://www.ncsa.illinois.edu/news/story/blue_waters_enables_massive_flu_simulations

slide-20
SLIDE 20

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

GPUs are critical for visualization and analysis

Large memory GPU-accelerated workstations can be accessed remotely from our facility today, but for future machines must be embedded at supercomputer centers.

1 Gigabit Network Compressed Video

slide-21
SLIDE 21

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Remote Visualization Now

  • TACC Stampede supports this today

– Includes nodes with 1TB memory – Not virtualized, allocate full dedicated node – New Maverick cluster added

  • Blue Waters – no visualization resource
  • Titan – Rhea adds large-memory GPU nodes
  • NIH Center - using NICE DCV for remote access
slide-22
SLIDE 22

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

NAMD is based on Charm++

Complete info at charmplusplus.org

slide-23
SLIDE 23

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Charm++ Used by NAMD

  • Parallel C++ with data driven objects.
  • Asynchronous method invocation.
  • Prioritized scheduling of messages/execution.
  • Measurement-based load balancing.
  • Portable messaging layer.
slide-24
SLIDE 24

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

  • Spatially decompose data and

communication.

  • Separate but related work

decomposition.

  • “Compute objects” facilitate

iterative, measurement-based load balancing system.

NAMD Hybrid Decomposition

Kale et al., J. Comp. Phys. 151:283-312, 1999.

slide-25
SLIDE 25

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Phillips et al., SC2002.

Offload to GPU

Objects are assigned to processors and queued as data arrives.

NAMD Overlapping Execution

slide-26
SLIDE 26

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Overlapping GPU and CPU with Communication

Remote Force Local Force GPU CPU Other Nodes/Processes Local Remote x f f f f Local x x Update One Timestep x

Phillips et al., SC2008

slide-27
SLIDE 27

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Phillips et al., SC14

Torus Adaptation

  • Job partitioning for multiple

copy sampling algorithms

  • Mapping NAMD spatial

decomposition domains onto machine torus

  • Mapping particle-mesh

Ewald (PME) electrostatics

  • nto spatial decomposition

Additional Techniques

  • Coarsening of PME grid to

reduce long-range communication

  • Offloading of PME

interpolation onto GPUs

  • Removal of implicit

synchronization in pressure control algorithm

slide-28
SLIDE 28

NAMD 2.11 Release (December 2015)

  • Improved GPU kernels from Antti-Pekka Hynninen
  • Reciprocal forces - avoid duplicate calculations
  • 30% faster explicit solvent
  • 100% faster implicit solvent
  • Improved scaling for GPU-accelerated simulations
  • Stream results from GPU
  • Better CPU-side parallelization of PME
  • Described in GTC 2015 talk
  • Asynchronous multi-copy Tcl scripting interface
  • Enables central work queue, workflow-style programming
  • Enables multiplexing replicas, hiding load imbalance
slide-29
SLIDE 29

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Streaming CPU Results to CPU

  • Allows incremental results from a single grid to be

processed on CPU before grid finishes on GPU

  • Allows merging and prioritizing of remote and local work
  • GPU side:

– Write results to host-mapped memory (also without streaming) – __threadfence_system() and __syncthreads() – Atomic increment for next output queue location – Write result index to output queue

  • CPU side:

– Poll end of output queue (int array) in host memory

slide-30
SLIDE 30

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Non-Streaming Kernel

slide-31
SLIDE 31

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Streaming Kernel

slide-32
SLIDE 32

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016 Non-bonded local kernel Non-bonded remote kernel Results from GPU Incoming positions Integration Bonded

  • n CPU
slide-33
SLIDE 33

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

slide-34
SLIDE 34

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

slide-35
SLIDE 35

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Streaming Kernel Performance Impact - GTC 2015

(2fs timestep)

4 8 16 32 64 256 512 1024 2048 4096 Number of Nodes 21M atoms Performance (ns per day) Blue Waters XK7 (new streaming) Blue Waters XK7 (no streaming)

+10% +30% +10-30%

slide-36
SLIDE 36

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Performance Comparison - GTC 2015

(2fs timestep)

0.25 0.5 1 2 4 8 16 32 64 256 512 1024 2048 4096 8192 16384 Number of Nodes 21M atoms 224M atoms Performance (ns per day) Blue Waters XK7 (GTC15) Titan XK7 (GTC15) Edison XC30 (SC14) Blue Waters XE6 (SC14)

+50-200% +100-200% +70%

slide-37
SLIDE 37

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Topology-Aware Scheduling on Blue Waters

  • Map jobs to convex sets to

avoid network interference

  • NCSA, Cray, Adaptive
  • Enabled January 2015
  • Most likely explanation for

Blue Waters performance advantage over Titan

  • See Enos et al., CUG 2014
slide-38
SLIDE 38

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

NAMD 2.11 Performance - GTC 2016

(2fs timestep)

0.25 0.5 1 2 4 8 16 32 64 256 512 1024 2048 4096 8192 16384 Number of Nodes 21M atoms 224M atoms Performance (ns per day) Blue Waters XK7 (GTC15) Titan XK7 (GTC16) Titan XK7 (GTC15) Edison XC30 (SC14) Blue Waters XE6 (SC14)

+50-200% +100-200% +70%

slide-39
SLIDE 39

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

NAMD 2.11 Performance - GTC 2016

(2fs timestep)

0.25 0.5 1 2 4 8 16 32 64 256 512 1024 2048 4096 8192 16384 Number of Nodes 21M atoms 224M atoms Performance (ns per day) Blue Waters XK7 (GTC16) Titan XK7 (GTC16) Edison XC30 (SC14) Blue Waters XE6 (SC14)

+50-200% +100-200% +70%

slide-40
SLIDE 40

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Looking forward to Summit

  • Highest throughput: Volta GPU
  • Fastest single-thread: Power 9
  • Fastest data transfer: NVLink
  • Fewer, fatter nodes: Only 3,400
  • Five times Titan performance
  • Potential for remote visualization
  • “Molecular Machinery of the Brain”

early science project

40

Synaptic vesicle and pre-synaptic membrane

slide-41
SLIDE 41

Biomedical Technology Research Center for Macromolecular Modeling and Bioinformatics Beckman Institute, University of Illinois at Urbana-Champaign - www.ks.uiuc.edu

GTC 2016

Thanks to: NIH, NSF, ORNL (Antti-Pekka Hynninen), ANL (Brian Radak),
 NVIDIA (Sarah Tariq, Patric Zhao, Sky Wu, Justin Luitjens, Nikolai Sakharnykh),
 Cray (Sarah Anderson, Ryan Olson), NCSA (Robert Brunner),
 PPL (Eric Bohm, Yanhua Sun, Gengbin Zheng, Nikhil Jain)
 and 20 years of NAMD and Charm++ developers and users.

James Phillips

Beckman Institute, University of Illinois http://www.ks.uiuc.edu/Research/namd/