GPU Clusters for HPC Edward Seidel Director Bill Kramer Blue - PowerPoint PPT Presentation

GPU Clusters for HPC Edward Seidel Director Bill Kramer Blue Waters Project Director National Center for Supercomputing Applications University of Illinois at Urbana- Champaign National Center for Supercomputing Applications University of Illinois at Urbana-Champaign

National Center for Supercomputing Applications WHAT IS NCSA Imaginations unbound

National Center for Supercomputing Applications: 30 years of leadership • NCSA • R&D unit of the University of Illinois at Urbana-Champaign • One of original five NSF-funded supercomputing centers • Mission : Provide state-of-the-art computing capabilities (hardware, software, hpc expertise) to nation’ s scientists and engineers • The Numbers • Approximately 200 staff (160+ technical/professional staff) • Approximately 15 graduate students (+ new SPIN program), 15 undergrad students • Two major facilities (NCSA Building, NPCF) • Operating NSF’s most powerful computing system: Blue Waters • Managing NSF’s national cyberinfrastructure: XSEDE Source: Thom Dunning

Petas cale Computing Facility: Home to Blue Waters • Blue Waters • 13PF, 1500TB, 300PB • >1PF On real apps • NAMD, MILC, WRF, PPM, NWChem, etc • Energy Efficiency • Modern Data Center • LEED certified Gold • 90,000+ ft 2 total • Power Utilization Efficiency • 30,000 ft 2 raised floor = 1.1 – 1.2 20,000 ft 2 machine room gallery Source: Thom Dunning

Data Intensive Computing Personalized Medicine w/ Mayo LSST, DES Source: Thom Dunning

NCSA’s Industrial Partners Source: Thom Dunning

NCSA and NVIDIA PARTNERSHIP FOR INNOVATION Imaginations unbound

NCSA, NVIDIA and GPUs • NCSA and NVIDIA have been partners for over a decade, building the expertise, experience and technology. • The efforts were at first exploratory and small scale, but have now blossomed into providing the largest GPU production resource in the US academic cyber- infrastructure • Today, we are focusing on helping world class science and engineering teams decrease their time to insight for some of the world’s most important and challenging computational and data analytical problems Imaginations unbound

Innovative Systems Lab 1.0 Research • Evaluation of emerging computing architectures • Reconfigurable computing • Many-core (GPU) architecture • Heterogeneous clusters • Systems software research and development • Run-time systems • GPU accelerator cluster management • Tools and utilities: GPU memory test, power profiling, etc. • QP: first GPU cluster at NCSA Power Measurement and 16 HP xw9400 workstations efficiency 32 Quadro Plex Computing Servers Imaginations unbound

How might this be used for Science?? • Focus on Application development for emerging computing architectures – not benchmarking • Cuda Center of Excellence in the Coordinated Systems MD Simulations and Computational Laboratory Microscope; see John Stone! • Computer Science efforts in GPUs • A variety of discipline departments • Computational chemistry (electronic structure, MD) • Computational physics (QCD) • Cosmology and astronomy • Data mining Turbulence simulations; Paul Woodward Imaginations unbound

FROM PIONEERING TO PRODUCTION Imaginations unbound

Blue Waters Computing S ys tem Aggregate Memory – 1.6 PB >1 TB/sec IB Switch 10/40/100 Gb External Servers Ethernet Switch 100 GB/sec 120+ Gb/sec Spectra Logic: 300 usable PB Sonexion: 26 usable PB 100-300 Gbps WAN Imaginations unbound

Original Blue Waters Goals • Deploy a computing system capable of sustaining more than one petaflops or more for a broad range of applications • Cray system achieves this goal using a well defined metrics • Enable the Science Teams to take full advantage of the sustained petascale computing system • Blue Waters Team has established strong partnership with Science Teams, helping them to improve the performance and scalability of their applications • Enhance the operation and use of the sustained petascale system • Blue Waters Team is developing tools, libraries and other system software to aid in operation of the system and to help scientists and engineers make effective use of the system • Provide a world-class computing environment for the petascale computing system • The NPCF is a modern, energy-efficient data center with a rich WAN environment (100-400 Gbps) and data archive (>300 PB) • Exploit advances in innovative computing technology • Proposal anticipated the rise of heterogeneous computing and planned to help the computational community transition to new modes for computational and data-driven science and engineering Imaginations unbound

Details of Blue Waters Imaginations unbound

Production Computation Science with XK nodes • The Computational Microscope • PI – Klaus Schulten • Simulated flexibility of ribosome trigger factor complex at full length and obtained better starting configuration of trigger factor model (simulated to 80ns) • 100ns simulation of cylindrical HIV 'capsule’ of CA proteins revealed it is stabilized by hydrophobic interactions between CA hexamers; maturation involves detailed remodeling rather than disassembly/re-assembly of CA lattice, as had been proposed. • 200ns simulation of CA pentamer surrounded by CA hexamers suggested interfaces in hexamer-hexamer and hexamer-pentamer pairings involve different patterns of interactions • Simulated photosynthetic membrane of a chromatophore in bacterium Rps. photometricum for 20 ns -- simulation of a few hundred nanoseconds will be needed Images from Klaus Schulten and John Stone, University of Illinois at Urbana-Champaign Imaginations unbound

XK7 For Visualization on Blue Waters • Many visualization utilities rely on the OpenGL API for hardware-accelerated rendering • Unsupported by default XK7 system software • Enabling NVIDIA’s OpenGL required that we: • Change operating mode of the XK7 GPU firmware • Develop a custom X11 stack • Work with Cray to acquire alternate driver package from NVIDIA • Blue Waters is the first Cray to offer this functionality which has been distributed to other systems now Imaginations unbound

Impact: VMD • Molecular dynamics analysis and visualization tool used by “The Computational Microscope” science team (PI Klaus Schulten) • 10X to 50X rendering speedup in VMD • Interactive rate visualization • Drastic reduction in required time to fine tune parameters for production visualization Imaginations unbound

Summary • NVIDIA GPUs are critical to the nation’s most powerful academic supercomputer • Partnership with NCSA, Cray, NVIDIA • Science, engineering, private sector partner development all depend on it • Enable research that cannot be done in any other way, on any other system • See John Stone presentation, Bill Kramer presentation for more details Imaginations unbound