and Extreme Scale Research Computing D. Karres, Beckman Institute - - PowerPoint PPT Presentation
NSF XSEDE Campus Champions and Extreme Scale Research Computing D. Karres, Beckman Institute J. Alameda, National Center for Supercomputing Applications S. Kappes, National Center for Supercomputing Applications Outline Research
– National Science Foundation Investments
– Research IT
– Campus Champions
– National Science Foundation Investments
– Research IT
– Campus Champions
PEARC18, July 25, 2018
– Leadership HPC
– Innovative HPC
– Services
– National Science Foundation Investments
– Research IT
– Campus Champions
Slides adapted from:
Linda Akli, SURA
Assistant Director, Education, Training, Outreach Manager, XSEDE Broadening Participation Program
Fall 2018
Foundation for a National CI Ecosystem
services that federates with other high-end facilities and campus-based resources
Unprecedented Integration of Diverse Advanced Computing Resources
possible the continuous addition of new technology capabilities and services
Mission: Accelerate scientific discovery
generation scholars, researchers, and engineers
Strategic Goals:
Total Research Funding Supported by XSEDE 2.0
10
$1.97 billion in research supported by XSEDE 2.0
September 2016 - April 2018
Research funding only. XSEDE leverages and integrates additional infrastructure, some funded by NSF (e.g. “Track 2” systems) and some not (e.g. Internet2).
NSF, 754.3, 38% NIH, 432.0, 22% DOE, 325.0, 16% DOD, 175.1, 9% DOC, 55.2, 3% NASA, 38.2, 2% All Others , 187.7, 10%
Earthquake Science Molecular Dynamics Nanotechnology Plant Science Storm Modeling Epidemiology Particle Physics Economic Analysis of Phone Network Patterns Large Scale Video Analytics (LSVA) Decision Making Theory Library Collection Analysis
Replicating Brain Circuitry to Direct a Realistic Prosthetic Arm XSEDE researchers visualize massive Joplin, Missouri tornado
A collaboration of social scientists, humanities scholars and digital researchers harnessed the power of high-performance computing to find and understand the historical experiences of black women by searching two massive databases of written works from the 18th through 20th centuries.
Visualization
Visualization Portal
based visualization
integration
Storage
All compute/visualization allocations include access to limited disk and scratch space
resource file systems to accomplish project goals
storage on XSEDE systems is used for large-scale persistent storage requested in conjunction with compute and visualization resources.
alone storage allows storage allocations independent of a compute allocation.
13
Bridges: Featuring interactive on-demand access, tools for gateway building, and virtualization. Comet: hosting a variety of tools including Amber, GAUSSIAN, GROMACS, Lammps, NAMD, and VisIt. Jetstream: A self-provisioned, scalable science and engineering cloud environment Stampede-2: Intel's new innovative MIC technology on a massive scale Super Mic: Equipped with Intel's Xeon Phi
compute nodes. Wrangler: Data Analytics System combines database services, flash storage and long- term replicated storage, and an analytics
HADOOP Service Reservations, and Database instances.
The CIPRES science gateway: A NSF investment launching thousands of scientific publications with no sign of slowing down. https://sciencenode.org/feature/cipres-one-facet-in-bold-nsf-vision.php?clicked=title
XSEDE High Throughput Computing Partnership
16
supporting usage of ~2,000,000 core hours per day
data dependencies are a good fit for OSG
with US institutions and who are funded by US funding agencies
Open Science Grid
Education Research
17
Champion Startup
Technical information Training Help Desk/Consultants Extended Collaborative Support Service
XSEDE Training Course Catalog with all materials in a single location Course Calendar for viewing a listing of and registering for upcoming training events and a registration Online Training on materials relevant to XSEDE users Badges available for completing selected training Some events provide participation documentation Training Roadmaps
pearc19.pearc.org
July 28 - Aug 1, 2019 Chicago, IL
Blue Waters
campus
National Center for Supercomputing Applications at the University of Illinois
24
Goal of the project Ensure researchers and educators can advance discovery in all fields of study
Blue Waters System
Top-ranked system in all aspects of its capabilities Emphasis on sustained performance
(Gigabit per second)
26
Blue Waters System
Processors, Memory, Interconnect, Online Storage, System Software, Programming Environment
Software
Visualization, analysis, computational libraries, etc.
SEAS: Software Engineering and Application Support
Petascale Applications
Computing Resource Allocations
User and Production Support
WAN Connections, Consulting, System Management, Security, Operations, …
National Petascale Computing Facility EOT
Education, Outreach, and Training
GLCPC
Great Lakes Consortium for Petascale Computing
Hardware
External networking, IDS, back-up storage, import/export, etc
Industry partners
Blue Waters Ecosystem
Blue Waters Computing System
Sonexion: 26 usable PB
>1 TB/sec 100 GB/sec
Spectra Logic: 200 usable PB 400+ Gb/sec WAN
Scuba Subsystem: Storage Configuration for User Best Access
1.66 PB
10/40/100 Gb Ethernet Switch IB Switch External Servers
13.34 PFLOPS
Blue Waters Allocations: ~600 Active Users
NSF PRAC, 80%
Illinois, 7%
GLCPC, 2%
Education, 1%
Industry Innovation and Exploration, 5% Broadening Participation, a new category for underrepresented communities
28
Blue Waters Allocations: ~600 Active Users
NSF PRAC, 80%
Illinois, 7%
GLCPC, 2%
Education, 1%
Industry Innovation and Exploration, 5% Broadening Participation, a new category for underrepresented communities
29
Usage by Discipline and User
Data From Blue Waters 2016-2017 Annual Report
Biophysics 10.8% Physics 12.3% Astronomical Sciences 10.4% Earth Sciences 13.3% Stellar Astronomy and Astrophysics 7.4% Molecular Biosciences 7.6% Atmospheric Sciences 6.4% Chemistry 5.2% Fluid, Particulate, and Hydraulic Systems 4.5% Engineering 4.9% Extragalactic Astronomy and Cosmology 2.4% Planetary Astronomy 2.5% Galactic Astronomy 2.1% Materials Research 2.5% Nuclear Physics 1.3% Biochemistry and Molecular Structure and Function 1.5% Neuroscience Biology 0.8% Computer and Computation Research 1.0% Biological Sciences 1.5% Magnetospheric Physics 0.5% Chemical, Thermal Systems 0.3% Design and Computer- Integrated Engineering 0.3% Climate Dynamics 0.1% Environmental Biology 0.1% Social, Behavioral, and Economic Sciences 0.1% Other 7.5%
31
LIGO binary-blackhole observation verification 160-million-atom flu virus EF5 Tornado Simulation Arctic Elevation Maps Earthquake rupture
Support for Python and Containers
Waters users use Python.
Python packages and two Python versions.
ML/DL, etc.
containers using Shifter.
access to native driver.
container.
32
33
Currently available libraries
In the Pipeline
Data challenge: large training datasets
Data Science and Machine Learning
Blue Waters Summary
Outstanding Computing System
reliability/maintainability
Most balanced system in the open community
memory, storage, compute, or network intensive or any combination.
NCSA is a leader in developing and deploying these technologies as well as contributing to community efforts.
34
Texas Advanced Computing Center
– Production mid-2019 – More to come soon
– National Science Foundation Investments
– Research IT
– Campus Champions
https://researchit.illinois.edu/
– Aggregation of many sources of Research IT resources – Some provided as services to campus wide community – Many contributed by campus units
– Partnership between Research Data Service (University Library), NCSA and Tech Services – Mid-scale storage of actively used data
storage-overview/
– Cloud services platform – Access coordinated by Tech Services
– Consultation with library based subject experts for Data Management Plans, required for most funding proposals – Offered by University Library
Access to Learning and Scholarship
– Digital repository for research and scholarship produced at the University of Illinois – Offered by University Library – https://www.ideals.illinois.edu/
– High performance computing cluster available at Illinois – Offered by Research IT – Investor-based access – On-demand Research Computing as a Service access – https://campuscluster.illinois.edu/
– Training opportunities from many sources aggregated on Research IT portal – Combination of live events and asynchronous training opportunities
page=1
Software Development Consulting
– Allocated resource; collaborate with one or more experts for up to 1 year on your research computing project – Apply online at https://researchit.illinois.edu/initiatives
– New computational resource – Currently in pilot phase
– National Science Foundation Investments
– Research IT
– Campus Champions
Slides adapted from: Dana Brunson XSEDE Campus Engagement co-manager
Director, High Performance Computing Center Oklahoma State University
With support from:
their local researchers to use CI, especially large scale and advanced computing.
user support, trainers, coordinators, research software engineers, etc.).
(XSEDE, Internet2, Globus, BDHubs, national HPC centers, etc).
47
Research computing community facilitating computing- and data-intensive research and education
48
local researchers and educators find and use the advanced digital services that best meet their needs; and,
challenges and solutions
With support from: https://www.xsede.org/web/site/community-engagement/campus-champions
TeraGrid 2004-2011
formed XSEDE 1: 2011-2016
initiated
XSEDE 2: 2016- present
and EPSCoR jurisdiction
Current numbers:
–Academic institutions in EPSCoR jurisdictions : 74 –Minority Serving Institutions: 48
49
100 200 300 400 500 600 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
Monthly Discussions
– Guest speakers – Community updates
Other Calls and Meetings
51
52
53
Longest Threads in 2017 1. Data Management Initiatives on YOUR Campus - TGR/TGW? 2. Adjusted Peak Performance for HPC clusters 3. OS flavors in HPC 4. HPC systems login access only with VPN -- good idea? 5. AMD EPYC and Intel Skylake Pricing Extremes 6. Theoretical Peak Performance 7. Champions-style job board? 8. CephFS for HPC? 9. Successful Scheduling
Slack
Email list:
how to help someone with their work and they take you seriously.
cutting edge is and can interact with them.
no hesitation about sending something to the champions list or the slack channel even if it’s a dumb question
not the Champions community
Campus Champions & Friends: Community of Communities
55
Photo Credit: Tiffany Jolley
56