Far more than Petaflops: The Jlich Supercomputing Centre ScicomP 15 - PowerPoint PPT Presentation

Mitglied der Helmholtz-Gemeinschaft Far more than Petaflops: The Jülich Supercomputing Centre ScicomP 15 & SP-XXL Thomas Lippert Barcelona Supercomputing Centre Institute for Advanced Simulation May 20, 2009 Jülich Supercomputing Centre

Supercomputing Drives Basic Sciences New Physics Nuclear Physics Plasma Physics Geophysics Astrophysics Astrophysik Particle Physics Solid State Physics Kosmologie Cosmology Structure of Matter Chemistry

Supercomputing Drives Applied Science Environment Ageing Society Materials Energy Weather/Climatology Medicine Spintronics Plasma Physics Pollution / Ozone Hole Biology Nano-Science Fuel Cells

Supercomputing Drives Engineering and Business Competitiveness Reducing design costs by virtual prototyping:  faster time to market Allowing investigations where economics or ethics preclude experimentation  imperative of supercomputing

FROM JÜLICH TO EUROPE 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 5

Jülich in Brief Largest civilian research centre in Europe 360 Mio. Euro/a budget 4.300 staff members  1.200 scientists  700 guest scientists from 50 countries 9 Departments (Institutes) (Institute for Advanced Simulation) 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 6

You might have heard of …. 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 7

Jülich Supercomputing Centre (JSC) 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 8

IAS Organisation IAS Institute for Advanced Simulation Nano/Material Soft Matter Science Hadron Physics Biophysics Jülich Supercomputing Centre (JSC) 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 9

Milestones 1961 Zentralinstitut für Angewandte Mathematik (ZAM) 1987 Höchstleistungsrechenzentrum HLRZ 1998 HLRZ  John von Neumann Institut für Computing (NIC) 2007 ZAM  Jülich Supercomputing Centre (JSC) Member of Gauss Centre for Supercomputing 2008 Institute for Advanced Simulation Coordinator of the PRACE Project 2010 European Supercomputing Centre 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 10

 German Research School for Simulation Science  Co-funded by NRW, BMBF and Helmholtz Association  PhD and Master students in two-years course 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 12

Organization JSC Organization JSC (current staff assignment) Grid & Computational Science Large Scale Facility Infrastructures Mathematical Methods Technology Communication HPC Application Mathematics Computational Distributed Systems HPC & Grid Computing Systems Support Systems & Education Science SimLab UNICORE Technology HPC User/Project Modeling & JuNet & Operations Management Biology Development Development Ext. Networks Methods HPC System SimLab Grid HPC File- & Archive Numerical SL Operation Development Plasma Phys. Research Systems Networking Algorithms HPC Data Distributed D-Grid Program SimLab Mathematical Security Systems Operation Management Optimization Software Mol. Systems Network Technical Programming European HPC Research Group Education Infrastructure Infrastructure Technologies Environments Quant.-Inf. Public Applied Relations Visualization Performance Complex NIC Director MATSE Coordination Analysis Systems Secretaries, Administration Education Helmholtz NIC Young Research Group Investigators Group 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 13

HPC Systems

20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 15

Supercomputers 1956 First Computer in Jülich 1989 Cray YMP 0.003 Teraflop/s 1996 Cray T3E 0.8 Teraflop/s 2003 IBM p690 9 Teraflop/s 2006 BGL: JUBL 46 Teraflop/s 2008 BGP: JUGENE 223 Teraflop/s 2009 JuRoPA 200 Teraflop/s HPC-FF 100 Teraflop/s BGP: JUGENE 1000 Teraflop/s 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 16

Developing Supercomputers @ JSC IBM Power 4+ 2004 JUMP, 9 TFlop/s IBM Blue Gene/L 2005/6 JUBL, 45 TFlop/s IBM Power 6 IBM Blue Gene/P 2007/8 JUMP, 9 TFlop/s JUGENE, 223 TFlop/s File Server GPFS Intel Nehalem Clusters 2009 JUROPA 200 TFlop/s File Server IBM Blue Gene/P HPC-FF GPFS, Lustre JUGENE, 1 PFlop/s 100 TFlop/s General-Purpose Highly-Scalable 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 17

JUGENE: Jülich’s Scalable Petaflop System IBM Blue Gene/P JUGENE  32-bit PowerPC 450 core 850 MHz, 4-way SMP  72 racks, 294,912 procs  1 Petaflop/s peak  144 TByte main memory  connected to a Global Parallel File System (GPFS) with 5 PByte online disk capacity and up to 25 PByte offline tape capacity  Torus network First Petaflop system in Europe 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 18

Juropa 2208 compute nodes  2 Intel Nehalem-EP quad-core processors  2.93 GHz  SMT (Simultaneous Multithreading)  24 GB memory (DDR3, 1066 MHz)  IB QDR HCA (via Network Express Module) 17664 cores, 207 TF peak  Sun Microsystems Blade SB6048  Infiniband QDR with non-blocking Fat Tree topology  ParaStation Cluster-OS 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 20

HPC-FF 1080 compute nodes  2 Intel Nehalem-EP quad-core processors  2.93 GHz  SMT (Simultaneous Multithreading)  24 GB memory (DDR3, 1066 MHz) 8640 cores, 101 TF peak  Bull NovaScale R422-E2  Infiniband QDR with non-blocking Fat Tree topology  ParaStation Cluster-OS 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 21

Infiniband Topology 4 Compute Sets (CS) with 15 Compute Cells (CC) 23 x 4 QNEM modules, 24 ports each each 6 x M9 switches, 648 ports max. each, CC with 18 Compute Nodes (CN) and 1 Mellanox 468/276 links used MTS3600 (Shark) switch each Mellanox MTS3600 switches (Shark), 36 ports, Virtual 648-port switches constructed for service nodes from 54x/44x Mellanox MTS3600 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 22

JUST – Jülich Storage Cluster GPFS Storage Cluster for all our Supercomputers Supercomputers are Remote Clusters for GPFS 1 PB Capacity today, expansion to 6 PB in Q4 2009 20 GB/sec Bandwidth, expansion to 66 GB/sec in Q4 Tivoli Storage Manager (TSM) for backup, archive and HSM 2 SUN tape libraries used with TSM  16 PB capacity today  Can be expanded to 32 PB next year 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 23

Information and Technology Deputy Director JSC

Preparing Infrastructure for …. 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 25

Emerging multi- & many-core Architectures Accelerators promise exciting performances at low power  Cell Broadband Engine: 200 / 100 GFlop/s (100 W)  nVIDIA Tesla T10: 1000 / 80 GFlop/s (200 W)  AMD FireStream 9270: 1200 / 240 GFlop/s (220 W)  Programming paradigm CUDA, Brook, Cell-SDK, CellSS, RapidMind, OpenCL, ...  Application kernels have to be adapted by hand 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 26

Many-core prototypes @ JSC QPACE / eQPACE  Special purpose computer for lattice QCD Main design goal: energy- and cost-efficiency Developed by SFB/TR “Hadron Physics” 3-D torus network based on FPGA – SPE-to-SPE comm. Ultra-dense packaging: 25,6 TFlop/s per rack  Explore broader purpose capabilities within PRACE WP8 Enhanced communication: Beyond nearest neighbor & MEM-to- MEM Support of standard communication layers (MPI) JUICEnext QS22 cluster Cell based computational platform and test facility 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 27

Future Developments around JuRoPA Cluster Management  ParaStation (incl. MPI)  GridMonitor Operating System  SUSE SLES 11 Fighting Operating System Jitter 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 28

Building the D-Grid to the other UNICORE sites Client Internet DMZ Firewall UNICORE 5 UNICORE 6 Gateway Registry Globus LCG/gLite UNICORE6 UNICORE5 UNICORE6 UNICORE5 JUGGLE SoftComp 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 29

Communication Systems PRACE Project Manager

High-speed Supercomputer Connectivity 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 31

Pan-European Supercomputer Network Research and Provisioning DEISA: • DEISA design and operation of a pan-European 10 Gbit/s network • LOFAR: planning and operation of German-Dutch Phosphorus peering LOFAR • Phosphorus: R&D in on-demand optical networking 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 32

Data Communication – JuNet by Numbers • JSC: overall responsibility for Campus network JuNet & external connections JuNet Supercomputing centre • •  94 Ethernet switches,  95 Ethernet switches, 1,5 Tbit/s >8 Tbit/s  9.500 ports in 60 buildings  6.000 ports in 2 buildings  300 WLAN Access Points  Infiniband, proprietary networks External Connectivity •  5 Gbit/s X-WiN (redundant)  Project network operation: DEISA, LOFAR, Phosphorus  Dark fibres to RWTH, TZJ,  VPN and dial-in services FhG-Birlinghoven 20.5.2009 SP-XXL, Barcelona ThomasLippert, IAS/JSC 33

Towards the High-End HPC Service for European Science Thomas Eickermann, PRACE Project Coordination@FZ-Jülich