for Convergence of Extreme Computing and Big Data Satoshi Matsuoka - - PowerPoint PPT Presentation
TSUBAME2.0, 2.5 towards 3.0 for Convergence of Extreme Computing and Big Data Satoshi Matsuoka Professor Global Scientific Information and Computing (GSIC) Center Tokyo Institute of Technology Fellow, Association for Computing Machinery (ACM)
Satoshi Matsuoka Professor Global Scientific Information and Computing (GSIC) Center Tokyo Institute of Technology Fellow, Association for Computing Machinery (ACM)
HP-CAST SC2014 Presentation New Orleans, USA 20141114
2 2
32nm 40nm >400GB/s Mem BW 80Gbps NW BW ~1KW max
>1.6TB/s Mem BW >12TB/s Mem BW 35KW Max >600TB/s Mem BW 220Tbps NW Bisecion BW 1.4MW Max
TSUBAME 2.0 New Development
HP S HP SL390G7 L390G7 (De (Developed f eloped for
TS TSUB UBAM AME 2.0, Modified f E 2.0, Modified for
2.5) GPU: NVIDIA Kepler K20X x 3 1310GFlops, 6GByte Mem Mem(per GPU)
CPU: Intel Westmere-EP 2.93GHz x2 Multi I/O chips, 72 PCI-e (16 x 4 + 4 x 2) lanes --- 3GPUs + 2 IB QDR Memory: 54, 96 GB DDR3-1333 SSD：60GBx2, 120GBx2
Thin Node
Infiniband QDR x2 (80Gbps)
Productized as HP ProLiant
SL390s
Modified for TSUABME2.5
Peak Perf. 4.08 Tflops ~800GB/s Mem BW 80GBps NW ~1KW max NVIDIA Fermi M2050 1039/515 GFlops NVIDIA Kepler K20X 3950/1310 GFlops
Phase-field simulation for Dendritic Solidification [Shimokawabe, Aoki et. al.] Gordon Bell 2011 Winner
solidification required for the evaluation of new materials.
Weak scaling on TSUBAME (Single precision) Mesh size（1GPU+4 CPU cores）:4096 x 162 x 130
TSUBAME 2.0 2.000 PFlops (4,000 GPUs+16,000 CPU cores) 4,096 x 6,480 x 13,000
TSUBAME 2.5 3.444 PFlops (3,968 GPUs+15,872 CPU cores) 4,096 x 5,022 x 16,640
Developing lightweight strengthening material by controlling microstructure Low-carbon society
Application TSUBAME2.0 Performance TSUBAME2.5 Performance Boost Ratio
Top500/Linpack 4131 GPUs (PFlops)
1.192 2.843 2.39
Green500/Linpack 4131 GPUs (GFlops/W)
0.958 3.068 3.20
Semi-Definite Programming Nonlinear Optimization 4080 GPUs (PFlops)
1.019 1.713 1.68
Gordon Bell Dendrite Stencil 3968 GPUs (PFlops)
2.000 3.444 1.72
LBM LES Whole City Airflow 3968 GPUs (PFlops)
0.592 1.142 1.93
Amber 12 pmemd 4 nodes 8 GPUs (nsec/day)
3.44 11.39 3.31
GHOSTM Genome Homology Search 1 GPU (Sec)
19361 10785 1.80
MEGADOC Protein Docking 1 node 3GPUs (vs. 1CPU core)
37.11 83.49 2.25
2013/11 Green 500 #6 in the world
2012/12 2013/12
18% Power Reduction
11.4 Petaflops SFP/DFP $1400mil 6 years (incl. power) x30 TSUBAME2 TS TSUB UBAM AME2.0 .0(2010) → TS TSUB UBAM AME2.5(2013) 17.1 Petaflops SFP 5.76 Petaflops DFP $45mil / 6 years (incl. power)
(TSUBAME2 Deploying State-of-Art Tech.)
TSUBAME2.5 BG/Q Sequoia K Computer Single Precision FP 17.1 Petaflops 20.1 Petaflops 11.3 Petaflops Green500 (MFLOPS/W) Nov. 2013 3,068.71 (6th) 2,176.58 (26th) 830.18 (123rd) Operational Power (incl. Cooling) ~0.8MW 5~6MW? 10~11MW Hardware Architecture Many-Core (GPU) + Multi-Core Hetero Multi-Core Homo Multi-Core Homo Maximum HW Threads > 1 Billion ~6 million ~700,000 Memory Technology GDDR5+DDR3 DDR3 DDR3 Network Technology Luxtera Silicon Photonics Standard Optics Copper Non Volatile Memory / SSD SSD Flash all nodes ~250TBytes None None Power Management Node/System Active Power Cap
Rack-level measurement only Rack-level measurement only
Virtualization KVM (G & V queues,
Resource segragation)
None None
– Latest power control, efficient liquid cooling, energy recovery
– Bigger capacity than the entire global Intenet (several 100Tbps)
– Petabytes of NVM, several Terabytes/s BW, several 100 million IOPS – Next generation “scientific big data” support
design, VM & container-based dynamic deployment…
Algorithms (CSRA)
Algorithms… (SC14 paper Tue 14:00-14:30 “Fail in Place Network…”)
Search/Sort, Deep Learning
Hybrid Algorithms
programming substrates,
for deep memory hierarchy, for fault tolerance, …
TSUBAME KFC
Rates and Volumes are extremely immense
Social NW – large graph processing
– 〜1 billion users – Average 130 friends – 30 billion pieces of content shared per month
– 500 million active users – 340 million tweets per day
– 300 million new websites per year – 48 hours of video to YouTube per minute – 30,000 YouTube videos played per second
Genomics advanced sequence matching Social Simulation
Lincoln Stein, Genome Biology, vol. 11(5), 2010
Sequencing data (bp)/$
x4000 per 5 years c.f., HPC x33 in 5 years
– Target Area: Planet
(Open Street Map)
– 7 billion people
– Road Network for Planet: 300GB (XML) – Trip data for 7 billion people 10KB (1trip) x 7 billion = 70TB – Real-Time Streaming Data
(e.g., Social sensor, physical data)
– 700TB
Weather – real time large data assimilation
①30-sec Ensemble Forecast Simulations 2 PFLOP ②Ensemble Data Assimilation 2 PFLOP Himawari 500MB/2.5min シミュレーション データ シミュレーション データ Ensemble Forecasts 200GB
Phased Array Radar 1GB/30sec/2 radars
シミュレーション データ シミュレーション データ Ensemble Analyses 200GB A-1. Quality Control A-2. Data Processing B-1. Quality Control B-2. Data Processing Analysis Data 2GB ③30-min Forecast Simulation 1.2 PFLOP 30-min Forecast 2GB Repeat every 30 sec.
NOT simply mining Tbytes Silo Data Peta~Zetabytes Data Ultra High-BW Data Stream Highly Unstructured, Irregular Complex correlations between data from multiple sources Extreme Capacity, Bandwidth, Compute All Required
Kronecke ecker graph h BSP Problem blem
A: 0.57, B: 0.19 C: 0.19, D: 0.05
November 15, 2010 Graph 500 Takes Aim at a New Kind of HPC Richard Murphy (Sandia NL => Micron) “ I expect that this ranking may at times look very different from the TOP500 list. Cloud architectures will almost certainly dominate a major chunk of part of the list.”
The 8th Graph500 List (June2014): K Computer #1, TSUBAME2 #12
Koji Ueno, Tokyo Institute of Technology/RIKEN AICS
#1 K Computer #12 TSUBAME2
Reality: Top500 Supercomputers Dominate No Cloud IDCs at all
A Major Northern Japanese Cloud Datacenter (2013)
Juniper EX8208 Juniper EX8208 2 zone switches (Virtual Chassis)
Juniper EX4200
Zone (700 nodes)
Juniper EX4200 Juniper EX4200
Zone (700 nodes)
Juniper EX4200 Juniper EX4200
Zone (700 nodes)
Juniper EX4200
Juniper MX480 Juniper MX480 10GbE 10GbE 10GbE 10GbE
LACP
the Internet
8 zones, Total 5600 nodes, Injection 1GBps/Node Bisection 160Gigabps
Advanced Silicon Photonics 40G single CMOS Die 1490nm DFB 100km Fiber
Supercomputer Tokyo Tech. Tsubame 2.0 #4 Top500 (2010)
~1500 nodes compute & storage Full Bisection Multi-Rail Optical Network Injection 80GBps/Node Bisection 220Terabps
Entire Global Internet Average Data BW ~200 Tbps (source: CISCO)
Supercomputers Compute&Batch-Oriented More fragile Cloud IDC Very low BW & Efficiency Highly available, resilient Convergent Architecture (Phases 1~4) Large Capacity NVM, High-Bisection NW
PCB TSV Interposer High Powered Main CPU Low Power CPU DRAM DRAM DRAM NVM/Fla sh NVM/Fla sh NVM/Fla sh Low Power CPU DRAM DRAM DRAM NVM/Flas h NVM/Flas h NVM/Flas h2Tbps HBM 4~6HBM Channels 1.5TB/s DRAM & NVM BW 30PB/s I/O BW Possible 1 Yottabyte / Year
EBD System Software
Large Scale Metagenomics Massive Sensors and Data Assimilation in Weather Prediction
Ultra Large Scale Graphs and Social Infrastructures
Exascale Big Data HPC Co-Design
Graph Store EBD Bag
Co-Design
日本地図 ページ 日本地図KV S KV S KV S
EBD KVS Cartesian Plane
Co-Design
Given a top-class supercomputer, how fast can we accelerate next generation big data c.f. Clouds? Issues regading Architectural, algorithmic, and system software evolution? Use of GPUs?
– Increase in Parallelism, Heterogeneity, Density, BW
– NVM (Non-Volatile Memory), SCM (Storage Class Memory) FLASH, PCM, STT-MRAM, ReRAM, HMC, etc. – Next-gen HDDs (SMR), Tapes (LTFS), Cloud
Network Locality Productivity FT Algorithm Power Storage Hierarchy I/O
Problems
Heterogeneity Scalability
KVS KVS KVS
Graph Store
Cloud Datacenter
Large Scale Genomic Correlation Data Assimilation in Large Scale Sensors and Exascale Atmospherics Large Scale Graphs and Social Infrastructure Apps
100,000 Times Fold EBD “Convergent” System Architecture
TSUBAME 3.0 TSUBAME-GoldenBox
EBD Bag EBD KVS Cartesian Plane
MapReduce for EBD Workflow/Scripting Languages for EBD Interconnect
(InfiniBand 100GbE)
EBD Abstract Data Models
(Distributed Array, Key Value, Sparse Data Model, Tree, etc.)
EBD Algorithm Kernels
(Search/ Sort, Matching, Graph Traversals, , etc.) NVM
(FLASH, PCM, STT-MRAM, ReRAM, HMC, etc.)
HPC Storage EBD File System EBD Data Object
Akiyama Group Suzumura Group Miyoshi Group Matsuoka Group Tatebe Group
SQL for EBD Graph Framework Message Passing (MPI, X10) for EBD PGAS/Global Array for EBD Network
(SINET5)
Intercloud / Grid (HPCI)
Web Object Storage EBD Burst I/O Buffer EBD Network Topology and Routing
Koibuchi Group
17
Programming Layer Basic Algorithms Layer System SW Layer Big Data & SC HW Layer Big Data & SC HW Layer
List
Rank GTEPS
Implementation
November 2013
4 5524.1 2
Top-down only June 2014
1 17977. 05
Efficient hybrid
200 400 600 800 1000 1200 64 nodes (Scale 30) 65536 nodes (Scale 40) Elapsed Time (ms) Communicaton Computation
*Problem size is weak scaling 73% total exec time wait in communication 1236
MTEPS/node
274
MTEPS/node
Red uce Map Red uce Map Red uce Shuf fle Shuf fle Sort Scan Initialization Operation on GPU Operation on GPU GPU CPU Memcpy (H2D, D2H) Processing for each chunk Map Red uce Sort Map
Problem: GPU memory capacity limits scalable large-scale graph processing Emergence of large-scale graphs
→ Need for fast graph processing on supercomputers
500 1000 1500 2000 2500 3000 500 1000 1500 Performance [MEdges/sec] Number of Compute Nodes
1CPU (S23 per node) 1GPU (S23 per node) 2CPUs (S24 per node) 2GPUs (S24 per node) 3GPUs (S24 per node)
2.10x
(3 GPU vs 2CPU)
Weak scaling on TSUBAME2.5
Experimental Results: performance improvement over CPUs
speedup
Proposal: Out-of-core GPU memory management on MapReduce
EBD Programming Framework
Motivation
Effectiveness of sorting for large-scale GPU- based heterogeneous systems remains unclear
Approach
Offload local sort, the most time-consuming phase, to GPU accelerators
10 20 30 500 1000 1500 2000
# of proccesses (2 proccesses per node) Keys/second(billions)
HykSort 1thread HykSort 6threads HykSort GPU + 6threads
GPU-HykSort achieves 2.2x performance improvement with 50GB/s CPU-GPU interconnect
Performance of weak scaling Performance prediction
1.4x 3.6x 0.25 TB/s ~1024 nodes ~2048 GPUs
unsorted sorted sorted sorted sorted unsorted unsorted unsorted unsorted
Separate an unsorted array Transfer an unsorted chunk to GPU memory Transfer a sorted chunk to DRAM Merge sorted chunks into a sorted array
sorted GPU
Iter 1 Iter 2 Iter 3 Iter 4 Iter 1 Iter 2 Iter 3 Iter 4 Sort a chunk on GPU
Implementation
unsorted unsorted unsorted unsorted sorted sorted sorted sorted merged merged merged merged local sort select splitters data transfer merge Process 0 Process 1 Process 2 Process 3
EBD Algorithm Kernels
Aleksandr Drozd, Miquel Pericàs, Satoshi Matsuoka. Efficient String Sorting on Multi- and Many- Core Architectures. in Proceedings of IEEE 3rd International Congress on Big Data. Anchorage, USA, August 2014 Aleksandr Drozd, Miquel Pericàs, Satoshi Matsuoka. MSD Radix String Sort on GPU: Longer Keys, Shorter Alphabets in proceedings of 第142回ハイパフォーマンスコンピューティング合同 研究発表会 (HOKKE-21)
long/variable-length keys (like strings)
characters (based on MSD Radix sort algorithm)
combining data-parallel and task- parallel stages
apple apricot banana kiwi 70 M keys/second sorting throughput
EBD Algorithm Kernels
CPU Intel Xeon E5-2690 2690 × 2 DRAM
256 256 GB
GB NVM NVM EBD-I/O 2TB × 2
0.0 0.0 1.0 1.0 2.0 2.0 3.0 3.0 4.0 4.0 5.0 5.0 6.0 6.0 23 23 24 24 25 25 26 26 27 27 28 28 29 29 30 30 31 31
SCAL CALE( E(# v # vert ertic ices es = 2 = 2SCALE)
DRAM + EBD-I/O DRAM Only Limit of DRAM Only 3.8 3.8 4.1
Med edian an Gi Giga gaTE TEPS PS
(Giga Traversed Edges Per Seconds) (Giga Traversed Edges Per Seconds)
mSATA-SSD
RAID Card
RAID Card (RAID 0) 0) mSATA SSD SSD
× 8
mSATA SSD SSD ・・・
www.adaptec.com www.crucial.com/
4 times larger graph with
6.9 % of degradation DRAM NVM
Load highly accessed graph data before BFS Holds full size of Graph Holds highly accessed data
[BigData2014]
Top-down Bottom-up
# of frontiers:nfrontier，# of all vertices:nall, Parameter : α, β
Switching two approaches
Tokyo’s Institute of Technology G raphCREST-Custom # 1 is ranked No.3 in the Big Data category of the G reen G raph 500 Ranking of Supercomputers with 35.21 MTEPS/ W on Scale 31Ranked 3rd
in Green Graph500 (June 2014)
EBD Algorithm Kernels
JST-CREST project, 2012-2018, PI Toshio Endo
Comm/BW reducing algorithms System software for mem hierarchy mgmt
HPC Architecture with hybrid memory devices
HMC, HBM O(GB/s) Flash Next-gen NVM
Target: Realizing extremely Fast&Big simulations of {O(100PF/s) or O(10PB/s)} & O(10PB) around 2018
Caution: Simply “swapping out” to larger host memory is disastrously slow PCIe traffic is too large! Keys are “Communication Avoiding & Locality Improvement” Algorithms
GPU mem 6GB Host memory 54GB
L2$ 1.5MB
GPU cores
250GB/s 8GB/s TSUBAME2.5 node GPU card
CPU cores
block, before proceeding to the next block
– Originally proposed to improve cache locality [Kowarschik 04] [Datta 08]
s-step updates at once
Step 1 Step 2 Step 3 Step 4
Simulated time
Redundant computation is introduced due to data dependency with neighbor
Introducing “larger halo” Step 1 Step 2 Step 3 Step 4 Redundancy can be removed when blocks are computed sequentially [Demmels 12]
Multi-level TB to reduce both
20 40 60 80 100 120 140 160 180 200 500 1000 1500 2000 Speed (GFlops) Size of Each Dimension Common Naïve Basic-TB Opt-TB
with little overhead!!! A step towards extremely fast&big simulations
3D 7point stencil on a K20X GPU (6GB GPU mem) Bigger Faster
5.3GB 52GB Version 1 Version 3
support larger domains
– Complex loop structure, complex border handling
– HHRT (Hybrid Hierarchical Runtime) – Physis DSL, by Maruyama, RIKEN
Compute node
GPU mem Host mem Process’s data
at granuarity of processes
reducing algorithms HHRT (Hybrid hierarchical RT) is for GPU supercomputers and MPI+CUDA user applications
Compute node Compute node
Larger Faster 3D 7point stencil on a single K20X GPU
20 40 60 80 100 120 140 10 20 30 Speed (GFlops) Problem Size (GB) Hand-TB NoTB HHRT-TB
Beyond GPU memory efficient execution w/ moderate programming cost
Weak scalability on TSUBAME2.5 Small: 3.4GB per GPU Large: “16GB” per GPU (>6GB!)
5 10 15 20
50 100 150 200 Speed (TFlops)
The number of GPUs
Small Large
14TFlops with 3TB Problem
1/500 Size, 1/150 Power, 1/500 Cost, x5 DRAM+ NVM Memory 10 Petaflops, 10 Petabyte Hiearchical Memory (K: 1.5PB), 10K nodes 50GB/s Interconnect (200-300Tbps Bisection BW) (Conceptually similar to HP “The Machine”)
PCB TSV Interposer
Optical SW & Launch Pad
Low Power CPU DRAM DRAM DRAM NVM/Flash NVM/Flash NVM/Flash Low Power CPU DRAM DRAM DRAM NVM/Flash NVM/Flash NVM/Flash
2Tbps HBM 4~6HBM Channels 2TB/s DRAM & NVM BW
Direct Chip-Chip Interconnect with DWDM optics
~11TFlops SFP
SSD, ~7GB/s I/O
immersive cooling
TSUBAME2
2022: x10 Flops, x10 Mem Bandwidth, silicon photonics, x10 NVM, x10 node density, with new device and packaging technologies
OUR GOALS
MPI abstractions Hardware Layer Application Layer MPI Layer
Performance Analysis Process View
② Non-intrusively profile low- level metrics ③ Flexible hardware-centric performance analysis ① Portably expose MPI’s internal performance
① ② ③
1 2 3 4 5 1 2 4 8 16 32 64 128 256 512 1,024 2,048 4,096 8,192 16,384 32,768
Message size (bytes)
Overhead of our profiler (named ibprof): NAS Parallel FT Benchmark
Runtime overhead = less than 0.02% (12.1919s -> 12.1935s)
MPI_Alltoall
Network visualization of TSUBAME 2.5 running the Graph500 benchmark on 512 nodes
Avg: 2.06%
EBD Interconnect
Take advantage of NVM new capabilities: atomic
writes, huge sparse address space, direct access to NVM device natively as a KVS
Enable range-queries support for KVS running
natively on NVM like FusionIO ioDrive
Provide optional persistence to the BPTree
structure and also snapshots
NVM-BPTree is a Key-Value Stores (KVS) running natively over Non-Volatile- Memory (NVM), like flash, supporting range-queries.
OpenNVM like Key-value store Interface NVM (Fusion-io flash device) KVS on NVM supporting range-queries In-memory B+Tree
(512B)
performance compared with original KVS
EBD NVM System Software
LLNL-PRES-654744
Extreme scale I/O for Burst Buffer
Extreme scale C/R modeling
k
p0(t +ck) t0(t +ck)
k
pi(t +ck) ti(t +ck)
i
k k
i
p0(r
k)pi(r
k)p0(r
k)t0(r
k)ti(r
k)Duration
t +ck r
kNo failure Failur e
li : i -level checkpoint time
: c -level checkpoint time
r
c : c -level recovery timecc
t : Interval
p0(T) t0(T)
: No failure for T
: Expected time pi(T)
ti(T) : i - level failure for T : Expected time
mSATA ☓ 8
Adaptec RAID ☓ 1
mSATA mSATA mSATA mSATA mSATA mSATA mSATA mSATA0.5 1 1.5 2 2.5 3 3.5 4 4.5 2 4 6 8 10 12 14 16 Read/Write throughput (GB/sec) # of Processes
Read - Peak Read - Local Read - IBIO Read - NFS Write - Peak Write - Local Write - IBIO Write - NFS Chunk buffers Compute node 1 Compute node 2 Compute node 3 Compute node 4 IBIO client IBIO client IBIO client IBIO server thread file4 Storage file3 file2 file1 3 fd1 fd2 fd3 fd4 2 Writer thread Writer thread Writer thread Writer thread Writer threads chunk 1 4 5 IBIO clientIBIO write: four IBIO clients and one IBIO server
Compute node Burst buffer node Application IBIO Client IBIO Server Write threadsEBD I/O
Li = Ci + Ei Oi = Ci + Ei (Sync.) Ii (Async.) Ci or Ri =
< C/R date size / node >☓ <# of C/R nodes per Si* >
< write perf. ( wi ) > or <read perf. ( ri ) >
Hi
Compute node
Si
i = 0 i > 0
1 2
mi
Hi-1 Hi-1 Hi-1
Storage Model: HN {m1, m2, . . . , mN }
EBD NVM System Software
Storage 1
I/O node I/O node I/O node Compute node Compute node Compute nodeSystem 1
I/O node I/O node I/O node Compute node Compute node Compute nodeI/O Bursting Buffer Nodes System 2 WAN LAN LAN LAN LAN Buffer Queue I/O node
Cloud-based I/O Burst Buffer Architecture (I/O Burst Buffer) In collaboration talks with Amazon EC2
20 40 60 80 100 120 140 160 1 2 3 4 5 6 7 8 Throughput (MB/s) # of node
read experiment result read simulation result write experiment result write simulation result read without IOnode write without IOnode 100 200 300 400 500 600 1 2 3 4 Throughput(MB/s) # of I/O nodes=# of clients write experiment result read experiment result write simulation result read simulation result read without IOnode write without IOnode
7X!
Main idea: using several compute nodes in public cloud as I/O nodes
Buffer I/O data in the main memory of I/O nodes All I/O nodes maintain a on-memory buffer queue dynamic burst buffer, # of I/O nodes can be dynamically decided
Taking advantage of high throughput of LAN inside public cloud
One Client Performance Multi Client Performance
1.7X!
8X! Cloud Supercomputer