: Welcome the Second Spring of Dataflow and Parallel Computing -- - - PowerPoint PPT Presentation
: Welcome the Second Spring of Dataflow and Parallel Computing -- Toward a Path of Convergence for Ecosystems of Extreme-Scale HPC, Big Data and Beyond Guang R. Gao ACM Fellow and IEEE Fellow Endowed Distinguished Professor, University of
:
Welcome the Second Spring of Dataflow and Parallel Computing --
Toward a Path of Convergence for Ecosystems of
Extreme-Scale HPC, Big Data and Beyond Guang R. Gao
ACM Fellow and IEEE Fellow Endowed Distinguished Professor, University of Delaware And Founder of ETI
A&M 05-16-2016 1
Divergence or Convergence ?
A&M 05-16-2016 2
The Pessimism over our field..
field ?
setback ….
A&M 05-16-2016 3
AIST-03-01-2016 演讲 4
IPDPS2005-Keynote 5
State of Parallel Computer Architecture Innovations
– “…researchers basked in parallel-computing
parallel algorithms for every applicable sequential
structure to interconnect thousands of processors…”
– But “.. The market for massively parallel computers
has collapsed, and many companies have gone
[IEEE Computer, Nov. 1994, pp 74-75]
IPDPS2005-Keynote 6
State of Parallel Computer Architecture Innovations
has become pejorative. For computer designers, the revolution is over and
[“End of Architecture”, Burton Smith 1990s]
5/25/2016
A&M 05-16-2016 7
Corporations Vanishing
(1985 – 2005)
1990 1992 1994 1996 1985 2000 1998 2005 1999 Sequent 1994 Thinking Machines 1992 Meiko Scientific 1995 Pyramid 1998 DEC 1989 ETA MasPar 1996 Convex Computer 1994 nCube 2005 Kendall Square Resarch 1996 ESCD 1990 Multiflow 1990 Cray Research 1996 BBN 1997 Myrias 1991
Keynote at the 2005 IPDPS Conference Denver, CO
“Is Parallel Computing Dead ?”
AIST-03-01-2016 演讲 8
“The announcement that Thinking Machines would seek Chapter 11 bankruptcy protection, although not unexpected, sent shock waves through the high- performance computing community. Coupled with the well-publicized problems of Kendall Square Research and the rumored problems of Intel Supercomputer Systems Division, this event has led many people to question the long- term viability of the parallel computing industry and even parallel computing itself. Meanwhile, the dramatic strides in the performance of scientific workstations continues to squeeze the market for parallel supercomputing. On several recent occasions, I have been asked whether parallel computing will soon be relegated to the trash heap reserved for promising technologies that never quite make it. Washington certainly seems to be looking in the other direction--agency program managers, if they talk of high-performance computing at all, seem to view it as a small and relatively unimportant subcomponent of the National Information Infrastructure.
Divergence or Convergence
A&M 05-16-2016 9
宋代诗人陆游的作品 《游山西村》
A&M 05-16-2016 10
2005-Present A Second Spring of HPC Parallel Computing
– Heat wall – Memory wall – Other walls
powewrful alternative to beat the walls
in the past decade
A&M 05-16-2016 11
2016/5/25
IPDPS2005-Keynote 12
2016/5/25
IPDPS2005-Keynote 13
Communication Ports for 3D Mesh Inter-Chip Network UPC+ / - Co- array Fortran OpenMP-XN EARTH-C + / - MPI……
Application Programming API Cyclops Thread Virtual Machine
Thread Managem ent Shared Mem ory Operations Thread Creation & Termination Scheduling Dynamic memory management Put / get with syncacquire / release
fibers async function invocationKcc/ gcc Com piler Tool chain
Cyclops-64 Programming Models and System Software Supports
Cyclops-64 ISA
Fine- Grain Multithreading Thread Synchronization Load Balancing Others Put / get Location Consistency System Software PercolationAdvanced Execution/ Programming Model Infrastructure and Tools
Simulation / Emulation Analytical ModelingBase Execution Model
Fine-Grain Multithreading (e.g. EARTH, CARE)24x24
24 PC cards in 1 shishkebab
1 PetaFlops
A-Switch
Crossbar Network…
MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK…
TU TU SP SP FPU 4 GB/sec * 6 4 GB/sec 50 MB/sec 1 Gbit/s ethernetOff-Chip Memory
Other Chips via 3D meshOff-Chip Memory Off-Chip Memory Off-Chip Memory
IDE HDD 4 GB/sec 6 SP SP SP SP SP SP SP SP TU TU SP SP FPU TU TU SP SP FPU TU TU SP SP FPU A-switch DMA 6A-Switch
Crossbar Network Crossbar Network…
MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK MEMORY BANK…
TU TU SP SP FPU TU TU TU TU SP SP SP SP FPU FPU 4 GB/sec * 6 4 GB/sec 50 MB/sec 1 Gbit/s ethernetOff-Chip Memory Off-Chip Memory
Other Chips via 3D meshOff-Chip Memory Off-Chip Memory Off-Chip Memory Off-Chip Memory Off-Chip Memory Off-Chip Memory
IDE HDD 4 GB/sec 6 SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP SP TU TU SP SP FPU TU TU TU TU SP SP SP SP FPU FPU TU TU SP SP FPU TU TU TU TU SP SP SP SP FPU FPU TU TU SP SP FPU TU TU TU TU SP SP SP SP FPU FPU A-switch DMA A-switch DMA 6 Convergence
A&M 05-16-2016 14
High-Performance Computing: The term "high-performance computing" refers to systems that, through a combination of processing capability and storage capacity, can solve computational problems that are beyond
the capability of small- to medium-scale systems.
[Obama’s Executive Order]
Gao-03-07-2016 MEXT 演讲 15
Big data is a broad term for data sets so large or complex that traditional data
processing applications are inadequate.
search, sharing, storage, transfer,visualization, querying and information privacy.
analytics or certain other advanced methods to extract value from data, and seldom to a particular size of data set.
Gao-03-07-2016 MEXT 演讲 16
Data analytics and computing ecosystem compared
Courtesy by “Exscale Computing and Big Data”, DANIEL A. REED AND JACK DONGARRA , CACM July 2015
Mahout: machine learning tool Hive: data warehouse software Pig: provide high level language for big data Sqoop: exchange data with traditional database Flume: log management Zookeeper: maintaining consistency Storm: real-time computation system. Hbase: a distributed, scalable big data store. AVRO: data serialization system.
Data Analytic
FORTRAN,C,C++: languages PAPI: performance and debugging tool MPI/OpenMP: multi-core parallel model SLURM: batch scheduler Lustre: parallel file system
Computational Science
NOTE: The Divergence of Big Data and HPC Eco-Systems!
Waseda-01-26-2016 演讲 17
Key Insights
analytics have diverged, to the detriment of both; unification
is essential to address a spectrum of major research domains
complicated functions on that data, or store a substantial part of it;
new approaches are required to meet these challenges
advanced computer architectures and global standards for processing data, even as international competition complicates the openness of the scientific process
Courtesy by “Exscale Computing and Big Data”, DANIEL A. REED AND JACK DONGARRA , CACM July 2015
Waseda-01-26-2016 演讲 18
Divergence or Convergence
A&M 05-16-2016 19
Actors (dataflow) ?
A&M 05-16-2016 20
21
Coarse-Grain vs. Fine-Grain Multithreading
CPU Memory
Fine-Grain non-preemptive thread- The “hotel” model
Thread Unit A Thread Pool CPU Memory Executor Locus A Single Thread
Coarse-Grain thread- The family home model
Thread Unit A&M 05-16-2016
A unit of computation which interacts with the global state only at its entrance and exit points
I do not like to use the term “functional” here – which usually means “stateless”!
22 A&M 05-16-2016
Operational Semantics of Codelets Enabling/Firing Rules
Consider a Codelet graph G – with an assignment of events on some of its edges:
– An event is present on each of its input edges; – none of the output edges may have any events.
(i.e. fired). The firing of a Codelet will remove all input events (one from each input), and will produce output events, one on each output.
23 A&M 05-16-2016
The Codelet: A Fine-Grain Piece of Computing
Codelet
Result Object Data Objects
A&M 05-16-2016 24
5/25/2016
Repeatability, Composability, among others.
This Looks Like Data Flow !
Evolution of Multithreaded Execution and Architecture Models
Non-dataflow based
CDC 6600 1964 MASA
Halstead 1986
HEP
1978
Cosmic Cube
Seiltz 1985
J-Machine
Dally 1988-93
M-Machine
Dally 1994-98
Dataflow model inspired
MIT TTDA
Arvind 1980
Manchester
Gurd & Watson 1982
*T/Start-NG
MIT/Motorola 1991-
SIGMA-I
Shimada 1988
Monsoon
Papadopoulos & Culler 1988
P-RISC
Nikhil & Arvind 1989
EM-5/4/X
RWC-1 1992-97 Iannuci’s 1988-92
Others: Multiscalar (1994), SMT (1995), etc.
Flynn’s Processor
1969
CHoPP’77 CHoPP’87
TAM
Culler 1990
Tera
1990-
Alwife
Agarwal 1989-96
Cilk
Leiserson
LAU
Syre 1976
Eldorado CASCADE Static Dataflow
Dennis 1972 MIT
Arg-Fetching Dataflow
DennisGao 1987-88
MDFA
Gao 1989-93
EARTH
Hum et al. 1993-2006
HTVM/TNT-X
DelCuvillo and Gao 2000-2010
Codelet Model
Gao et. al. 2009-
An early version of this slide was presented in my invited talk at Turing Award Winner Fran Allen’s Retirement Party 2002
A&M 05-16-2016 25
26 A&M 05-16-2016
27 A&M 05-16-2016
Dataflow Model of Computation (pioneered by J.B. Dennis, Early 1970s)
(PXM) where a program unit is enabled for execution upon the availability of its input data at runtime.
性的）departure from the classical von Neumann computation model (often referred as a control-driven or control- flow PXM).
A&M 05-16-2016 28
General purpose parallel machines based on a dataflow graph model of computation 2013年，因其在操作系统和 数据流领域的重大贡献 荣获IEEE John von Neumann Medal
29 A&M 05-16-2016
Dataflow Model Superioity --
Breaking “两墙一锁”
parallelism exploitation due to the von- Neumann computation model [Arvind 1982]
bottle-neck: provide a tight/smooth coupling of the processing and data [Backus 1977 ACM Turing Award]
[SPARK latest news Deep Dive Into Databricks’ Big
Speedup Plans for Apache Spark, May, 2015]
30 A&M 05-16-2016
First Second Third Finally
A&M 05-16-2016 31
A&M 05-16-2016 32
A&M 05-16-2016 33
Execution Model API SWift Abstract Runtime Machine
SWARM
Programming Environment Platforms
Users Users
SWARM Execution Model
Programming Models
High-Level Programming API (MPI, Open MP, CnC, Xio, Chapel, etc.) Software packages Program libraries Utility applications Compilers Tools/SDK
Exascale Hardware Architecture
SWARM Runtime
Language Runtime A&M 05-16-2016 34
Abstract Codelet Machine (ACM)
small business based on Delaware
license and agreement
A&M 05-16-2016 35
36
D-TEC
http://www.dtec-xstack.org
XPRESS
http://xstack.sandia.gov/xpress
XTUNE http://ctop.cs.utah.edu/ x-tune/
GVR http://gvr.cs.uchicago.edu
Traleika Glacier (https://sites.google.com/
site/traleikaglacierxstack/
DynAX http://www.etinternational .com/xstack
SLEEC https://engineering.purdu e.edu/ ~milind/sleec/
DEGAS
http://crd.lbl.gov/groups-depts/ future-technologies-group/ projects/DEGAS/
CORVETTE
http://crd.lbl.gov/groups- depts/ future-technologies-group/ projects/corvette/
Codelet based
US DOE X-Stack Program --- 9 Awardees
A&M 05-16-2016
Event driven tasks (inspired by Delaware codelet model): Dataflow inspired codelets (self contained/”atomic”). Non blocking, no preemption. Programming model: Separation of concerns: Domain specification & HW mapping. Express data locality with hierarchical tiling. Global, shared, non-coherent address space. Optimization and auto generation of EDTs (HW specific). Execution model: Dynamic, event-driven scheduling, non-blocking. Dynamic decision to move computation to data. Observation based adaption (self-awareness). Implemented in the runtime environment. Separation of concerns: User application, control, and resource management.
Programming & Execution Model
A&M 05-16-2016 37
Convergence ?
A&M 05-16-2016 38
Data analytics and computing ecosystem compared
Courtesy by “Exscale Computing and Big Data”, DANIEL A. REED AND JACK DONGARRA , CACM July 2015
Mahout: machine learning tool Hive: data warehouse software Pig: provide high level language for big data Sqoop: exchange data with traditional database Flume: log management Zookeeper: maintaining consistency Storm: real-time computation system. Hbase: a distributed, scalable big data store. AVRO: data serialization system.
Data Analytic
FORTRAN,C,C++: languages PAPI: performance and debugging tool MPI/OpenMP: multi-core parallel model SLURM: batch scheduler Lustre: parallel file system
Computational Science
NOTE: The Divergence of Big Data and HPC Eco-Systems!
Applications and Community Codes Mahout, R, and Applications
Application Level
Hive Pig Sqoop Flume Storm Map-Reduce AVRO Hbase Big Table (key- value store) HDFS (Hadoop File System) Zookeeper (coordination) Cloud Services (e.g. AWS
Application Level Middleware and Management
Virtual Machines and Cloud Services (optional) Linux OS variant FORTRAN, C, C++, and IDEs Domain-specific Libraries MPI/OpenMP + Accelerator Tools Numerical Libraries Performance and Debugging Lustre (Parallel File System) Batch Scheduler (such as SLURM) System Monitoring Tools Linux OS variant Ethernet Switches Local Node Storage Commodit y X86 Racks Infiniband + Ethernet Switches SAN + Local Node Storage X86 Racks + GPUs or Accelerators
System Software Cluster Software Data Analytics Ecosystem Computational Science Ecosystem
A&M 05-16-2016 39
Issues and Challenges
eco-systems of extreme-scale computation and big data ?
– data vs. knowledge – Knowledge vs. “$$”s
innovation and entrepreneurship to bridge the gaps ?
40 A&M 05-16-2016
Innovation and Entrepreneurship – Jointly Funded by Public and Private Sectors Partnerships
Obama and DoD are going to Silicon Volley and announced a "cooperation" between the private sector VCs to promote/fund entrepreneurship for US mission-driven innovations
ground-breaking R&D
side
A&M 05-16-2016 41
A&M 05-16-2016 42
A&M 05-16-2016 43
Case I：Cyclops 64 Project
Cyclops-64 1.1 Petaflops 13 TB memory Processor Rack 11.52 Teraflops 144 GB memory Mid-plane 3.84 Teraflops 48 GB memory Node Card 80 Gigaflops 1 GB memory Cyclops-64 ASIC 80 Procs + 16 iCaches + 96 Port xbar switch C64 Processor 2 Thread Units + 60 KB SRAM + 1FP Unit
– first generation of large-scale many-core chip technology is employed (160 core/chip, upto 10,000 chips/system)
ever erage e dat atal alow model el to brea eak the e OS ba barrier er，
TNT）PXM。
Ethernet Switches Local Node Storage Commodity X86 Racks
Hardware
Infiniband Ethernet Switches SAN Local Node Storage X86 Racks, GPUs or Accelerators
Big Data Ecosystem HPC Ecosystem
Hadoop
System Software
Map-Reduce based API Spark RDD API Spark Storm
CrAMER/HAMR
MPI OpenMP SWARM OpenCL EARTH Cilk Flowlet-based API Storm API Programming Model MPI API OpenMP API Codelet API TiNy-Threads API Data Analytic Machine Learning Real Time Processing Financial Application
Application
Computational Chemistry Bioinformatics Computational physics Domain-specific Application A&M 05-16-2016 44
Ethernet Switches Local Node Storage Commodity X86 Racks
Hardware
Infiniband Ethernet Switches SAN Local Node Storage X86 Racks, GPUs or Accelerators
Big Data Ecosystem HPC Ecosystem
Hadoop
System Software
Map-Reduce based API Spark RDD API Spark Storm HAMR MPI OpenMP SWARM OpenCL EARTH Cilk Flowlet-based API Storm API Programming Model MPI API OpenMP API Codelet API TiNy-Threads API Data Analytic Machine Learning Real Time Processing Financial Application
Application
Computational Chemistry Bioinformatics Computational physics Domain-specific Application
DataFlow Model Inspired Runtime System
45 A&M 05-16-2016
A&M 05-16-2016 46
critical for the success in the “second spring”!
lessons that we have learned in the 1st Spring!
A&M 05-16-2016 47
Rich List (2015)
Wang Xing cracks the top 100 richest in China for the first time. Wang created the largest "online-to-offline commerce company in China. Wang, a student of Prof. Guang R. Gao at University of Delaware, went back to China in 2005 and started his innovation and entrepreneur career.
http://www.forbes.com/profile/wang-xing/
Latest News : Chinese startup raises largest private funding round ever!
Published: Jan 19, 2016 9:53 a.m. ET
48 A&M 05-16-2016
Public Support + Private Industry Giant + Small Entrepreneur
A&M 05-16-2016 49
50 A&M 05-16-2016