Programming Models for Parallel Computing Katherine Yelick U.C. - - PowerPoint PPT Presentation

▶
programming models for parallel computing
SMART_READER_LITE
LIVE PREVIEW

Programming Models for Parallel Computing Katherine Yelick U.C. - - PowerPoint PPT Presentation

Nov 08, 2023 236 likes •613 views

Programming Models for Parallel Computing Katherine Yelick U.C. Berkeley and Lawrence Berkeley National Lab http://titanium.cs.berkeley.edu http://upc.lbl.gov 1 LCPC 2006 Kathy Yelick Parallel Computing Past Not long ago, the viability

slide-1
SLIDE 1

1

LCPC 2006 Kathy Yelick

Programming Models for Parallel Computing

Katherine Yelick U.C. Berkeley and Lawrence Berkeley National Lab http://titanium.cs.berkeley.edu http://upc.lbl.gov

slide-2
SLIDE 2

Kathy Yelick, 2 LCPC 2006

Parallel Computing Past

  • Not long ago, the viability of parallel computing was

questioned:

  • Several panels titled “Is parallel processing dead?”
  • “On several recent occasions, I have been asked whether

parallel computing will soon be relegated to the trash heap reserved for promising technologies that never quite make it.”

  • Ken Kennedy, CRPC Directory, 1994
  • But then again, there’s a history of tunnel vision
  • “I think there is a world market for maybe five computers.”
  • Thomas Watson, chairman of IBM, 1943.
  • “There is no reason for any individual to have a computer in their

home”

  • Ken Olson, president and founder of Digital Equipment Corporation,

1977.

  • “640K [of memory] ought to be enough for anybody.”
  • Bill Gates, chairman of Microsoft,1981.

Slide source: Warfield et al.

slide-3
SLIDE 3

Kathy Yelick, 3 LCPC 2006

Moore’s Law is Alive and Well

2X transistors/Chip Every 1.5 years

Called “Moore’s Law”

Moore’s Law

Microprocessors have become smaller, denser, and more powerful.

Gordon Moore (co-founder of Intel) predicted in 1965 that the transistor density of semiconductor chips would double roughly every 18 months.

Slide source: Jack Dongarra

slide-4
SLIDE 4

Kathy Yelick, 4 LCPC 2006

But Clock Scaling Bonanza Has Ended

  • Processor designers are forced to go

“multicore” due to

  • Heat density: faster clock means hotter chips
  • more cores with lower clock rates burn less power
  • Declining benefits of “hidden” Instruction Level Parallelism

(ILP)

  • Last generation of single core chips probably over-engineered
  • Lots of logic/power to find ILP parallelism, but it wasn’t in the apps
  • Yield problems
  • Parallelism can also be used for redundancy
  • IBM Cell processor has 8 small cores; a blade system with all 8

sells for $20K, whereas a PS3 is about $600 and only uses 7

slide-5
SLIDE 5

Kathy Yelick, 5 LCPC 2006

Power Density Limits Serial Performance

Clock Scaling Extrapolation:

slide-6
SLIDE 6

Kathy Yelick, 6 LCPC 2006

Revolution is Happening Now

  • Chip density is

continuing increase ~2x every 2 years

  • Clock speed is not
  • Number of

processor cores may double instead

  • There is little or no

hidden parallelism (ILP) to be found

  • Parallelism must

be exposed to and managed by software

Source: Intel, Microsoft (Sutter) and Stanford (Olukotun, Hammond)

slide-7
SLIDE 7

Kathy Yelick, 7 LCPC 2006

1 10 100 1000 10000 1978 1980 1982 1984 1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 2006

Performance (vs. VAX-11/780)

25%/year 52%/year ??%/year

Revolution in Hardware: Multicore

From Hennessy and Patterson, Computer Architecture: A Quantitative Approach, 4th edition, Sept. 15, 2006

Power density and ILP limits software-visible parallelism

3X

slide-8
SLIDE 8

Kathy Yelick, 8 LCPC 2006

Why Parallelism (2007)?

  • These arguments are no longer theoretical
  • All major processor vendors are producing multicore chips
  • Every machine will soon be a parallel machine
  • All programmers will be parallel programmers???
  • New software model
  • Want a new feature? Hide the “cost” by speeding up the code first
  • All programmers will be performance programmers???
  • Some may eventually be hidden in libraries, compilers, and high

level languages

  • But a lot of work is needed to get there
  • Big open questions:
  • What will be the killer apps for multicore machines?
  • How should the chips be designed: multicore, manycore, heterogenous?
  • How will they be programmed?
slide-9
SLIDE 9

Kathy Yelick, 9 LCPC 2006

100 1000 10000 100000 1E+06 1E+07 1E+08 1E+09 1E+10 1E+11 1E+12

1993 1996 1999 2002 2005 2008 2011 2014

SUM #1 #500

Petaflop with ~1M Cores By 2008

1Eflop/s 100 Pflop/s 10 Pflop/s 1 Pflop/s 100 Tflop/s 10 Tflops/s 1 Tflop/s 100 Gflop/s 10 Gflop/s 1 Gflop/s 10 MFlop/s

1 PFlop system in 2008

Slide source Horst Simon, LBNL

Data from top500.org 6-8 years

Common by 2015?

slide-10
SLIDE 10

Kathy Yelick, 10 LCPC 2006

Memory Hierarchy

  • With explicit parallelism, performance becomes a software problem
  • Parallelism is not the only way to get performance; locality is at least

as important

  • And this problem is growing, as off-chip latencies are relatively flat

(about 7% improvement per year) compared to processor performance

  • n-chip

cache registers datapath control processor Second level cache (SRAM) Main memory (DRAM) Secondary storage (Disk) Tertiary storage (Disk/Tape)

TB GB MB KB B Size 10sec 10ms 100ns 10ns 1ns Speed

slide-11
SLIDE 11

Kathy Yelick, 11 LCPC 2006

Predictions

  • Parallelism will explode
  • Number of cores will double every 12-24 months
  • Petaflop (million processor) machines will be common

in HPC by 2015 (all top 500 machines will have this)

  • Performance will become a software problem
  • Parallelism and locality are key will be concerns for

many programmers – not just an HPC problem

  • A new programming model will emerge for

multicore programming

  • Can one programming model (not necessarily one

language) cover games, laptops, and top500 space?

slide-12
SLIDE 12

12

LCPC 2006 Kathy Yelick

PGAS Languages: What, Why, and How

slide-13
SLIDE 13

Kathy Yelick, 13 LCPC 2006

Parallel Programming Models

  • Parallel software is still an unsolved problem !
  • Most parallel programs are written using either:
  • Message passing with a SPMD model
  • for scientific applications; scales easily
  • Shared memory with threads in OpenMP, Threads, or Java
  • non-scientific applications; easier to program
  • Partitioned Global Address Space (PGAS)

Languages

  • global address space like threads (programmability)
  • SPMD parallelism like MPI (performance)
  • local/global distinction, i.e., layout matters (performance)
slide-14
SLIDE 14

Kathy Yelick, 14 LCPC 2006

Partitioned Global Address Space Languages

  • Explicitly-parallel programming model with SPMD

parallelism

  • Fixed at program start-up, typically 1 thread per processor
  • Global address space model of memory
  • Allows programmer to directly represent distributed data structures
  • Address space is logically partitioned
  • Local vs. remote memory (two-level hierarchy)
  • Programmer control over performance critical decisions
  • Data layout and communication
  • Performance transparency and tunability are goals
  • Initial implementation can use fine-grained shared memory
  • Base languages UPC (C), CAF (Fortran), Titanium (Java)
  • New HPCS languages have similar data model, but

dynamic multithreading

slide-15
SLIDE 15

Kathy Yelick, 15 LCPC 2006

Partitioned Global Address Space

  • Global address space: any thread/process may directly

read/write data allocated by another

  • Partitioned: data is designated as local or global

Global address space

x: 1 y: l: l: l: g: g: g: x: 5 y: x: 7 y: 0 p0 p1 pn

By default:

  • Object heaps

are shared

  • Program

stacks are private

  • 3 Current languages: UPC, CAF, and Titanium
  • All three use an SPMD execution model
  • Emphasis in this talk on UPC and Titanium (based on Java)
  • 3 Emerging languages: X10, Fortress, and Chapel
slide-16
SLIDE 16

Kathy Yelick, 16 LCPC 2006

PGAS Language Overview

  • Many common concepts, although specifics differ
  • Consistent with base language, e.g., Titanium is strongly typed
  • Both private and shared data
  • int x[10]; and

shared int y[10];

  • Support for distributed data structures
  • Distributed arrays; local and global pointers/references
  • One-sided shared-memory communication
  • Simple assignment statements: x[i] = y[i];
  • r

t = *p;

  • Bulk operations: memcpy in UPC, array ops in Titanium and CAF
  • Synchronization
  • Global barriers, locks, memory fences
  • Collective Communication, IO libraries, etc.
slide-17
SLIDE 17

Kathy Yelick, 17 LCPC 2006

Private vs. Shared Variables in UPC

  • C variables and objects are allocated in the private memory space
  • Shared variables are allocated only once, in thread 0’s space

shared int ours; int mine;

  • Shared arrays are spread across the threads

shared int x[2*THREADS] /* cyclic, 1 element each, wrapped */ shared int [2] y [2*THREADS] /* blocked, with block size 2 */

  • Heap objects may be in either private or shared space

Shared Global address space Private

mine: mine: mine:

Thread0 Thread1 Threadn

  • urs:

x[0,n+1] y[0,1] x[1,n+2] y[2,3] x[n,2n] y[2n-1,2n]

slide-18
SLIDE 18

Kathy Yelick, 18 LCPC 2006

PGAS Language for Multicore

  • PGAS languages are a good fit to shared

memory machines

  • Global address space implemented as reads/writes
  • Current UPC and Titanium implementation uses threads
  • Working on System V shared memory for UPC
  • “Competition” on shared memory is OpenMP
  • PGAS has locality information that may be important when

we get to >100 cores per chip

  • Also may be exploited for processor with explicit local

store rather than cache, e.g., Cell processor

  • SPMD model in current PGAS languages is both an

advantage (for performance) and constraining

slide-19
SLIDE 19

Kathy Yelick, 19 LCPC 2006

PGAS Languages on Clusters: One-Sided vs Two-Sided Communication

  • A one-sided put/get message can be handled directly by a network

interface with RDMA support

  • Avoid interrupting the CPU or storing data from CPU (preposts)
  • A two-sided messages needs to be matched with a receive to

identify memory address to put data

  • Offloaded to Network Interface in networks like Quadrics
  • Need to download match tables to interface (from host)

address message id data payload data payload

  • ne-sided put message

two-sided message network interface memory host CPU

Joint work with Dan Bonachea

slide-20
SLIDE 20

Kathy Yelick, 20 LCPC 2006

One-Sided vs. Two-Sided: Practice

100 200 300 400 500 600 700 800 900 10 100 1,000 10,000 100,000 1,000,000 Size (bytes) Bandwidth (MB/s)

GASNet put (nonblock)" MPI Flood

Relative BW (GASNet/MPI)

1.0 1.2 1.4 1.6 1.8 2.0 2.2 2.4 10 1000 100000 10000000 Size (bytes)

  • InfiniBand: GASNet vapi-conduit and OSU MVAPICH 0.9.5
  • Half power point (N ½ ) differs by one order of magnitude
  • This is not a criticism of the implementation!

Joint work with Paul Hargrove and Dan Bonachea

(up is good)

NERSC Jacquard machine with Opteron processors

slide-21
SLIDE 21

Kathy Yelick, 21 LCPC 2006

GASNet: Portability and High-Performance

(down is good)

GASNet better for latency across machines

8-byte Roundtrip Latency 14.6 6.6 22.1 9.6 6.6 4.5 9.5 18.5 24.2 13.5 17.8 8.3

5 10 15 20 25

Elan3/Alpha Elan4/IA64 Myrinet/x86 IB/G5 IB/Opteron SP/Fed

Roundtrip Latency (usec)

MPI ping-pong GASNet put+sync

Joint work with UPC Group; GASNet design by Dan Bonachea

slide-22
SLIDE 22

Kathy Yelick, 22 LCPC 2006

(up is good)

GASNet at least as high (comparable) for large messages

Flood Bandwidth for 2MB messages

1504 630 244 857 225 610 1490 799 255 858 228 795 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Elan3/Alpha Elan4/IA64 Myrinet/x86 IB/G5 IB/Opteron SP/Fed

Percent HW peak (BW in MB)

MPI GASNet

GASNet: Portability and High-Performance

Joint work with UPC Group; GASNet design by Dan Bonachea

slide-23
SLIDE 23

Kathy Yelick, 23 LCPC 2006

(up is good)

GASNet excels at mid-range sizes: important for overlap

GASNet: Portability and High-Performance

Flood Bandwidth for 4KB messages

547 420 190 702 152 252 750 714 231 763 223 679 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Elan3/Alpha Elan4/IA64 Myrinet/x86 IB/G5 IB/Opteron SP/Fed

Percent HW peak

MPI GASNet

Joint work with UPC Group; GASNet design by Dan Bonachea

slide-24
SLIDE 24

Kathy Yelick, 24 LCPC 2006

Communication Strategies for 3D FFT

Joint work with Chris Bell, Rajesh Nishtala, Dan Bonachea

chunk = all rows with same destination pencil = 1 row

  • Three approaches:
  • Chunk:
  • Wait for 2nd dim FFTs to finish
  • Minimize # messages
  • Slab:
  • Wait for chunk of rows destined

for 1 proc to finish

  • Overlap with computation
  • Pencil:
  • Send each row as it completes
  • Maximize overlap and
  • Match natural layout

slab = all rows in a single plane with same destination

slide-25
SLIDE 25

Kathy Yelick, 25 LCPC 2006

NAS FT Variants Performance Summary

  • Slab is always best for MPI; small message cost too high
  • Pencil is always best for UPC; more overlap

200 400 600 800 1000 Myrinet 64 InfiniBand 256 Elan3 256 Elan3 512 Elan4 256 Elan4 512 MFlops per Thread Best MFlop rates for all NAS FT Benchmark versions Best NAS Fortran/MPI Best MPI Best UPC

100 200 300 400 500 600 700 800 900 1000 1100 Myrinet 64 InfiniBand 256 Elan3 256 Elan3 512 Elan4 256 Elan4 512 MFlops per Thread Best NAS Fortran/MPI Best MPI (always Slabs) Best UPC (always Pencils)

.5 Tflops

Myrinet Infiniband Elan3 Elan3 Elan4 Elan4 #procs 64 256 256 512 256 512 MFlops per Thread Chunk (NAS FT with FFTW) Best MPI (always slabs) Best UPC (always pencils)

slide-26
SLIDE 26

26

LCPC 2006 Kathy Yelick

Making PGAS Real: Applications and Portability

slide-27
SLIDE 27

Kathy Yelick, 27 LCPC 2006

AMR in Titanium

C++/Fortran/MPI AMR

  • Chombo package from LBNL
  • Bulk-synchronous comm:
  • Pack boundary data between procs

Titanium AMR

  • Entirely in Titanium
  • Finer-grained communication
  • No explicit pack/unpack code
  • Automated in runtime system

Code Size in Lines

4200* 6500 35000 C++/F/MPI 1500 Elliptic PDE solver 1200 AMR operations 2000 AMR data Structures Titanium

10X reduction in lines of code!

* Somewhat more functionality in PDE part of Chombo code

AMR Work by Tong Wen and Philip Colella

slide-28
SLIDE 28

Kathy Yelick, 28 LCPC 2006

Performance of Titanium AMR

Speedup

10 20 30 40 50 60 70 80 16 28 36 56 112 #procs speedup Ti Chombo

  • Serial: Titanium is within a few % of C++/F; sometimes faster!
  • Parallel: Titanium scaling is comparable with generic optimizations
  • optimizations (SMP-aware) that are not in MPI code
  • additional optimizations (namely overlap) not yet implemented

Comparable parallel performance

Joint work with Tong Wen, Jimmy Su, Phil Colella

slide-29
SLIDE 29

Kathy Yelick, 29 LCPC 2006

Particle/Mesh Method: Heart Simulation

  • Elastic structures in an incompressible fluid.
  • Blood flow, clotting, inner ear, embryo growth, …
  • Complicated parallelization
  • Particle/Mesh method, but “Particles” connected

into materials (1D or 2D structures)

  • Communication patterns irregular between particles

(structures) and mesh (fluid)

Joint work with Ed Givelberg, Armando Solar-Lezama, Charlie Peskin, Dave McQueen

2D Dirac Delta Function Code Size in Lines

8000 Fortran 4000 Titanium

Note: Fortran code is not parallel

slide-30
SLIDE 30

Kathy Yelick, 30 LCPC 2006

Immersed Boundary Method Performance

Hand-Optimized (planes, 2004)

10 20 30 40 50 1 2 4 8 16 32 64 128 procs time (secs)

256^3 on Power3/Colony 512^3 on Power3/Colony 512^2x256 on Pent4/Myrinet

Automatically Optimized (sphere, 2006)

0.5 1 1.5 2 1 2 4 8 16 32 64 128 procs time (secs)

128^3 on Power4/Federation 256^3 on Power4/Federation

Joint work with Ed Givelberg, Armando Solar-Lezama, Charlie Peskin, Dave McQueen

slide-31
SLIDE 31

Kathy Yelick, 31 LCPC 2006

slide-32
SLIDE 32

Kathy Yelick, 32 LCPC 2006

Dense and Sparse Matrix Factorization

Blocks 2D block-cyclic distributed Panel factorizations involve communication for pivoting Matrix- matrix multiplication used here. Can be coalesced

Completed part of U Completed part of L

A(i,j) A(i,k) A(j,i) A(j,k)

Trailing matrix to be updated Panel being factored

Joint work with Parry Husbands and Esmond Ng

slide-33
SLIDE 33

Kathy Yelick, 33 LCPC 2006

Matrix Factorization in UPC

  • UPC factorization uses a highly multithreaded style
  • Used to mask latency and to mask dependence delays
  • Three levels of threads:
  • UPC threads (data layout, each runs an event scheduling loop)
  • Multithreaded BLAS (boost efficiency)
  • User level (non-preemptive) threads with explicit yield
  • No dynamic load balancing, but lots of remote invocation
  • Layout is fixed (blocked/cyclic) and tuned for block size
  • Same framework being used for sparse Cholesky
  • Hard problems
  • Block size tuning (tedious) for both locality and granularity
  • Task prioritization (ensure critical path performance)
  • Resource management can deadlock memory allocator if not careful

Joint work with Parry Husbands

slide-34
SLIDE 34

Kathy Yelick, 34 LCPC 2006

UPC HP Linpack Performance

X1 UPC vs. MPI/HPL

200 400 600 800 1000 1200 1400 60 X1/64 X1/128 GFlop/s MPI/HPL UPC

Opteron cluster UPC vs. MPI/HPL

50 100 150 200 Opt/64 GFlop/s MPI/HPL UPC

Altix UPC. Vs. MPI/HPL

20 40 60 80 100 120 140 160 Alt/32 GFlop/s MPI/HPL UPC

  • Comparable to MPI HPL (numbers from HPCC database)
  • Faster than ScaLAPACK due to less synchronization
  • Large scaling of UPC code on Itanium/Quadrics (Thunder)
  • 2.2 TFlops on 512p and 4.4 TFlops on 1024p

Joint work with Parry Husbands

UPC vs. ScaLAPACK

20 40 60 80

2x4 pr oc gr i d 4x4 pr oc gr i d

GFlops

ScaLAPACK UPC

slide-35
SLIDE 35

Kathy Yelick, 35 LCPC 2006

PGAS Languages and Symbolic Computing

  • Most of these applications are numeric
  • Experience in parallel symbolic computing
  • Grobner basis completion procedure [CAD 92, PPoPP 93, RTA 93]
  • Compiling Verilog [IVC 95]
  • The Perfect Phylogeny Problem [Supercomputing 95]
  • Connected components
  • Mesh generation
  • What do these applications require?
  • Complex, irregular shared data structures
  • Not just distributed arrays
  • Ability to communicate/share data asynchronously
  • Not bulk-synchronous; not two-sided messaging
  • Fast low-overhead communication/sharing
  • Shared memory is ideal, remote procedure invocation useful
slide-36
SLIDE 36

Kathy Yelick, 36 LCPC 2006

Portability of Titanium and UPC

  • Titanium and the Berkeley UPC translator use a similar model
  • Source-to-source translator (generate ISO C)
  • Runtime layer implements global pointers, etc
  • Common communication layer (GASNet)
  • Both run on most PCs, SMPs, clusters & supercomputers
  • Operating Systems:
  • Linux, FreeBSD, Tru64, AIX, IRIX, HPUX, Solaris, Cygwin, MacOSX, Unicos, SuperUX
  • Supported CPUs:
  • x86, Itanium, Alpha, Sparc, PowerPC, PA-RISC, Opteron
  • GASNet communication:
  • Myrinet, Quadrics, Infiniband, IBM LAPI, Cray X1, SGI Altix, SHMEM, MPI and UDP
  • Specific platforms:
  • HP AlphaServer, Cray X1, IBM SP, NEC SX-6, Cluster X (Big Mac), SGI Altix 3000
  • Underway: Cray XT3, BG/L (both run over MPI)
  • Can be mixed with MPI, C/C++, Fortran
  • Several other compilers for UPC: HP, Cray, MTU, Intrepid, IBM

Also used by gcc/upc

Joint work with Titanium and UPC groups

slide-37
SLIDE 37

Kathy Yelick, 37 LCPC 2006

Conclusions

  • Parallel computing is the future
  • Time to think about parallelization strategies; think in long

term towards machine trends

  • Best time ever for a new parallel language
  • PGAS Languages
  • Good fit for shared and distributed memory
  • Control over locality and (for better or worse) SPMD
  • Support needs of symbolic and numeric communities
  • Offer incremental parallelism
  • Available for download
  • Berkeley UPC compiler: http://upc.lbl.gov
  • Titanium compiler: http://titanium.cs.berkeley.edu