[PDF] - G I A V C E I R H S C I T Y F O Mathematics and PDF Document

SLIDE 1 T uto rial

n

MPI The MessageP assing Interface Willi am Gropp

A R G O N N E N A T I O N A L L A B O R A T O R Y U N I V E R S I T Y O F C H I C A G O

Mathematics

and Computer Science Division Argonne National Lab

rato

ry Argonne IL

groppmcsanlgov

SLIDE 2 Course Outline

Background
n

P a rall el Computing

Getting

Sta rted

MPI

Basics

Intermediate

MPI

T
ls

fo r writing lib ra r i es

Final

comments Thanks to Rust y Lusk fo r some

f

the material in this tuto rial This tuto rial ma y b e used in conjunction with the b

k

Using MPI which contains detailed descriptions

f

the use

f

the MPI routines

Material

that b eings with this symb

l

is advanced and ma y b e skipp ed

n

a rst reading

SLIDE 3 Background

P

a rallel Computing

Communicating

with

ther

p ro cesses

Co
p

erative

p

erations

Onesided
p

erations

The

MPI p ro cess

SLIDE 4 P a rallel Computing

Sepa

rate w

rk

ers

r

p ro cesses

Interact

b y exchanging info rmation

SLIDE 5 T yp es

f

pa rallel computing All use dierent data fo r each w

rk

er Datapa rallel Same

p

erations

n

dierent data Also called SIMD SPMD Same p rogram dierent data MIMD Dierent p rograms dierent data SPMD and MIMD a re essentially the same b ecause any MIMD can b e made SPMD SIMD is also equivalent but in a less p ractical sense MPI is p rima ril y fo r SPMDMIMD HPF is an example

f

a SIMD interface

SLIDE 6 Communicating with

ther

p ro cesses Data must b e exchanged with

ther

w

rk

ers

Co
p

erative

all

pa rties agree to transfer data

One

sided

ne

w

rk

er p erfo rms transfer

f

data

SLIDE 7 Co

p

erative

p

erations Messagepassing is an app roach that mak es the exchange

f

data co

p

erative Data must b

th

b e explicitl y sent and received An advantage is that any change in the receivers memo ry is made with the receivers pa rticipation

SEND( data ) Process 0 Process 1 RECV( data )

SLIDE 8 Onesided

p

erations Onesided

p

erations b et w een pa rallel p ro cesses include remote memo ry reads and writes An advantage is that data can b e accessed without w aiting fo r another p ro cess

Process 0 Process 1 Process 0 Process 1 (Memory) PUT( data ) (Memory) GET( data )

SLIDE 9 Class Example T ak e a pad

f

pap er Algo rithm Initiali ze with the numb er

f

neighb

rs

y

u

have

Compute

average

f

y

ur

neighb

rs

values and subtract from y

ur

value Mak e that y

ur

new value

Rep

eat until done Questions

Ho

w do y

u

get values from y

ur

neighb

rs
Which

step

r

iteration do they co rresp

nd

to Do y

u

kno w Do y

u

ca re

Ho

w do y

u

decide when y

u

a re done

SLIDE 10 Ha rdw a re mo dels The p revious example illustrates the ha rdw a re mo dels b y ho w data is exchanged among w

rk

ers

Distributed

memo ry eg P a ragon IBM SPx w

rkstation

net w

rk
Sha

red memo ry eg SGI P

w

er Challenge Cra y TD Either ma y b e used with SIMD

r

MIMD soft w a re mo dels

All

memo ry is distributed

SLIDE 11 What is MPI

A

messagepassing lib ra ry sp ecication

messagepassing

mo del

not

a compiler sp ecication

not

a sp ecic p ro duct

F
r

pa rallel computers clusters and heterogeneous net w

rks
F

ullfeatured

Designed

to p ermit unleash the development

f

pa rall el soft w a re lib ra ri es

Designed

to p rovide access to advanced pa rallel ha rdw a re fo r

end

users

lib

ra ry writers

to
l

develop ers

SLIDE 12 Motivation fo r a New Design

Message

P assing no w mature as p rogramm i ng pa radigm

w

ell understo

d
ecient

match to ha rdw a re

many

applications

V

endo r systems not p

rtable
P
rtable

systems a re mostly resea rch p rojects

incomplete
lack

vendo r supp

rt
not

at most ecient level

SLIDE 13 Motivation cont F ew systems

er

the full range

f

desired features

mo

dula ri t y fo r lib ra r ies

access

to p eak p erfo rmance

p
rtabilit

y

heterogeneit

y

subgroups
top
logies
p

erfo rmance measurement to

ls

SLIDE 14 The MPI Pro cess

Began

at William sburg W

rkshop

in Ap ril

Organized

at Sup ercomputing

Novemb

er

F
llo

w ed HPF fo rmat and p ro cess

Met

every six w eeks fo r t w

da

ys

Extensive
p

en email discussions

Drafts

readings votes

Prenal

draft distributed at Sup ercomputing

Tw
month

public comment p erio d

Final

version

f

draft in Ma y

Widely

availabl e no w

n

the W eb ftp sites netlib httpww wm cs a nl go vm pi in de xh tm l

Public

implem entations availabl e

V

endo r implementati

ns

coming so

n

SLIDE 15 Who Designed MPI

Broad

pa rticipation

V

endo rs

IBM

Intel TMC Meik

Cra

y

Convex

Ncub e

Lib

ra ry writers

PVM

p Zip co de TCGMSG Chameleon Exp ress Linda

Application

sp ecialists and consultants Companies Lab

rato

ries Universities ARCO ANL UC Santa Ba rba ra Convex GMD Syracuse U Cra y Res LANL Michigan State U IBM LLNL Oregon Grad Inst Intel NO AA U

f

New Mexico KAI NSF Miss State U Meik

ORNL

U

f

Southampton NA G PNL U

f

Colo rado nCUBE Sandia Y ale U P a raSoft SDSC U

f

T ennessee Shell SRC U

f

Ma ryland TMC W estern Mich U U

f

Edinburgh Co rnell U Rice U U

f

San F rancisco

SLIDE 16 F eatures

f

MPI

General
Communicato

rs combine context and group fo r message securit y

Thread

safet y

P
inttop
int

communicati

n
Structured

buers and derived datat yp es heterogeneit y

Mo

des no rmal blo cking and nonblo cking synchronous ready to allo w access to fast p roto cols buered

Collective
Both

builtin and userdened collective

p

erations

La

rge numb er

f

data movement routines

Subgroups

dened directly

r

b y top

logy

SLIDE 17 F eatures

f

MPI cont

Applicationo

ri ented p ro cess top

logies
Builtin

supp

rt

fo r grids and graphs uses groups

Proling
Ho
ks

allo w users to intercept MPI calls to install their

wn

to

ls
Environmental
inquiry
erro

r control

SLIDE 18 F eatures not in MPI

Nonmessagepassing

concepts not included

p

ro cess management

remote

memo ry transfers

active

messages

threads
virtual

sha red memo ry

MPI

do es not address these issues but has tried to remain compatible with these ideas eg thread safet y as a goal intercommuni cato rs

SLIDE 19 Is MPI La rge

r

Small

MPI

is la rge

functions
MPIs

extensive functionalit y requires many functions

Numb

er

f

functions not necessa rily a measure

f

complexit y

MPI

is small

functions
Many

pa rall el p rograms can b e written with just

basic

functions

MPI

is just right

One

can access exibilit y when it is required

One

need not master all pa rts

f

MPI to use it

SLIDE 20 Where to use MPI

Y
u

need a p

rtable

pa rallel p rogram

Y
u

a re writing a pa rallel lib ra ry

Y
u

have irregula r

r

dynamic data relationships that do not t a data pa rallel mo del Where not to use MPI

Y
u

can use HPF

r

a pa rallel F

rtran
Y
u

dont need pa rallelism at all

Y
u

can use lib ra ri es which ma y b e written in MPI

SLIDE 21 Why lea rn MPI

P
rtable
Exp

ressive

Go
d

w a y to lea rn ab

ut

subtle issues in pa rallel computing

SLIDE 22 Getting sta rted

W

riting MPI p rograms

Compiling

and linking

Running

MPI p rograms

Mo

re info rmation

Using

MPI b y William Gropp Ewing Lusk and Anthony Skjellum

The

LAM companion to Using MPI b y Zdzisla w Meglicki

Designing

and Building P a rall el Programs b y Ian F

ster
A

T uto rialUsers Guide fo r MPI b y P eter P acheco ftpmathusfcaedupubMPImpiguideps

The

MPI standa rd and

ther

info rmation is available at httpww w mcs a nl go vm pi

Also

the source fo r several implementati

ns

SLIDE 23 W riting MPI p rograms include mpih include stdioh int main argc argv

int

argc char argv

MPIInit

argc argv

printf

Hello worldn

MPIFinalize

return

SLIDE 24 Commenta ry

include

mpih p rovides basic MPI denitions and t yp es

MPIInit

sta rts MPI

MPIFinalize

exits MPI

Note

that all nonMPI routines a re lo cal thus the printf run

n

each p ro cess

SLIDE 25 Compiling and linking F

r

simple p rograms sp ecial compiler commands can b e used F

r

la rge p rojects it is b est to use a standa rd Mak ele The MPICH implementation p rovides the commands mpicc and mpif as w ell as Makefile examples in usrlocalmpiexamplesMak efilein

SLIDE 26 Sp ecial compilation commands The commands mpicc

first

firstc mpif

firstf

firstff ma y b e used to build simple p rograms when using MPICH These p rovide sp ecial

ptions

that exploit the p roling features

f

MPI mpilog Generate log les

f

MPI calls mpitrace T race execution

f

MPI calls mpianim Realtime animation

f

MPI not available

n

all systems There a re sp ecic to the MPICH implementati

n
ther

implem entati

ns

ma y p rovide simil a r commands eg mpcc and mpxlf

n

IBM SP

SLIDE 27 Using Mak eles The le Makefilein is a template Mak ele The p rogram script mpireconfig translates this to a Mak ele fo r a pa rticula r system This allo ws y

u

to use the same Mak ele fo r a net w

rk
f

w

rkstations

and a massively pa rallel computer even when they use dierent compilers lib ra ri es and link er

ptions

mpireconfig Makefile Note that y

u

must have mpireconfig in y

ur

PATH

SLIDE 28 Sample Mak elein

User

configur ab le

ptions
ARCH
ARCH

COMM

COMM

INSTALLD IR

INSTAL

L DI R CC

CC

F

F

CLINKER

CLINKE

R FLINKER

FLINKE

R OPTFLAGS

OPTFLA

GS

LIBPATH
LINS

TA LL D IR

li

b

AR

CH

C

OM M FLIBPATH

FLIBPAT

H LE AD ER

I

NS TA LL D IR

li

b

ARC

H

C

OM M LIBLIST

LIBLI

ST

INCLUDED

IR

INCLUD

E PA TH

IINST

AL LD IR

in

cl ud e

End

User configur ab le

ptions

SLIDE 29 Sample Mak elein cont CFLAGS

CFLAGS
OPTFLA

GS

INCLUD

ED IR

DMPI

AR CH

FFLAGS
FFLAGS

INCLU DE D IR

OPTFLAG

S LIBS

LIBPA

TH

LIBLI

ST

FLIBS
FLIB

PA TH

LIBLI

ST

EXECS
hello

default hello all EXECS hello helloo INSTAL L DI R i nc lu de m pi h CLINK ER

OPTFLA

GS

hello

helloo

LIBP

AT H LIBL IS T lm clean binrm f

PI

EXECS

co

CC CFLAG S c c fo F FFLAGS

c

f

SLIDE 30 Running MPI p rograms mpirun np

hello

mpirun is not pa rt

f

the standa rd but some version

f

it is common with several MPI implementations The version sho wn here is fo r the MPICH implementation

f

MPI

Just

as F

rtran

do es not sp ecify ho w F

rtran

p rograms a re sta rted MPI do es not sp ecify ho w MPI p rograms a re sta rted

The
ption

t sho ws the commands that mpirun w

uld

execute y

u

can use this to nd

ut

ho w mpirun sta rts p rograms

n

y

r

system The

ption

help sho ws all

ptions

to mpirun

SLIDE 31 Finding

ut

ab

ut

the environment Tw

f

the rst questions ask ed in a pa rallel p rogram a re Ho w many p ro cesses a re there and Who am I Ho w many is answ ered with MPICommsize and who am I is answ ered with MPICommrank The rank is a numb er b et w een zero and size

SLIDE 32 A simple p rogram include mpih include stdioh int main argc argv

int

argc char argv

int

rank size MPIInit

argc

argv

MPIComm

r ank

MPICOMMW

ORL D rank

MPIComm

s ize

MPICOMMW

ORL D size

printf

Hello world Im d

f

dn rank size

MPIFina

li ze

return

SLIDE 33 Caveats

These

sample p rograms have b een k ept as simple as p

ssible

b y assuming that all p ro cesses can do

utput

Not all pa rallel systems p rovide this feature and MPI p rovides a w a y to handle this case

SLIDE 34 Exercise

Getting

Sta rted Objective Lea rn ho w to login write compile and run a simple MPI p rogram Run the Hello w

rld

p rograms T ry t w

dierent

pa rallel computers What do es the

utput

lo

k

lik e

SLIDE 35 Sending and Receiving messages

Process 0 Process 1 A: B: Send Recv

Questions

T
whom

is data sent

What

is sent

Ho

w do es the receiver identify it

SLIDE 36 Current MessageP assing

A

t ypical blo cking send lo

ks

lik e send dest type address length

where
dest

is an integer identier rep resenting the p ro cess to receive the message

type

is a nonnegative integer that the destination can use to selectively screen messages

address

length describ es a contiguous a rea in memo ry containing the message to b e sent and

A

t ypical global

p

eration lo

ks

lik e broadcas t type address length

All
f

these sp ecications a re a go

d

match to ha rdw a re easy to understand but to

inexible

SLIDE 37 The Buer Sending and receiving

nly

a contiguous a rra y

f

b ytes

hides

the real data structure from ha rdw a re which might b e able to handle it directly

requires

p repacking disp ersed data

ro

ws

f

a matrix sto red columnwise

general

collections

f

structures

p

revents communicati

ns

b et w een machines with dierent rep resentations even lengths fo r same data t yp e

SLIDE 38 Generalizing the Buer Description

Sp

ecied in MPI b y sta rting address

datat

yp e

and

count

where

datat yp e is

elementa

ry all C and F

rtran

datat yp es

contiguous

a rra y

f

datat yp es

strided

blo cks

f

datat yp es

indexed

a rra y

f

blo cks

f

datat yp es

general

structure

Datat

yp es a re constructed recursively

Sp

ecications

f

elementa ry datat yp es allo ws heterogeneous communication

Elim

i nati

n
f

length in favo r

f

count is clea rer

Sp

ecifying applicationo r i ented la y

ut
f

data allo ws maximal use

f

sp ecial ha rdw a re

SLIDE 39 Generalizing the T yp e

A

single t yp e eld is to

constraining

Often

verloaded

to p rovide needed exibilit y

Problems
under

user control

wild

ca rds allo w ed MPIANYT AG

lib

ra ry use conicts with user and with

ther

lib ra ri es

SLIDE 40 Sample Program using Lib ra ry Calls Sub and Sub a re from dierent lib ra ri es Sub Sub Suba and Subb a re from the same lib ra r y Suba Sub Subb Thanks to Ma rc Snir fo r the follo wing four examples

SLIDE 41 Co rrect Execution

f

Lib ra ry Calls

Process 0 Process 1 Process 2 recv(any) send(1) recv(any) send(0) recv(1) send(0) recv(2) send(1) send(2) recv(0) Sub1 Sub2

SLIDE 42 Inco rrect Execution

f

Lib ra ry Calls

Process 0 Process 1 Process 2 recv(any) send(1) recv(any) send(0) recv(1) send(0) recv(2) send(1) send(2) recv(0) Sub1 Sub2

SLIDE 43 Co rrect Execution

f

Lib ra ry Calls with P ending Communcication

Process 0 Process 1 Process 2 recv(any) send(1) send(0) send(0) recv(0) recv(any) send(1) send(2) recv(1) recv(2) Sub1a Sub2 Sub1b

SLIDE 44 Inco rrect Execution

f

Lib ra ry Calls with P ending Communication

Process 0 Process 1 Process 2 recv(any) send(1) send(0) send(0) recv(0) recv(any) send(1) send(2) recv(1) recv(2) Sub1a Sub2 Sub1b

SLIDE 45 Solution to the t yp e p roblem

A

sepa rate communicati

n

context fo r each family

f

messages used fo r queueing and matching This has

ften

b een simulated in the past b y

verloading

the tag eld

No

wild ca rds allo w ed fo r securit y

Allo

cated b y the system fo r securit y

T

yp es tags

in

MPI retained fo r no rmal use wild ca rds OK

SLIDE 46 Delimiting Scop e

f

Communication

Sepa

rate groups

f

p ro cesses w

rking
n

subp roblems

Merging
f

p ro cess name space interferes with mo dula ri t y

Lo

cal p ro cess identiers desirable

P

a rall el invo cation

f

pa rallel lib ra ri es

Messages

from application must b e k ept sepa rate from messages internal to lib ra r y

Kno

wledge

f

lib ra ry message t yp es interferes with mo dula rit y

Synchronizing

b efo re and after lib ra ry calls is undesirable

SLIDE 47 Generalizing the Pro cess Identier

Collective
p

erations t ypically

p

erated

n

all p ro cesses although some systems p rovide subgroups

This

is to

restrictive

eg need minimum

ver

a column

r

a sum across a ro w

f

p ro cesses

MPI

p rovides groups

f

p ro cesses

initial

all group

group

management routines build delete groups

All

communication not just collective

p

erations tak es place in groups

A

group and a context a re combined in a communicato r

Sourcedestination

in sendreceive

p

erations refer to rank in group asso ciated with a given communicato r

MPIANY

SO URC E p ermitted in a receive

SLIDE 48 MPI Basic SendReceive Thus the basic blo cking send has b ecome MPISend start count datatype

dest

tag comm

and

the receive MPIRecvs ta rt count datatype source tag comm status The source tag and count

f

the message actually received can b e retrieved from status Tw

simple

collective

p

erations MPIBcast st art

count

datatype

root

comm MPIReduce s tar t result count datatype

peration
root

comm

SLIDE 49 Getting info rmation ab

ut

a message MPIStat us status MPIRecv

status
statusMP

I TAG

statusMP

I SOU RC E MPIGet co unt

status

datatype

count
MPITAG

and MPISOURC E p rima ri l y

f

use when MPIANY TA G ando r MPIANY SOU RC E in the receive MPIGet co unt ma y b e used to determine ho w much data

f

a pa rticula r t yp e w as received

SLIDE 50 Simple F

rtran

example program main include mpifh

integer

rank size to from tag count i ierr integer src dest integer stsour ce

sttag

stcount integer status MP I ST AT US S IZ E double precisio n data

call

MPIINIT

ierr
call

MPICOMM R AN K MPICOM M WO RL D rank ierr

call

MPICOMM S IZ E MPICOM M WO RL D size ierr

print
Process
rank
f
size
is

alive dest

size
src
C

if rank eq src then to

dest

count

tag
do
i
datai
i

call MPISEN D data count MPIDOUBL E PR EC IS IO N to

tag

MPICOM M WOR LD

ierr
else

if rank eq dest then tag

MPIANY

TA G count

from
MPIANY

SO UR CE call MPIREC V da ta

count

MPIDOUB LE P RE CI SI ON

from
tag

MPICOMM W ORL D status ierr

SLIDE 51 Simple F

rtran

example cont call MPIGET C OU NT

status

MPIDOUBL E PR EC IS IO N

stcount
ierr
stsourc

e

statusM

PI S OU RC E sttag

statusM

PI T AG

C

print

Status

info source

stsourc

e

tag
sttag
count
stcoun

t print

rank
receive

d

datai

i

endif

call MPIFINA LI ZE

ierr
end

SLIDE 52 Six F unction MPI MPI is very simple These six functions allo w y

u

to write many p rograms MPI Init MPI Finalize MPI Comm size MPI Comm rank MPI Send MPI Recv

SLIDE 53 A taste

f

things to come The follo wing examples sho w a C and F

rtran

version

f

the same p rogram This p rogram computes PI with a very simple metho d but do es not use MPISend and MPIRecv Instead it uses collective

p

erations to send data to and from all

f

the running p ro cesses This gives a dierent sixfunction MPI set MPI Init MPI Finalize MPI Comm size MPI Comm rank MPI Bcast MPI Reduce

SLIDE 54 Broadcast and Reduction The routine MPIBcast sends data from

ne

p ro cess to all

thers

The routine MPIReduce combines data from all p ro cesses b y adding them in this case and returning the result to a single p ro cess

SLIDE 55 F

rtran

example PI program main include mpifh

double

precisio n PIDT paramet er PIDT

d
double

precisio n mypi pi h sum x f a integer n myid numprocs

i

rc c function to integrat e fa

d
d
aa

call MPIINIT

ierr
call

MPICOMM R AN K MPICOM M WO RL D myid ierr

call

MPICOMM S IZ E MPICOM M WO RL D numprocs

ierr
if
myid

eq

then

write

format

En te r the number

f

intervals

quits
read
n
formati
endif

call MPIBCAS T n

MP

I IN TE GE R

MPI

C OM M WO RL D ie rr

SLIDE 56 F

rtran

example cont c check for quit signal if

n

le

goto
c

calculate the interva l size h

dn

sum

d

do

i
myid

n numproc s x

h
dblei
d

sum

sum
fx
continue

mypi

h
sum

c collect all the partial sums call MPIRED UC E my pi p i

MP

I DO UB LE PR EC IS IO N MP I SU M

MPICOM

M WO RL D ie rr

c

node

prints

the answer if myid eq

then

write

pi

abspi

PIDT
format
pi

is approxi ma te ly

F
Error

is

F

endif goto

call

MPIFIN AL IZ E rc

stop

end

SLIDE 57 C example PI include mpih include mathh int mainargc a rg v int argc char argv ! " int done

n

myid numprocs

i

rc double PIDT

double

mypi pi h sum x a MPIInit #a rg c #a rg v

MPIComm

si ze M PI C OM M WO RL D #n um pr

c

s MPIComm ra nk M PI C OM M WO RL D #m yi d

SLIDE 58 C example cont while $done " if myid

"

printf E nt er the number

f

interval s

quits
scanf

%d

#n
&

MPIBcast # n

MPIINT
MPICOMM

WO RL D

if

n

break

h

double
n

sum

for

i

myid
i
n

i

numprocs
"

x

h
doubl

e i

sum
xx

& mypi

h
sum

MPIReduc e #m yp i #pi

MPIDOU

BL E MPISUM

MPICOM

M WO RL D

if

myid

printf

p i is approxi ma te ly %f Error is %fn

pi

fabspi

PIDT
&

MPIFinal iz e

&

SLIDE 59 Exercise

PI

Objective Exp eriment with sendreceive Run either p rogram fo r PI W rite new versions that replace the calls to MPIBcast and MPIReduce with MPISend and MPIRecv

The

MPI b roadcast and reduce

p

erations use at most log p send and receive

p

erations

n

each p ro cess where p is the size

f

MPI COMM WORLD Ho w many

p

erations do y

ur

versions use

SLIDE 60 Exercise

Ring

Objective Exp eriment with sendreceive W rite a p rogram to send a message a round a ring

f

p ro cesso rs That is p ro cesso r

sends

to p ro cesso r

who

sends to p ro cesso r

etc

The last p ro cesso r returns the message to p ro cesso r

Y
u

can use the routine MPI Wtime to time co de in MPI The statement t

MPI

Wtime returns the time as a double DOUBLE PRECISION in F

rtran

SLIDE 61 T

p
logies

MPI p rovides routines to p rovide structure to collections

f

p ro cesses This helps to answ er the question Who a re my neighb

rs

SLIDE 62 Ca rtesian T

p
logies

A Ca rtesian top

logy

is a mesh Example

f
Ca

rtesian mesh with a rro ws p

inting

at the right neighb

rs

(0,0) (1,0) (2,0) (3,0) (0,1) (1,1) (2,1) (3,1) (0,2) (1,2) (2,2) (3,2)

SLIDE 63 Dening a Ca rtesian T

p
logy

The routine MPICart cre at e creates a Ca rtesian decomp

sition
f

the p ro cesses with the numb er

f

dimensions given b y the ndim a rgument dims

dims
periods
false

periods

false

reorder

true

ndim

call

MPICARTC RE ATE

MPICOMMW

OR LD ndim dims

periods

reorder commd ierr

SLIDE 64 Finding neighb

rs

MPICart c rea te creates a new communicato r with the same p ro cesses as the input communicato r but with the sp ecied top

logy
The

question Who a re my neighb

rs

can no w b e answ ered with MPICart sh if t call MPICART S HIF T commd

nbrleft

nbrright ierr

call

MPICART S HIF T commd

nbrbottom
nbrtop

ierr

The

values returned a re the ranks in the communicato r commd

f

the neighb

rs

shifted b y

in

the t w

dimensions

SLIDE 65 Who am I Can b e answ ered with integer coords call MPICOMM RA NK

commd

myrank ierr

call

MPICART CO OR DS commd myrank

coords

ierr

Returns

the Ca rtesian co

rdinates
f

the calling p ro cess in coords

SLIDE 66 P a rtitioning When creating a Ca rtesian top

logy
ne

question is What is a go

d

choice fo r the decomp

sition
f

the p ro cesso rs This question can b e answ ered with MPIDims cr ea te

integer

dims dims

dims
call

MPICOMM SI ZE

MPICOMM

W ORL D size ierr

call

MPIDIMS CR EA TE size

dims

ierr

SLIDE 67 Other T

p
logy

Routines MPI contains routines to translate b et w een Ca rtesian co

rdinates

and ranks in a communicato r and to access the p rop erties

f

a Ca rtesian top

logy
The

routine MPIGraphcreate allo ws the creation

f

a general graph top

logy

SLIDE 68 Why a re these routines in MPI In many pa rallel computer interconnects some p ro cesso rs a re closer to than

thers

These routines allo w the MPI implementation to p rovide an

rdering
f

p ro cesses in a top

logy

that mak es logical neighb

rs

close in the physical interconnect

Some

pa rallel p rogrammers ma y rememb er hyp ercub es and the eo rt that w ent into assigning no des in a mesh to p ro cesso rs in a hyp ercub e through the use

f

Grey co des Many new systems have dierent interconnects

nes

with multiple paths ma y have notions

f

nea r neighb

rs

that changes with time These routines free the p rogrammer from many

f

these considerations The reorder a rgument is used to request the b est

rdering

SLIDE 69 The p erio ds a rgument Who a re my neighb

rs

if I am at the edge

f

a Ca rtesian Mesh

?

SLIDE 70 P erio dic Grids Sp ecify this in MPICart cr ea te with dims

dims
periods
TRUE

periods

TRUE

reorder

true

ndim

call

MPICARTC RE ATE

MPICOMMW

OR LD ndim dims

periods

reorder commd ierr

SLIDE 71 Nonp erio dic Grids In the nonp erio dic case a neighb

r

ma y not exist This is indicated b y a rank

f

MPIPROCNULL This rank ma y b e used in send and receive calls in MPI The action in b

th

cases is as if the call w as not made

SLIDE 72 Collective Communications in MPI

Communicati
n

is co

rdinated

among a group

f

p ro cesses

Groups

can b e constructed b y hand with MPI groupmanipul ati

n

routines

r

b y using MPI top

logydenition

routines

Message

tags a re not used Dierent communicato r s a re used instead

No

nonblo cking collective

p

erations

Three

classes

f

collective

p

erations

synchronization
data

movement

collective

computation

SLIDE 73 Synchronization

MPIBarriercomm
F

unction blo cks untill all p ro cesses in comm call it

SLIDE 74 Available Collective P atterns

P0 P1 P2 P3 P0 P1 P2 P3 P0 P1 P2 P3 P0 P1 P2 P3 P0 P1 P2 P3 P0 P1 P2 P3 P0 P1 P2 P3 P0 P1 P2 P3 A A A A A A B C D A B C D A B C D A B C D A B C D A B C D A B C D A0 A1 A2 A3 B0 B1 B2 B3 C0 C1 C2 C3 D0 D1 D2 D3 A0 B0 C0 D0 A1 B1 C1 D1 A2 B2 C2 D2 A3 B3 C3 D3 All to All All gather Scatter Gather Broadcast

Schematic rep resentation

f

collective data movement in MPI

SLIDE 75 Available Collective Computation P atterns

ABC ABCD AB A ABCD Reduce Scan P3 P3 P0 P1 P2 P0 P1 P2 A B C D P3 A B C D P3 P0 P1 P2 P0 P1 P2

Schematic rep resentation

f

collective data movement in MPI

SLIDE 76 MPI Collective Routines

Many

routines Allgather Allgatherv Allreduce Alltoall Alltoallv Bcast Gather Gatherv Reduce ReduceScatter Scan Scatter Scatterv

All

versions deliver results to all pa rticipating p ro cesses

V

versions allo w the chunks to have dierent sizes

Allreduce

Reduce ReduceScatter and Scan tak e b

th

builtin and userdened combination functions

SLIDE 77 Builtin Collective Computation Op erations MPI Name Op eration MPI MAX Maximum MPI MIN Minimum MPI PROD Pro duct MPI SUM Sum MPI LAND Logical and MPI LOR Logical

r

MPI LXOR Logical exclusive

r

xo r MPI BAND Bit wise and MPI BOR Bit wise

r

MPI BXOR Bit wise xo r MPI MAXLOC Maximum value and lo cation MPI MINLOC Minimum value and lo cation

SLIDE 78 Dening Y

ur

Own Collective Op erations MPIOpc re ate u ser f unc ti

n

commute

p

MPIOpf re eo p userfun ct ion i nve c inoutvec len datatype The user function should p erfo rm inoutvec i

inveci
p

inoutvec i

fo

r i from

to

len userfun ct ion can b e noncommutative eg matrix multipl y

SLIDE 79 Sample user function F

r

example to create an

p

eration that has the same eect as MPISUM

n

F

rtran

double p recision values use subrouti ne myfunc invec inoutvec len datatype

integer

len datatype double precision inveclen

inoutvecl

en integer i do

ilen
inoutvec

i

inveci

inoutveci

return

end T

use

just integer myop call MPIOpc re ate

myfunc

true myop ierr

call

MPIRedu ce

a

b

MPIDOUBL

E PRE CI SON

myop
The

routine MPIOpf ree destro ys userfunctions when they a re no longer needed

SLIDE 80 Dening groups All MPI communication is relative to a communicato r which contains a context and a group The group is just a set

f

p ro cesses

SLIDE 81 Sub dividin g a communicato r The easiest w a y to create communicato rs with new groups is with MPICOMMS PL IT F

r

example to fo rm groups

f

ro ws

f

p ro cesses

1 2 1 2 3 4 Column Row

use MPIComm s pli t

ldcomm

row

newcomm
T
maintain

the

rder

b y rank use MPIComm r ank

ldcomm

rank

MPIComm

s pli t

ldcomm

row rank newcomm

SLIDE 82 Sub dividin g cont Simil a rl y

to

fo rm groups

f

columns

1 2 1 2 3 4 Column Row

use MPIComm s pli t

ldcomm

column

newcomm
T
maintain

the

rder

b y rank use MPIComm r ank

ldcomm

rank

MPIComm

s pli t

ldcomm

column rank newcomm

SLIDE 83 Manipulating Groups Another w a y to create a communicato r with sp ecic memb ers is to use MPIComm c rea te

MPIComm

c rea te

ldcomm

group newcomm

The

group can b e created in many w a ys

SLIDE 84 Creating Groups All group creation routines create a group b y sp ecifying the memb ers to tak e from an existing group

MPIGroup

in cl sp ecies sp ecic memb ers

MPIGroup

ex cl excludes sp ecic memb ers

MPIGroup

ra nge i ncl and MPIGroup ra nge e xc l use ranges

f

memb ers

MPIGroup

un ion and MPIGroup in ter se cti

n

creates a new group from t w

existing

groups T

get

an existing group use MPIComm g rou p

ldcomm

group

F

ree a group with MPIGrou p fre e group

SLIDE 85 Buering issues Where do es data go when y

u

send it One p

ssibilit

y is

Local Buffer Local Buffer A: B: Process 1 Process 2 The Network

SLIDE 86 Better buering This is not very ecient There a re three copies in addition to the exchange

f

data b et w een p ro cesses W e p refer

B: A: Process 1 Process 2

But this requires that either that MPISend not return until the data has b een delivered

r

that w e allo w a send

p

eration to return b efo re completing the transfer In this case w e need to test fo r completion later

SLIDE 87 Blo cking and NonBlo cking communication

So

fa r w e have used blocking communication

MPI

Send do es not complete until buer is empt y available fo r reuse

MPI

Recv do es not complete until buer is full available fo r use

Simple

but can b e unsafe Pro cess

Pro

cess

Send

Send Recv Recv Completion dep ends in general

n

size

f

message and amount

f

system buering

Send

w

rks

fo r small enough messages but fails when messages get to

la

rge T

la

rge ranges from zero b ytes to s

f

Megab ytes

SLIDE 88 Some Solutions to the Unsafe Problem

Order

the

p

erations mo re ca refully Pro cess

Pro

cess

Send

Recv Recv Send

Supply

receive buer at same time as send with MPI Sendrecv

Pro

cess

Pro

cess

Sendrecv

Sendrecv

Use

nonblo cking

p

erations Pro cess

Pro

cess

Isend

Isend Irecv Irecv W aitall W aitall

Use

MPIBsend

SLIDE 89 MPIs NonBlo cking Op erations Nonblo cking

p

erations return immediatel y request handles that can b e w aited

n

and queried

MPI

Isendst art

count

datatype dest tag comm request

MPI

Irecvst art

count

datatype dest tag comm request

MPI

Waitreq ues t status One can also test without w aiting MPITest request flag status

SLIDE 90 Multiple completions It is

ften

desirable to w ait

n

multiple requests An example is a masterslave p rogram where the master w aits fo r

ne
r

mo re slaves to send it a message

MPI

Waitall cou nt

array
f

requests array

f

statuses

MPI

Waitany cou nt

array
f

requests index status

MPI

Waitsome in co unt

array
f

requests

utcount

array

f

indices array

f

statuses

There

a re co rresp

nding

versions

f

test fo r each

f

these

The

MPI WAITSOME and MPI TESTSOME ma y b e used to implem ent masterslave algo rithms that p rovide fair access to the master b y the slaves

SLIDE 91 F airness What happ ens with this p rogram include mpih include stdioh

int

mainargc

argv

int argc char argv " int rank size i buf ! MPIStatu s status MPIInit #argc #argv

MPIComm

ra nk

MPICOMM

W OR LD

#rank
MPIComm

si ze

MPICOMM

W OR LD

#size
if

rank

"

for i i si ze

i

" MPIRec v buf

MPIINT
MPIANYS

OU RC E MPIANY TA G MPICOM M WOR LD

#status
printf

Msg from %d with tag %dn status MP I SO UR CE

statusMP

I TA G

&

& else " for i i i MPISen d buf

MPIINT
i

MPICOM M WO RL D

&

MPIFinal iz e

return
&

SLIDE 92 F airness in messagepassing An pa rallel algo rithm is fair if no p ro cess is eectively igno red In the p receeding p rogram p ro cesses with lo w rank lik e p ro cess zero ma y b e the

nly
ne

whose messages a re received MPI mak es no gua rentees ab

ut

fairness Ho w ever MPI mak es it p

ssible

to write ecient fair p rograms

SLIDE 93 Providing F airness One alternative is define large

MPIReque

st request s la rg e!

MPIStatu

s statuse s la rg e!

int

indices l ar ge ! int buf lar ge ! for i isize i MPIIrecv

bufi
MPIINT

i MPIANY TA G MPICOM M WO RLD

#request

s i !

whilenot

done " MPIWaits

m

e size request s #ndone indices statuse s

for

i indone

i

" j

indices

i ! printf Msg from %d with tag %dn statuse s i! M PI S OU RC E statuse s i! M PI T AG

MPIIre

cv

bufj
MPIINT

j MPIANY TA G MPICOM MW OR LD

#request

s j!

&

&

SLIDE 94 Providing F airness F

rtran

One alternative is paramete r large

integer

requests l ar ge

integer

statuses M PI S TA TU S SI ZE l ar ge

integer

indices la rg e

integer

buflarg e

logical

done do

i
size
call

MPIIre cv

bufi
MPIINT

EG ER i

MPIANY

TA G MPICOM M WO RLD

requests

i

ierr
if

not done then call MPIWait so me

size

requests

ndone

indices statuse s ierr

do
i

ndone j

indices

i

print
Msg

from

statuse

s MPI S OU RC E i

with

tag

statuses

M PI T AG i

call

MPIIrec v bufj

MPIINTEG

ER

j

MPIANY TA G MPICOM MW OR LD

requests

j

ierr
done
continue

goto

endif

SLIDE 95 Exercise

F

airness Objective Use nonblo cking communications Complete the p rogram fragment

n

p roviding fairness Mak e sure that y

u

leave no uncompleted requests Ho w w

uld

y

u

test y

ur

p rogram

SLIDE 96 Mo re

n

nonblo cking communication In applications where the time to send data b et w een p ro cesses is la rge it is

ften

helpful to cause communicati

n

and computation to

verlap

This can easily b e done with MPIs nonblo cking routines F

r

example in a D nite dierence mesh moving data needed fo r the b

unda

ries can b e done at the same time as computation

n

the interio r MPIIrec v

each

ghost edge

MPIIsen

d

data

for each ghost edge

compute
n

interior while still some uncomplete d requests

MPIWaita

ny

requests
if

request is a receive

compute
n

that edge

Note

that w e call MPIWaita ny several times This exploits the fact that after a request is satised it is set to MPIREQU ES TN UL L and that this is a valid request

bject

to the w ait and test routines

SLIDE 97 Communication Mo des MPI p rovides mulitple mo des fo r sending messages

Synchronous

mo de MPI Ssend the send do es not complete until a matching receive has b egun Unsafe p rograms b ecome inco rrect and usually deadlo ck within an MPISsen d

Buered

mo de MPI Bsend the user supplies the buer to system fo r its use User supplies enough memo ry to mak e unsafe p rogram safe

Ready

mo de MPI Rsend user gua rantees that matching receive has b een p

sted
allo

ws access to fast p roto cols

undened

b ehavio r if the matching receive is not p

sted

Nonblo cking versions MPI Issend MPI Irsend MPI Ibsend Note that an MPIRecv ma y receive messages sent with any send mo de

SLIDE 98 Buered Send MPI p rovides a send routine that ma y b e used when MPIIsen d is a wkw a rd to use eg lots

f

small messages MPIBsen d mak es use

f

a userp rovided buer to save any messages that can not b e immedi atel y sent int bufsize char buf

mallocb

ufs iz e MPIBuff er at ta ch buf bufsize

MPIBsen

d

same

as MPISend

MPIBuff

er de ta ch buf bufsize

The

MPIBuffe r det ac h call do es not complete until all messages a re sent

The

p erfo rmance

f

MPI Bsend dep ends

n

the implem entati

n
f

MPI and ma y also dep end

n

the size

f

the message F

r

example making a message

ne

b yte longer ma y cause a signicant drop in p erfo rmance

SLIDE 99 Reusing the same buer Consider a lo

p

MPIBuff er at ta ch buf bufsize

while

done

MPIBsend
where

the buf is la rge enough to hold the message in the MPIBsen d This co de ma y fail b ecause the

void

buf int bufsize MPIBuff er de ta ch buf bufsize

MPIBuff

er at ta ch buf bufsize

SLIDE 100 Other P

inttoP
int

F eatures

MPISENDRECV

MPISENDRECVREPLACE

MPICANCEL
P

ersistent communication requests

SLIDE 101 Datat yp es and Heterogenit y MPI datat yp es have t w

main

purp

ses
Heterogenit

y

pa

rallel p rograms b et w een dierent p ro cesso rs

Noncontiguous

data

structures

vecto rs with nonunit stride etc Basic datat yp e co rresp

nding

to the underlying language a re p redened The user can construct new datat yp es at run time these a re called derived datat yp es

SLIDE 102 Datat yp es in MPI Elementa ry Languagedened t yp es eg MPIINT

r

MPIDOUBLEPRECISION

V

ecto r Sepa rated b y constant stride Contiguous V ecto r with stride

f
ne

Hvecto r V ecto r with stride in b ytes Indexed Arra y

f

indices fo r scattergather Hindexed Indexed with indices in b ytes Struct General mixed t yp es fo r C structs etc

SLIDE 103 Basic Datat yp es F

rtran

MPI datat yp e F

rtran

datat yp e MPIINTEGER INTEGER MPIREAL REAL MPIDOUBLEPRECISION DOUBLE PRECISION MPICOMPLEX COMPLEX MPILOGICAL LOGICAL MPICHARACTER CHARACTER MPIBYTE MPIPACKED

SLIDE 104 Basic Datat yp es C MPI datat yp e C datat yp e MPICHAR signed char MPISHORT signed short int MPIINT signed int MPILONG signed long int MPIUNSIGNEDCHAR unsigned char MPIUNSIGNEDSHORT unsigned short int MPIUNSIGNED unsigned int MPIUNSIGNEDLONG unsigned long int MPIFLOAT float MPIDOUBLE double MPILONGDOUBLE long double MPIBYTE MPIPACKED

SLIDE 105 V ecto rs

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35

T

sp

ecify this ro w in C

rder

w e can use MPIType v ect

r
count

blocklen

stride
ldtype

newtype

MPIType

c

mm

it

newtype
The

exact co de fo r this is MPIType v ect

r
!
MPIDOUBL

E newtype

MPIType

c

mm

it

newtype

SLIDE 106 Structures Structures a re describ ed b y a rra ys

f
numb

er

f

elements arrayofle n

displacement
r

lo cation arrayofdi spl s

datat

yp e arrayofty pe s MPIType s tru ct ure

count

arrayof l en arrayof d isp ls

arrayof

t ype s newtype

SLIDE 107 Example Structures struct " char display

!
Name
f

display

int

maxiter

max
f

iteratio ns

double

xmin ymin

lower

left corner

f

rectangl e

double

xmax ymax

upper

right corner

int

width

f

display in pixels

int

height

f

display in pixels

&

cmdline

set

up

blocks
int

blockcou nt s !

"
&

MPIDatat yp e types !

MPIAint

displs ! MPIDatat yp e cmdtype

initiali

ze types and displs with addresse s

f

items

MPIAddre

ss

#cmdline

d is pl ay

#displs

!

MPIAddre

ss

#cmdline

m ax it er

#displs

!

MPIAddre

ss

#cmdline

x mi n #displs !

MPIAddre

ss

#cmdline

w id th

#displs

!

types !
MPICHAR
types !
MPIINT

types !

MPIDOUB

LE

types !
MPIINT

for i

i
i

displs i!

displs

! MPIType st ru ct

blockco

un ts

displs

types #cmdtype

MPIType

co mm it

#cmdtype

SLIDE 108 Strides The extent

f

a datat yp e is no rmally the distance b et w een the rst and last memb er

LB UB EXTENT Memory locations specified by datatype

Y

u

can set an a rticial extent b y using MPIUB and MPILB in MPITypestruct

SLIDE 109 V ecto rs revisited This co de creates a datat yp e fo r an a rbitra ry numb er

f

element in a ro w

f

an a rra y sto red in F

rtran
rder

column rst int blens displs

MPIData

ty pe types rowtype blens

blens
displs
displs
numberi

nc

l

umn

sizeofdo

ub le

types
MPIDOUB

LE types

MPIUB

MPIType s tru ct

blens

displs types rowtype

MPIType

c

mm

it

rowtype
T
send

n elements y

u

can use MPISend

buf

n rowtype

SLIDE 110 Structures revisited When sending an a rra y

f

a structure it is imp

rtant

to ensure that MPI and the C compiler have the same value fo r the size

f

each structure The most p

rtable

w a y to do this is to add an MPIUB to the structure denition fo r the end

f

the structure In the p revious example this is

initiali

ze types and displs with addresse s

f

items

MPIAddre

ss

#cmdline

d is pl ay

#displs

!

MPIAddre

ss

#cmdline

m ax it er

#displs

!

MPIAddre

ss

#cmdline

x mi n #displs !

MPIAddre

ss

#cmdline

w id th

#displs

!

MPIAddre

ss

#cmdline
#displs

!

types !
MPICHAR
types !
MPIINT

types !

MPIDOUB

LE

types !
MPIINT

types !

MPIUB

for i

i
i

displs i!

displs

! MPIType st ru ct

blockco

un ts

displs

types #cmdtype

MPIType

co mm it

#cmdtype

SLIDE 111 Interleaving data By moving the UB inside the data y

u

can interleave data Consider the matrix

1 2 3 4 5 6 7 8 9 10 11 12 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 13

W e wish to send

and
to

p ro cess

and
to

p ro cess

etc

Ho w can w e do this with MPIScatterv

SLIDE 112 An interleaved datat yp e MPIType v ect

r
"

MPIDOUBL E vec

denes

a blo ck

f

this matrix blens

blens
types
vec

types

MPIUB

displs

displs
sizeofd
ub

le

MPIType

s tru ct

blens

displs types block

denes

a blo ck whose extent is just

entries

SLIDE 113 Scattering a Matrix W e set the displacements fo r each blo ck as the lo cation

f

the rst element in the blo ck This w

rks

b ecause MPIScat te rv uses the extents to determine the sta rt

f

each piece to send scdispls

scdispls
scdispls
scdispls
#

MPIScat te rv sendbuf sendcount s scdispls block recvbuf nx

ny

MPIDOUBLE

MPICOMMW

ORL D

Ho

w w

uld

use use the top

logy

routines to mak e this mo re general

SLIDE 114 Exercises

datat

yp es Objective Lea rn ab

ut

datat yp es

W

rite a p rogram to send ro ws

f

a matrix sto red in columnmajo r fo rm to the

ther

p ro cesso rs Let p ro cesso r

have

the entire matrix which has as many ro ws as p ro cesso rs Pro cesso r

sends

ro w i to p ro cesso r i Pro cesso r i reads that ro w into a lo cal a rra y that holds

nly

that ro w That is p ro cesso r

has

a matrix AN

M
while

the

ther

p ro cesso rs have a ro w B M

a

W rite the p rogram to handle the case where the matrix is squa re b W rite the p rogram to handle a numb er

f

columns read from the terminal C p rogrammers ma y send columns

f

a matrix sto red in ro wmajo r fo rm if they p refer If y

u

have time try

ne
f

the follo wing If y

u

dont have time think ab

ut

ho w y

u

w

uld

p rogram these

W

rite a p rogram to transp

se

a matrix where each p ro cesso r has a pa rt

f

the matrix Use top

logies

to dene a Dimensional pa rtitioning

SLIDE 115

f

the matrix across the p ro cesso rs and assume that all p ro cesso rs have the same size submatrix a Use MPISend and MPIRecv to send the blo ck the transp

se

the blo ck b Use MPISend re cv instead c Create a datat yp e that allo ws y

u

to receive the blo ck already transp

sed
W

rite a p rogram to send the ghostp

ints
f

a Dimensional mesh to the neighb

ring

p ro cesso rs Assume that each p ro cesso r has the same size subblo ck a Use top

logies

to nd the neighb

rs

b Dene a datat yp e fo r the ro ws c Use MPISend re cv

r

MPIIRecv and MPISend with MPIWaital l d Use MPIIsen d and MPIIrecv to sta rt the communicati

n

do some computation

n

the interio r and then use MPIWaita ny to p ro cess the b

unda

ries as they a rrive The same app roach w

rks

fo r general datastructures such as unstructured meshes

Do
but

fo r Dimensional meshes Y

u

will need MPIType Hv ec to r

SLIDE 116 T

ls

fo r writing lib ra ries MPI is sp ecically designed to mak e it easier to write messagepassing lib ra ri es

Communicato

rs solve tagsource wildca rd p roblem

A

ttributes p rovide a w a y to attach info rmation to a communicato r

SLIDE 117 Private communicato rs One

f

the rst thing that a lib ra ry should no rmally do is create p rivate communicato r This allo ws the lib ra ry to send and receive messages that a re kno wn

nly

to the lib ra ry

MPICommdup
ldcomm

newcomm

SLIDE 118 A ttributes A ttributes a re data that can b e attached to

ne
r

mo re communicato rs A ttributes a re referenced b y k eyval Keyvals a re created with MPIKEYVALCREATE A ttributes a re attached to a communicato r with MPIAttrput and their values accessed b y MPIAttrget

Op

erations a re dened fo r what happ ens to an attribute when it is copied b y creating

ne

communicato r from another

r

deleted b y deleting a communicato r when the k eyval is created

SLIDE 119 What is an attribute In C an attribute is a p

inter
f

t yp e void

Y
u

must allo cate sto rage fo r the attribute to p

int

to mak e sure that y

u

dont use the address

f

a lo cal va riable In F

rtran

it is a single INTEGER

SLIDE 120 Examples

f

using attributes

F
rcing

sequential

p

eration

Managing

MPI_Send PMPI_Send MPI_Bcast MPI_Send PMPI_Send MPI_Send MPI_Bcast User Program MPI Library Profile Library

static int nsend

int

MPISend start count datatyp e dest tag comm

"

nsend return PMPISen d start count datatyp e dest tag comm

&

SLIDE 142 W riting p roling routines The MPICH implem entati

n

contains a p rogram fo r writing wrapp ers This description will write

ut

each MPI routine that is called ifdef MPIBUIL DP RO FIL IN G undef MPIBUIL DP RO FIL IN G endif include stdioh include mpih fnall fnname vardec l int llrank PMPICom mr an k MPICOMMW OR LD

llrank
printf

d Starting fnname

n
llrank
fflush

stdout

callfn
printf

d Ending fnname

n
llrank
fflush

stdout

endfna

ll

The

command wrapperg en w tracew

tracec

converts this to a C p rogram The complie the le tracec and insert the resulting

bject

le into y

ur

link line cc

aout

ao

traceo

lpmpi lmpi

SLIDE 143 Another p roling example This version counts all calls and the numb er

f

b ytes sent with MPISend MPIBse nd

r

MPIIse nd

include

mpih ""foreach fn fnname MPISen d MPIBsen d MPIIsend && static long ""fnna me && n by te s "" fi le no && "" en df

r

ea ch fn && ""forallf n fnname MPIInit MPIFin al iz e MPIWti me && in t ""fnname && n ca ll s "" fi le no &&

""endfora

ll fn && ""fnall thisfn n am e MPIFina li ze && printf ""this f n na me && is being calledn

""callfn&

& ""thisfn n am e& & nc al ls " "f il en

&

&

""endfnal

l& & ""fn fnname MPISend MPIBse nd MPIIse nd& & ""vardecl int typesiz e& & ""callfn& & MPIType si ze

""dataty

pe &&

MPIAin

t #""ty pe si ze &&

""fnname

&& n by te s "" fi le no &&

""

ty pe siz e& & "" co un t& & ""fnname && n ca ll s "" fi le no &&

""endfn&&

SLIDE 144 Another p roling example cont ""fn fnname MPIFina li ze && ""forallf n disfn&& if ""disf n& & nc al ls " "f il en

&

& " printf ""dis fn &&

%d

callsn

""disfn

&& n ca ll s "" fi le no& &

&

""endfora ll fn && if MPISen d nc al ls " "f il en

&

& " printf %d bytes sent in %d calls with MPISen d n

MPISend

n by te s "" fi le no && MPISend nc al ls " "f il en

&

&

&

""callfn& & ""endfn&&

SLIDE 145 Generating and viewing log les Log les that contain a histo ry

f

a pa rallel computation can b e very valuable in understanding a pa rallel p rogram The upshot and nupshot p rograms p rovided in the MPICH and MPIF implementations ma y b e used to view log les

SLIDE 146 Generating a log le This is very easy with the MPICH implementation

f

MPI Simply replace lmpi with llmpi lpmpi lm in the link line fo r y

ur

p rogram and relink y

ur

p rogram Y

u

do not need to recompile On some systems y

u

can get a realtime animation b y using the lib ra ri es lampi lmpe lm lX lpmpi Alternately

y
u

can use the mpilog

r

mpianim

ptions

to the mpicc

r

mpif commands

SLIDE 147 Connecting several p rograms together MPI p rovides supp

rt

fo r connection sepa rate messagepassing p rograms together through the use

f

intercommunicato rs

SLIDE 148 Sending messages b et w een dierent p rograms Programs sha re MPICOMMWORLD Programs have sepa rate and disjoint communicato rs

Comm1 Comm2 App1 App2 MPI_COMM_WORLD Comm_intercomm

SLIDE 149 Exchanging data b et w een p rograms

F
rm

intercommunicato r MPIINTERCOMMCREATE

Send

data MPISend

intercomm
MPIRecv

buf

intercomm
MPIBcast

buf

localcomm
Mo

re complex p

inttop
int
p

erations can also b e used

SLIDE 150 Collective

p

erations Use MPIINTERCOMMMERGE to create an intercommunicato r

SLIDE 151 Final Comments Additional features

f

MPI not covered in this tuto rial

P

ersistent Communication

Erro

r handling

SLIDE 152 Sha rable MPI Resources

The

Standa rd itself

As

a T echnical rep

rt

U

f

T ennessee rep

rt
As

p

stscript

fo r ftp at infomcs an lg

v

in pubmpim pi re po rt ps

As

hyp ertext

n

the W

rld

Wide W eb httpww w mcs a nl go v mpi

As

a journal a rticle in the F all issue

f

the Journal

f

Sup ercomputing Applications

MPI

F

rum

discussions

The

MPI F

rum

email discussions and b

th

current and ea rlier versions

f

the Standa rd a re availabl e from netlib

Bo
ks
Using

MPI P

rtable

P a rallel Programmi ng with the MessageP assing Interface b y Gropp Lusk and Skjellum MIT Press

MPI

Annotated Reference Manual b y Otto et al in p repa ration

SLIDE 153 Sha rable MPI Resources continued

Newsgroup
comppara

ll el mp i

Mailing

lists

mpicomm$

mc sa nl g

v

the MPI F

rum

discussion list

mpiimpl$

mc sa nl g

v

the implem ento rs discussion list

Implementations

availabl e b y ftp

MPICH

is availabl e b y anonymous ftp from infomcs an lg

v

in the directo ry pubmpim pi ch le mpichta rZ

LAM

is available b y anonymous ftp from tbagosc ed u in the directo ry publam

The

CHIMP version

f

MPI is availabl e b y anonymous ftp from ftpepcc ed ac u k in the directo ry pubchim pr el eas e

T

est co de rep

sito

ry

ftpinf
mcs

a nl go v pub m pi mp it es t

SLIDE 154 MPI

The

MPI F

rum

with

ld

and new pa rticipants has b egun a follo won series

f

meetings

Goals
cla

rify existing draft

p

rovide features users have requested

mak

e extensions not changes

Majo

r T

pics

b eing considered

dynamic

p ro cess management

clientserver
realtim

e extensions

nesided

communication putget active messages

p
rtable

access to MPI system state fo r debuggers

language

bindings fo r C and F

rtran
Schedule
Dynamic

p ro cesses clientserver b y SC

MPI

complete b y SC

SLIDE 155 Summa ry

The

pa rallel computing communit y has co

p

erated to develop a fullfeatured standa rd messagepassing lib ra ry interface

Implementati
ns

ab

und
Applications

b eginning to b e develop ed

r

p

rted
MPI

p ro cess b eginning

Lots
f

MPI material availabl e