SLIDE 1
07-03-2006 Alexandre David, MVP'06 2
Today
Scatter and Gather (4.4). All-to-All Personalized Communication
(4.5).
Circular Shift (4.6). Improving the Speed of Some
Basic Communication Operations (cont.) Alexandre David B2-206 - - PowerPoint PPT Presentation
Basic Communication Operations (cont.) Alexandre David B2-206 - - PowerPoint PPT Presentation
Basic Communication Operations (cont.) Alexandre David B2-206 Today Scatter and Gather (4.4). All-to-All Personalized Communication (4.5). Circular Shift (4.6). Improving the Speed of Some Communication Operations (4.7).
SLIDE 2
SLIDE 3
07-03-2006 Alexandre David, MVP'06 3
Scatter and Gather
Scatter: A node sends a unique message
to every other node – unique per node.
Gather: Dual operation but the target node
does not combine the messages into one.
1 2 … 1 2 …
M0 M0 M1 M1 M2 M2
Scatter Gather
SLIDE 4
07-03-2006 Alexandre David, MVP'06 4
SLIDE 5
07-03-2006 Alexandre David, MVP'06 5
Cost Analysis
Number of steps: logp. Size transferred: pm/2, pm/4,…,m.
Geometric sum
Cost T=tslogp+twm(p-1).
) 2 2 2 ( 1 ) 2 1 1 ( 2 ) 2 1 1 ( 2 2 ... 4 2 2 1 1 2 1 1 2 ... 4 2
log 1 1 1 1
p p p p p p p p p p p p p p p
p n n n n n
= = − = − − = − − = + + + − − = + + + +
+ + + +
SLIDE 6
07-03-2006 Alexandre David, MVP'06 6
All-to-All Personalized Communication
Each node sends a distinct message to
every other node.
1 2 … 1 2 …
M0,0 M0,1 M0,2 M0,0 M0,1 M0,2 M1,0 M1,1 M1,2 M1,0 M1,1 M1,2 M2,0 M2,1 M2,2 M2,0 M2,1 M2,2
SLIDE 7
07-03-2006 Alexandre David, MVP'06 7
Example: Transpose
SLIDE 8
07-03-2006 Alexandre David, MVP'06 8
Total Exchange on a Ring
5 2 1 3 4 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 0 1 2 3 4 5 1 2 3 4 5 0 2 3 4 5 0 1 3 4 5 0 1 2 4 5 0 1 2 3 5 0 1 2 3 4 1 2 3 4 5
SLIDE 9
07-03-2006 Alexandre David, MVP'06 9
Total Exchange on a Ring
5 2 1 4 3 1 1 2 2 3 3 4 4 5 5 0 1 4 5 0 1 2 5 0 1 2 3 1 2 3 4 2 3 4 5 0 3 4 5 1 2 3 4 5
SLIDE 10
07-03-2006 Alexandre David, MVP'06 10
Cost Analysis
Number of steps: p-1. Size transmitted: m(p-1),m(p-2)…,m.
) 1 )( 2 / ( ) 1 (
1 1
− + = + − =
∑
− =
p mp t t m it p t T
w s p i w s
Optimal
SLIDE 11
07-03-2006 Alexandre David, MVP'06 11
Total Exchange on a Mesh
1 2 3 4 5 6 7 8
SLIDE 12
07-03-2006 Alexandre David, MVP'06 12
Total Exchange on a Mesh
1 2 3 4 5 6 7 8
SLIDE 13
07-03-2006 Alexandre David, MVP'06 13
Total Exchange on a Mesh
1 2 3 4 5 6 7 8
SLIDE 14
07-03-2006 Alexandre David, MVP'06 14
Cost Analysis
Substitute p by √p (number of nodes per
dimension).
Substitute message size m by m√p. Cost is the same for each dimension. T=(2ts+twmp)(√p-1)
SLIDE 15
07-03-2006 Alexandre David, MVP'06 15
Total Exchange on a Hypercube
Generalize the mesh algorithm to logp
steps = number of dimensions, with 2 nodes per dimension.
Same procedure as all-to-all broadcast.
SLIDE 16
07-03-2006 Alexandre David, MVP'06 16
Total Exchange on a Hypercube
1 2 3 4 5 6 7
SLIDE 17
07-03-2006 Alexandre David, MVP'06 17
Total Exchange on a Hypercube
1 2 3 4 5 6 7
SLIDE 18
07-03-2006 Alexandre David, MVP'06 18
Total Exchange on a Hypercube
1 2 3 4 5 6 7
SLIDE 19
07-03-2006 Alexandre David, MVP'06 19
Total Exchange on a Hypercube
1 2 3 4 5 6 7
SLIDE 20
07-03-2006 Alexandre David, MVP'06 20
Cost Analysis
Number of steps: logp. Size transmitted per step: pm/2. Cost: T=(ts+twmp/2) logp. Optimal?
Each node sends and receives m(p-1) words.
Average distance = ( logp)/2. Total traffic = p*m(p-1)* logp/2.
Number of links = p logp/2. Time lower bound = twm(p-1).
NO
SLIDE 21
07-03-2006 Alexandre David, MVP'06 21
An Optimal Algorithm
Have every pair of nodes communicate
directly with each other – p-1 communication steps – but without congestion.
At jth step node i communicates with node
(i xor j) with E-cube routing.
SLIDE 22
07-03-2006 Alexandre David, MVP'06 22
Total Exchange on a Hypercube
1 2 3 4 5 6 7
SLIDE 23
07-03-2006 Alexandre David, MVP'06 23
Total Exchange on a Hypercube
1 2 3 4 5 6 7
SLIDE 24
07-03-2006 Alexandre David, MVP'06 24
Total Exchange on a Hypercube
1 2 3 4 5 6 7
SLIDE 25
07-03-2006 Alexandre David, MVP'06 25
Total Exchange on a Hypercube
1 2 3 4 5 6 7
SLIDE 26
07-03-2006 Alexandre David, MVP'06 26
Total Exchange on a Hypercube
1 2 3 4 5 6 7
SLIDE 27
07-03-2006 Alexandre David, MVP'06 27
Total Exchange on a Hypercube
1 2 3 4 5 6 7 Etc…
SLIDE 28
07-03-2006 Alexandre David, MVP'06 28
Cost Analysis
Remark: Transmit less, only what is
needed, but more steps.
Number of steps: p-1. Transmission: size m per step. Cost: T=(ts+twm)(p-1). Compared withT=(ts+twmp/2) logp. Previous algorithm better for small
messages.
SLIDE 29
07-03-2006 Alexandre David, MVP'06 29
Circular Shift
It’s a particular permutation. Circular q-shift: Node i sends data to node
(i+q) mod p (in a set of p nodes).
Useful in some matrix operations and
pattern matching.
Ring: intuitive algorithm in min{q,p-q}
neighbor to neighbor communication
- steps. Why?
SLIDE 30
07-03-2006 Alexandre David, MVP'06 30
q mod √p on rows compensate ⎣q / √p⎦ on colums
Circular 5-shift
- n a mesh.
SLIDE 31
07-03-2006 Alexandre David, MVP'06 31
Circular Shift on a Hypercube
Map a linear array with 2d nodes onto a
hypercube of dimension d.
Expand q shift as a sum of powers of 2
(e.g. 5-shift = 20+22).
Perform the decomposed shifts. Use bi-directional links for “forward” (shift
itself) and “backward” (rotation part)… logp steps.
SLIDE 32
07-03-2006 Alexandre David, MVP'06 32
Or better: Direct E-cube routing. q-shifts on a 8-node hypercube.
SLIDE 33
07-03-2006 Alexandre David, MVP'06 33
Improving Performance
So far messages of size m were not split. If we split them into p parts:
One-to-all broadcast = scatter + all-to-all
broadcast of messages of size m/p.
All-to-one reduction = all-to-all reduce +
scatter of messages of size m/p.
All-reduce = all-to-all reduction + all-to-all