SLIDE 1
21-02-2006 Alexandre David, MVP'06 2
Today
Preliminaries (3.1). Decomposition Techniques (3.2). Surprise.
Principle of Parallel Algorithm Design Alexandre David B2-206 - - PowerPoint PPT Presentation
Principle of Parallel Algorithm Design Alexandre David B2-206 - - PowerPoint PPT Presentation
Principle of Parallel Algorithm Design Alexandre David B2-206 Today Preliminaries (3.1). Decomposition Techniques (3.2). Surprise. 21-02-2006 Alexandre David, MVP'06 2 Overview Introduction to parallel algorithms. Tasks
SLIDE 2
SLIDE 3
21-02-2006 Alexandre David, MVP'06 3
Overview
Introduction to parallel algorithms.
Tasks and decomposition. Processes and mapping. Processes vs. processors.
Decomposition techniques.
Recursive decomposition. Exploratory decomposition. Hybrid decomposition.
SLIDE 4
21-02-2006 Alexandre David, MVP'06 4
Introduction
Parallel algorithms have the added
dimension of concurrency.
Typical tasks:
Identify concurrent works. Map them to processors. Distribute inputs, outputs, and other data. Manage shared resources. Synchronize the processors.
SLIDE 5
21-02-2006 Alexandre David, MVP'06 5
Decomposing Problems
Decomposition into concurrent tasks.
No unique solution. Different sizes. Decomposition illustrated as a directed graph:
Nodes = tasks. Edges = dependency.
Task dependency graph
SLIDE 6
21-02-2006 Alexandre David, MVP'06 6
Example: Matrix * Vector
N tasks, 1 task/row: Matrix Vector Task dependency graph?
SLIDE 7
21-02-2006 Alexandre David, MVP'06 7
Example: Database Query Processing
MODEL = ``CIVIC'' AND YEAR = 2001 AND (COLOR = ``GREEN'' OR COLOR = ``WHITE)
SLIDE 8
21-02-2006 Alexandre David, MVP'06 8
A Solution
SLIDE 9
21-02-2006 Alexandre David, MVP'06 9
Another Solution
SLIDE 10
21-02-2006 Alexandre David, MVP'06 10
Granularity
Number and size of tasks.
Fine-grained: many small tasks. Coarse-grained: few large tasks.
Related: degree of concurrency.
Maximal degree of concurrency. Average degree of concurrency.
SLIDE 11
21-02-2006 Alexandre David, MVP'06 11
Coarser Matrix * Vector
N tasks, 3 task/row: Matrix Vector
SLIDE 12
21-02-2006 Alexandre David, MVP'06 12
Granularity
Average degree of concurrency if we take
into account varying amount of work?
Critical path = longest directed path
between any start & finish nodes.
Critical path length = sum of the weights
- f nodes along this path.
Average degree of concurrency = total
amount of work / critical path length.
SLIDE 13
21-02-2006 Alexandre David, MVP'06 13
Database Example
Critical path (3). Critical path (4). Critical path length = 27. Critical path length = 34.
- Av. deg. of concurrency = 63/27.
- Av. deg. of conc. = 64/34.
2.33 1.88
SLIDE 14
21-02-2006 Alexandre David, MVP'06 14
Interaction Between Tasks
Tasks often share data. Task interaction graph:
Nodes = tasks. Edges = interaction. Optional weights.
Task dependency graph is a sub-graph of
the task interaction graph.
SLIDE 15
21-02-2006 Alexandre David, MVP'06 15
Example: Sparse Matrix Multiplication
SLIDE 16
21-02-2006 Alexandre David, MVP'06 16
Processes and Mapping
Tasks run on processors. Process: processing agent executing the
- tasks. Not exactly like in your OS course.
Mapping = assignment of tasks to
processes.
API expose processes and binding to
processors not always controlled.
SLIDE 17
21-02-2006 Alexandre David, MVP'06 17
Mapping Example
SLIDE 18
21-02-2006 Alexandre David, MVP'06 18
Processes vs. Processors
Processes = logical computing agent. Processor = hardware computational unit. In general 1-1 correspondence but this
model gives better abstraction.
Useful for hardware supporting multiple
programming paradigms.
Now remains the question: How do you decompose?
SLIDE 19
21-02-2006 Alexandre David, MVP'06 19
Decomposition Techniques
Recursive decomposition.
Divide-and-conquer.
Data decomposition.
Large data structure.
Exploratory decomposition.
Search algorithms.
Speculative decomposition.
Dependent choices in computations.
SLIDE 20
21-02-2006 Alexandre David, MVP'06 20
Recursive Decomposition
Problem solvable by divide-and-conquer:
Decompose into sub-problems.
Do it recursively.
Combine the sub-solutions.
Do it recursively.
Concurrency: The sub-problems are solved
in parallel.
SLIDE 21
21-02-2006 Alexandre David, MVP'06 21
Quicksort Example
<5≤ <3≤ <9≤ <7≤ <10≤ <11≤
SLIDE 22
21-02-2006 Alexandre David, MVP'06 22
Minimal Number
4 9 1 7 8 11 2 12
SLIDE 23
21-02-2006 Alexandre David, MVP'06 23
Data Decomposition
2 steps:
Partition the data. Induce partition into tasks.
How to partition data? Partition output data:
Independent “sub-outputs”.
Partition input data:
Local computations, followed by combination.
SLIDE 24
21-02-2006 Alexandre David, MVP'06 24
Matrix Multiplication
SLIDE 25
21-02-2006 Alexandre David, MVP'06 25
Intermediate Data Partitioning
Linear combination
- f the intermediate
results.
SLIDE 26
21-02-2006 Alexandre David, MVP'06 26
Owner Compute Rule
Process assigned to some data
is responsible for all computations associated
with it.
Input data decomposition:
All computations done on the (partitioned)
input data are done by the process.
Output data decomposition:
All computations for the (partitioned) output
data are done by the process.
SLIDE 27
21-02-2006 Alexandre David, MVP'06 27
Exploratory Decomposition
15-puzzle example
SLIDE 28
21-02-2006 Alexandre David, MVP'06 28
Search
SLIDE 29
21-02-2006 Alexandre David, MVP'06 29
Anomalous Behavior Possible
Work depends on the order of the search!
SLIDE 30
21-02-2006 Alexandre David, MVP'06 30
Speculative Decomposition
Dependencies between tasks are not
known a-priori.
How to identify independent tasks? Conservative approach: identify tasks that are
guaranteed to be independent.
Optimistic approach: schedule tasks even if we
are not sure – may roll-back later.
SLIDE 31
21-02-2006 Alexandre David, MVP'06 31