QR Factorization of Tall and Skinny Matrices in a Grid Computing - - PowerPoint PPT Presentation

QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment

Emmanuel AGULLO (INRIA / LaBRI) Camille COTI (Iowa State University) Jack DONGARRA (University of Tennessee) Thomas H´

ERAULT (U. Paris Sud / U. of Tennessee / LRI / INRIA)

Julien LANGOU (University of Colorado Denver) IPDPS, Atlanta, USA, April 19-23, 2010

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 1

Introduction

Question

Can we speed up dense linear algebra applications using a computational grid ?

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 2

Introduction

Building blocks

Tremendous computational power of grid infrastructures

⋆ BOINC: 2.4 Pflop/s, ⋆ Folding@home: 7.9 Pflop/s.

MPI-based linear algebra libraries

⋆ ScaLAPACK; ⋆ HP Linpack.

Grid-enabled MPI middleware

⋆ MPICH-G2; ⋆ PACX-MPI; ⋆ GridMPI. Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 3

Introduction

Past answers

Can we speed up dense linear algebra applications using a computational grid ?

⋆ GrADS project [Petitet et al., 2001]:

Grid enables to process larger matrices; For matrices that can fit in the (distributed) memory of a cluster, the use of a single cluster is optimal.

⋆ Study on a cloud infrastructure [Napper et al., 2009]

Linpack on Amazon EC2 commercial offer:

Under-calibrated components; Grid costs too much

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 4

Introduction

Our approach

Principle

Confine intensive communications (ScaLAPACK calls) within the different geographical sites.

Method

Articulate:

⋆ Communication-Avoiding algorithms [Demmel et al., 2008]; ⋆ with a topology-aware middleware (QCG-OMPI).

Focus

⋆ QR factorization; ⋆ Tall and Skinny matrices. Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 5

Introduction

Outline

• 1. Background
• 2. Articulation of TSQR with QCG-OMPI
• 3. Experiments

ScaLAPACK performance TSQR performance TSQR vs ScaLAPACK performance

• 4. Conclusion and future work

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 6

Background

Outline

• 1. Background
• 2. Articulation of TSQR with QCG-OMPI
• 3. Experiments

ScaLAPACK performance TSQR performance TSQR vs ScaLAPACK performance

• 4. Conclusion and future work

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 7

Background TSQR / CAQR

Communication-Avoiding QR (CAQR) [Demmel et al., 2008]

Tall and Skinny QR (TSQR) CAQR

R

TSQR

UPDATES

Examples of applications for TSQR

⋆ panel factorization in CAQR; ⋆ block iterative methods (iterative methods with multiple

right-hand sides or iterative eigenvalue solvers);

⋆ linear least squares problems with a number of equations

extremely larger than the number of unknowns.

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 8

Background TSQR / CAQR

Communication-Avoiding QR (CAQR) [Demmel et al., 2008]

Tall and Skinny QR (TSQR) CAQR

R

TSQR

UPDATES

Examples of applications for TSQR

⋆ panel factorization in CAQR; ⋆ block iterative methods (iterative methods with multiple

right-hand sides or iterative eigenvalue solvers);

⋆ linear least squares problems with a number of equations

extremely larger than the number of unknowns.

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 8

Background TSQR / CAQR

Communication-Avoiding QR (CAQR) [Demmel et al., 2008]

Tall and Skinny QR (TSQR) CAQR

R

TSQR

UPDATES

Examples of applications for TSQR

⋆ panel factorization in CAQR; ⋆ block iterative methods (iterative methods with multiple

right-hand sides or iterative eigenvalue solvers);

⋆ linear least squares problems with a number of equations

extremely larger than the number of unknowns.

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 8

Background TSQR / CAQR

Communication-Avoiding QR (CAQR) [Demmel et al., 2008]

Tall and Skinny QR (TSQR) CAQR

R

TSQR

UPDATES

Examples of applications for TSQR

⋆ panel factorization in CAQR; ⋆ block iterative methods (iterative methods with multiple

right-hand sides or iterative eigenvalue solvers);

⋆ linear least squares problems with a number of equations

extremely larger than the number of unknowns.

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 8

Background QCG-OMPI

Topology-aware MPI middleware for the Grid

MPICH-G2

⋆ description of the topology through the concept of colors:

used to build topology-aware MPI communicators; the application has to adapt itself to the discovered topology;

⋆ based on MPICH.

QCG-OMPI

⋆ resource-aware grid meta-scheduler (QosCosGrid); ⋆ allocation of resources that match requirements expressed in a

“JobProfile” (amount of memory, CPU speed, network properties between groups of processes, . . . )

application always executed on an appropriate resource topology.

⋆ based on OpenMPI. Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 9

Background QCG-OMPI

Topology-aware MPI middleware for the Grid

MPICH-G2

⋆ description of the topology through the concept of colors:

used to build topology-aware MPI communicators; the application has to adapt itself to the discovered topology;

⋆ based on MPICH.

QCG-OMPI

⋆ resource-aware grid meta-scheduler (QosCosGrid); ⋆ allocation of resources that match requirements expressed in a

“JobProfile” (amount of memory, CPU speed, network properties between groups of processes, . . . )

application always executed on an appropriate resource topology.

⋆ based on OpenMPI. Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 9

Background QCG-OMPI

Topology-aware MPI middleware for the Grid

MPICH-G2

⋆ description of the topology through the concept of colors:

used to build topology-aware MPI communicators; the application has to adapt itself to the discovered topology;

⋆ based on MPICH.

QCG-OMPI

⋆ resource-aware grid meta-scheduler (QosCosGrid); ⋆ allocation of resources that match requirements expressed in a

“JobProfile” (amount of memory, CPU speed, network properties between groups of processes, . . . )

application always executed on an appropriate resource topology.

⋆ based on OpenMPI. Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 9

Articulation of TSQR with QCG-OMPI

Outline

• 1. Background
• 2. Articulation of TSQR with QCG-OMPI
• 3. Experiments

ScaLAPACK performance TSQR performance TSQR vs ScaLAPACK performance

• 4. Conclusion and future work

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 10

Articulation of TSQR with QCG-OMPI Communication pattern

Communication pattern (M-by-3 matrix)

ScaLAPACK (panel factorization routine) - non optimized tree

Domain 2,4 Domain 1,1 Domain 1,2 Domain 1,3 Cluster 1 Domain 1,4 Domain 1,5 Domain 2,1 Domain 2,2 Domain 2,3 Cluster 2 Domain 3,1 Domain 3,2 Cluster 3 Illustration of ScaLAPACK PDEGQRF without reduce affinity

25 inter-cluster communications

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 11

Articulation of TSQR with QCG-OMPI Communication pattern

Communication pattern (M-by-3 matrix)

ScaLAPACK (panel factorization routine) - non optimized tree

25 inter-cluster communications

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 11

Articulation of TSQR with QCG-OMPI Communication pattern

Communication pattern (M-by-3 matrix)

ScaLAPACK (panel factorization routine) - optimized tree

Domain 1,1 Domain 1,2 Domain 1,3 Cluster 1 Domain 1,4 Domain 1,5 Domain 2,1 Domain 2,2 Domain 2,3 Cluster 2 Domain 2,4 Domain 3,1 Domain 3,2 Cluster 3 Illustration of ScaLAPACK PDEGQRF with reduce affinity

10 inter-cluster communications

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 11

Articulation of TSQR with QCG-OMPI Communication pattern

Communication pattern (M-by-3 matrix)

TSQR - optimized tree

2 inter-cluster communications

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 11

Articulation of TSQR with QCG-OMPI Communication pattern

Communication pattern (M-by-3 matrix)

TSQR - optimized tree

2 inter-cluster communications

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 11

Articulation of TSQR with QCG-OMPI Communication pattern

Communication pattern (M-by-3 matrix)

ScaLAPACK TSQR

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 11

Articulation of TSQR with QCG-OMPI Articulation of TSQR with QCG-OMPI

Articulation of TSQR with QCG-OMPI

ScaLAPACK-based TSQR

⋆ each domain is factorized with a ScaLAPACK call; ⋆ the reduction is performed by pairs of communicators; ⋆ the number of domains per cluster may vary from 1 to 64

(number of cores per cluster).

QCG JobProfile

⋆ processes are split into groups of equivalent computing power; ⋆ good network connectivity inside each group (low latency, high

bandwidth);

⋆ lower network connectivity between the groups.

→ Classical cluster of clusters approach (with a constraint on the relative size of the clusters to facilitate load balancing).

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 12

Articulation of TSQR with QCG-OMPI Articulation of TSQR with QCG-OMPI

• Comm. and computation breakdown (critical path)

Computing R

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 2N log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C TSQR log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C + 2/3 log2(C)N3

Computing Q and R (on C clusters)

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 4N log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C TSQR 2 log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C + 4/3 log2(C)N3 ⋆ C: number of clusters; ⋆ 1 domain per cluster is assumed for these tables. Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 13

Articulation of TSQR with QCG-OMPI Articulation of TSQR with QCG-OMPI

• Comm. and computation breakdown (critical path)

Computing R

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 2N log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C TSQR log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C + 2/3 log2(C)N3

Computing Q and R (on C clusters)

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 4N log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C TSQR 2 log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C + 4/3 log2(C)N3 ⋆ C: number of clusters; ⋆ 1 domain per cluster is assumed for these tables. Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 13

Articulation of TSQR with QCG-OMPI Articulation of TSQR with QCG-OMPI

• Comm. and computation breakdown (critical path)

Computing R

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 2N log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C TSQR log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C + 2/3 log2(C)N3

Computing Q and R (on C clusters)

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 4N log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C TSQR 2 log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C + 4/3 log2(C)N3 ⋆ C: number of clusters; ⋆ 1 domain per cluster is assumed for these tables. Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 13

Articulation of TSQR with QCG-OMPI Articulation of TSQR with QCG-OMPI

• Comm. and computation breakdown (critical path)

Computing R

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 2N log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C TSQR log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C + 2/3 log2(C)N3

Computing Q and R (on C clusters)

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 4N log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C TSQR 2 log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C + 4/3 log2(C)N3 ⋆ C: number of clusters; ⋆ 1 domain per cluster is assumed for these tables. Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 13

Articulation of TSQR with QCG-OMPI Articulation of TSQR with QCG-OMPI

• Comm. and computation breakdown (critical path)

Computing R

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 2N log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C TSQR log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C + 2/3 log2(C)N3

Computing Q and R (on C clusters)

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 4N log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C TSQR 2 log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C + 4/3 log2(C)N3 ⋆ C: number of clusters; ⋆ 1 domain per cluster is assumed for these tables. Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 13

Articulation of TSQR with QCG-OMPI Articulation of TSQR with QCG-OMPI

• Comm. and computation breakdown (critical path)

Computing R

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 2N log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C TSQR log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C + 2/3 log2(C)N3

Computing Q and R (on C clusters)

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 4N log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C TSQR 2 log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C + 4/3 log2(C)N3 ⋆ C: number of clusters; ⋆ 1 domain per cluster is assumed for these tables. Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 13

Articulation of TSQR with QCG-OMPI Articulation of TSQR with QCG-OMPI

• Comm. and computation breakdown (critical path)

Computing R

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 2N log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C TSQR log2(C) log2(C)(N2/2) (2MN2 − 2/3N3)/C + 2/3 log2(C)N3

Computing Q and R (on C clusters)

# inter-cluster msg inter-cluster vol. exchanged # FLOPs ScaLAPACK QR2 4N log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C TSQR 2 log2(C) 2 log2(C)(N2/2) (4MN2 − 4/3N3)/C + 4/3 log2(C)N3 ⋆ C: number of clusters; ⋆ 1 domain per cluster is assumed for these tables. Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 13

Experiments

Outline

• 1. Background
• 2. Articulation of TSQR with QCG-OMPI
• 3. Experiments

ScaLAPACK performance TSQR performance TSQR vs ScaLAPACK performance

• 4. Conclusion and future work

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 14

Experiments Experimental environment

Experimental environment: Grid’5000

⋆ Four clusters – 32 nodes per cluster – 2 cores per node. ⋆ AMD Opteron 246 (2 GHz/ 1MB L2 cache) up to AMD Opteron

2218 (2.6 GHz / 2MB L2 cache).

⋆ Linux 2.6.30 – ScaLAPACK 1.8.0 – GotoBlas 1.26. ⋆ 256 cores total (theoretical peak 2048 Gflop/s – dgemm

upperbound 940 Gflop/s).

!"#$% &'"()$*+ ,'*-'*#) .'/01$2 3451/'-1#

Network

Latency (ms) Orsay Toulouse Bordeaux Sophia Orsay 0.07 7.97 6.98 6.12 Toulouse 0.03 9.03 8.18 Bordeaux 0.05 7.18 Sophia 0.06 Throughput (Mb/s) Orsay Toulouse Bordeaux Sophia Orsay 890 78 90 102 Toulouse 890 77 90 Bordeaux 890 83 Sophia 890

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 15

Experiments ScaLAPACK performance

Outline

• 1. Background
• 2. Articulation of TSQR with QCG-OMPI
• 3. Experiments

ScaLAPACK performance TSQR performance TSQR vs ScaLAPACK performance

• 4. Conclusion and future work

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 16

Experiments ScaLAPACK performance

ScaLAPACK - N = 64

5 10 15 20 25 30 35 100000 1e+06 1e+07 1e+08

Gflop/s Number of rows (M)

4 sites (128 nodes) 2 sites (64 nodes) 1 site (32 nodes)

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 17

Experiments ScaLAPACK performance

ScaLAPACK - N = 128

10 20 30 40 50 60 100000 1e+06 1e+07 1e+08

Gflop/s Number of rows (M)

4 sites (128 nodes) 2 sites (64 nodes) 1 site (32 nodes)

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 18

Experiments ScaLAPACK performance

ScaLAPACK - N = 256

10 20 30 40 50 60 100000 1e+06 1e+07

Gflop/s Number of rows (M)

4 sites (128 nodes) 2 sites (64 nodes) 1 site (32 nodes)

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 19

Experiments ScaLAPACK performance

ScaLAPACK - N = 512

10 20 30 40 50 60 70 80 90 100000 1e+06 1e+07

Gflop/s Number of rows (M)

4 sites (128 nodes) 2 sites (64 nodes) 1 site (32 nodes)

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 20

Experiments TSQR performance

Outline

• 1. Background
• 2. Articulation of TSQR with QCG-OMPI
• 3. Experiments

ScaLAPACK performance TSQR performance TSQR vs ScaLAPACK performance

• 4. Conclusion and future work

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 21

Experiments TSQR performance - effect of the number of domains per cluster

TSQR - N = 64 - one cluster 5 10 15 20 25 30 35 40 64 32 16 8 4 2 1 Gflop/s Number of domains M = 8 388 608 M = 1 048 576 M = 131 072 M = 65 536

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 22

Experiments TSQR performance - effect of the number of domains per cluster

TSQR - N = 64 - all four clusters

20 40 60 80 100 64 32 16 8 4 2 1

Gflop/s Number of domains per cluster

M = 33 554 432 M = 4 194 304 M = 524 288 M = 131 072

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 23

Experiments TSQR performance - effect of the number of domains per cluster

TSQR - N = 512 - one cluster

20 40 60 80 100 64 32 16 8 4 2 1

Gflop/s Number of domains

M = 2 097 152 M = 1 048 576 M = 131 072 M = 65 536

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 24

Experiments TSQR performance - effect of the number of domains per cluster

TSQR - N = 512 - all four clusters

50 100 150 200 250 300 350 64 32 16 8 4 2 1

Gflop/s Number of domains per cluster

M = 8 388 608 M = 2 097 152 M = 524 288 M = 262 144

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 25

Experiments TSQR performance (optimum configuration)

TSQR - N = 64

20 40 60 80 100 100000 1e+06 1e+07 1e+08

Gflop/s Number of rows (M)

4 sites (128 nodes) 2 sites (64 nodes) 1 site (32 nodes)

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 26

Experiments TSQR performance (optimum configuration)

TSQR - N = 128

20 40 60 80 100 120 140 100000 1e+06 1e+07 1e+08

Gflop/s Number of rows (M)

4 sites (128 nodes) 2 sites (64 nodes) 1 site (32 nodes)

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 27

Experiments TSQR performance (optimum configuration)

TSQR - N = 256

20 40 60 80 100 120 140 160 180 100000 1e+06 1e+07

Gflop/s Number of rows (M)

4 sites (128 nodes) 2 sites (64 nodes) 1 site (32 nodes)

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 28

Experiments TSQR performance (optimum configuration)

TSQR - N = 512

50 100 150 200 250 300 100000 1e+06 1e+07

Gflop/s Number of rows (M)

4 sites (128 nodes) 2 sites (64 nodes) 1 site (32 nodes)

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 29

Experiments TSQR vs ScaLAPACK performance

Outline

• 1. Background
• 2. Articulation of TSQR with QCG-OMPI
• 3. Experiments

ScaLAPACK performance TSQR performance TSQR vs ScaLAPACK performance

• 4. Conclusion and future work

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 30

Experiments TSQR vs ScaLAPACK performance

TSQR vs ScaLAPACK - N = 64

10 20 30 40 50 60 70 80 90 100000 1e+06 1e+07 1e+08

Gflop/s Number of rows (M)

TSQR (best) ScaLAPACK (best)

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 31

Experiments TSQR vs ScaLAPACK performance

TSQR vs ScaLAPACK - N = 128

20 40 60 80 100 120 100000 1e+06 1e+07 1e+08

Gflop/s Number of rows (M)

TSQR (best) ScaLAPACK (best)

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 32

Experiments TSQR vs ScaLAPACK performance

TSQR vs ScaLAPACK - N = 256

20 40 60 80 100 120 140 160 180 100000 1e+06 1e+07

Gflop/s Number of rows (M)

TSQR (best) ScaLAPACK (best)

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 33

Experiments TSQR vs ScaLAPACK performance

TSQR vs ScaLAPACK - N = 512

50 100 150 200 250 300 100000 1e+06 1e+07

Gflop/s Number of rows (M)

TSQR (best) ScaLAPACK (best)

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 34

Conclusion and future work

Outline

• 1. Background
• 2. Articulation of TSQR with QCG-OMPI
• 3. Experiments

ScaLAPACK performance TSQR performance TSQR vs ScaLAPACK performance

• 4. Conclusion and future work

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 35

Conclusion and future work Conclusion

Conclusion

Can we speed up dense linear algebra applications using a computational grid ? Yes, at least for applications based on the QR factorization

• f Tall and Skinny matrices.

Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 36

Conclusion and future work Future directions

Future directions

⋆ What about square matrices (CAQR) ? ⋆ LU and Cholesky factorizations ? ⋆ Can we benefit from recursive kernels ? Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 37

Conclusion and future work current investigations

N = 64 - one cluster

10 20 30 40 50 60 70 80 5e+06 1e+07 1.5e+07 2e+07 2.5e+07 3e+07 3.5e+07 GFlops Matrix height (M) Comparison of QR algorithms, N = 64, 1 clusters Cholesky QR A v0 Householder A v2 PDGEQRF Householder A v1 CGS v0 CGS v1 Householder A v0 Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 38

Conclusion and future work current investigations

N = 64 - two clusters

20 40 60 80 100 120 140 1e+07 2e+07 3e+07 4e+07 5e+07 6e+07 7e+07 GFlops Matrix height (M) Comparison of QR algorithms, N = 64, 2 clusters Cholesky QR A v0 Householder A v2 PDGEQRF Householder A v1 CGS v0 CGS v1 Householder A v0 Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 39

Conclusion and future work current investigations

N = 64 - all four clusters

50 100 150 200 250 300 2e+07 4e+07 6e+07 8e+07 1e+08 1.2e+08 1.4e+08 GFlops Matrix height (M) Comparison of QR algorithms, N = 64, 4 clusters Cholesky QR A v0 Householder A v2 PDGEQRF Householder A v1 CGS v0 CGS v1 Householder A v0 Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 40

Conclusion and future work current investigations

N = 512 - one cluster

20 40 60 80 100 120 140 500000 1e+06 1.5e+06 2e+06 2.5e+06 3e+06 3.5e+06 4e+06 4.5e+06 GFlops Matrix height (M) Comparison of QR algorithms, N = 512, 1 cluster Cholesky QR A v0 Householder A v2 PDGEQRF Householder A v1 CGS v0 CGS v1 Householder A v0 Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 41

Conclusion and future work current investigations

N = 512 - two clusters

50 100 150 200 250 1e+06 2e+06 3e+06 4e+06 5e+06 6e+06 7e+06 8e+06 9e+06 GFlops Matrix height (M) Comparison of QR algorithms, N = 512, 2 clusters Cholesky QR A v0 Householder A v2 PDGEQRF Householder A v1 CGS v0 CGS v1 Householder A v0 Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 42

Conclusion and future work current investigations

N = 512 - all four clusters

50 100 150 200 250 300 350 400 450 500 2e+06 4e+06 6e+06 8e+06 1e+07 1.2e+07 1.4e+07 1.6e+07 1.8e+07 GFlops Matrix height (M) Comparison of QR algorithms, N = 512, 4 clusters Cholesky QR A v0 Householder A v2 PDGEQRF Householder A v1 CGS v0 CGS v1 Householder A v0 Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 43

Conclusion and future work current investigations

Thanks

⋆ Questions ? Agullo - Coti - Dongarra - H´ erault - Langou QR Factorization of Tall and Skinny Matrices in a Grid Computing Environment 44