FaultTolerantLinearAlgebra: goalsandmethods. - - PowerPoint PPT Presentation
FaultTolerantLinearAlgebra: goalsandmethods. JulienLangou,UniversityofColoradoDenver FaulttolerantLinearAlgebra:GoalsandMethods. 0GOALS
0‐ GOALS 0.1‐ ERRASURE OR ERROR? 1‐ METHODS 1.1‐ ERRASURE: DISKLESS CHECKPOINTING AND ROLLBACK 1.2‐ ERRASURE & ERROR: ABFT: ALGORITHM BASED FAULT TOLERANCE 1.3‐ OTHERS 2‐ ERROR: DETECTING/LOCATING/CORRECTING IN FLOATING‐POINT ARITHMETIC 3‐ NOVEL ABFT‐ALGORITHM (GEMM, LU, QR, ETC.) (ERRASURE OR ERROR) 4‐ ABFT‐BLAS LIBRARY 5‐ ABFT‐BLAS EXPERIMENTS (ERRASURE) Fault‐tolerant Linear Algebra: Goals and Methods.
0‐ GOALS 0.1‐ ERRASURE OR ERROR? 1‐ METHODS 1.1‐ ERRASURE: DISKLESS CHECKPOINTING AND ROLLBACK 1.2‐ ERRASURE & ERROR: ABFT: ALGORITHM BASED FAULT TOLERANCE 1.3‐ OTHERS 2‐ ERROR: DETECTING/LOCATING/CORRECTING IN FLOATING‐POINT ARITHMETIC 3‐ NOVEL ABFT‐ALGORITHM (GEMM, LU, QR, ETC.) (ERRASURE OR ERROR) 4‐ ABFT‐BLAS LIBRARY 5‐ ABFT‐BLAS EXPERIMENTS (ERRASURE) Fault‐tolerant Linear Algebra: Goals and Methods.
P1
2
P2
4
P3
6
P4
8
P1
1+1
P2
2+2
P3
3+3
P4
4+4 4 processors available
P1
2
P2
4
P3
6
P4
8
P1
1+1
P2
2+2
P3
3+3
P4
4+4 4 processors available
P1
2
P2
4
P3
6
P4
8
P1
1+1
P2
2+2
P3
3+3
P4
4+4 4 processors available Lost processor 2
P1
2
P2
5
P3
6
P4
8
P1
1+1
P2
2+2
P3
3+3
P4
4+4 4 processors available Processor 2 returns an incorrect result
P1
2
P2
4
P3
6
P4
8
P1
1+1
P2
2+2
P3
3+3
P4
4+4 4 processors available
‐ we know whether there is an errasure or not,
P1
2
P2
5
P3
6
P4
8
P1
1+1
P2
2+2
P3
3+3
P4
4+4 4 processors available ‐ we do not know if there is an error, Lost processor 2 Processor 2 returns an incorrect result
P1
2
P2
4
P3
6
P4
8
P1
1+1
P2
2+2
P3
3+3
P4
4+4 4 processors available
‐ we know whether there is an errasure or not, ‐ we know where the errasure is,
P1
2
P2
5
P3
6
P4
8
P1
1+1
P2
2+2
P3
3+3
P4
4+4 4 processors available ‐ we do not know if there is an error, ‐ assuming we know that an error occurs, we do not know where it is Lost processor 2 Processor 2 returns an incorrect result
P1
2
P2
4
P3
6
P4
8
P1
1+1
P2
2+2
P3
3+3
P4
4+4 4 processors available
‐ we know whether there is an errasure or not, ‐ we know where the errasure is, ‐ so we only need to recover
P1
2
P2
5
P3
6
P4
8
P1
1+1
P2
2+2
P3
3+3
P4
4+4 4 processors available ‐ we do not know if there is an error, ‐ assuming we know that an error occurs, we do not know where it is ‐ we also need to recover Lost processor 2 Processor 2 returns an incorrect result
0‐ GOALS 0.1‐ ERRASURE OR ERROR? 1‐ METHODS 1.1‐ ERRASURE: DISKLESS CHECKPOINTING AND ROLLBACK 1.2‐ ERRASURE & ERROR: ABFT: ALGORITHM BASED FAULT TOLERANCE 1.3‐ OTHERS 2‐ ERROR: DETECTING/LOCATING/CORRECTING IN FLOATING‐POINT ARITHMETIC 3‐ NOVEL ABFT‐ALGORITHM (GEMM, LU, QR, ETC.) (ERRASURE OR ERROR) 4‐ ABFT‐BLAS LIBRARY 5‐ ABFT‐BLAS EXPERIMENTS (ERRASURE) Fault‐tolerant Linear Algebra: Goals and Methods.
P1 P2 P3 P4 4 processors available
P1 P2 + P3 P4 + + Pc P1 P2 P3 P4 P1 P2 P3 P4 Pc 4 processors available Add a 5th one and perform a checksum (MPI_Reduce) Ready for the computaFons … … …
P1 P2 + P3 P4 + + Pc P1 P2 P3 P4 Pc P1 P2 P3 P4 P1 P3 P4 Pc P1 P2 P3 P4 Pc 4 processors available Add a 5th one and perform a checksum (MPI_Reduce) Ready for the computaFons … … … Lost a processor
P1 P2 + P3 P4 + + Pc P1 P2 P3 P4 Pc P1 P2 P3 P4 P1 P3 P4 Pc Pc P1 ‐ P3 P4 ‐ ‐ P2 P1 P2 P3 P4 Pc P1 P2 P3 P4 Pc 4 processors available Add a 5th one and perform a checksum (MPI_Reduce) Ready for the computaFons … … … Lost a processor Recover the processor (FT‐MPI) Recover the data (MPI_Reduce) Ready for the computaFons
0 0.5 1 1.5 2 2.5 64 81 100 121 256 122.0 MB 68.7 MB 30.5 MB 7.6 MB # of processors Fme (sec)
0‐ GOALS 0.1‐ ERRASURE OR ERROR? 1‐ METHODS 1.1‐ ERRASURE: DISKLESS CHECKPOINTING AND ROLLBACK 1.2‐ ERRASURE & ERROR: ABFT: ALGORITHM BASED FAULT TOLERANCE 1.3‐ OTHERS 2‐ ERROR: DETECTING/LOCATING/CORRECTING IN FLOATING‐POINT ARITHMETIC 3‐ NOVEL ABFT‐ALGORITHM (GEMM, LU, QR, ETC.) (ERRASURE OR ERROR) 4‐ ABFT‐BLAS LIBRARY 5‐ ABFT‐BLAS EXPERIMENTS (ERRASURE) Fault‐tolerant Linear Algebra: Goals and Methods.
X1 X2 X3 X4 Y1 Y2 Y3 Y4 Z1 Z2 Z3 Z4 + + + + We want to perform z = λx+μy. Proc 1 Proc 2 Proc 3 Proc 4 λ λ λ λ μ μ μ μ
X1 X2 + X3 X4 + + Y1 Y2 + Y3 Y4 + + Proc 1 Proc 2 Proc 3 Proc 4 We want to perform z = λx+μy.
X1 X2 + X3 X4 + + Xc Y1 Y2 + Y3 Y4 + + Yc Proc 1 Proc 2 Proc 3 Proc 4 Proc c checkX checkY We want to perform z = λx+μy.
X1 X2 X3 X4 Xc Y1 Y2 Y3 Y4 Yc Proc 1 Proc 2 Proc 3 Proc 4 Proc c checkX checkY We want to perform z = λx+μy.
X1 X2 X3 X4 Xc Y1 Y2 Y3 Y4 Yc Z1 Z2 Z3 Z4 + + + + Proc 1 Proc 2 Proc 3 Proc 4 Proc c checkX checkY λ λ λ λ μ μ μ μ We want to perform z = λx+μy.
X1 X2 X3 X4 Xc Y1 Y2 Y3 Y4 Yc Z1 Z2 + Z3 Z4 + + Zc + + + + Proc 1 Proc 2 Proc 3 Proc 4 Proc c checkX checkY checkZ λ λ λ λ μ μ μ μ We want to perform z = λx+μy.
X1 X2 X3 X4 Xc Y1 Y2 Y3 Y4 Yc Z1 Z2 Z3 Z4 Zc + + + + + Proc 1 Proc 2 Proc 3 Proc 4 Proc c checkX checkY checkZ No overhead to compute the checksum of Z. Property used: (λx1+μy1)+(λx2+μy2)=λ(x1+x2)+μ(y1+y2), distribuFvity of external mulFplicaFon over intenal addiFon, associaFvity of internal addiFon. λ λ λ λ μ μ μ μ λ μ We want to perform z = λx+μy.
0‐ GOALS 0.1‐ ERRASURE OR ERROR? 1‐ METHODS 1.1‐ ERRASURE: DISKLESS CHECKPOINTING AND ROLLBACK 1.2‐ ERRASURE & ERROR: ABFT: ALGORITHM BASED FAULT TOLERANCE 1.3‐ OTHERS 2‐ ERROR: DETECTING/LOCATING/CORRECTING IN FLOATING‐POINT ARITHMETIC 3‐ NOVEL ABFT‐ALGORITHM (GEMM, LU, QR, ETC.) (ERRASURE OR ERROR) 4‐ ABFT‐BLAS LIBRARY 5‐ ABFT‐BLAS EXPERIMENTS (ERRASURE) Fault‐tolerant Linear Algebra: Goals and Methods.
|| Cx ‐ α( A* ( B x )) + ( βCin x ) || < ε
0‐ GOALS 0.1‐ ERRASURE OR ERROR? 1‐ METHODS 1.1‐ ERRASURE: DISKLESS CHECKPOINTING AND ROLLBACK 1.2‐ ERRASURE & ERROR: ABFT: ALGORITHM BASED FAULT TOLERANCE 1.3‐ OTHERS 2‐ ERROR: DETECTING/LOCATING/CORRECTING IN FLOATING‐POINT ARITHMETIC 3‐ NOVEL ABFT‐ALGORITHM (GEMM, LU, QR, ETC.) (ERRASURE OR ERROR) 4‐ ABFT‐BLAS LIBRARY 5‐ ABFT‐BLAS EXPERIMENTS (ERRASURE) Fault‐tolerant Linear Algebra: Goals and Methods.
=
=
=
= ‐
=
3 1 3 1 0 1 7 9 2 5 8 2 1 1 1 3 = 3 3
3 1 3 1 0 1 7 9 2 5 8 2 1 1 1 3 0 0 0 0 0 3 0 0 = 3 3
1 1 1 1 1 1 1 1 1 2 3 4 5 6 7 8 = 2 8
1 1 1 1 1 1 1 1 1 2 3 4 5 6 7 8 0 0 0 2 0 0 0 0 = 2 8
3 1 3 1 0 1 7 9 2 5 8 2 1 1 1 3 0 0 0 0 0 1 0 = 4
1 4 9 3 0 0 5 0 ‐7 ‐4 ‐1
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
1 2 3 4 5 6 7 8
n, size of x Time in sec
n, size of x max( | xi – yi|/|xi|)
n size of x n, size of x max( | xi – yi|/|xi|)
n, size of x n, size of x max( | xi – yi|/|xi|) Time in sec
0‐ GOALS 0.1‐ ERRASURE OR ERROR? 1‐ METHODS 1.1‐ ERRASURE: DISKLESS CHECKPOINTING AND ROLLBACK 1.2‐ ERRASURE & ERROR: ABFT: ALGORITHM BASED FAULT TOLERANCE 1.3‐ OTHERS 2‐ ERROR: DETECTING/LOCATING/CORRECTING IN FLOATING‐POINT ARITHMETIC 3‐ NOVEL ABFT‐ALGORITHM (GEMM, LU, QR, ETC.) (ERRASURE OR ERROR) 4‐ ABFT‐BLAS LIBRARY 5‐ ABFT‐BLAS EXPERIMENTS (ERRASURE) Fault‐tolerant Linear Algebra: Goals and Methods.
A Ar Ac Af Bc
T
A I I Br StarFng from the encoding: =
A Ar Ac Af Bc
T
A = Ur Lc 0 = U L = Ac A = Ar Br Bc
T
A = Br (1) (2) (3) Bc
T
= Lc = Ur Br (1) (2) U L P End Af
A Ar Ac Af Bc
T
A = Ur Lc 0 = U L = Ac A = Ar Br Bc
T
A = Br (1) (2) (3) Bc
T
= Lc = Ur Br (1) (2) U L Ur Af Bc
T
= = Ac = Ar Br = (1) (2) (3) U L Lc Ar Ac A U L (4) (5) L Bc
T
= L P P Br = U A A Bc
T
A Br For j=1:nb:n, End Af Af Lc Ur
0‐ GOALS 0.1‐ ERRASURE OR ERROR? 1‐ METHODS 1.1‐ ERRASURE: DISKLESS CHECKPOINTING AND ROLLBACK 1.2‐ ERRASURE & ERROR: ABFT: ALGORITHM BASED FAULT TOLERANCE 1.3‐ OTHERS 2‐ ERROR: DETECTING/LOCATING/CORRECTING IN FLOATING‐POINT ARITHMETIC 3‐ NOVEL ABFT‐ALGORITHM (GEMM, LU, QR, ETC.) (ERRASURE OR ERROR) 4‐ ABFT‐BLAS LIBRARY 5‐ ABFT‐BLAS EXPERIMENTS (ERRASURE) Fault‐tolerant Linear Algebra: Goals and Methods.
* α + β A B C C α + β A B Cin Cout x ‐ x
x
int rc; struct Vector v; struct Matrix a; struct Dataworld worldmpi; struct Global_ddata normv; struct Global_idata nbr_iter; … rc = MPI_Init(&argc, &argv); rc = init_world(&worldmpi, p, q, rc); rc = get_info_on_grid(&worldmpi, &me, &myrow, &mycol, &nprow, &npcol); … rc = allocate_vector(&v, POS_ROW, 0, nb_n, &worldmpi, "v"); rc = allocate_matrix(&a, m, n, nb_m, nb_n, &worldmpi, "a"); rc = allocate_dglobal(&normv, 1, &worldmpi); rc = allocate_iglobal(&nbr_iter, 1, &worldmpi); … if (!worldmpi.recovering) { ... here goes the user code to iniWalize objects ... rc = make_checksum_matrix(&a, &worldmpi); rc = make_checksum_vector(&v, &worldmpi); } if (worldmpi.user_state == 0) { rc = gdnrm2(&worldmpi, &v, normv.data); worldmpi.user_state = 1; } if (worldmpi.user_state == 1) { ... here goes any call to the ABFT‐BLAS numerical rouWnes ... worldmpi.user_state = 2; } free_vector(&v); free_matrix(&a); free_dglobal(&normv); free_iglobal(&nbr_iter); exit(0);
0‐ GOALS 0.1‐ ERRASURE OR ERROR? 1‐ METHODS 1.1‐ ERRASURE: DISKLESS CHECKPOINTING AND ROLLBACK 1.2‐ ERRASURE & ERROR: ABFT: ALGORITHM BASED FAULT TOLERANCE 1.3‐ OTHERS 2‐ ERROR: DETECTING/LOCATING/CORRECTING IN FLOATING‐POINT ARITHMETIC 3‐ NOVEL ABFT‐ALGORITHM (GEMM, LU, QR, ETC.) (ERRASURE OR ERROR) 4‐ ABFT‐BLAS LIBRARY 5‐ ABFT‐BLAS EXPERIMENTS (ERRASURE) Fault‐tolerant Linear Algebra: Goals and Methods.
PDGEMM‐SUMMA
A B * + C C For k = 1:nb:n, End For PDGEMM :
PDGEMM‐SUMMA ABFT‐PDGEMM‐SUMMA
A B * + C C For k = 1:nb:n, End For ABFT‐PDGEMM :
0 0.5 1 1.5 2 2.5 3 3.5 4 64 81 100 121 256 454 0 0.5 1 1.5 2 2.5 3 3.5 4 64 81 100 121 256 454 4000 3000 2000 1000 # of processors # of processors GFLOPs/sec/proc GFLOPs/sec/proc
0 0.5 1 1.5 2 2.5 3 3.5 64 81 100 121 256 484 FT off FT on FT on,1 fault GFLOPs/sec/proc # of processors
0 0.5 1 1.5 2 2.5 3 3.5 64 81 100 121 256 484 FT off FT on FT on,1 fault 0 200 400 600 800 1000 1200 1400 1600 64 81 100 121 256 484 GFLOPs/sec/proc # of processors # of processors GFLOPs/sec
0 0.5 1 1.5 2 2.5 3 3.5 4 100 400 1024 Model SUMMA Model ABFT Measured SUMMA Measured ABFT # of processors GFLOPs/sec/proc
0 0.5 1 1.5 2 2.5 3 3.5 4 4 100 400 1024 nloc = 60,000 SUMMA nloc = 40,000 SUMMA nloc = 30,000 SUMMA nloc = 20,000 SUMMA nloc = 10,000 SUMMA # of processors GFLOPs/sec/proc
0 0.5 1 1.5 2 2.5 3 3.5 4 4 100 400 1024 nloc = 60,000 SUMMA nloc = 60,000 ABFT nloc = 40,000 SUMMA nloc = 30,000 ABFT nloc = 30,000 SUMMA nloc = 30,000 ABFT2 nloc = 20,000 SUMMA nloc = 20,000 ABFT nloc = 10,000 SUMMA nloc = 10,000 ABFT # of processors GFLOPs/sec/proc ABFT represents the only known alternaFve to address fault tolerance in strong scalability
0 0.5 1 1.5 2 2.5 3 3.5 4 4 100 400 1024 nloc = 60,000 SUMMA nloc = 60,000 ABFT nloc = 40,000 SUMMA nloc = 30,000 ABFT nloc = 30,000 SUMMA nloc = 30,000 ABFT2 nloc = 20,000 SUMMA nloc = 20,000 ABFT nloc = 10,000 SUMMA nloc = 10,000 ABFT # of processors GFLOPs/sec/proc ABFT represents the only known alternaFve to address fault tolerance in strong scalability
– Important for n/n operaFons (e.g. FFT) – Important for MM with small n
– No need to guess parameters
– No need for explicit synchronizaFon for example
– CancellaFon, ill‐condiFonned matrices, etc.