Preliminary Investigations on Resilient Parallel Numerical Linear - - PowerPoint PPT Presentation
SIAM EX14 Workshop July 7, Chicago - IL Preliminary Investigations on Resilient Parallel Numerical Linear Algebra Solvers Luc Giraud joint work with E. Agullo , P. Salas , E. F. Yetkin , M. Zounon funded by ANR RESCUE and G8-ECS HiePACS Inria
joint work with
funded by ANR RESCUE and G8-ECS
HiePACS Inria Project Joint Inria-CERFACS lab INRIA Bordeaux Sud-Ouest
2/ 25
Resilience: Ability to compute a correct output in presence of faults
◮ Context: Numerical linear algebra ◮ Goal: Keep converging in presence of fault ◮ Method: Recover-restart strategy without Checkpoint ◮ HPC systems are not fault-free ◮ A faulty components (node, core, memory) loses
all its data
◮ Simulations at exascale have to be resilient
3/ 25
Faults in HPC Systems
4/ 25
Faults in HPC Systems
◮ Mean Time Between Failure (MTBF): less than one hour ◮ Checkpoint time might be larger than MTBF
5/ 25
Faults in HPC Systems
◮ Mean Time Between Failure (MTBF): less than one hour ◮ Checkpoint time might be larger than MTBF
◮ Explore fault-tolerant schemes with less/no overhead ◮ Numerical algorithms to deal with overhead issue
5/ 25
Faults in HPC Systems
◮ Mean Time Between Failure (MTBF): less than one hour ◮ Checkpoint time might be larger than MTBF
◮ Explore fault-tolerant schemes with less/no overhead ◮ Numerical algorithms to deal with overhead issue
◮ Detected corrupted memory space (node crashes, damaged
5/ 25
Sparse linear systems
6/ 25
Sparse linear systems
7/ 25
x b A =
◮ Stationary methods (Jacobi, Gauss-Seidel, . . . ) ◮ Krylov subspace methods (CG, GMRES, Bi-CGStab, . . . ) ◮ Krylov methods have attractive potential for Extreme-scale
Interpolation methods
8/ 25
Interpolation methods
9/ 25
x b A P P P P
1 2 3 4
=
◮ Static data ◮ Dynamic data
Interpolation methods
9/ 25
x b A P P P P
1 2 3 4
=
◮ Static data ◮ Dynamic data
Interpolation methods
9/ 25
x b A P P P P
1 2 3 4
=
◮ Static data ◮ Dynamic data
Interpolation methods
9/ 25
b A P P P P
1 2 3 4
Static data Dynamic data
=
◮ Static data ◮ Dynamic data
Interpolation methods
9/ 25
b A P P P P
1 2 3 4
Static data Dynamic data
=
◮ Static data ◮ Dynamic data
Interpolation methods
9/ 25
b A P P P P
1 2 3 4
Static data Dynamic data Lost data
=
◮ Static data ◮ Dynamic data
Interpolation methods
9/ 25
b A P P P P
1 2 3 4
Static data Dynamic data Lost data
=
◮ Static data ◮ Dynamic data
◮ Failed processor is replaced ◮ Static data are restored
Interpolation methods
9/ 25
b A P P P P
1 2 3 4
Static data Dynamic data Lost data
=
◮ Static data ◮ Dynamic data
◮ Failed processor is replaced ◮ Static data are restored
Interpolation methods
9/ 25
b A P P P P
1 2 3 4
Static data Dynamic data Lost data Interpolatedv data
=
◮ Static data ◮ Dynamic data
◮ Failed processor is replaced ◮ Static data are restored
Interpolation methods
Overview of our fault tolerant algorithm
10/ 25
P P P P
1 2 3 4
Time
◮ Sequential simulations ◮ Simulation of parallel
environment
Interpolation methods
Overview of our fault tolerant algorithm
10/ 25
P P P P
1 2 3 4
Time
Fault
◮ Sequential simulations ◮ Simulation of parallel
environment
◮ Generation of fault trace ◮ Realistic probability distribution
Interpolation methods
Overview of our fault tolerant algorithm
10/ 25
P P P P
1 2 3 4
Time
Fault Successful iteration
◮ Sequential simulations ◮ Simulation of parallel
environment
◮ Generation of fault trace ◮ Realistic probability distribution
Interpolation methods
Overview of our fault tolerant algorithm
10/ 25
P P P P
1 2 3 4
Time
Fault Successful iteration
◮ Sequential simulations ◮ Simulation of parallel
environment
◮ Generation of fault trace ◮ Realistic probability distribution
Interpolation methods
Overview of our fault tolerant algorithm
10/ 25
P P P P
1 2 3 4
Time
Fault Successful iteration Failed iteration
◮ Sequential simulations ◮ Simulation of parallel
environment
◮ Generation of fault trace ◮ Realistic probability distribution
Interpolation methods
Overview of our fault tolerant algorithm
10/ 25
P P P P
1 2 3 4
Time
Fault Successful iteration Failed iteration Interpolation
◮ Sequential simulations ◮ Simulation of parallel
environment
◮ Generation of fault trace ◮ Realistic probability distribution
Interpolation methods
Overview of our fault tolerant algorithm
10/ 25
P P P P
1 2 3 4
Time
Fault Successful iteration Failed iteration Interpolation Restart
◮ Sequential simulations ◮ Simulation of parallel
environment
◮ Generation of fault trace ◮ Realistic probability distribution
Interpolation methods
Overview of our fault tolerant algorithm
10/ 25
P P P P
1 2 3 4
Time
Fault Successful iteration Failed iteration Interpolation Restart
◮ Sequential simulations ◮ Simulation of parallel
environment
◮ Generation of fault trace ◮ Realistic probability distribution
Interpolation methods
Overview of our fault tolerant algorithm
10/ 25
P P P P
1 2 3 4
Time
Fault Successful iteration Failed iteration Interpolation Restart
◮ Sequential simulations ◮ Simulation of parallel
environment
◮ Generation of fault trace ◮ Realistic probability distribution
Interpolation methods
Overview of our fault tolerant algorithm
10/ 25
P P P P
1 2 3 4
Time
Fault Successful iteration Failed iteration Interpolation Restart
◮ Sequential simulations ◮ Simulation of parallel
environment
◮ Generation of fault trace ◮ Realistic probability distribution
Interpolation methods
Overview of our fault tolerant algorithm
10/ 25
P P P P
1 2 3 4
Time
Fault Successful iteration Failed iteration Interpolation Restart
◮ Sequential simulations ◮ Simulation of parallel
environment
◮ Generation of fault trace ◮ Realistic probability distribution
Interpolation methods
11/ 25
Interpolation methods
11/ 25
Interpolation methods
[Langou, Chen, Bosilca, Dongarra, SISC, 2007]
11/ 25
Interpolation methods
[Langou, Chen, Bosilca, Dongarra, SISC, 2007]
x
11/ 25
Interpolation methods
12/ 25
Interpolation methods
[LI might not be defined for non-SPD matrices as diagonal blocks might be singular]
12/ 25
Interpolation methods
[LI might not be defined for non-SPD matrices as diagonal blocks might be singular]
12/ 25
Numerical experiments
13/ 25
Numerical experiments
1e-11 1e-10 1e-09 1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 140 280 420 560 700 840 980 1120 1260 1400
||(b-Ax)||/||b|| Iteration
Reset LI LSI SC REF
Figure: 4 faults
14/ 25
Numerical experiments
1e-11 1e-10 1e-09 1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 140 280 420 560 700 840 980 1120 1260 1400
||(b-Ax)||/||b|| Iteration
Reset LI LSI SC REF
Figure: 8 faults
14/ 25
Numerical experiments
1e-11 1e-10 1e-09 1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 140 280 420 560 700 840 980 1120 1260 1400
||(b-Ax)||/||b|| Iteration
Reset LI LSI SC REF
Figure: 17 faults
14/ 25
Numerical experiments
1e-11 1e-10 1e-09 1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 140 280 420 560 700 840 980 1120 1260 1400
||(b-Ax)||/||b|| Iteration
Reset LI LSI SC REF
Figure: 40 faults
14/ 25
Numerical experiments
1e-11 1e-10 1e-09 1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 98 196 294 392 490 588 686 784 882 980
||(b-Ax)||/||b|| Iteration
Reset LI LSI SC REF
Figure: 3 % data lost
15/ 25
Numerical experiments
1e-11 1e-10 1e-09 1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 98 196 294 392 490 588 686 784 882 980
||(b-Ax)||/||b|| Iteration
Reset LI LSI SC REF
Figure: 0.8 % data lost
15/ 25
Numerical experiments
1e-11 1e-10 1e-09 1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 98 196 294 392 490 588 686 784 882 980
||(b-Ax)||/||b|| Iteration
Reset LI LSI SC REF
Figure: 0.2 % data lost
15/ 25
Numerical experiments
1e-11 1e-10 1e-09 1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 98 196 294 392 490 588 686 784 882 980
||(b-Ax)||/||b|| Iteration
Reset LI LSI SC REF
Figure: 0.001 % data lost
15/ 25
Numerical experiments
◮ Recover-restart strategy ◮ When restarting, we lose the Krylov subspace built before the
◮ Consequence: delay of convergence due to restart ◮ Restarting mechanism is naturally implemented in GMRES to
◮ CG does not need to be restarted
16/ 25
Numerical experiments
1e-13 1e-12 1e-11 1e-10 1e-09 1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 83 166 249 332 415 498 581 664 747 830
A-norm(error) Iterations
Reset LI LSI SC
Figure: PCG on a 7-point stencil 3D Poisson equation with 70 faults - 5 % data lost
17/ 25
Numerical experiments
1e-13 1e-12 1e-11 1e-10 1e-09 1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 83 166 249 332 415 498 581 664 747 830
A-norm(error) Iterations
Reset LI LSI SC REF
Figure: PCG on a 7-point stencil 3D Poisson equation with 70 faults - 5 % data lost
17/ 25
Resilience in eigensolvers
18/ 25
Resilience in eigensolvers
19/ 25
Resilience in eigensolvers
x
19/ 25
Resilience in eigensolvers
20/ 25
x A = x
◮ λ : eigenvalue ◮ x : eigenvector ◮ (λ, x) : eigenpair
◮ Fixed Point Methods (Power Method, Subspace iteration) ◮ Subpace Methods (Jacobi-Davidson, Arnoldi, IRA/Krylov
Schur)
Resilience in eigensolvers
21/ 25
Resilience in eigensolvers
1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1 1 1e+01 24 48 72 96 120 144 168 192 216 240
||(Ax - lambda*x)||/||lambda|| Iteration
1 2 2 3 4 LSI REF
Figure: Jacobi-Davidson method with 5 faults - 1 % lost data. Convergence history using LSI and Checkpoint of current iterate
22/ 25
Concluding remarks and perspectives
23/ 25
Concluding remarks and perspectives
◮ We have designed techniques to interpolate meaningfull lost
◮ Our techniques preserve some of the key monotonicy of Krylov
◮ The restarting effect remains reasonable within the GMRES
◮ No fault, no overhead ◮ These techniques can be adpated to multiple faults ◮ What about silent soft-error - CGPOP preliminary
24/ 25