ACCELERATING CFD AND RESERVOIR SIMULATIONS WITH ALGEBRAIC MULTI GRID Chris Gottbrath, Nov 2016
Challenges What is Algebraic Multi-Grid (AMG)? Why use AMG? AGENDA When to use AMG? NVIDIA AmgX Results 2
CHALLENGES Computational Fluid Dynamics and Reservoir Simulations Large Volumes Complex Geometry High Velocities Turbulence Multiple fluids 3
ACCURACY & PERFORMANCE MATTERS Quality Matters Designing critical systems Oil field management -- $$$ Time to solution matters Limits size and accuracy Limits coverage 4
WHAT IS AMG – ALGEBRAIC MULTI GRID A way to efficiently solve very large systems System: Ax – b = 0 Start with a proposed solution vector x Iterative process – reduces |r| A = matrix - system Trick: approximate solutions x,b = vector - state Make a small system “based on” the big one Residual Function : r= b-Ax Solve that to get a good approximation Project back to full system Fast, powerful, general technique! 5
ALGEBRAIC MULTI GRID – THE V CYCLE Full Size Full Size System System Coarsen + Prolong+ Smooth Smooth Medium Medium System System Coarsen + Prolong+ Smooth Smooth Tiny Tiny Solve System System Repeat a few times 6
Coarsening 7
Coarsening 8
Why use AMG? A Powerful Solver Handles Complex Geometries Complex Physics Huge Systems High Resolution Fast Algorithm – Each iteration reduces |r| by 2-10x. Converge with 6-20 iterations It runs really well on NVIDIA GPUs! 9
NVIDIA AmgX > 15x Speedup vs HYPRE Unstructured Implicit Linear Systems - Solved HYPRE K40 M40 P100-SXM2 The AmgX library provides a configurable 18x 17x 16x 15x 16x and scalable GPU accelerated algebraic 15x 14x 14x Speedup vs. HYPRE 14x multi-grid solver for large sparse linear 12x 10x systems. 8x 6x CFD and Reservoir Simulation 4x 2x 0x Scales up to hundreds of GPUs Rich collection of algorithms Accelerate existing simulations • Florida Matrix Collection; Total Time to Solution • HYPRE AMG Package (http://acts.nersc.gov/hypre) on Intel Xeon E5-2697 v4@2.3GHz, 3.6GHz Turbo, Hyperthreading off • AmgX on K40m, M40, P100; Base clocks • Host system: Intel Xeon Haswell single-socket 16-core E5-2698 v3 @ 2.3GHz, 3.6GHz Turbo • CentOS 7.2 x86-64 with 128GB System Memory 10 https://developer.nvidia.com/amgx
ANSYS Mechanical 16.0 on Tesla K80 THE Industry Standard CFD is accelerated with AmgX Simulation productivity (with HPC Pack) 371 V15sp-4 Model V15sp-5 Model ANSYS Mechanical jobs/day 2.3x 247 1.8x Higher Ball Grid Array is geometry Turbine geometry Better 159 3,200,000 DOF 6,000,000 DOF 135 Static, nonlinear SOLID187 FEs Distributed ANSYS 16.0 Static, nonlinear Direct sparse solver Distributed ANSYS 16.0 Direct sparse solver 8 CPU cores 8 CPU cores 6 CPU cores + 6 CPU cores + K80 GPU K80 GPU V15sp-5 Model V15sp-4 Model 11 Distributed ANSYS Mechanical 16.0 with Ivy Bridge (Xeon E5-2697 V2 2.7 GHz) 8-core CPU and a Tesla K80 GPU.
AmgX in Reservoir Simulation Application Time (seconds) 1500 1150 1000 Lower is Better 500 197 98 0 CPU GPU AmgX 3-phase Black Oil Reservoir Simulation. 400K Custom grid blocks solved fully implicitly. CPU: Intel Xeon CPU E5-2670 GPU: NVIDIA Tesla K10 12
Minimal Example With Config //One header AMGX_vector_create(&x,res,mode); solver(main)=FGMRES main:max_iters=100 #include “ amgx_c.h ” AMGX_vector_create(&b,res,mode); main:convergence=RELATIVE_MAX //Read config file //Read coefficients from a file main:tolerance=0.1 main:preconditioner(amg)=AMG AMGX_create_config(&cfg, cfgfile); AMGX_read_system(&A,&x,&b, amg:algorithm=AGGREGATION matrixfile); amg:selector=SIZE_8 //Create resources based on config amg:cycle=V //Setup and Solve Loop amg:max_iters=1 AMGX_resources_create_simple(&res, cfg); AMGX_solver_setup(solver,A); amg:max_levels=10 amg:smoother(smoother)=BLOCK_JACOBI //Create solver object, A,x,b, set AMGX_solver_solve(solver, b, x); amg:relaxation_factor= 0.75 precision amg:presweeps=1 //Download Result amg:postsweeps=2 AMGX_solver_create(&solver, res, amg:coarsest_sweeps=4 mode, cfg); AMGX_download_vector(&x) determinism_flag=1 AMGX_matrix_create(&A,res,mode); 13
AmgX GPU acceleration for your simulation Flexible & powerful technique Simple to adopt Get results 15x faster with Pascal Scales out to handle large systems http://developer.nvidia.com/amgx 14
Thanks! Questions? http://developer.nvidia.com/amgx cgottbrath@nvidia.com
Numerical Models There are a lot of different ways to model AmgX is only used for PDE type models Event Based PDE based models Models It works best with unstructured implicit code. Grid • • Lattice Other code can be • Structured Boltzman Accelerated using the CUDA Unstructured • • Monte Carlo libraries such as • Method Finite State • Implicit • Machine cuRAND, cuBLAS, cuFFT, Explicit • • N-body / SPH cuSPARSE, and cuSOLVER 16
Second Order PDEs Hyperbolic Many physical problem domains can Use Other be modeled using 2 nd order PDE Solvers These can be classified based on Pure Hyperbolic how they behave mathematically • Parabolic – Solutions smooth out over time. • Heat transfer Elliptic – Smooth within volume, • Use AmgX potentially discontinuous boundary values. Helmholtz Subsonic fluid flow • Hyperbolic – Discontinuities • Parabolic Elliptic (shocks) will persist • Wave equation Helmholtz problems are currently Supersonic fluid flow • not well suited 17
KEY FEATURES Classical and Aggregation AMG High level API with composition through configuration Robust Aggressive Coarsening MPI support with consolidation Algorithms Multi-precision Krylov methods: CG, GMRES, BiCGStab, IDR Adaptors for: HYPRE, PETSc, Trilinos Smoothers / Solvers Python bindings Jacobi-L1,Block-Jacobi, ILU[0,1,2], DILU, Dense LU, KPZ- Polynomial, Chebyshev 18
Recommend
More recommend
