eXtended Discrete Element Method (XDEM) Research on UL HPC platform - - PowerPoint PPT Presentation

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

eXtended Discrete Element Method (XDEM) Research on UL HPC platform

• A. Rousset
• X. Besseron
• W. Mainassara

LuXDEM Research Team, RUES

• Prof. Bernhard Peters

http://luxdem.uni.lu

UL HPC School 2017 June 12, 2017

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 1 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

XDEM = Extended Discrete Element Method

XDEM software is multi-physics simulation toolbox modeling granular materials and processes

Particle Motion

Snow, Sand, ...

Chemical Reactions

Coke, Iron ore, Biomass, ... + coupling external libraries: CFD with OpenFoam, FEM with Diffpack

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 2 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

Domains of Applications

• Storage and transport of granular material
• Mining and its machinery
• Agriculture and its machinery
• Processing industry: Fluidised beds, fixed and moving bed

reactors for

• Drying
• Thermal conversion (combustion, gasification)
• Processing of raw materials
• Pharmaceutical industry e.g. coating, drug production
• Food industry (transport, coating, processing)
• Material science
• Additive manufacturing

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 3 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

XDEM Research Examples

Blast furnace Rolling tire on snow

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 4 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

XDEM Workflow (Dynamic phase)

Distance/Overlap

Force, Torque Species fractions Heat flux, etc. Resultants on particle A Resultants on particle B Particle A at time t+Δt Particle B at time t New acceleration, velocity, position temperature, species concentration volume, mass, etc. for particle A New acceleration, velocity, position temperature, species concentration volume, mass, etc. for particle B Particle A at time t Particle B at time t+Δt

Timestep t Timestep t+Δt Interaction step

− Impact − Attraction − Bond − Radiation − Chemical reaction

Integration step

− Dynamics solvers − Conversion solvers

Interaction/Collision/Contact Integration Integration Update Update eXtended Discrete Element Method (XDEM) Research on UL HPC platform 5 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

Why using UL HPC facilities?

Granular dynamics applications

• Computation intensive = Time consumming
• Require the use of UL HPC facilities

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 6 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

Usage of UL HPC platform

XDEM model studies

• model validation (using parameter studies)
• execution of full simulations

Parallel/distributed code development

• Debugging, validation
• Scalability studies

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 7 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

XDEM - Parallel code design

Domain Decomposition

• Domain = whole simulation space
• Cell = regular subdivision with a fixed size
• Partition = subset of cells assiged to a given process

Partitioner and Load-Balancer

• Responsible for assigning a cell to a process
• Balance the workload
• Based on Orthogonal Recursive Bisection (ORB), Zoltan, PHG, etc...

MPI Parallel Simulation Driver

• Schedule the modules on all the processes
• Exchange the data between processes (Ghost particles in ghost cells)
• Independent of the module implementation
• Based on Message Passing Interface (MPI)

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 8 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

Testcase - Tire

Sinkage of a Tire

• Simulate the vertical motion
• f the tire tread
• The tire is a rigid surface
• f 18,707 triangles
• Soil layer is made of

43,624 sand grains

• Sand grains are described

as spherical particles

• f different radii

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 9 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

Testcase - Tire: Scalability

Parallel execution from 1 process to 64 processes (8 processes/node)

10 20 30 40 50 60 70 80 10 20 30 40 50 60 70 Iteration Time (s) Number of processes T1/N DPM execution

2 processes → 1.3X speedup 8 processes → 4.6X speedup 64 processes → 17.6X speedup

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 10 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

Testcase - Hopper

Hopper discharge

• Simulate hopper discharge
• The hopper is a rigid surface
• f 31,500 triangles
• Spherical particles
• f different radii

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 11 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion eXtended Discrete Element Method (XDEM) Research on UL HPC platform 12 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

Testcase - Hopper: Scalability

Parallel execution from 1 process to 48 processes (24 processes/node)

• 500

1000 10 20 30 40 50

Number of processes Simulation time[s]

ORB RIB RCB

Xdem parallel results

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 13 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

Testcase - Hopper discharge: Scalability

• 10

20 30 40 50 10 20 30 40 50

Number of processes Speed Up

ORB RIB RCB Ideal Speed Up

Xdem parallel results 48 processes → 21X speedup

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 14 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

Conclusion

Discrete Particle Method (DPM) sofware

• Multi-physics simulation toolbox
• Flexible and extensible design
• Parallel execution using MPI: 21x speedup with 48 processes

Future work

• Hydrid approach: MPI + (OpenMP or GPU)
• OpenMP or GPU: collision detection

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 15 / 16

XDEM HPC usages Enhanced Design Parallel Experiments/Results Conclusion

Thank you for your attention

eXtended Discrete Element Method (XDEM) Research on UL HPC platform 16 / 16