1 of 1

Dynamic AMR for a Relativistic Electron Drift-Kinetic Solver and �Scalable PETSc-p4est Implementation and Implicit Time Stepping

Research Details

We developed a new parallel data management (DM) in PETSc that interfaces AMR capabilities (via p4est library) with physics simulations that require adaptivity
AMR reduces simulation errors and computational cost by increasing the degrees of freedom only where needed
Dynamic AMR coarsens and refines meshes to adapt over time as the solution evolves through a dynamical processes
Fully-implicit time stepping (via PETSc TS) enables accurate solution of stiff dynamical systems

Scientific Achievement

New dynamic adaptive mesh refinement (AMR) in PETSc enables runaway electron simulations (Fokker-Planck PDE) at several orders of magnitude higher resolutions

Significance and Impact

New physical processes for runaway electrons can be simulated, hence new science problems are possible to better understand disruption processes in tokamaks / plasma fusion

ANL: Johann Rudi, Max Heldman, Emil Constantinescu

LANL: Qi Tang, Xianzhu Tang

Parallel octree-based AMR. Left: Forest-of-trees topology with 2 trees and leaves are cells of the mesh. Right: Space filling curve to sequentialize cells of mesh. (Credit: p4est)

Dynamic AMR in parallel. Each color represents one of 1024 MPI ranks.

The aggressive adaptivity required by the application results in 12 levels of difference in refinement, which corresponds to 3 orders of magnitude difference in cell size.

Technical Contacts:

Emil Constantinescu (econsta@anl.gov, ANL MCS)

Johann Rudi (jrudi@anl.gov, ANL MCS -> moved to Virginia Tech)

��Supported by: �TDS SciDAC (PI Xianzhu Tang)��Notes:

Adaptive mesh refinement (AMR) reduces simulation errors and computational cost by increasing the degrees of freedom only where needed
Dynamic AMR coarsens and refines meshes to adapt over time as the solution evolves through the dynamical processes
p4est is a parallel octree-based AMR library developed for aggressive adaptivity and extreme-scale applications (O(100,000) MPI ranks)
We developed a new data management (DM) in PETSc that interfaces p4est with applications that require adaptivity
Leverage PETSc’s scalability and built-in functionality:

composable linear solvers and preconditioners, nonlinear solvers, approximations of Jacobians for Newton’s method, coloring for finite difference gradients
advanced time integration in PETSc: implicit time stepping, adaptive time steps, multirate time stepping
parallel mesh and associated data managed by PETSc DM; discretization and physics are handled in application code

Runaway electron (Fokker-Planck PDE) simulations are implemented by using the new DM, leveraging direct access to fast algorithms in p4est

Future directions:

Preconditioning with algebraic and geometric multigrid (AMG & GMG)
Develop new conservative multirate implicit-explicit: multirate explicit address AMR and implicit deal with stiffness
Develop new refinement criteria
Add new physics to the PDE models to improve fidelity
Numerical schemes (interpolation) for conservation of all critical quantities
Profiling and performance analysis
Utilizing heterogenous architectures (GPUs) and portability across hardware platforms