Doubt and Redundancy Kill Soft Errors �Towards Detection and Correction of Silent Data Corruption in Task-based Numerical Software�

Philipp Samfass¹, Tobias Weinzierl², Anne Reinarz², Michael Bader¹

¹Technical University of Munich, Germany�²Durham University, UK

This project has received funding from the ExCALIBUR grants ExaClaw (EP/V00154X/1) and the project Exposing Parallelism: Task Parallelism (grant ESA 10 CDEL).

Motivation

• Decreasing reliability with increasing system scale
• Undetected soft errors: silent data corruption (SDC)
• Problem for numerical simulations: SDC hard to spot
• Replication possible, but typically expensive
• Non-functional requirements for resilience:
• Little synchronization
• Little additional computation and communication
• Little I/O
• Little additional memory consumption

​

​

​

​

​

​

​

2

Reinarz et al.,

Influence of A-Posteriori Subcell Limiting on Fault Frequency in Higher-Order DG Schemes, FTXS ‘18

Context and Research Goal

• ChEESE flagship code ExaHyPE
• ExaHyPE:
• Engine for solving systems of hyperbolic PDEs:
• Explicit global time stepping on tree-structured adaptive meshes
• ADER-DG numerical algorithm
• MPI+TBB parallel, task-based implementation
• Goal: detect and correct silent errors at as little as possible performance costs

​

​

​

​

​

​

​

​

3

Outline and Main Ingredients

• Transparent process replication (two-fold) as asynchronously running teams with teaMPI¹
• Task-based error criteria: how likely?
• Task outcome sharing¹ between replica ranks
• Comparisons between „dubious“ and redundantly computed task outcomes

​

​

4

TeaMPI‘s logo

¹ Samfass et al., TeaMPI—Replication-Based Resilience Without the (Performance) Pain, ISC ‘20

Combine these ingredients

for task-based error detection and correction

without full overhead of redundant computation!

TeaMPI

5

• PMPI wrapper library for application-transparent replication as teams
• Teams:
• Group of ranks
• Run single application instance
• Independent of each other
• No synchronization between teams, no lock-stepping
• Asynchronous communication between teams for outcome sharing

​

​

Task Outcome Sharing

6

Algorithmic framework

• Independent tasks
• Yield task outcome for some input data
• Silent error yields flawed outcome
• Error criterion :
• : no error
• : outcome likely correct
• : outcome is clearly wrong
• : is more likely correct

​

​

​

​

​

You audience will listen to you or read the content, but won’t do both.

​

7

Algorithmic framework (cnt’d)

• Combine multiple error criteria with tolerances
• Error indicator (boolean): is outcome dubious?
• Local cache of task outcomes and error criterion values:
• Buffers task outcomes from other team
• Buffers local task outcomes until we know they are valid (i.e., no SDC has occurred)

​

​

​

​

8

Task Execution Control Flow

• Case 1: matching outcome not available from other team

​

​

​

9

yes

no

Task Execution Control Flow

• Case 2: matching outcome is available from other team

​

​

​

10

yes

no

Checking and Correcting Task Outcomes

11

no

yes

Realization in ExaHyPE

• ADER-DG in a nutshell:
• Space-time predictor (STP): polynomial solution approximation locally per cell
• Riemann solves: influence between neighbouring cells
• Corrector: combines Riemann + STP to new solution
• Focus on resilience for STP tasks only:
• Compute-intense
• Consume bulk part of CPU time
• Independent tasks
• Not immediately time-critical

​

​

12

Benchmark¹: Warm Rising Bubble (Compressible Navier Stokes)

​

​

• Perturbation in the potential temperature field over a background state in hydrostatic balance

​

13

¹ Krenz et al.: A High-Order Discontinuous Galerkin Solver with Dynamic Adaptive Mesh Refinement to Simulate Cloud Formation Processes, PPAM ‘19

Error criteria

• Arithmetic corruption ( ):
• Checks for NaN values
• Boolean criterion
• Define if outcome contains NaN values
• otherwise
• Physical admissibility ( ):
• Checks for physical plausibility (e.g., negative density)
• Boolean criterion
• Define if outcome violates these checks
• otherwise

14

Error criteria (2)

• Dubious time step size changes ( ):
• Time step size depends on maximum eigenvalue
• Detects drastic changes in time step sizes/eigenvalue

​

​

​

15

Error criteria (3)

• Solution smoothness ( ):
• Based on second derivatives in interpolation points

​

​

• Summed up over directions:
• Large changes are suspicious
• Can happen “normally” for wave stiffening but is rare

​

​

​

​

16

Error Indicators & Combination of Error Criteria

• Rigorous evaluation
• Computes all criteria
• Dubious (i.e., ) if any criterion surpasses tolerance

​

​

• Lazy evaluation
• Evaluate lazily if any of the other “pre-filtering“ criteria are violated
• Dubious only if at least one pre-filtering criterion and are violated

​

17

• Prioritized comparison for selecting more likely outcome

​

​

Error Injection

• Manual error injection for deterministic studies
• Usually one error per run in random cell and time step
• Alter one randomly chosen coefficient in Lagrangian polynomial (i.e., in random sample point)
• Obtain flawed outcome
• Test different sizes of errors (positive and negative values)
• Sensitivity = #corrected runs / #total runs

18

Sensitivity

19

Sensitivity

20

Sensitivity

21

Sensitivity – Combined Criteria

22

Performance-Sensitivity Tradeoff

23

Upscaling of Different Configurations

24

Conclusion & Outlook

• Resilience with task-based error detection and correction
• Error criteria and indicators
• Only requires two-fold redundancy
• Local recovery
• Replication of data but only partially replicated computations
• Outlook
• Offload checking to smart NICs
• Dynamic tuning of tolerances

25