1 of 1

Analysis of GPU Data Access Patterns on Complex Geometries for D3Q19 Lattice Boltzmann Algorithm

Scientific Achievement

Examine memory access schemes for the Lattice Boltzmann method (LBM) on GPUs via empirical testing and find addressing and memory layout schemes performing better than state-of-the-art practices.

Significance and Impact

Present the first near-optimal strong results for LBM with arterial geometries run on GPU systems (Titan and Summit), leading to an increased computational speed and memory reductions.

G. Herschlag, S. Lee, J. S. Vetter, and A. Randles. Analysis of GPU Data Access Patterns on Complex Geometries for D3Q19 Lattice Boltzmann Algorithm. IEEE Transactions on Parallel and Distributed Systems (TPDS), Vol. 32, No. 10, 2021

Resolution versus Performance on NVIDIA GPUs (K40, P100, V100) for the Aorta Geometry. Results show that semi-direct methods typically outperforms indirect methods, and locally-direct addressing is consistently outperformed by indirect and semi-direct addressing.

Technical Approach

Examine the computational cost of different data storage strategies for solving LBM on complex geometries with GPUs.
Find strong evidence that semi-direct addressing is superior for arterial and porous media geometries, and the CSoA memory layout consistently provides computational acceleration with minimal coding effort and negligible memory increase.

TALKING POINTS:

GPU performance of the lattice Boltzmann method (LBM) depends heavily on memory access patterns.
When implemented with GPUs on complex domains, typically, geometric data is accessed indirectly and lattice data is accessed lexicographically.
Although there are a variety of other options, no study has examined the relative efficacy between them.
Here, researchers at Oak Ridge National Laboratory and Duke University examine a suite of memory access schemes via empirical testing and performance modeling.
We find that our recommended addressing and memory layout schemes lead to better performance than state-of-the-art practices.
We demonstrate that our results hold across multiple GPUs on two leadership class systems (Titan and Summit), and present the first near-optimal strong results for LBM with arterial geometries run on various GPUs (NVIDIA K40, P100, and V100).
A paper on the preliminary evaluation of this study is published in the IEEE Transactions on Parallel and Distributed Systems (TPDS) journal.

METADATA:

Project: The RAPIDS2 Institute, Scientific Discovery through Advanced Computing (SciDAC) Program.

PI name: Robert B. Ross (ANL)
Name of the program manager: ASCR research

CITATIONS:

G. Herschlag, S. Lee, J. S. Vetter, and A. Randles. Analysis of GPU Data Access Patterns on Complex Geometries for D3Q19 Lattice Boltzmann Algorithm. IEEE Transactions on Parallel and Distributed Systems (TPDS), Vol. 32, No. 10, 2021

DOI: 10.1109/TPDS.2021.3061895