1 of 1

Analysis of GPU Data Access Patterns on Complex Geometries for D3Q19 Lattice Boltzmann Algorithm

Scientific Achievement

We examine a suite of memory access schemes for the Lattice Boltzmann method (LBM) on GPUs via empirical testing and performance modeling and find that our recommended addressing and memory layout schemes lead to better performance than state-of-the-art practices.

Significance and Impact

We present the first near-optimal strong results for LBM with arterial geometries run on GPUs, and we also demonstrate that the proposed recommendations hold across multiple GPUs on two leadership class systems (Titan and Summit), leading to an increased computational speed and memory reductions.

G. Herschlag, S. Lee, J. S. Vetter, and A. Randles. Analysis of GPU Data Access Patterns on Complex Geometries for D3Q19 Lattice Boltzmann Algorithm. IEEE Transactions on Parallel and Distributed Systems (TPDS), Vol. 32, No. 10, October 2021

Resolution versus Performance on NVIDIA GPUs (K40, P100, V100) for the Aorta Geometry. Results show that semi-direct methods typically outperforms indirect methods, and locally-direct addressing is consistently outperformed by indirect and semi-direct addressing.

Research Details

Examine the computational cost of different data storage strategies for solving LBM on complex geometries with GPUs.
Find strong evidence that semi-direct addressing is superior for arterial and porous media geometries, and the CSoA memory layout consistently provides computational acceleration with minimal coding effort and negligible memory increase.

an Office of Basic Energy Sciences Energy Frontier Research Center

Notes: GPU performance of the lattice Boltzmann method (LBM) depends heavily on memory access patterns. When implemented with GPUs on complex domains, typically, geometric data is accessed indirectly and lattice data is accessed lexicographically. Although there are a variety of other options, no study has examined the relative efficacy between them. Here, researchers at Oak Ridge National Laboratory and Duke University examine a suite of memory access schemes via empirical testing and performance modeling. We find that our recommended addressing and memory layout schemes lead to better performance than state-of-the-art practices. We demonstrate that our results hold across multiple GPUs on two leadership class systems (Titan and Summit), and present the first near-optimal strong results for LBM with arterial geometries run on various GPUs (NVIDIA K40, P100, and V100).

A paper on the preliminary evaluation of this study is published in the IEEE Transactions on Parallel and Distributed Systems (TPDS) journal.