1 of 23

VTK-M

VISUALIZATION FOR THE EXASCALE ERA AND BEYOND

SAND2023-07725C

THE PREMIER CONFERENCE & EXHIBITION ON COMPUTER GRAPHICS & INTERACTIVE TECHNIQUES

2 of 23

ACKNOWLEDGEMENTS

This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under Award Numbers 10-014707, 12-015215, and 14-017566.
This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of two U.S. Department of Energy organizations (Office of Science and the National Nuclear Security Administration) responsible for the planning and preparation of a capable exascale ecosystem, including software, applications, hardware, advanced system engineering, and early testbed platforms, in support of the nation’s exascale computing imperative.
Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC., a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA-0003525.
Thanks to many, many partners in labs, universities, and industry.

2

3 of 23

TERMINOLOGY

FLOP – Floating Point Operations per Second
teraflop – 10¹² floating point operations per seconds
petaflop – 10¹⁵ FLOPS
exaflop – 10¹⁸ FLOPS
Exascale – A system capable of achieving an exaflop (measured by the HPLinpack benchmark)

For Comparison:

Current generation AMD/Intel CPUs – 100s of gigaflops
Current generation AMD/NVidia GPUs - ~10s - 100 teraflops

3

Let’s have some common terminology so that we’re all on the same page.

<click>

A FLOP is a single floating point operation per second, e.g., addition or multiplication. Now, to the rest of the world a FLOP can mean different things, in our world of HPC, it mostly means in double-precision. So, none of this namby-pamby 16-bit floats used for AI and ML, but, revel in 64-bit gloriousness.

A teraflop is a trillion floating point operations per second, or 10^12 flops, A petaflop is 10^15 flops, and even though flop is an acronym, you never see it capitalized. An exaflop is 10^18 flops, and an exascale system is one that can sustain computations at a rate of > than an exaflop. In order to achieve that magic exascale designator, the determination is through the HPLinpack benchmark. It’s not the best measure of performance, but hey, it’s what we use.

<click>

To give you an idea of an exaflop, most modern CPUs are in the 100’s of gigaflops, and GPUs, in double-precision, are in the 10s to 100s of teraflops.

4 of 23

DOE EXASCALE CLASS COMPUTING SYSTEMS

AURORA (ARGONNE)

FRONTIER (OAK RIDGE)

4

9472 AMD Epyc CPUs (>600,000 cores)
37,888 AMD MI250X GPUs (> 8.3 million cores)
74 Cabinets; Power consumption 21 MW
World’s first exascale computer

1.1 exaflops sustained
1.67 exaflops peak

21,248 Intel Xeon CPU Max Series
63,744 Intel Data Center GPU Max Series
Theoretical Peak: > 2 exaflops
Fun facts:

300 miles of optical cable
44,000 gallons of water for cooling
Weighs > 600 tons

The US Dept of Energy has a long history of supporting HPC. In 2016 the Exascale Computing Project was founded to create a “capable exascale system”, which meant a system roughly 50x faster than the current state of the art with power consumption in the 10’s of megawatts (While that’s a huge power draw, extrapolating from the current generation of H/W at that time, the power consumption would have been in the 10s – 100s of gigawatts.

<click>

Counting of nodes on Frontier is a little weird. The numbers above are for the full system specs. For exascale, it’s really 9408 compute nodes, and 37632 AMD GPUs. 1 Frontier compute node has 1 Epyc CPU, and 4 GPUs. But each GPU consists of 2 Graphics Compute Dies (GCD), so a node presents itself has having 8 GPUs. Later in the talk you’ll see results referring to 70k+ GPUs, and that’s counting each GCD as a GPU.

<click>

Here's some statistics about Aurora. The final system was assembled just last month, so we have a bit to wait before we get to run on the full system

5 of 23

ORIGINS OF VTK-M

Goals

A single place for the visualization community to collaborate, contribute, and leverage massively threaded algorithms.
Reduce the challenges of writing highly concurrent algorithms by using data parallel algorithms.
Make it easier for simulation codes to take advantage these parallel visualization and analysis tasks on a wide range of current and next-generation hardware.

5

6 of 23

ORIGINS OF VTK-M

Combined the strengths of three different projects

Extreme-scale Analysis and Visualization Library (EAVL), ORNL

New mesh layouts, memory efficiency, parallel algorithms, zero-copy for In-Situ support

Data Analysis Toolkit for Extreme Scale (Dax), SNL

ParaView plugin, large volumes through streaming to threaded filters

Piston, LANL

Data-parallel algorithms for many/multi-core, in-situ focused

6

7 of 23

VTK-M ARCHITECTURE

7

Use

Develop

Research

Data Model

Filters

Worklets

Device Algorithms

Execution

Arrays

Piston

EAVL

Dax

EAVL

Piston

Dax

Piston

Dax

EAVL

8 of 23

8

Contour

Streams

Clip

Render

CUDA

Xeon Phi

CPU

AMD ROCm

Intel GPU

Surface

Normals

Ghost Cells

Warp

…

Vis algorithms, isosurfacing, clip planes, … are written to the VTK-m API, and VTK-m takes care of the data transfer and scheduling on many-core architecures. Our initial targets were NVidia GPUs using CUDA (the leading supercomputer at that time was Summit, an IBM Power 9 system with NVidia GPUs), the Xeon Phi using Intel's threaded building blocks, and CPUs using either Intel TBB, or pthreads.

ECP then announced the architectures for the two proposed exascale systems, and surprise, they weren’t anything the team had anticipated, with the two systems being based on AMD GPUs, frontier, and Intel GPUs, Aurora. The team did not have the man power to generate two completely new backends, but the Kokkos team at Sandia was in the process of developing a C++ library for executing generic C++ on these new systems. So the team decided to go all in and build a Kokkos backend for any new architecture that came our way.

9 of 23

VTK-M INTERNALS

THE PREMIER CONFERENCE & EXHIBITION ON COMPUTER GRAPHICS & INTERACTIVE TECHNIQUES

10 of 23

MCD³ ARCHITECTURE

Meta-DPPs: which are parallel processing patterns that involve one or more DPPs. The word choice of “meta” is meant to evoke its definition of “denoting something of a higher or second-order kind.”
Convenience routines: which encapsulate common operations for scientific visualization.
DPPs: which provide parallel processing patterns.
Data management: which insulates algorithms from data layout complexities. These complexities range from how data is organized (e.g., structure-of-arrays vs array-of-structures) to different types of meshes (e.g., unstructured, rectilinear, etc.) to different memory spaces (e.g., host memory, device memory, or unified managed memory)
Devices: which enable code to run on a given hardware architecture.

10

K. Moreland, R. Maynard, D. Pugmire, A. Yenpure, A. Vacanti, M. Larsen, and H. Childs. Minimizing Development Costs for Efficient Many-Core Visualization Using MCD3. Parallel Computing, 108:102834, Dec. 2021.

Guy Blelloch’s dissertation considered parallel

vector models as a way to unify theory, language, and architecture.

In his thesis, he introduced two important primitives,

scan and segmented instructions, and demonstrated that diverse tasks could be parallelized using these primitives. These two primitives are part of a larger group referred to as

“data-parallel primitives” (DPPs).

DPPs operate on arrays (vectors) of data. They need to operate quickly when supplied with sufficient parallelism — if

an array is of size N and if there are at least N cores, then a DPP needs to complete in O(log N) time. Many operations can achieve this time bound, including Reduce, Scan, Transform, and Sort.

The MCD3 design relies on the five constructs listed here

Meta-DPPs are effectively templates for parallel processing patterns, while convenience routines are essentially sub-routines.

Meta- DPPs and convenience routines work through a data management layer to call actual DPPs. DPPs then provide hardware-specific implementations through the device layer.

11 of 23

MAP FIELD

11

Functor

Meta-DPPs are the most important part of the MCD3 system. They identify what type of data to parallelize over, cells, points, or over elements in an array.

Meta-DPPs then provide infrastructure that organizes input data (arrays or data sets) for execution on many-core devices based on the parallelization. On the algorithm developer side, the developer provides a functor to carry out the operation for a single instance. For example, if the meta-DPP parallelizes over cells, then an algorithm developer would write a functor that operates on a single cell.

Map Field parallelizes over the elements in a field array.

This method can take any number of arrays as input and any number of arrays as output.

Conceptually, the operation of Map Field is quite simple.

This meta-DPP calls the functor once for every element in the

array as shown here. Values from the appropriate index

are pulled from the arrays and provided to the functor rather

than giving the functor direct access to the arrays.

In terms of utility, Map Field is used to implement mathematical

expressions on fields, for example finding the magnitude

of a vector field.

12 of 23

POINT NEIGHBORHOOD

12

Functor

13 of 23

REDUCE BY KEY

13

Functor

14 of 23

VISIT POINT WITH CELLS

14

Functor

15 of 23

VISIT CELL WITH POINTS

15

Functor

16 of 23

DOES IT SCALE?

16

17 of 23

18 of 23

ACCELERATING PARAVIEW

18

19 of 23

ACCELERATING POINCARÉ PLOTS

19

20 of 23

VOLUME RENDERING WITH SHADOWS

20

M. Mathai, M. Larsen, and H. Childs. A Distributed-Memory Parallel Approach for Volume Rendering with Shadows. To appear at the IEEE Symposium for Large Data Analysis and Visualization (LDAV), October 2023.

21 of 23

VOLUME RENDERING WITH SHADOWS

21

M. Mathai, M. Larsen, and H. Childs. A Distributed-Memory Parallel Approach for Volume Rendering with Shadows. To appear at the IEEE Symposium for Large Data Analysis and Visualization (LDAV), October 2023.

22 of 23

RENDERING AT SCALE ON FRONTIER

22

80 trillion cells
9400 nodes
74,088 GPUs
Render Time: 300ms!

23 of 23

CONCLUSIONS

VTK-m has been proven that it can run at scale
VTK-m is currently the only software that is capable of visualizing simulations on exascale class systems
DoE is currently starting the process for the next set of supercomputers, based on AI/ML for physics
Currently, ParaView/VTK have ~10 filters with VTK-m overrides

Interested in working at the extreme-edge of computing and graphics?

Send a team member a resume, or apply to the intern programs at any of the labs

23