GPU Profiling and Tools
Jumanazarova Mardonbek
Plan:
In this chapter, we'll explore tools and various workflows that can be used to accelerate application development. We'll show you how GPU profiling tools can be useful. We'll also discuss overcoming the challenges of using profiling tools while running on a remote HPC cluster. Because profiling tools continue to evolve and improve, we'll focus on the methodology rather than the specifics of any one tool. The main takeaway from this chapter will be understanding how to create a productive workflow when using powerful GPU profiling tools.
1. Overview of profiling tools
Profiling tools allow you to speed up optimization, improve hardware utilization, and gain a deeper understanding of application performance and hotspots. We'll discuss how profiling tools reveal bottlenecks and help you improve hardware utilization. The following bulleted list lists the most commonly used GPU profiling tools. We've specifically shown NVIDIA's tools for use with their GPU processors because these tools have been around the longest. If your system has a GPU from a different vendor, consider substituting their tools into your workflow. Don't forget about standard Unix profiling tools, such as gprof, which we'll use later in Section 13.4.2.
We recommend consulting the examples provided in this chapter. The included source code is located at http://github.com/EssentialsOfParallelComputing/Chapter13, which shows installation examples of software packages for tools from various hardware vendors. There are detailed lists of all software and information that can be installed for each vendor. You'll likely want to install tools for the corresponding hardware.
1. Overview of profiling tools
NOTE: While a third-party tool may partially work on your system, its full functionality will be impaired.
1. Overview of profiling tools
2. How to choose a good workflow
Before tackling any complex task, you must choose the appropriate workflow, or workflow. You might be working on a website with an excellent connection, off-site with a slow home network, or somewhere in between. Each situation requires a different workflow. In this section, we'll discuss four potential and effective workflows for these different scenarios.
Figure 13.1 shows a visual representation of the four different workflows. Connection availability and speed are determining factors in deciding which method you ultimately use. You can run GUI tools directly on the system, remotely in client-server mode, or simply avoid problems by using command-line (console) tools.
Using profiling tools from a remote server often introduces significant latency in GUI rendering and responsiveness. Client-server mode separates the GUI so that it runs locally on your system. It then communicates with the server on the remote website to execute commands. This helps maintain the tool's graphical interface's responsiveness. For example, profiling tools like NVVP can experience high latency when used on a remote server.
2. How to choose a good workflow
Figure 13.1 There are several different methods of using profiling tools that provide alternatives for your application development situation.
2. How to choose a good workflow
Having to wait minutes after each mouse click isn't very productive. Fortunately, NVIDIA and many other tools provide several options for solving this problem. We'll explore the different workflows in more detail in the discussion below.
2. How to choose a good workflow
It's important to understand that even if you don't have a fast connection to a computing website, you have several options when using development tools. Regardless of the method you use for porting and performance analysis, you must ensure that the versions of the software and information systems you use match. This is especially important for CUDA and NVIDIA tools like nvprof and NVVP.
3. Sample Problem: Shallow Water Simulation
In this section, we'll work with a realistic example to demonstrate the source code migration process and the use of several tools. We'll use the problem from Figure 1.9, where a volcanic eruption or earthquake can trigger a tsunami. Tsunamis can travel thousands of miles across oceans with a height of just a few feet, but by the time they reach shore, they can reach hundreds of meters. These types of simulations are typically performed post-event due to the time required to set up and run the simulation. We'd prefer to simulate it in real time to be able to warn those who might be affected. Accelerating the simulation by running it on a GPU could provide this capability.
First, we'll examine the physics involved in this scenario and then translate it into equations to simulate the problem numerically. We want to represent a specific scenario: the collapse of a large island or other land mass falling into the ocean, as shown in Figure 13.2. This event actually occurred on the uninhabited volcanic island of Anak Krakatau ("Child of Krakatau") in Indonesia in December 2018.
In this December event, the landslide on the western slope of Krakatau Island had a volume of approximately 0.2 cubic kilometers. This was smaller than previously estimated risk forecasts. Furthermore, the wave height was estimated to be over 100 meters. Due to the short distance from the source to the shore, there was little warning for those in the area, and, causing over 400 deaths, the event received worldwide news coverage.
3. Sample Problem: Shallow Water Simulation
Отложения сползают с вулкан
Цунами удаляется от вулкана
Fig. 13.2 The tsunami wave that occurred at Anak Krakatau on December 22, 2018, was caused by a landslide of sediments from the volcanic island
3. Sample Problem: Shallow Water Simulation
Scientists conducted numerous simulations before this event and even more after it. You can see several visualizations and an analysis of this event at http://mng.bz/4Mqw . How were the simulations conducted? The required basic physics represents only a small step in the complexity of the stencil calculations we've discussed in this book. A full-fledged simulation source code would require much more sophisticated rigmarole, but simple physics will suffice for a long time. So, let's look at the necessary physics underlying the simulations.
The mathematical equations for a tsunami are relatively simple. They are conservation of mass and conservation of momentum. The latter is essentially Newton's first law of motion: "An object at rest remains at rest, and an object in motion remains in motion." The momentum equation uses the second law of motion: "Force equals change in momentum." For the mass conservation equation, we essentially have that the change in mass for a computational cell over a small period of time is equal to the sum of the mass crossing the cell boundaries, as shown below:
3. Sample Problem: Shallow Water Simulation
(Conservation of mass),
Where ¶M/¶l is the change in mass with respect to time, and vx M/¶x и vy M/¶y are the mass fluxions (vector velocity × mass) along the x- and y-directions. Furthermore, since water is incompressible, its density can be considered constant. The mass of a cell is equal to its volume × density. If our cells are 1 × 1 m, then the volume is equal to height × 1 m × 1 m. Putting it all together, everything is constant except height, so we can substitute the height variable for mass:
Mass = Volume × Density =
Height ×1 m × 1 m × Density = Constant × Height.
3. Sample Problem: Shallow Water Simulation
Also using u = vx and v = vy , we now obtain the standard form of the conservation law for the shallow water equations:
Conservation of momentum is similar, where momentum is substituted for mass or height. We only show the x terms to keep the equation on the page, as shown below:
(Conservation of mass).
(Conservation of momentum x).
3. Sample Problem: Shallow Water Simulation
The additional term 1/2gh2 is due to the work exerted on the system by gravity. According to Newton's second law, an external force creates additional momentum (F = ma). We'll look at the generation of this term using calculus and without it. First, the acceleration in this case is represented by gravity, and it causes a force acting on the water column, as shown in Figure 13.3. Each additional meter of water height creates what's called hydrostatic pressure, leading to an increase in pressure along the entire water column. Using calculus, we would integrate the pressure along the column to obtain the momentum created. This integration over the vertical of the barrel (z) from 0 to the wave height (h) is expressed as follows:
(Integrate the force over depth z).
3. Sample Problem: Shallow Water Simulation
There's also a much simpler derivation of the formula. In this case, the pressure is a linear function (Figure 13.4). If we look at the midpoint of the height and then apply the pressure difference at that midpoint to the entire column, we can obtain the same solution. In this case, we sum all the pressure forces under the curve. In mathematical terms, this summation is nothing more than integrating a function or performing a Riemann sum, where you break the area under the curve into columns and then add them up. But all of this would be overkill. The area under the curve is a triangle, and we can use the area of the triangle or A = 1/2 bh.
(Using hydrostatic pressure at mid-height).
Fig. 13.3 The force of gravity acting on a column of water creates flow and momentum
3. Sample Problem: Shallow Water Simulation
Fig. 13.4 Hydrostatic pressure due to gravity is a linear function of depth
3. Sample Problem: Shallow Water Simulation
Our resulting set of equations is as follows:
If you're observant, notice the cross terms of the momentum fluxes for momentum y in the equation for momentum x and momentum x in the equation for momentum y. With momentum x conserved, the third term has momentum x (hu), which moves along the y face with vector velocity y (v). This can be described as the advection (transport), or fluxion, of momentum x with velocity in the y direction along the upper and lower faces of the computational cell. The fluxion of momentum x (hu) along the x faces with vector velocity u is found in the second term as hu2.
3. Sample Problem: Shallow Water Simulation
We also see that the newly generated momentum is distributed across two momentum equations, with the new momentum x in the x-momentum equation and the new momentum y in the y-momentum equation. These equations are then implemented as three stencil operations in our shallow water simulation source code, where for simplicity we use H = h, U = hu, and V = hv. Now we have a simple scientific application that can be used for demonstrations.
We have one more implementation detail. We use a numerical method that estimates properties such as mass and momentum at the edges of each cell at the midpoint of a time step. We then use these estimates to calculate the amount of mass and momentum that moves into the cell during the time step. This gives us a bit more precision in the numerical solution.
Congratulations if you've made it through this exposition and gained some understanding. You've now seen how we take simple laws of physics and build a scientific application from them. You should always strive to understand the physics and numerical method behind the problem, rather than viewing the source code as a set of loops.
4. Sample profiling workflow
Next, we move on to profiling the shallow water simulation application. For this, we created a shallow water simulation application based on the mathematical and physical equations presented in Section 13.3. In many ways, the source code is simply three stencil calculations for the mass equation and two momentum equations. Since Chapter 1, we've been working with a single simple stencil equation, and an example of its source code is available at https://github.com/EssentialsofParallelComputing/Chapter13.
Running the Shallow Water Simulation Application
In this section, we'll show you how to run the shallow water simulation application source code. We'll use this source code to walk you through a sample workflow for porting your source code to the GPU. First, a few platform notes:
4. Sample profiling workflow
If you have a GPU on your local system, you can use a local worker. If not, you'll likely be working remotely on a compute cluster and transferring files back for analysis.
4. Sample profiling workflow
If you want to use graphics, you'll need to install several additional packages. On Ubuntu, this can be done with the following commands. The first command installs OpenGL and freeglut for real-time graphics. The second installs ImageMagick® for processing output into graphics files, which we can use for image captures. Image captures can also be converted into videos. The README.graphics file on GitHub contains additional information about graphics formats and scripts in the examples accompanying this chapter.
sudo apt-get install libglu1-mesa-dev freeglut3-dev mesa-common-dev -y
sudo apt install cmake imagemagick libmagickwand-dev
We've found that real-time graphics can speed up development and debugging of source code, so we've included an example of its use in the source code samples accompanying this chapter. For example, real-time graphics uses OpenGL to display water height in the computational grid, providing immediate visual feedback. The real-time graphics source code can also be easily extended to respond to keyboard and mouse interactions within the real-time graphics window.
4. Sample profiling workflow
This example is coded with OpenACC, so it's best to use the PGI compiler. A limited subset of the examples work with the GCC compiler due to its still-developing OpenACC support. Compiling the example source code is straightforward. We simply use CMake and make.
1. To build the makefile, type:
mkdir build && cd build
cmake ..
2. To enable graphics, type:
cmake -DENABLE_GRAPHICS=1
3. Set the graphics file format with:
export GRAPHICS_TYPE=JPEG
make
4. Then execute the sequential code with:
./ShallowWater
4. Sample profiling workflow
If you can't get the graphical output working, this program will work fine without it. However, if configured correctly, real-time graphics will display a graphics window similar to the one shown in Figure 13.5. The graphics are updated every 100 iterations. This figure shows a computational grid smaller than the hardcoded size in the sample source code. The lines represent computational cells, where the wave height is higher on the left. The wave moves to the right, with its height decreasing as it moves. The wave crosses the computational domain and reflects off the right edge. It then moves back and forth across the computational grid. In a real simulation, the computational grid would contain objects (such as shorelines).
If you have a system that can run OpenACC, the ShallowWater_par1-ShallowWater_par4 executables will also be built. You can use these for subsequent profiling exercises.
4. Sample profiling workflow
Fig. 13.5. Real-time graphics output from the shallow water simulation application. The red bars on the left indicate the wave's origin, where the landslide enters the water. The wave moves to the right, crossing the ocean: orange, yellow, green, and blue. If you're reading this in black and white,
the left shaded area corresponds to red, and the rightmost shaded area corresponds to blue. The lines are the contours of computational cells.
4. Sample profiling workflow
Profiling CPU Source Code to Develop an Action Plan
We described the parallel development cycle back in Chapter 2 as follows:
1. Profiling;
2. Planning;
3. Implementation;
4. Committing.
The first step is profiling our application. For most applications, we recommend using a high-level profiler, such as Cachegrind, which we introduced in Section 3.3.1. Cachegrind shows the most time-consuming paths in the source code and displays the results in an easy-to-interpret, visual format. However, for a simple program like the shallow-water simulation application, function-level profilers like Cachegrind are ineffective. Cachegrind shows that 100% of the time is spent in the main function, which isn't very helpful. In our specific situation, we need a line-by-line profiler. For this purpose, we use gprof, the most well-known profiler on Unix systems. Later, once we have the source code running on the GPU, we'll use the NVIDIA NVVP profiling tool to gather performance statistics. To get started, we just need a simple tool for profiling an application running on the CPU.
4. Sample profiling workflow
Example: Profiling with gprof
1. Edit CMakeLists.txt, adding the -pg flag to the compiler flags (the diff output marks the original line in CMakeLists with the - character and the new line with the + character):
-set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g -O3")
+set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -g -O3 -pg")
2. Edit ShallowWater.c and increase the computational grid size:
- int nx = 500, ny = 200;
+ int nx = 5000, ny = 2000;
3. Rebuild the ShallowWater executable by typing make.
4. Run the ShallowWater executable by typing ./ShallowWater. You should get an output file called gmon.out.
5. Run the post-processing step by typing gprof -l -pg ./ShallowWater.
4. Sample profiling workflow
The output from gprof shows the cycles that take up most of the time in the shallow water simulation application (see the next figure below).
The output from gprof. The loop on line 207 takes up most of the time and will be a good starting point for porting to the GPU.
4. Sample profiling workflow
We look at the cycles for each row number in the profiling results in the figure above and find that they correspond to the following operations:
This tells us that we should focus our initial efforts on computing the second pass at the end of the main computation loop and work our way up to the top of the loop. There's a tendency to try to do everything at once, but a safer approach is to work cycle by cycle, ensuring the result is still correct. Focusing on the most expensive cycles first will yield some performance gains sooner.
4. Sample profiling workflow
Adding OpenACC Compute Directives to Begin the Implementation Step
Now that we've finished profiling the application and developing a plan, the next step of the parallel development cycle is to implement the plan. In this step, we begin the long-awaited modification of the source code.
The implementation begins with porting the source code to the GPU by moving the compilation loop. We follow the same procedure for porting the source code to the GPU as in Section 11.2.2. The computation is moved by inserting the acc parallel loop pragma before each loop, as shown on line 95 in the listing below.
4. Sample profiling workflow
We also need to replace the pointer exchange in line 191 at the end of the loop with a data copy. This isn't ideal because it involves more data movement and is slower than a pointer exchange. However, implementing a pointer exchange in OpenACC is difficult because the pointers on the host and device must be swapped simultaneously.
4. Sample profiling workflow
You'll get a more accurate picture of your application's performance from a visual representation. At each step of the process, we execute the NVVP profiling tool to generate a graphical performance trace.
Example: Generating a visual performance profile using the NVIDIA Visual Profiler (NVVP)
To generate a visual performance timeline, we execute the source code using the command:
nvprof --export-profile ShallowWater_par1_timeline.prof
./ShallowWater_par1
Using the nvprof command saves the profiling timeline in the current directory:
nvvp ShallowWater_par1_timeline.prof
4. Sample profiling workflow
The nvvp command then imports the profile into the NVIDIA Visual Profiler, producing the graphical output shown in the figure below. If you wish, you can copy the profile back to your local computer between these steps and view it locally.
First, let's examine the visual profile to get a quick, color-coded overview of the relative performance of our memory copies and compute cores. The timeline is shown at the top of the Visual Profiler window. Here, pay particular attention to the MemCpy (HtoD) and MemCpy (DtoH) lines, which show data transfers from host to device and device to host. The Guided Analysis and OpenACC Details panels, located at the bottom of this window, are discussed in Section 13.4.5.
4. Sample profiling workflow
The NVIDIA NVVP profiler output shows the timeline of a single computational cycle. You can see memory copies between the device and the hardware, and vice versa. The highlighted line also shows the computational regions.
If your network connection prevents you from using the graphical tool directly or transferring profile data to your computer, you can always revert to using nvprof in text mode. The same information can be obtained from the text output, but there are always some observations that become clearer in a visual representation.
4. Sample profiling workflow
Figure 13.6 demonstrates the ability to zoom in on specific cores to more precisely identify performance metrics in certain computational cycles. Specifically, we've zoomed in on line 95 of Listing 13.1 to show individual memory copies.
Fig. 13.6. Using NVIDIA NVVP, you can zoom in on specific copies in Timeline mode. Here, you see a magnified version of individual memory copies in each cycle. Zooming allows you to see the rows they are located on, helping you easily navigate the application.
4. Sample profiling workflow
Adding Data Movement Directives
The next step in porting source code to the GPU is adding data movement directives. This allows us to further improve application performance by eliminating expensive memory copies. In this section, we'll show you how.
The visual profiler, NVVP, helps us understand where we need to focus our efforts. Start by finding large temporary MemCpy blocks and eliminating them one by one. As you remove data movement costs, your code will begin to show speedups, recovering the performance lost during the implementation of the compute directives in Section 13.4.4.
Listing 13.3 shows an example of the data movement directives we've added. At the beginning of the data section, we use the acc enter data create directive to mark the beginning of a region of dynamic data. As a result, the data will persist on the device until we encounter the acc exit data directive. For each loop, we add a present statement to inform the compiler that the data is already on the device. Please refer to the example code in Chapter 13 of OpenACC/ShallowWater/ShallowWater_par2.c for any changes made to manage data movement.
4. Sample profiling workflow
Applying the data movement directives from Listing 13.3 and rerunning the profiler yields new performance results in Figure 13.7, where data movement is reduced. By reducing data transfer time, the overall application execution time is significantly reduced. In a larger application, you should continue to identify other data transfer operations that can then be eliminated to further speed up the source code.
4. Sample profiling workflow
Figure 13.7. This timeline from NVIDIA's NVVP visual profiler shows four iterations of the computation, but now with data movement optimizations. What's interesting about this figure isn't so much what's there, but what's not. The data movement that occurred in the previous figure has been dramatically reduced or no longer exists.
4. Sample profiling workflow
Guided analysis can provide several suggested improvements.
For deeper insights, the NVVP tool provides guided analysis functionality (Figure 13.8). In this section, we'll discuss how to use this functionality.
You should evaluate guided analysis suggestions based on your knowledge of your application. In our example, we have few data transfers, so we can't achieve the memory copy/compute overlay mentioned in the top suggestion about low Memcpy/compute overlay in Figure 13.8. This applies to most of the other suggestions as well. For example, with low core concurrency, we only have one core, so we can't achieve concurrency. Although our application is small and may not need these additional optimizations, they are worth considering, as they can be useful for larger applications.
4. Sample profiling workflow
Figure 13.8 NVVP also offers a guided analysis section. Here, the user can receive hints for further optimization. Note that the highlighted section shows low computational efficiency.
4. Sample profiling workflow
Additionally, Figure 13.8 shows a low GPU utilization for our application. This is not unusual. Such low GPU utilization is more indicative of the enormous computing power available on the GPU and how much more it can accomplish. Referring briefly to the performance measurements and performance analysis performed by our mixbench tool (Section 9.3.4), we have a bandwidth-limited compute core, so at best, we'll use 1-2% of the GPU's floating-point capacity. In this light, a 0.1% GPU utilization isn't all that bad.
Another feature of the NVVP tool is the OpenACC details window, which displays the timings for each operation. This is best used by displaying before-and-after timings, as shown in Figure 13.9. Side-by-side comparisons give you a clear picture of the improvements achieved by data movement directives.
4. Sample profiling workflow
Figure 13.9. The OpenACC details window in the NVVP tool displays information about each OpenACC core and the cost of each operation. The data transfer costs can be seen in the left window for version 1 of the source code, compared to the time for optimized data movement in version 2 on the right.
4. Sample profiling workflow
When the OpenACC details window opens, you'll notice that the line numbers move within the profile. If we look at line 166 in the ShallowWater_par1 listing (on the left in Figure 13.10), it takes up 4.8% of the execution time. The breakdown of operations shows that most of this time is due to data transfer overhead. The corresponding line of code in the ShallowWater_par2 listing is number 181 (on the right in Figure 13.10) and contains the addition of the present data expression. We see that the time for line 181 is only 0.81%, and that this is largely due to the elimination of data transfer overhead. The compute construct in both cases takes approximately the same time of 0.16 ms, as shown in the line labeled acc_compute_construct just below the highlighted line.
4. Sample profiling workflow
Figure 13.10 Side-by-side comparison of source codes showing that line 166
in version 1 of the shallow water simulation source code is now line 181, which has an additional present statement.
4. Sample profiling workflow
The NVIDIA Nsight Toolkit Can Be a Powerful Development Aid
NVIDIA is replacing its Visual Profiler tools (NVVP and nvprof) with the Nsight™ toolkit. This toolkit is powered by two integrated development environments (IDEs).
1. Nsight Visual Studio Edition supports CUDA and OpenCL development within the Microsoft Visual Studio IDE.
2. Nsight Eclipse Edition adds the CUDA language to the popular open-source Eclipse development environment.
Figure 13.11 shows our shallow water simulation application in the Nsight Eclipse Edition development toolkit.
The Nsight Toolkit also includes single-function components that can be downloaded by registered NVIDIA developers. These profilers incorporate the functionality of the NVIDIA Visual Profiler and add additional capabilities. The two components are:
4. Sample profiling workflow
Figure 13.11 NVIDIA Nsight Eclipse Edition is a source code development tool. This tool window shows the ShallowWater_par1 application.
4. Sample profiling workflow
CodeXL for the AMD GPU Processor Ecosystem
AMD also offers source code development and performance analysis capabilities in its CodeXL toolkit. As shown in Figure 13.12, this application development tool is a fully functional source code simulator. CodeXL also includes a profiling component (under the "Profile" menu) that helps optimize source code for AMD GPUs.
These new tools from NVIDIA and AMD are still being rolled out. The availability of fully functional tools, including debuggers and profilers, will be a huge boost to GPU source code development.
4. Sample profiling workflow
Fig. 13.12 The CodeXL development tool supports compilation, execution, debugging, and profiling
5. Don't get lost in the swamp: Focus on important metrics
As with many profiling and performance measurement tools, the amount of information is initially overwhelming. You should focus on the most important metrics you can obtain using hardware counters and other measurement tools. The number of hardware counters is steadily increasing in the latest processors, giving you deeper insight into many aspects of processor performance that were previously hidden. We suggest the following three aspects as the most important: occupancy, issuance efficiency, and memory bandwidth.
Occupancy: Is There Enough Work?
The concept of occupancy is often cited as a priority for GPU processors. We first discussed this measure in Section 10.3. For good GPU performance, there must be enough work to keep the compute units (CUs) busy. Additionally, we need alternative work to cover bottlenecks when work groups encounter memory waits (Figure 13.13). As a reminder, CU units in OpenCL terminology in CUDA are called streaming multiprocessors (SMs). The actual achieved utilization is reported by measurement counters. If you encounter low utilization, you can adjust the workgroup size and resource consumption in the compute cores to try to improve this factor. Higher utilization is not always better. The utilization simply needs to be high enough to provide the compute units with alternative work.
5. Don't get lost in the swamp: Focus on important metrics
Figure 13.13 GPU processors have a large number of compute units (CUs), also called streaming multiprocessors (SMs). To keep the CUs busy, we need to create a lot of work with enough overhead to handle bottlenecks.
5. Don't get lost in the swamp: Focus on important metrics
Issue Efficiency: Are Your Warps Preempting Too Often?
Issue efficiency is a measure of instructions issued per cycle compared to the maximum instructions per cycle. To be able to issue instructions, each CU scheduler must have an acceptable wavefront, or warp, ready to execute. An acceptable wavefront is an active wavefront that does not stall. In a sense, this is an important result of having sufficiently high occupancy, resulting in many active wavefronts. Instructions can be floating-point operations, integers, or memory operations. Poorly written compute cores with a high number of stalls lead to low efficiency, even if occupancy is high. There are a number of reasons why compute cores experience stalls. Additionally, there are counters that can identify specific causes of stalls. Some possible causes include:
5. Don't get lost in the swamp: Focus on important metrics
5. Don't get lost in the swamp: Focus on important metrics
Achieved Throughput: It Always Comes Down to Throughput
Throughput is an important metric to understand because most applications are bandwidth-limited. The best starting point is always to look at throughput. There are a number of memory counters that allow you to drill down as deeply as you like. Comparing your throughput measurements with the theoretical and measured throughput performance of your architecture from Sections 9.3.1–9.3.3 can give you an assessment of the performance of your application. You can use memory measurement results to determine whether it would be beneficial to consolidate memory loads, store values in local memory (scratchpad), or restructure your source code to reuse data values.
6. Containers and virtual machines provide workarounds
You're on a flight to nowhere and just want to get some of your GPU source code working. The latest version of the firmware doesn't work on your company-issued laptop. A workaround is to use a container or virtual machine (VM) to run a different operating system or a different compiler version.
Docker Containers as a Workaround
Each chapter includes a sample Dockerfile and instructions for using it. The Dockerfile contains commands for building the base operating system and then installing the necessary firmware it requires.
6. Containers and virtual machines provide workarounds
Example: Building a Docker Image Using the Provided Dockerfile
The top-level directory of most chapters contains a Dockerfile. In the directory for the chapter you're interested in, run the Docker build command and create a Docker container, using the -t option to name it. In this example, we'll build the Chapter 2 Docker container and name it chapter2:
docker build -t essentialsofparallelcomputing/chapter2
Now run the Docker container with the following command:
docker run -it --entrypoint /bin/bash
essentialsofparallelcomputing/chapter2
Alternatively, use:
./docker_run.sh
Some chapters contain both a text and a graphical Dockerfile. To use the text file, delete the Dockerfile and link to the text version using this command:
ln -s Dockerfile.Ubuntu20.04 Dockerfile
6. Containers and virtual machines provide workarounds
A Docker container is useful for running software that doesn't run natively on your operating system. For example, for software that only runs on Linux, you can install a container on your Mac or Windows laptop running Ubuntu 20.0.4. Using a container is well suited for text-based console software running on the command line.
Containers also limit access to hardware devices, such as GPU processors. One option is to run device kernels on the CPU for GPU languages that support this capability. This allows us to at least test our software. If this isn't sufficient for our needs, we can take a few additional steps to try and get graphics and GPU computing working. We'll start by addressing the graphics side of things. Running a GUI from a Docker build requires a bit more effort.
6. Containers and virtual machines provide workarounds
Example: Running a Docker Image with a GUI on macOS
Mac laptops do not have an X Window client built into their default operating system. Therefore, you will need to install an X Window client on your Mac if you haven't already. XQuartz is an open-source version of the original X Window client included with older versions of macOS. You can install it using the brew package manager as follows:
brew cask install xQuartz
Now launch XQuartz and look for the XQuartz menu bar at the top of the screen. If you don't see it, you may also need to launch an X Window application, such as xterm, by right-clicking the XQuartz icon. Then:
1. Select the XQuartz menu bar, then Preferences;
2. Go to the Security tab and select Allow Connections from Network Clients;
3. Restart your system to apply the changes you've made;
4. Launch XQuartz again.
6. Containers and virtual machines provide workarounds
Chapters that require a graphical interface for tools or graphs have slightly different instructions. We use Virtual Network Computing (VNC) software to provide graphical capabilities via a web interface and VNC client viewers. You must use the dock-er_run.sh script to start the VNC server, then launch the VNC client on your local system. You can use one of the many VNC client packages, or open the graphics file in some browsers by entering the following in the website name field in the browser toolbar:
http://localhost:6080/vnc.html?resize=downscale&autoconnect=1&password=<password>"
To test a GUI application like NVVP, type nvvp. Alternatively, you can test graphics using a simple X Window application such as xclock or xterm. We can also try accessing the GPUs for computation. GPUs can be accessed using the --gpus option or the older
--device=/dev/<device name> option. This option is a relatively new addition to Docker and currently only applies to NVIDIA GPUs.
6. Containers and virtual machines provide workarounds
Example: Accessing GPUs for Computing
To access GPUs for computing, add the --gpu option with an integer argument for the number of GPUs to be available, or all for all GPUs:
docker run -it --gpus all --entrypoint /bin/bash chapter13
For Intel GPUs, you can try:
docker run -it --device=/dev/dri --entrypoint /bin/bash chapter13
Most chapters have pre-built Docker containers. You can access each chapter's containers at https://hub.docker.com/u/essentialsofparallelcomputing. You can retrieve a chapter's container using the following command:
docker run -p 4000:80 -it --entrypoint /bin/bash
essentialsofparallelcomputing/chapter2
6. Containers and virtual machines provide workarounds
NVIDIA also offers a pre-built Docker container that can be used as a starting point for your own Docker images. Visit their website at https://github.com/NVIDIA/nvidia-docker for the latest instructions. NVIDIA has another website with comprehensive container options at https://ngc.nvidia.com/catalog/containers. ROCm has detailed Docker container instructions at https://github.com/RadeonOpenCompute/ROCm-docker. Intel also has a website dedicated to setting up their oneAPI software in containers at https://github.com/intel/oneapi-containers. Some of their base containers are large and require a good internet connection.
The PGI compiler is essential for OpenACC source code development, as well as for some other GPU source code development tasks. If you need a PGI compiler for your work, the container website for PGI compilers is located at https://ngc.nvidia.com/catalog/containers/hpc:pgi-compilers . As you can see from the websites mentioned here, there are a number of resources available for creating production environments using Docker containers. But this capability is also rapidly evolving.
6. Containers and virtual machines provide workarounds
Virtual Machines Using VirtualBox
Using a virtual machine (VM) allows a user to create a guest OS on their own computer. A regular operating system is called the host, and a virtual machine is called a guest. You can run multiple VMs as a guest. They use a more restrictive environment for the guest operating system than container implementations. GUIs are often easier to configure than containers. Unfortunately, access to the GPU for computing is difficult or impossible. You may find VMs useful for GPU languages that have an option to support computing on the host CPU.
Let's look at the process of setting up an Ubuntu guest operating system in VirtualBox. The example below configures a sample shallow water simulation application running on the CPU with the PGI compiler in VirtualBox with graphics.
6. Containers and virtual machines provide workarounds
Example: Setting Up an Ubuntu Guest OS in VirtualBox
To set up your system under VirtualBox, you need to:
1. Download VirtualBox for your system and install it;
2. Download the Ubuntu desktop and save it to your local drive
[ubuntu-20.04-desktop-amd64.iso]
3. Download the VBoxGuestAdditions.iso file, which may already be included in your VirtualBox download.
Next, we configure the Ubuntu guest system. An automated script, autovirtualbox.sh, is included in this chapter's examples for automating the setup of Ubuntu guests in VirtualBox at https://github.com/EssentialsOfParallelComputing/Chapter13.git . Most of the other chapters have similar scripts. To configure the Ubuntu guest system, follow these steps:
6. Containers and virtual machines provide workarounds
1. Launch VirtualBox and click New;
2. Enter a name (e.g., chapter13);
3. Select Linux, then the 64-bit version of Ubuntu;
4. Select the memory size (e.g., 8192);
5. Create a virtual hard disk;
6. Select the VirtualBox Disk Image;
7. Select Fixed Size Disk;
8. Select 50 GB.
Your new virtual machine should now be added to the list.
Now we're ready to install Ubuntu. This process is similar to setting up Ubuntu on your desktop computer.
6. Containers and virtual machines provide workarounds
Example: Installing Ubuntu
To install Ubuntu, follow these steps:
1. Start the Ubuntu virtual machine by clicking the green Start arrow;
2. Select the previously saved ISO file by typing ubuntu-20.04-desktop-amd64.iso from the options provided;
3. Select Install Ubuntu;
4. Select your keyboard and click Continue;
5. Select Minimal Install, download updates and install third-party software and information, then click Continue;
6. Select Erase Disk and install Ubuntu, then click Install and select your time zone;
7. Type the following in the text fields: your name (e.g., chapter13), your computer name (chapter13-virtualbox), user name (chapter13), and password (chapter13);
8. Select Require My Password to Log In, and then select Continue.
6. Containers and virtual machines provide workarounds
Time for a coffee. Once the installation is complete, reboot your computer and follow these steps:
1. Log back in;
2. Click the "What's New" link;
3. Select the dots in the lower left corner and launch a terminal;
4. Edit the "Sudo Authorized Users" configuration file using
sudo -i
visudo
and add the following to any empty line: %vboxsf ALL=(ALL) ALL, then log out.
You may need to wait for updates or reboot and log back in. After logging in, install the essential build tools using sudo apt install build-essential dkms git -y . Then:
6. Containers and virtual machines provide workarounds
1. Make the VirtualBox window active and select the Devices drop-down menu from the window menu at the top of the screen;
2. Set Shared Clipboard to Bidirectional;
3. Set Drag and Drop to Bidirectional;
4. Install the Guest Additions by selecting the virtualbox-guest-additions-iso menu option;
5. Eject the optical drive: right-click on the desktop and eject the device, or in the VirtualBox window, select Devices > Optical Disk and eject the drive from the virtual drive;
6. Reboot and test by copying and pasting (Copy on Mac is Command-C, and Paste on Ubuntu is Shift-Ctrl-v).
Your Ubuntu guest system is now ready to download and install software and information.
6. Containers and virtual machines provide workarounds
Each chapter has instructions for setting up virtual machines with examples from those chapters. For this chapter, log in and install the examples for that chapter:
git clone –recursive https://github.com/essentialsofparallelcomputing/Chapter13.git
cd Chapter13 && sh -v README.virtualbox
The commands in the README.virtualbox file install the software and information, build, and run the shallow water simulation application. Real-time graphics output should also work. You can also try the nvprof utility to profile the shallow water simulation application.
7. Cloud Options: Flexible and Portable Capabilities
When access to a specific GPU is limited (no supercomputer, laptop, or desktop GPU, or remote server available), you can use cloud computing[1]. Cloud computing refers to servers provided by large data centers. While most of these services are aimed at more general users, websites focused on HPC-style services are beginning to appear. One such website is http://mng.bz/Q2YG. The Fluid Numerics cloud cluster (fluid-slurm-gcp), configured on the Google Cloud Platform (GCP), features the Slurm packet scheduler and MPI. NVIDIA GPU processors can also be scheduled. Getting started may be a bit challenging. The Fluid Numerics website at http://mng.bz/XYwv has some information to help with this process.
[1] Refer to the README.cloud file in the examples in this chapter for the latest information on using the cloud.
7. Cloud Options: Flexible and Portable Capabilities
The benefits of on-demand hardware resources are often significant. Google Cloud offers a $300 trial credit, which should be more than enough to explore this service. Other cloud providers and additional services may provide exactly what you need, or you can customize your environment. Intel has launched a cloud service for testing Intel GPU processors to give developers access to software and hardware for its oneAPI initiative and its DPCPP compiler, which implements SYCL. You can try it out by visiting https://software.intel.com/en-us/oneapi and registering for it.
8. Exercises
1. Run the nvprof tool on the streaming triad example. You can try the CUDA version from Chapter 12 or the OpenACC version from Chapter 11. Which workflow did you use for your hardware resources? If you don't have access to an NVIDIA GPU, can you use a different profiling tool?
2. Create a trace from nvprof and import it into NVVP. Where is the execution time spent? What could you do to optimize it?
3. Download a pre-built Docker container from the appropriate vendor for your system. Run the container and run one of the examples from Chapters 11 or 12.