GPU architectures and concepts
Jumanazarov Mardonbek
Plan:
Why should we care about GPUs in high-performance computing? GPUs provide a massive source of parallel operations that can significantly exceed that available in more traditional CPU architectures. To effectively utilize their capabilities, it is important to understand GPU architecture. While GPUs have often been used for graphics processing, they are also used for general-purpose parallel computing. This chapter provides an overview of GPU-accelerated platform hardware.
What systems today qualify as GPU-accelerated? Virtually every computing system delivers the powerful graphics capabilities expected by modern users. These GPUs range from small components within the main CPU to large peripheral cards that occupy a significant portion of the desktop computer chassis. HPC systems are increasingly equipped with multiple GPUs. Occasionally, even personal computers used for simulation or gaming may sometimes connect two GPUs to boost graphics performance. In this chapter, we present a conceptual model that defines the key hardware components of a GPU-accelerated system. These components are shown in Figure 9.1.
Fig. 9.1. Structural diagram of a GPU-accelerated system
using a dedicated GPU. The CPU and GPU each have their own memory and communicate via the PCI bus.
Inconsistent terminology used within the community adds complexity to understanding GPU processors. We will use the terminology established by the OpenCL standard, as it has been agreed upon by several GPU vendors. We will also note alternative terminology that is widely used, for example, by NVIDIA. Before continuing our discussion, let's look at a few definitions:
We'll introduce each component in a GPU-accelerated system and show how to calculate theoretical performance for each. We'll then examine their actual performance using microbenchmarking applications. This will help us understand how certain hardware components can create bottlenecks that prevent GPU-accelerated applications from being accelerated. Armed with this information, we'll conclude the chapter with a discussion of the types of applications that benefit most from GPU acceleration and what your goals should be to see performance improvements when porting an application to GPU processors. The source code for this chapter can be found at https://github.com/EssentialsofParallelComputing/Chapter9.
1. CPU-GPU system as an accelerated computing platform
GPUs are everywhere. They can be found in mobile phones, tablets, personal computers, consumer workstations, game consoles, high-performance computing centers, and cloud computing platforms. GPUs provide additional computing power on most modern hardware and accelerate many operations you might not even realize are possible. As the name suggests, GPUs were designed for graphics-related computations. Consequently, GPU design focuses on parallel processing of large blocks of data (triangles or polygons), a prerequisite for graphics applications. Compared to CPUs, which can process dozens of parallel threads or processes per clock cycle, GPUs are capable of processing thousands of parallel threads simultaneously. This design allows GPUs to deliver significantly higher theoretical peak performance, potentially reducing both the time to solution and the power footprint of an application.
Computer scientists, always striving for computing power, gravitated toward GPUs, which could handle more general-purpose computing tasks. Because GPUs were designed for graphics, the languages originally developed for their programming, such as OpenGL, were oriented toward graphics operations. To implement algorithms on GPUs, programmers had to refactor their algorithms to accommodate these operations, which was time-consuming and error-prone. The extension of GPUs to non-graphics workloads became known as general-purpose graphics processing units (GPGPUs).
1. CPU-GPU system as an accelerated computing platform
The continued interest and success of GPGPU computing has led to the emergence of a flurry of GPGPU languages. The first widely adopted language was CUDA (Compute Unified Device Architecture) for NVIDIA GPU processors, which was first introduced in 2007. The dominant open standard language for GPGPU computing is Open Computing Language (OpenCL), developed by a group of vendors led by Apple and released in 2009. We discuss CUDA and OpenCL in Chapter 12.
Despite, or perhaps because of, the continued adoption of GPGPU languages, many computing professionals have found the original, native GPGPU languages difficult to use. Consequently, higher-level approaches using directive-based APIs have gained widespread adoption, spurring corresponding vendor development efforts. We'll look at examples of directive-based languages like OpenACC and OpenMP (with the new target directive) in Chapter 11. For now, we can simply summarize that the new directive-based GPGPU languages, OpenACC and OpenMP, have been an unqualified success. These languages and APIs have allowed programmers to focus more on developing their applications rather than expressing their algorithms in terms of graphics operations. Ultimately, this has often resulted in huge speedups in scientific and data science applications.
1. CPU-GPU system as an accelerated computing platform
GPU processors are best described as accelerators, long used in the computing world. First, let's define what we mean by an accelerator.
DEFINITION: An accelerator (or accelerator hardware) is a highly specialized device that supports a main general-purpose CPU by accelerating certain operations.
A classic example of an accelerator is the original personal computer, which shipped with the 8088 processor. It had an option and socket for an 8087 coprocessor, which could perform floating-point operations in hardware rather than software. Today, the most common hardware accelerator is the GPU, which can be either a separate hardware component or integrated into the main CPU. An accelerator is distinguished by its highly specialized focus, rather than being a general-purpose device, but this distinction is not always clearly defined. A GPU is an additional hardware component that can perform operations in conjunction with the CPU. GPU processors come in two varieties:
Integrated GPUs are built directly into the CPU chip. Integrated GPUs share RAM (random access memory) with the CPU. Dedicated GPUs are attached to the motherboard via a PCI (Peripheral Component Interconnect) slot, which is used to connect peripheral components. The PCI slot is the physical component that allows data transfer between the CPU and GPU. It is commonly referred to as the PCI bus.
1. CPU-GPU system as an accelerated computing platform
Integrated GPUs: An Underutilized Option in Commodity Systems
Intel® has long included integrated GPUs in its CPUs designed for the budget market. They fully expected that users seeking real performance would buy a discrete GPU. Intel's integrated GPUs have historically been relatively weak compared to integrated versions from AMD (Advanced Micro Devices, Inc.). This recently changed when Intel announced that the integrated graphics in its Ice Lake processor matched AMD's integrated GPUs.
AMD's integrated GPUs are called Accelerated Processing Units (APUs). They are a tightly coupled combination of a CPU and GPU. The GPU design was originally inspired by AMD's acquisition of graphics card maker ATI in 2006. The CPU and GPU in an AMD accelerated processing unit (APU) share the same processor memory. These GPUs are smaller than discrete GPUs but still (proportionally) provide the graphics (and computing) performance of a GPU-based processor. AMD's real goal with APUs is to offer a more cost-effective yet high-performance system for the mainstream market. Shared memory is also attractive because it eliminates PCI bus data transfers, which are often a significant performance bottleneck.
The ubiquitous nature of integrated GPUs is significant. For us, this means that many commodity desktops and laptops are now capable of accelerating computing. The goal of these systems is a relatively modest performance boost and perhaps reduced power consumption or improved battery life. But when it comes to extreme performance, discrete GPUs remain the undisputed performance champions.
1. CPU-GPU system as an accelerated computing platform
Dedicated GPUs: The Workhorse
In this chapter, we will primarily focus on GPU-accelerated platforms with dedicated GPUs, also referred to as discrete GPUs. Dedicated GPUs typically provide higher computing power than integrated GPUs. Furthermore, these GPUs can be isolated to perform general-purpose computing tasks. Figure 9.1 conceptually illustrates a CPU-GPU system with a dedicated GPU. The CPU has access to its own memory space (RAM in the CPU) and is connected to the GPU via a PCI bus. It is able to send data and commands over the PCI bus to the GPU, which has its own memory space, separate from the CPU's memory space.
In order to execute work on the GPU, data must at some point be transferred from the CPU to the GPU. When the work is complete and the results are ready to be written to a file, the GPU must send the data back to the CPU. The commands the GPU needs to execute are also transferred from the CPU to the GPU. Each of these transactions is mediated by the PCI bus. While this chapter won't discuss approaches to performing these operations, we will discuss the hardware performance limits of the PCI bus. Due to these limitations, a poorly designed GPU-based application can potentially have lower performance than if it were written using CPU-only code. We will also discuss the internal GPU architecture and GPU performance in terms of memory and floating-point operations.
2. GPU and streaming engine
For those of us who have spent years programming threads (aka virtual cores) on CPUs, the GPU is like the perfect threading engine. The components of this engine are:
Let's examine the GPU hardware architecture to get a sense of how it works this magic. To illustrate the conceptual model of the GPU, we abstract elements common to different GPU processor vendors and even common between design variants from the same vendor. It's worth remembering that some hardware variations are not captured by these abstract models. Add to this the abundance of terminology currently in use, and it's no wonder that a newcomer to the field has a difficult time understanding GPU hardware and programming languages. Nevertheless, this terminology is relatively sound compared to the graphics world with vertex shaders, texture mapping units, and fragment generators. Table 9.1 summarizes the approximate equivalence of terminology, but keep in mind that since hardware architectures are not exactly the same, the equivalence in terminology varies depending on the context and user.
2. GPU and streaming engine
Table 9.1 Equipment terminology: rough correlation
2. GPU and streaming engine
The last row of Table 9.1 shows the hardware level at which the single instruction, multiple data model, commonly referred to as SIMD[2], is implemented. Strictly speaking, NVIDIA hardware does not have vector hardware or SIMD, but emulates it using a collection of threads in what NVIDIA calls a warp in the single instruction, multiple thread (SIMT) model. You may want to revisit our initial discussion of parallel categories in Section 1.4 to refresh your memory of these different approaches. Other GPUs are also capable of performing SIMT operations on what OpenCL and AMD call subgroups, equivalent to NVIDIA warps. We discuss this topic in more detail in Chapter 10, which covers GPU programming models in detail. However, this chapter will focus on the GPU hardware, its architecture, and concepts.
GPU processors often also have hardware replication units, some of which are listed in Table 9. 9.2 to simplify scaling their hardware designs to larger numbers of modules. These replicating modules are easy to manufacture and often appear in specification lists and discussions.
Table 9.2 GPU hardware replication modules by vendor
2. GPU and streaming engine
Figure 9.2 shows a simplified block diagram of a single-node system with one multiprocessor CPU and two GPUs. A single node can have a wide range of configurations, consisting of one or more multiprocessor CPUs with an integrated GPU and one to six discrete GPUs. In OpenCL nomenclature, each GPU is a compute unit. However, CPUs in OpenCL can also be compute units.
DEFINITION A compute unit in OpenCL is any computing hardware that can perform computations and supports OpenCL. This can include GPU processors, CPUs, or even more exotic hardware such as embedded processors or field-programmable gate arrays (FPGAs).
[1] Depending on the vendor, the term "warp" means:
1) a set of threads executing the same instruction (on different data elements); in Nvidia parlance, this is a SIMT warp; 2) threads that run along a woven fabric. – Translator's note.
[2] In the case of GPUs, this means that all warp threads simultaneously execute a single common command. – Translator's note.
2. GPU and streaming engine
Figure 9.2 A simplified block diagram of a GPU system, showing two computing devices, each with a separate GPU, GPU memory, and multiple compute units (CUs). NVIDIA CUDA terminology refers to CUs as streaming multi-processors (SMs).
2. GPU and streaming engine
The simplified diagram in Figure 9.2 is our model for describing GPU components and is also useful for understanding how a GPU processes data. A GPU consists of:
CUs have their own internal architecture, often referred to as a microarchitecture. Commands and data received from the CPU are processed by the workload balancer. This balancer coordinates the execution of commands and the movement of data to and from the compute units. Achievable GPU performance depends on:
global memory bandwidth;
compute unit (CU) bandwidth;
the number of compute units.
In this section, we explore each component of our GPU-based model. We also discuss theoretical peak throughput models for each component. We also demonstrate how to use microbenchmarking tools to measure the actual performance of the components.
2. GPU and streaming engine
The compute unit is a streaming multiprocessor (or subslice).
The GPU processor's compute unit has multiple compute units (CUs). (The term "compute unit" was agreed upon by the community for the OpenCL standard.) NVIDIA calls them streaming multiprocessors (SMs), while Intel calls them subslices.
The processing elements are individual processors.
Each CU contains multiple GPUs, called processing elements (PEs) in OpenCL, or CUDA cores (or compute cores) at NVIDIA. Intel calls them execution units (EUs), while the graphics community calls them shader processors.
Figure 9.3 shows a simplified conceptual diagram of processing elements. These processors are not equivalent to CPUs; by design, they are simpler constructs that require graphics operations. However, the operations required for graphics include almost all the arithmetic operations a programmer would use on a conventional CPU.
Multiple operations performed on data by each processing element
Within each PE, an operation can be performed on multiple data elements. Depending on the details of the GPU microprocessor architecture and the GPU vendor, these are called SIMT, SIMD, or vector operations. Similar functionality can be achieved by bundling processing elements together.
2. GPU and streaming engine
Calculating Peak Theoretical Flops for Some Leading GPUs
With an understanding of GPU hardware, we can now calculate peak theoretical flops for some of the latest GPUs. These include the NVIDIA V100, AMD Vega 20, AMD Arcturus, and the integrated Gen11 GPU on the Intel Ice Lake processor. Table 9.3 lists the specifications of these GPUs. We'll use these specifications to calculate the theoretical performance of each device. Knowing the theoretical performance will then allow you to compare the performance of each device. This will help you make decisions or evaluate how much faster or slower another GPU is in your calculations. Hardware specifications for many graphics cards can be found on the TechPowerUp website: https://www.techpowerup.com/gpu-specs/.
For NVIDIA and AMD, GPU processors targeted at the HPC market have hardware cores that can perform one double-precision operation for every two single-precision operations. This relative flop capability can be expressed as a 1:2 ratio, where double precision is 1:2 to single precision on high-end GPUs. This ratio is important because it indicates the ability to roughly double performance by reducing your need for double-precision precision to single precision. For many GPUs, half-precision has a 2:1 ratio to single precision, or double flop capability. Intel's integrated GPU has a 1:4 ratio of double precision to single precision, and some standard GPUs have 1:8 ratios of double precision to single precision. GPU processors with such low double-precision ratios are targeted at the graphics market or machine learning. To obtain these ratios, take the FP64 row and divide it by the FP32 row.
2. GPU and streaming engine
Table 9.3 Technical specifications of the latest discrete GPUs from NVIDIA, AMD and integrated GPUs from Intel
2. GPU and streaming engine
Peak theoretical flops can be calculated by multiplying the clock frequency by the number of processors, multiplied by the number of floating-point operations per cycle. The number of flops per cycle provides the capability for fused multiply-add (FMA), which performs two operations in a single cycle.
Peak theoretical flops (GFLOPS/s) = Clock frequency (MHz) × Compute units × Processing units × flops/cycle.
Both the NVIDIA V100 and AMD Vega 20 deliver impressive floating-point performance. Ampere demonstrates some additional improvement in floating-point performance, but it is memory performance that promises a greater increase. AMD's MI100 demonstrates a larger leap in floating-point performance. Intel's integrated GPU is also impressive, considering it is limited by available silicon space and the lower nominal TDP of the processor. Given Intel's plans to develop discrete graphics cards for multiple market segments, this GPU is expected to have even more options in the future.
2. GPU and streaming engine
Example: Peak theoretical flops for some leading GPUs
Theoretical peak flops for NVIDIA V100:
Theoretical peak flops for NVIDIA Ampere:
Theoretical peak flops for AMD Vega 20 (MI50):
Theoretical peak flops for AMD Arcturus (MI100):
Theoretical peak flops for Intel Integrated Gen 11 on Ice Lake:
3. Characteristics of GPU memory spaces
A typical GPU has different types of memory. Using the correct memory space can have a significant impact on performance. Figure 9.4 shows these memory types in a conceptual diagram. This helps visualize the physical locations of each memory level to understand its behavior. Although the vendor can place GPU memory anywhere, it must behave as shown in this diagram.
Figure 9.4. The rectangles represent each GPU component and the memory located at each hardware level. The host writes to and reads global and persistent memory. Each compute unit (CU) can read and write data from global memory and read data from persistent memory.
3. Characteristics of GPU memory spaces
The list of GPU memory types and their properties is as follows:
One of the factors that makes GPU processors fast is that they use specialized global memory (RAM), which provides higher bandwidth, whereas current GPUs use DDR4 memory and are only now moving to DDR5. GPU processors use a special version of GDDR5, which delivers higher performance. The latest GPUs are now moving to High Bandwidth Memory (HBM2), which delivers even higher throughput. In addition to increasing throughput, HBM also reduces power consumption.
3. Characteristics of GPU memory spaces
Calculating Theoretical Peak Memory Bandwidth
Theoretical peak memory bandwidth for a GPU can be calculated based on the GPU's memory clock speed and the memory transaction width in bits. Table 9.4 shows several higher values for each memory type. We also need to multiply by two to account for the double data rate at which memory is fetched both at the top and bottom of the cycle. Some DDR memory can even perform more transactions per cycle. Table 9.4 also shows several transaction multipliers for different types of graphics memory.
Table 9.4: Specifications of Common GPU Memory Types
3. Characteristics of GPU memory spaces
To calculate theoretical memory bandwidth, the memory clock frequency is multiplied by the number of transactions per cycle, then multiplied by the number of bits fetched with each transaction:
Theoretical Bandwidth = Memory Clock Frequency (GHz) × Memory Bus (bits) × (1 byte/8 bits) × Transaction Multiplier.
Some specifications list the memory transaction rate in GB/s rather than the memory clock frequency. This rate is the number of transactions per cycle multiplied by the clock frequency. Given this specification, the bandwidth equation becomes:
Theoretical Bandwidth = Memory Transaction Rate (GB/s) × Memory Bus (bits) × (1 byte/8 bits).
Example: Calculating theoretical bandwidth
Theoretical bandwidth = 0.876 × 4096 × 1/8 × 2 = 897 GB/s.
Theoretical bandwidth = 1.000 × 4096 × 1/8 × 2 = 1024 GB/s.
3. Characteristics of GPU memory spaces
Measuring GPU Performance with the STREAM Benchmark App
Since most of our applications scale with memory bandwidth, the STREAM Benchmark app, which measures memory bandwidth, is one of the most important benchmark microbenchmarks. We introduced this benchmark to measure CPU throughput in Section 3.2.4. The benchmarking process is similar for GPU processors, but we need to rewrite the stream computing kernels in GPU languages. Fortunately, this has been done by Tom Deakin of the University of Bristol for various languages and GPU hardware in his Babel STREAM Benchmark app.
The source code for the Babel STREAM Benchmark app measures the throughput of various hardware and programming languages. We use it here to measure NVIDIA GPU throughput using CUDA. Versions in OpenCL, HIP, OpenACC, Kokkos, Raja, SYCL, and OpenMP with GPU targets are also available. These are all different languages that can be used for GPU hardware such as NVIDIA, AMD, and Intel GPUs.
3. Characteristics of GPU memory spaces
Exercise: Measuring Throughput with the Babel STREAM Benchmark Application
The steps for using the Babel STREAM Benchmark application for CUDA on NVIDIA GPUs are as follows:
1. Clone the Babel STREAM Benchmark application using
git clone git@github.com:UoB-HPC/BabelStream.git
2. Then run:
make -f CUDA.make
./cuda-stream
The results for the NVIDIA V100 GPU are as follows:
3. Characteristics of GPU memory spaces
This process is similar for AMD GPUs.
1. Edit the OpenCL.make file and add the paths to your OpenCL header files and libraries.
2. Then run:
make -f OpenCL.make
./ocl-stream
For AMD Vega 20, the GPU throughput is slightly lower than that of NVIDIA GPUs.
Using OpenCL device gfx906+sram-ecc
3. Characteristics of GPU memory spaces
Roofline Performance Model for GPUs
We introduced a roofline performance model for CPUs in Section 3.2.4. This model takes into account both memory bandwidth and system performance limits. This model is similarly useful for GPUs and helps understand their performance limits.
Exercise: Measuring Bandwidth using the Empirical Roofline Toolkit
For this exercise, you will need access to an NVIDIA and/or AMD GPU. A similar process can be used for other GPUs.
1. Get the Empirical Roofline Toolkit using
git clone https://bitbucket.org/berkeleylab/cs-roofline-toolkit.git
2. Then type:
cd cs-roofline-toolkit/Empirical_Roofline_Tool-1.1.0
cp Config/config.voltar.uoregon.edu Config/config.V100_gpu
3. Characteristics of GPU memory spaces
3. Edit the following settings in the Config/config.V100_gpu file:
ERT_RESULTS Results.V100_gpu
ERT_PRECISION FP64
ERT_NUM_EXPERIMENTS 5
4. Run:
tests ./ert Config/config.V100_gpu
5. View the file Results.config.V100_gpu/Run.001/roofline.ps
cp Config/config.odinson-ocl-fp64.01 Config/config.Vega20_gpu
6. Edit the following settings in the Config/config. Vega20_gpu:
ERT_RESULTS Results.Vega20_gpu
ERT_CFLAGS -O3 -x c++ -std=c++11 -Wno-deprecated-declarations
-I<path to OpenCL headers>
ERT_LDLIBS -L<path to OpenCL libraries> -lOpenCL
7. Run:
tests ./ert Config/config.Vega20_gpu
8. View the results in the file Results.config.Vega20_gpu/Run.001/roofline.ps.
Figure 9.5 shows the results of comparative tests based on the roofline model for the NVIDIA V100 GPU and the AMD Vega20 GPU.
3. Characteristics of GPU memory spaces
Figure 9.5 Roof contour plots for the NVIDIA V100 and AMD Vega 20, showing the throughput and flops limits for the two GPUs
3. Characteristics of GPU memory spaces
Using a Mixed Benchmarking Tool to Select the Best GPU for Your Workload
There are many GPU options available for cloud services and in the HPC server market. Is there a way to determine the optimal GPU for your application? We'll explore a performance model that can help you select the best GPU for your workload.
By changing the independent variable in the roof contour plot from arithmetic intensity to memory bandwidth, we can highlight the application's performance limits relative to each GPU device. The Mixed Benchmarking Tool (mixbench) was designed to reveal the performance differences between different GPU devices. This information is actually no different from what's shown in the roof contour model, but visually, it has a very different impact. Let's walk through an exercise using the Mixed Benchmarking Tool to demonstrate what you can learn.
3. Characteristics of GPU memory spaces
Exercise: Obtaining Peak Flop Rate and Throughput with the Mixbench Benchmarking Tool
1. Get the Mixbench benchmarking tool code:
git clone https://github.com/ekondis/mixbench.git
2. Check for CUDA or OpenCL and install if necessary:
cd mixbench; edit Makefile
3. Adjust the path to the CUDA and/or OpenCL installations.
4. Specify the executables to build. The CUDA installation path can be overridden with the command
make CUDA_INSTALL_PATH=<path>
5. Do one of the following:
./mixbench-cuda-ro
./mixbench-ocl-ro
Benchmarking results are presented as compute speed in GFlops/s relative to memory bandwidth in GB/s (Figure 9.6). Essentially, the benchmark results show a horizontal line at peak flop rate and a vertical drop at the memory bandwidth limit. The maximum of each of these values is taken and used to plot a single dot in the upper right corner, which reflects both the GPU's peak flop rate and its peak memory bandwidth.
3. Characteristics of GPU memory spaces
Figure 9.6: The output from a mixed benchmark run on the V100 (shown as a line graph). Maximum throughput and floating-point speed are detected and used to plot the V100 performance point in the upper-right corner of the graph.
3. Characteristics of GPU memory spaces
We can run a mixed benchmarking tool on a wide variety of GPU devices and obtain their peak performance characteristics. GPU processors designed for the HPC market feature high-precision floating-point hardware, while GPU processors for other markets, such as graphics and machine learning, target single-precision hardware.
We can plot each GPU device in Figure 9.7 as a line representing the arithmetic or operational intensity of the application. Most typical applications have an intensity of approximately 1 flop/load. On the other hand, matrix multiplication has an arithmetic intensity of 65 flops/load. We show a slanted line for both of these application types in Figure 9.7. If a GPU point is above the application line, we draw a vertical line down to the application line to determine the level of achievable application performance. For devices to the right and below the application line, we use a horizontal line to find the performance limit.
This graph clarifies whether the GPU device characteristics meet the application requirements. For a typical 1-flop/load application, GPUs like the GeForce GTX 1080Ti, designed for the graphics market, are well suited. The V100 GPU is more suitable for the Linpack benchmark used to rank the TOP500 large computing systems, as it primarily consists of matrix multiplication. GPUs like the V100 are specialized hardware designed specifically for the HPC market and command a high price premium. For some less arithmetic-intensive applications, commodity GPU processors designed for the graphics market may be more cost-effective.
3. Characteristics of GPU memory spaces
Figure 9.7. A collection of performance points for GPU devices (shown in the graph on the right) along with the application's arithmetic intensity (shown as straight lines). Values above the line indicate that the application is memory-bound, while values below the line indicate that it is compute-bound.
4. PCI bus: overhead of data transfer from CPU to GPU
The PCI bus, shown in Figure 9.1, is necessary for transferring data from the CPU to the GPU and back. The cost of data transfer can significantly limit the performance of operations offloaded to the GPU. It is often crucial to limit the amount of data transferred back and forth to accelerate GPU operation.
The current version of the PCI bus is called PCI Express (PCIe). At the time of this writing, it has been revised several times, from generations 1.0 to 6.0. To understand the performance limits of the PCIe bus, you must know the generation number of your system. In this section, we will demonstrate two methods for estimating PCI bus throughput:
The theoretical peak performance model is useful for quickly estimating what to expect from a new system. The advantage of this model is that you don't need to run any applications on the system. This is useful when you are just starting a project and want to quickly manually assess potential performance bottlenecks. Additionally, the benchmark example shows that the peak throughput you can achieve depends on how you use your hardware.
4. PCI bus: overhead of data transfer from CPU to GPU
PCI Bus Theoretical Bandwidth
On platforms with dedicated GPUs, all data exchange between the GPU and CPU occurs over the PCI bus. This makes it a critical hardware component that can significantly impact the overall performance of your application. In this section, we'll cover the key PCI bus features and descriptors you need to know to calculate the theoretical PCI bus bandwidth. Calculating this number on the fly is useful for assessing potential performance limitations when porting your application to GPU processors.
The PCI bus is the physical component that maps dedicated GPU processors to CPUs and other devices. It enables data exchange between the CPU and GPU. This exchange occurs across multiple PCIe lanes. We'll begin by introducing the theoretical bandwidth formula and then explain each term.
Theoretical Bandwidth (Gbps) = Lanes × Transfer Rate (GT/s) × Overhead Factor
(GT/GT) × Bytes/8 bits. Theoretical throughput is measured in gigabytes per second (Gbps). It is calculated by multiplying the number of lanes by the maximum data rate for each lane, then converting from bits to bytes. This conversion remains in the formula because transfer rates are typically specified in gigatransfers per second (GT/s). The overhead factor is due to the encoding scheme used to ensure data integrity, which reduces the effective transfer rate. For Generation 1.0 devices, the encoding scheme cost 20%, so the overhead factor was 100 - 20%, or 80%. From Generation 3.0 onward, the encoding scheme overhead drops to a mere 1.54%, so the achieved throughput becomes essentially the same as the transfer rate. Let's now examine each term in the throughput equation.
4. PCI bus: overhead of data transfer from CPU to GPU
PCIe Lanes
You can find the number of PCI bus lanes by reviewing the manufacturer's specifications, or you can use a number of tools available on Linux platforms. Keep in mind that some of these tools may require root privileges. If you don't have these privileges, it's best to consult your system administrator to obtain this information. However, we will present two options for determining the number of PCIe lanes.
The lspci utility is widely used on Linux systems. This utility lists all components attached to the motherboard. We can use the grep search tool with regular expressions to isolate only the PCI bridge. The following command will display vendor information and the device name along with the number of PCIe lanes. In this example, (x16) in the results indicates 16 lanes.
$ lspci -vmm | grep "PCI bridge" -A2
Class: PCI bridge
Vendor: Intel Corporation
Device: Sky Lake PCIe Controller (x16)
The dmidecode command provides similar information as an alternative:
$ dmidecode | grep "PCI"
PCI is supported
Type: x16 PCI Express
4. PCI bus: overhead of data transfer from CPU to GPU
Determining the Maximum Transfer Rate
The maximum data transfer rate for each lane in the PCIe bus can be directly determined by its generation. A generation is a specification of the required hardware performance, much like 4G, the industry standard for mobile phones. The PCI Special Interest Group (PCI SIG) represents industry partners and develops the PCIe specification, which is commonly referred to as a generation for short. Table 9.5 shows the maximum data transfer rate for PCI lanes and directions.
Table 9.5: PCI Express (PCIe) Specifications by Generation
4. PCI bus: overhead of data transfer from CPU to GPU
If you don't know the generation of your PCIe bus, you can use the lspci command to obtain this information. Among the lspci output, we look for the link capacity for the PCI bus. This value is abbreviated as LnkCap below:
$ sudo lspci -vvv | grep -E 'PCI|LnkCap'
Output:
00:01.0 PCI bridge:
Intel Corporation Sky Lake PCIe Controller (x16) (rev 07)
LnkCap: Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s
Now that we know the maximum transfer rate from this information, we can use it in the throughput formula. It's also useful to know that this rate can be aligned with the generation. In this case, the information indicates that we are working with a PCIe Gen3 system.
NOTE: On some systems, the output from lspci and other system utilities may not provide much information. The output is system-dependent and simply reports the identification of each device. If these utilities are unable to determine the characteristics, a backup option is to use the PCI benchmark source code provided in Section 9.4.2 to determine the capabilities of your system.
4. PCI bus: overhead of data transfer from CPU to GPU
Overhead Levels
Transferring data over the PCI bus incurs additional overhead. Generation 1 and 2 standards stipulate that 10 bytes are transferred for every 8 bytes of useful data. Starting with Generation 3, 130 bytes are transferred for every 128 bytes of data. The overhead ratio is the ratio of the number of bytes used to the total number of bytes transferred (Table 9.5).
Benchmark Data for PCIe Theoretical Peak Throughput
Now that we have all the necessary information, let's estimate the theoretical throughput using an example, using the output shown in the previous sections.
Example: Estimating Theoretical Throughput
We've determined that we have a Gen3 PCIe system with 16 lanes. Gen3 systems have a maximum data transfer rate of 8.0 GT/s and an overhead ratio of 0.985. As shown below, with 16 lanes, the theoretical throughput is 15.75 Gbps.
Theoretical throughput (Gbps) = 16 lanes × 8.0 GT/s × 0.985 (Gb/GT) × bytes/8 bits = 15.75 Gbps.
4. PCI bus: overhead of data transfer from CPU to GPU
PCI Bandwidth Benchmarking Application
The theoretical PCI bandwidth equation yields the expected peak throughput. In other words, it is the maximum possible throughput that an application can achieve on a given platform. In practice, the achieved throughput can depend on a number of factors, including the operating system, system drivers, other hardware components of the compute node, the GPU programming API, and the PCI bus data block size. On most systems you have access to, it's likely that all but the last two variables are beyond your control. During application development, you control the programming API and the PCI bus data block size.
This leaves us with the question: how does the data block size affect the achieved throughput? This type of question is typically answered using a microbenchmark. A microbenchmark, or benchmark, is a small program designed to execute a single process or piece of hardware used by a larger application. Microbenchmarks help obtain some metrics about system performance.
4. PCI bus: overhead of data transfer from CPU to GPU
In our scenario, we want to develop a benchmark that copies data from a CPU to a GPU and vice versa. Since data copying is expected to take place in microseconds to tens of microseconds, we will measure the time required to complete 1000 data copies. This time will then be divided by 1000 to obtain the average data copy time between the CPU and GPU.
We will walk through this step-by-step using a benchmarking application to measure PCI throughput. Listing 9.1 shows the source code that copies data from the host to the GPU. Writing the code in the CUDA GPU programming language will be covered in Chapter 10, but the basic operation is clear from the function names. In Listing 9.1, we accept as input the size of the flat one-dimensional array we want to copy from the CPU (host) to the GPU (device).
NOTE: The referenced source code is available in the PCI_Bandwidth_Benchmark subdirectory at https://github.com/EssentialsofParallelComputing/Chapter9.
4. PCI bus: overhead of data transfer from CPU to GPU
4. PCI bus: overhead of data transfer from CPU to GPU
In Listing 9.1, the first step is to allocate memory for the host copy and the device copy. The procedure at line 40 in this listing uses cudaMallocHost, allocating pinned memory on the host for faster data transfer. For a procedure that uses regular paged memory, standard malloc and free calls are used. The cudaMemcpy procedure transfers data from the host CPU to the GPU. The call to cudaDeviceSynchronize waits for the copy to complete. Before the loop in which we repeat the host-to-device copy, we record the start time. Then, we perform the host-to-device copy 1000 times in a row and record the current time again. The average host-to-device copy time is then calculated by dividing by 1000. For accuracy, we free the space occupied by the host and device arrays.
Knowing the time required to transfer an array of size N from the host to the device, we can now call this procedure repeatedly, changing N each time. However, we are more interested in evaluating the achieved throughput.
Recall that throughput is the number of bytes transferred per unit of time. At this point, we know the number of array elements and the time required to copy the array between the CPU and GPU. The number of bytes transferred depends on the type of data stored in the array. For example, in an array of size N containing real numbers (4 bytes), the amount of data copied between the CPU and GPU is 4N. If 4N bytes are transferred in time T, then the achieved throughput is
B = 4N/T.
This allows us to construct a dataset showing the achieved throughput as a function of N. The routine in the following listing requires the maximum array size to be specified and then returns the throughput measured for each experiment.
4. PCI bus: overhead of data transfer from CPU to GPU
4. PCI bus: overhead of data transfer from CPU to GPU
Here, we loop through the array sizes and obtain the average copy time from the host to the device for each size. The throughput is then calculated as the number of bytes divided by the time required to copy. The array contains floating-point numbers, each with four bytes for each array element. Now let's look at an example of using a benchmarking application to characterize the performance of your PCI bus.
Gen3 x16 Performance on a Laptop
We ran the PCI throughput microbenchmarking application on a GPU-accelerated laptop. On this system, the lspci command shows that the specified system is equipped with a Gen3 x16 PCI bus:
$ sudo lspci -vvv | grep -E 'PCI|LnkCap'
00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x16) (rev 07)
LnkCap: Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s
For this system, the theoretical peak PCI bus throughput is 15.8 Gb/s. Figure 9.8 shows a graph of the achieved throughput in our microbenchmarking application (curved lines) compared to the theoretical peak throughput (horizontal dashed lines). The shaded area around the achieved bandwidth indicates the standard deviation of ±1 in throughput.
4. PCI bus: overhead of data transfer from CPU to GPU
Figure 9.8 shows the theoretical peak throughput (horizontal lines) and the empirically measured throughput output from the microbenchmarking application for a PCIe Gen3 x16 system. The figure also shows results for both pinned and paged memory.
4. PCI bus: overhead of data transfer from CPU to GPU
First, note that when transferring small chunks of data over the PCI bus, the achievable throughput is low. Beyond array sizes of 107 bytes, the achievable throughput approaches a maximum of approximately 11.6 GB/s. Also note that the throughput with pinned memory is much higher than with pageable memory, and pageable memory has a much wider range of performance results for each memory size. It's helpful to understand the difference between pinned and pageable memory to understand the reason for this difference.
Allocating pinned memory reduces the amount of memory available to other processes because the operating system kernel can no longer page the memory out to disk for other processes to use. Pinned memory is allocated from standard DRAM for the processor. The allocation process takes slightly longer than a regular allocation. When using pageable memory, it must be copied to the pinned memory cell before transferring it. Pinned memory prevents it from being paged out to disk during memory transfers. The example in this section shows that larger data transfers between the CPU and GPU can lead to increased throughput. Furthermore, in this system, the maximum achievable throughput only reaches approximately 72% of the theoretical peak performance.
5. Platforms with multiple GPUs and MPI
Now that we've introduced the basic components of a GPU-accelerated platform, let's discuss the more exotic configurations you might encounter. These exotic configurations are a result of the adoption of multiple GPUs. Some platforms offer multiple GPUs per node, connected to one or more CPUs. Others offer connectivity to multiple compute nodes via networking hardware.
On platforms with multiple GPUs (Figure 9.9), an MPI+GPU approach is typically required to achieve parallelism. To achieve data parallelism, each MPI rank is assigned to one GPU. Let's consider several possibilities:
Fig. 9.9 Here we illustrate a platform with multiple GPUs. A single computing node can have multiple GPUs and multiple CPUs. Furthermore, multiple nodes can be connected via a network.
5. Platforms with multiple GPUs and MPI
Some early GPU software and hardware failed to handle efficient multiplexing, resulting in poor performance. With many performance issues addressed in recent software, multiplexing MPI ranks across GPU processors is becoming increasingly attractive.
Optimizing Data Transfer between GPU Processors over a Network
To use multiple GPUs, we must send data from one GPU to another. Before we can discuss optimization, we need to describe the standard data transfer process.
1. Copy data from the GPU to the host processor.
a. Move data over the PCI bus to the processor.
b. Store data in the CPU's DRAM.
5. Platforms with multiple GPUs and MPI
2. Send data in an MPI message to another processor.
a. Store data from CPU memory to the processor.
b. Move data over the PCI bus to the network interface card (NIC).
c. Store data from the processor to CPU memory.
3. Copy data from the second processor to the second GPU.
a. Load data from CPU memory to the processor.
b. Send data over the PCI bus to the GPU.
As shown in Figure 9.10, this process involves significant data movement and will significantly limit application performance.
5. Platforms with multiple GPUs and MPI
Figure 9.10. The top section shows a standard data move for sending data from a GPU to other GPUs. The bottom section bypasses the CPU when moving data from one GPU to another.
5. Platforms with multiple GPUs and MPI
In NVIDIA GPUDirect®, the CUDA language adds the ability to send data as messages. AMD has a similar capability, called DirectGMA, for exchanging data between GPUs in the OpenCL language. While a pointer to the data must still be passed, the message itself is sent over the PCI bus directly from one GPU to another, thereby reducing memory footprint.
A Higher-Performance Alternative to the PCI Bus
There is little argument that the PCI bus is the primary limitation for compute nodes with multiple GPUs. While it is primarily a problem for large applications, it also impacts large workloads such as machine learning on small clusters. NVIDIA introduced NVLink® in its Volta P100 and V100 series of GPU processors to replace GPU-to-GPU and GPU-to-CPU connections. Starting with NVLink 2.0, data transfer rates can reach 300 GB/s. AMD's new GPUs and CPUs incorporate Infinity Fabric to accelerate data transfer. Intel has also been using this technology for several years to accelerate data transfer between CPUs and memory.
6. Potential Benefits of GPU-Accelerated Platforms
When is it worth offloading processing to the GPU? By this point, you've seen the theoretical peak performance of modern GPUs and how it compares to CPUs. Compare the GPU roof contour plot in Figure 9.5 with the CPU roof contour plot (from Section 3.2.4) for floating-point calculations and memory bandwidth limits. In practice, many applications don't reach these peak performance values. However, as the ceilings rise, GPUs have the potential to outperform CPU architectures in several metrics. These include application execution time, power consumption, cloud computing costs, and scalability.
Reducing Time to Solution
Suppose you have code that runs on CPU processors. You've spent a lot of time adding OpenMP or MPI to it to utilize all the CPU cores. You feel the code is well-tuned, but a colleague tells you that you could get more benefit by offloading it to GPU processors. Your application has over 10,000 lines of code, and you know that running it on GPU processors will require significant effort. At this point, you're interested in the prospect of working on GPU processors because you enjoy learning new things and trust your colleague's intuition. Now you need to pitch this idea to your colleagues and your boss.
6. Potential Benefits of GPU-Accelerated Platforms
An important measure of your application's performance is the reduction in time to solution for jobs that run for multiple days in a row. The best way to convey the impact of time-to-solution reduction is with an example. In this study, we'll use the Cloverleaf application as a proxy.
Example: Considering an Upgrade to Your Current, Heavily Used System
Below are the steps for evaluating Cloverleaf's performance on a baseline system. We're using an Intel Ivybridge system (E5-2650 v2 @ 2.60 GHz) with 16 physical cores. On average, your application runs for approximately 500,000 cycles. You can obtain a runtime estimate for a short trial run using the following steps.
1. Clone Cloverleaf.git with
git clone --recursive git@github.com:UK-MAC/CloverLeaf.git CloverLeaf
2. Then enter the following commands:
cd CloverLeaf_MPI make COMPILER=INTEL
sed -e '1,$s/end_step=2955/end_step=500/' InputDecks/clover_bm64.in\
>clover.in
mpirun -n 16 --bind-to core ./clover_leaf
The execution time for 500 cycles is 615.1 seconds, or 1.23 seconds per cycle. This gives a runtime of 171 hours, or 7 days and 3 hours.
6. Potential Benefits of GPU-Accelerated Platforms
Now let's get the runtimes for several possible replacement platforms.
CPU swap example: Skylake Gold 6152 with 2.10 GHz and 36 physical cores
The only difference for this run is that we increase the 16 processors to 36 for mpirun using the following command:
mpirun -n 36 --bind-to core ./clover_leaf
The Skylake system's runtime is 273.3 seconds, or 0.55 seconds per cycle. This would give a typical application runtime of 76.4 hours, or 3 days and 4 hours.
Not bad! Less than half the time you calculated earlier. You're almost ready to buy this system, but then you think maybe you should check out those GPU processors you've heard about.
6. Potential Benefits of GPU-Accelerated Platforms
GPU swap example: V100
CloverLeaf has a version of CUDA that runs on the V100. You measure performance using the following steps:
1. Clone Cloverleaf.git with
git clone --recursive git@github.com:UK-MAC/CloverLeaf.git CloverLeaf
2. Then type:
cd CloverLeaf_CUDA
3. Add the CUDA architecture flags for Volta to the makefile in the current CODE_GEN architecture list:
CODE_GEN_VOLTA=-arch=sm_70
4. Add the path to the CUDA CUDART library, if necessary:
make COMPILER=GNU NV_ARCH=VOLTA
sed -e '1,$s/end_step=2955/end_step=500/' clover_bm64.in >clover.in
./clover_leaf
Wow! The run is so fast, you don't even have time for a cup of coffee. But here it is: 59.3 seconds! That's 0.12 seconds per cycle, or 16.5 hours for the full test case. Compared to the Skylake system, that's 4.6 times faster and 10.4 times faster than the original Ivy Bridge system. Just imagine how much more work you could accomplish overnight instead of over several days!
6. Potential Benefits of GPU-Accelerated Platforms
Is this performance gain typical for GPU processors? Performance gains for applications span a wide range, but these results are not unusual.
Reducing Power Consumption with GPU Processors
Energy costs are becoming increasingly important for parallel applications. Where computer power consumption was once unconcerned, the energy costs of operating computers, storage drives, and cooling systems are now rapidly approaching the same levels as the hardware acquisition costs over the lifetime of the computing system.
In the race for exoscale computing, one of the biggest challenges is maintaining the power requirements of an exoscale system at approximately 20 MW. For comparison, this is roughly enough to power 13,000 homes. Data centers simply don't have enough installed capacity to go beyond this level. At the other end of the spectrum, smartphones, tablets, and laptops operate on batteries with limited available energy (between charges). On these devices, it can be beneficial to focus on reducing computing power consumption to extend battery life. Fortunately, aggressive efforts to reduce power consumption have kept power consumption at a reasonable rate of increase.
6. Potential Benefits of GPU-Accelerated Platforms
Accurately calculating an application's power consumption is difficult without direct power consumption measurements. However, you can obtain a higher cost estimate by multiplying the manufacturer's thermal design power (TDP) by the application's runtime and the number of processors used. TDP is the rate at which energy is consumed under typical workloads. Power consumption for your application can be estimated using the formula:
Energy = (N Processors) × (R Watts/Processor) × (T Hours),
where Energy is the power consumption, N is the number of processors, R is the TDP, and T is the application's runtime. Let's compare a GPU-based system with a roughly equivalent CPU-based system (Table 9.6). We'll assume our application is memory-bound, so we'll calculate the power consumption and consumption for a 10 TB/s system.
6. Potential Benefits of GPU-Accelerated Platforms
Table 9.6: Designing a System Based on 10 Tbps GPUs and CPUs
To calculate the energy consumption for one day, we take the specifications from Table 9.6 and calculate the nominal power costs.
Example: Thermal Design Power (TDP) for a 22-Core Intel Xeon Gold 6152 CPU
The 22-core Intel Xeon Gold 6152 CPU has a Thermal Design Power (TDP) of 140 W. Let's assume your application uses 15 of these processors for 24 hours to fully complete its work. The estimated power consumption of your application is:
Energy = (45 Processors) × (140 W/Processors) × (24 hours) = 151.2 kWh.
This is a significant amount of energy! Your power consumption in this calculation is enough to power seven homes for the same 24 hours.
6. Potential Benefits of GPU-Accelerated Platforms
In general, GPUs have a higher thermal design power (TDP) than CPUs (300 W versus 140 W in Table 9.6), so they consume power at a higher rate. However, GPUs can potentially reduce execution time or require very little power to complete your computations. You can use the same formula as before, where N is now considered the number of GPUs.
Example: TDP for a Multi-GPU Platform
Suppose you migrate your application to a multi-GPU platform. You can now run your application on four NVIDIA Tesla V100 GPUs in 24 hours. NVIDIA's Tesla V100 GPU has a maximum TDP of 300 W. The estimated power consumption of your application is kWh:
Power = (12 GPUs) × (300 W/GPUs) × (24 hours) = 86.4 kWh.
In this example, the GPU-accelerated application runs with significantly lower power consumption compared to the CPU-only version. Note that in this case, even though the time to solution remains the same, power consumption is reduced by approximately 40%! We also see a 20% reduction in initial hardware costs, but we haven't taken into account the cost of the host CPU for GPU processors.
6. Potential Benefits of GPU-Accelerated Platforms
We can now see that a GPU-based system has great potential, but the nominal values we used are quite far from reality. We could further refine this estimate by obtaining measured performance and power consumption metrics for our algorithm.
Achieving power savings with GPU-based accelerators requires that the application exhibit sufficient parallelism and that the device resources are used efficiently. In this hypothetical example, we were able to halve the power consumption when running on 12 GPUs in the same amount of time as running on 45 fully subscribed CPUs. The power consumption formula also suggests other power reduction strategies. We'll discuss these strategies later, but it's important to note that, in general, a GPU consumes more power per unit of time than a CPU. We'll begin by inspecting the power consumption between a single CPU and a single GPU.
Listing 9.3 shows how to plot the power consumption and utilization of a V100 GPU.
6. Potential Benefits of GPU-Accelerated Platforms
Example: Monitoring GPU Power Consumption Throughout the Life of an Application
Let's return to the CloverLeaf task we ran earlier on the V100 GPU. We used the nvidia-smi (NVIDIA System Management Interface) tool to collect runtime performance metrics, including GPU power and utilization. To do this, we ran the following command before running our application:
nvidia-smi dmon -i 0 --select pumct -c 65 --options DT\
--filename gpu_monitoring.log &
The table below shows the nvidia-smi command options.
6. Potential Benefits of GPU-Accelerated Platforms
The abbreviated data output to the file looks like this:
6. Potential Benefits of GPU-Accelerated Platforms
6. Potential Benefits of GPU-Accelerated Platforms
6. Potential Benefits of GPU-Accelerated Platforms
Figure 9.11 shows the resulting graph. We also integrate the area under the curve to obtain power consumption. Note that even at 100% efficiency, the power is only about 61% of the GPU's nominal power specification. At idle, GPU power consumption is about 20% of the nominal value. This shows that the real power consumption of GPU processors is significantly lower than estimates based on nominal specifications. The processors are also below their nominal capacity, but likely not by such a large percentage of their nominal level.
Figure 9.11 Power consumption for the CloverLeaf task running on a V100. We integrate under the power curve to get 10.4 kJ for a run that lasted about 60 seconds. This power consumption is about 61% of the rated power specification for the V100 GPU.
6. Potential Benefits of GPU-Accelerated Platforms
When Will Multi-GPU Platforms Save You Energy?
Generally, parallel performance decreases as you add more CPUs or GPUs (remember Amdahl's Law from Section 1.2?), and the cost of computing increases. Sometimes there are fixed costs (such as storage) associated with the overall execution time of a job, which are reduced if the job completes early and the data can be transferred or deleted. However, in a typical situation, you have a set of jobs to execute, with a choice of processors for each job. The following example illustrates the tradeoffs in this situation.
Example: Set of Parallel Jobs to Execute
You have 100 jobs to execute, all of roughly the same type and length. You can choose to execute these jobs on either 20 processors or 40 processors. On 20 processors, each job takes 10 hours to complete. Parallel performance in this range of processors is approximately 80%. The cloud cluster you have access to has 200 processors. Let's consider two scenarios.
6. Potential Benefits of GPU-Accelerated Platforms
Case 1. Executing 10 tasks at a time with 20 processors gives us the following solution:
Total execution time for the entire set = 10 hours × 100/10 = 100 hours.
Case 2. Executing 5 tasks at a time with 40 processors.
Increasing the number of processors to 40 reduces the execution time to 5 hours if the parallel efficiency is ideal. However, it is only 80% efficient, so the execution time is 6.25 hours. How did we arrive at this number? Let's assume that 20 processors is the base case. For doubling the number of processors, the parallel efficiency formula is:
Pefficiency = S/Pmultiply = 80%.
S is the speedup of the problem. The processor multiplier, Pmultiply, is equal to 2. Solving the speedup equation yields:
S = 0.8 × Pmultiply = 0.8 × 2 = 1.6.
6. Potential Benefits of GPU-Accelerated Platforms
Now we use the speedup equation to calculate the new time (TN):
S = Tbasis / Tnew.
For this form of the speedup equation, we use Tbasis instead of Tserial. Parallel efficiency often decreases as processors are added, so we want to have a relationship at this point on the efficiency curve. Solving the equation for Tnew, we get:
Tnew = Tbasis /S = 10/1.6 = 6.25 h.
Considering that this execution time applies to 40 processors, we can now calculate the total execution time of the suite:
Total suite time = 6.25 h × 100/5 = 125 h.
Thus, the scenario used in Case 1 completes the same amount of work much faster.
6. Potential Benefits of GPU-Accelerated Platforms
This example shows that if we're optimizing execution time for a large set of tasks, it's often better to use less parallelism. Conversely, if we're more concerned with the execution time of a single task, it's better to use more processors.
Reducing Cloud Computing Costs by Using GPU Processors
Cloud computing services from Google and Amazon allow you to combine your workloads with a wide range of compute server types and requirements.
As computing costs become more visible with cloud computing services, optimizing your application's performance becomes a higher priority. The advantage of cloud computing is that it gives you access to a wider range of hardware than you might have on-premises, and provides more options for combining hardware with your workload.
7. When to use GPU processors
GPUs are not general-purpose processors. They are most suitable in situations where the compute workload is similar to a graphics workload—a large number of identical operations. GPU processors still perform poorly in some areas, although technological solutions are being found in some areas with each iteration of GPU hardware and software development.
8. Exercises
1. Table 9.7 shows the achievable application performance at one flop per load. Look at current GPU market prices and fill in the last two columns to get the flop per dollar for each GPU. What does the best value for money look like? If your application's execution time is the most important criterion, which GPU is best to buy?
Table 9.7: Achievable Application Performance at One Flop per Load with Different GPUs
2. Measure the streaming throughput of your GPU or another selected GPU. How does it compare to those presented in this chapter?
3. Use the likwid performance tool to find out the CPU requirements for the CloverLeaf application on a system where you have access to hardware power meters.