���������File Operations for a Parallel World
Jumanazarov Mardonbek
Plan:
File systems create an organized workflow for retrieving, storing, and updating data. In any computational work, the product is the result, whether it is data, graphics or statistics. This includes not only final results, but also intermediate results related to graphics, checkpointing and analysis. Checkpoint state capture is especially necessary in large HPC systems with long-running calculations that can take days, weeks, or months.
DEFINITION Checkpointing state is the practice of periodically saving the state of a computation to disk to be able to restart the calculation in the event of system failures or due to finite durations of executions in the batch system.
When processing data from highly parallel applications, you need a secure and efficient way to read and store data at runtime. This explains the need to understand file operations in a parallel world. Considerations such as correctness, reduction of duplication of results and productivity must be taken into account.
It is always important to remember that the scaling of file system performance has not kept pace with other computer hardware. We scale computes to billions of cells or particles, which places serious demands on file systems. With the advent of machine learning and data science, more and more applications need big data, which requires large sets of files and complex workflows with intermediate file storage.
In this chapter, we will discuss how to modify file operations for parallel applications to ensure that you can write data efficiently and make the best use of available hardware. We believe it is the foundation needed for today's parallel applications. You will learn how to speed up the file writing operation by orders of magnitude while maintaining correctness. We'll also look at the different information and hardware tools that are commonly used for large HPC systems. We will use an example of recording data from a circular cell regular lattice decomposition using different software and information for parallel files. We recommend that you consult the examples in this chapter at https://github.com/EssentialsOfParal-lelComputing/Chapter16.git .
1. Components of a High-Performance File System
First, we'll look at the hardware that makes up a high-performance file system. Traditionally, file operations store data on a hard disk drive through a mechanism that writes a series of bits to a magnetic substrate. Like many other parts of HPC systems, storage hardware has become more complex, with deeper hardware hierarchies and varying performance characteristics. This evolution of storage hardware is analogous to the deepening of cache hierarchies in processors as their performance increases. Storage hierarchies also help cover the large differences in throughput at the processor level compared to mechanical disk storage. This is because mechanical components are much more difficult to scale down than electrical circuits. The introduction of solid-state drives (SSDs) and other solid-state devices has provided a workaround for scaling physical spinning disks.
1. Components of a High-Performance File System
Let's first define what might make up an HPC storage system, as shown in Figure 16.1. Typical storage hardware components include the following:
1. Components of a High-Performance File System
Figure 16.1 A diagram showing the location of burst buffer hardware between compute resources and disk storage. Burst buffers can be local to nodes or shared across the network.
1. Components of a High-Performance File System
The storage diagram in Figure 16.1 illustrates the storage hierarchy between the compute and storage systems. Burst buffers are inserted between the compute hardware and the main disk storage to cover the widening performance gap. Burst buffers can be located on each node or on I/O nodes and shared across the network with other compute nodes.
With the rapid advancement of solid-state storage technology, burst buffer designs will evolve in the near future. In addition to closing the gap in latency and throughput performance, new storage designs are increasingly driven by the need to reduce power requirements as systems grow larger. Magnetic tape has traditionally been used for long-term storage, and some designs have even turned to "dark disk," where spinning disks are used but powered down when not needed.
2. Standard File Operations: Interface between Parallel and Sequential Processing
Let's first look at standard file operations. In our parallel applications, the conventional file processing interface is still a serial operation. It's impractical to have a hard drive for every processor. Even a per-process file is only viable in limited situations and on a small scale. Consequently, for each file operation, we transition from parallel to serial. A file operation should be viewed as a reduction (or read extension) to the number of processes that require special processing for parallel applications. This parallelism can be handled with a few simple modifications to standard file input and output (IO).
A large portion of the modifications for parallel applications are in the file operations interface. First, we should revisit our previous examples related to file operations. The file input example in Section 8.3.2 showed how to read data in one process and then pass it on to other processes. In Section 8.3.4, we used the collect operation within MPI to ensure that the results of processes are written in a deterministic order.
2. Standard File Operations: Interface between Parallel and Sequential Processing
(Pro Tip) To avoid further complications, the first step you should take when parallelizing an application is to review the source code and insert an if (rank == 0) statement before each I/O instruction. By reviewing the code, you should identify file operations that require additional treatment. These operations include the following (shown in Figure 16.2):
2. Standard File Operations: Interface between Parallel and Sequential Processing
Fig. 16.2 Modifications for a parallel application for working with a standard file system. All file operations are performed from rank 0.
2. Standard File Operations: Interface between Parallel and Sequential Processing
A common inefficiency is opening a file for every process; imagine this as the equivalent of a dozen people trying to open a door at the same time. While your program may not crash, it will cause problems on a large scale (imagine 1000 people opening the same door). There is a lot of contention for file metadata and the resulting lock installation to ensure correctness, which can take minutes with a larger number of processes. This contention can be avoided by opening the file in only one process. By adding parallel data calls at each transition point from serial to parallel and from parallel to serial processing, we can give modest parallel applications the ability to operate using standard files. This is sufficient for the vast majority of parallel applications.
As our applications grow in size, we can no longer easily collect or distribute data within a single process. Our biggest limit is memory; We don't have enough memory resources for a single process to consolidate data from thousands of other processes into a single one. Therefore, we must have a different, more scalable approach to file operations. This is the topic of the next two sections, which are devoted to MPI file operations, called MPI-IO, and the Hierarchical Data Format v5 (HDF5). In these sections, we will show how these two libraries allow a parallel application to treat file operations in a parallel style. There are also other parallel file processing libraries, which we will mention in Section 16.5.
3. MPI File Operations (MPI-IO) for a More Parallel World
The best way to learn about the MPI-IO library is by seeing how it is used in a realistic scenario. We'll look at an example of writing a regular computational grid distributed across halo-cell processors using MPI-IO. This example will introduce you to the basic structure used in MPI-IO and several of the most common function calls.
The first parallel file operations were added to MPI in the MPI-2 standard in the late 1990s. The first widely available implementation of MPI file operations, called ROMIO, was led by Rajeev Thakur of Argonne National Laboratory (ANL). ROMIO can be used with any MPI implementation. Most MPI distributions include ROMIO as a standard part of their software release. MPI-IO has a large number of functions, all of which begin with the prefix MPI_File. In this section, we will only cover a subset of the most frequently used operations (see Table 16.1).
3. MPI File Operations (MPI-IO) for a More Parallel World
There are different ways to use MPI-IO. We are interested in the highly parallel, collective form, in which processes work together, writing to their own section of the file. For this, we will take advantage of the new MPI data type, which was first introduced in Section 8.5.1.
The MPI-IO library has both a shared file pointer for all processes and independent file pointers for each process. Using a shared pointer places a lock on each process and serializes file operations. To avoid locks, we use independent file pointers for higher performance.
File operations are divided into collective and non-collective operations. Collective operations use collective MPI data exchange calls, and all participants in the communicator must make the call, otherwise the communicator will hang. Non-collective calls are sequential operations that are called separately for each process. Table 1 shows the table. Figure 16.1 shows several common operations and the corresponding commands for each.
3. MPI File Operations (MPI-IO) for a More Parallel World
Table 16.1 Common MPI file routines
3. MPI File Operations (MPI-IO) for a More Parallel World
The file opening and closing operations are self-explanatory. The seek operation moves a separate file pointer to the specified location for each process. The MPI_File_set_info procedure can be used to pass both general and vendor-specific hints. There is also the MPI_File_delete procedure, but it is a non-collective call. In this case, we assume that a non-collective call is sequential: each process deletes the file. For C and C++ programs, the remove function works equally well. Calling the MPI_File_set_size procedure with the expected file size can be more efficient than gradually increasing the file size with each write.
We'll start by considering independent file operations for read and write operations. When each process operates on its own independent file pointer, this is called an independent file operation. Independent file operations are useful for writing replicated data between processes. For these common data, they can be written from one rank using the procedures given in Table 16.2.
3. MPI File Operations (MPI-IO) for a More Parallel World
Table 16.2 MPI File Independent Procedures
You must write distributed data using collective operations (Table 16.3). When processes operate on a file collectively, it is called a collective file operation. The corresponding write and read functions are similar to independent file operations, but _all is appended to the function name. To make best use of collective operations, we need to create complex MPI data types. The MPI_File_set_view function is used to set the data layout of a file.
3. MPI File Operations (MPI-IO) for a More Parallel World
Table 16.3 MPI Collective File Procedures
In this example, we'll split the source code into four blocks. (The complete source code for this example is included in the chapter's source code.) First, we must create an MPI data type for the memory layout and another for the file layout; these are called the memspace and filespace, respectively. Figure 16.3 shows these data types for a scaled-down 4x4 version of our example. For simplicity, we show only four processes, each with a 4x4 grid surrounded by a single-cell halo. The depth of the halo in the figure is ng (from "number of ghost cells").
3. MPI File Operations (MPI-IO) for a More Parallel World
Figure 16.3: 4x4 data blocks from each process, written without halo cells to contiguous sections of the output file. The top row is the memory layout within the process, called the memspace. The middle row is the memory within the file with halo cells removed, called the filespace. The memory within the file is actually linear, so it takes the form shown in the last row.
3. MPI File Operations (MPI-IO) for a More Parallel World
The first block of code in Listing 16.1 shows the creation of the two data types. This only needs to be done once, at the beginning of the program. The data types must then be freed at the end of the program in the finalization routine.
3. MPI File Operations (MPI-IO) for a More Parallel World
3. MPI File Operations (MPI-IO) for a More Parallel World
In this first step, we created the two data types shown in Figure 16.1. Now we need to write these data types to a file. The writing process consists of four steps, as shown in Listing 16.2.
1. Create a file.
2. Set the view on the file.
3. Write each array using a collective call.
4. Close the file.
3. MPI File Operations (MPI-IO) for a More Parallel World
3. MPI File Operations (MPI-IO) for a More Parallel World
3. MPI File Operations (MPI-IO) for a More Parallel World
During open, several optimizations can be performed using hints in the MPI_Info object (line 53). The hint could be that file operations should be performed using collective operations, collective_buffering, as in line 55. Or the hint could be a file system specific for striping across eight hard drives, striping_factor = 8, as in line 56. We discuss hints in more detail in Section 16.6.1.
We can also pre-allocate file space, as shown in line 61, so that it does not have to be increased during writes. Reading a file consists of the same four steps as the write process shown earlier and is shown in the following listing.
3. MPI File Operations (MPI-IO) for a More Parallel World
3. MPI File Operations (MPI-IO) for a More Parallel World
The read operation requires fewer hints and settings than the write operation. This is because some of the read settings are determined from the file. So far, these MPI-IO file operations have been written in a general form that can be called to solve any problem. Now let's look at the main application source code in the following listing, which specifies the calls.
3. MPI File Operations (MPI-IO) for a More Parallel World
3. MPI File Operations (MPI-IO) for a More Parallel World
3. MPI File Operations (MPI-IO) for a More Parallel World
This configuration requires a little explanation. The source code above supports writing more than one MPI data file. This is commonly referred to as writing to an N × M file, where N processes write M files and where M is greater than one but much smaller than the number of processes (Figure 16.4). The reason for this technique is that for larger problem sizes, writing to a single file does not always scale well.
3. MPI File Operations (MPI-IO) for a More Parallel World
We can partition the processes into groups by color, as shown in Figure 16.4. In lines 17–22 of Listing 16.4, we set up a new communicator based on M colors, where M is the number of files. The number of files is specified in line 19, and our color is calculated in lines 20 and 21. ranks_per_file is a floating-point type to handle the uneven division of ranks. We then obtain a new rank within our color. Each communication group on the right side of Figure 16.4 has 4096 processes, or ranks. The rank order is the same as in the global communication group. If more than one file exists, the file names include the color number on lines 59–64. Currently, this source code specifies only one color and writes only one file, as shown on the left side of Figure 16.4, but it is written to support more files.
3. MPI File Operations (MPI-IO) for a More Parallel World
Fig. 16.4. For larger processes, they can be split into communication groups by color so that they write to separate files. The subgroup ranks are arranged in the same order as the ranks in the original communicator.
3. MPI File Operations (MPI-IO) for a More Parallel World
We also need to know the locations of the initial x and y values of each process. For data decompositions that contain the same number of rows and columns per process, the computation only needs to know the location of the process in the global set. But when the number of rows and columns varies across processes, we need to sum all the dimensions below our position. As we discussed earlier in Section 5.6, this operation is a common parallel pattern called a scan. To perform this computation, in lines 22–34, we create communicators for each row and column. These perform an exclusive scan operation to obtain the initial x and y locations per process. In the source code above, we subdivide the data only in the x-coordinate direction to make things a little simpler. The global and process dimensions of the array subdivisions are specified in lines 27–44. This includes the data offsets calculated using exclusive scans.
Now that we have all the necessary information about the data layout, we can call our mpi_io_file_init routine on line 52 to set up the MPI data types for the memory layout and file system. This only needs to be done once, at startup. We are then free to call our writing, write_mpi_io_file, and reading, read_mpi_io_file, routines on lines 63 and 65. We can call them as many times as needed during execution. In our example source code, we then validate the read data, compare it with the original data, and print an error if one occurs. Finally, we open the file in one process and use standard C binary reading to show how the data is laid out in the file. This is done by sequentially reading each value from the file and printing it.
3. MPI File Operations (MPI-IO) for a More Parallel World
Now let's compile and run this example. The build is a standard CMake build, and we're running it on four processors.
mkdir build && cd build cmake ..
make
mpirun -n 4 ./mpi_io_block2d
Figure 16.5 shows the output from a standard C binary read for a 10x10 grid on each processor.
3. MPI File Operations (MPI-IO) for a More Parallel World
Figure 16.5. The output of a small binary read source code for MPI-IO shows the file contents. When using the MPI-IO library, we had to write a small utility to check the file contents.
4. HDF5 as a self-describing format for better data management
In traditional data file formats, data is meaningless without the program code used to write and read the file. Hierarchical Data Format (HDF) version 5 takes a different approach. HDF5 provides a self-describing parallel data format. HDF5 is self-describing because the name and characteristics are stored in the data file. With HDF5, having a description of the data contained in the file means you no longer need the source code and can read the data simply by querying the file. HDF5 also has a rich set of command-line utilities (such as h5ls and h5dump) that can be used to query the file's contents. You'll find these utilities helpful in verifying the correctness of your files.
We want to write data in binary format for speed and accuracy. However, because it's in binary format, it's difficult to verify the correctness of the written data. If we read the data back, a problem may arise during the reading process. A utility that can query a file provides a way to examine a write operation separately from a read operation. In Figure 16.4 in the previous section on MPI-IO, we needed a small program to read the file contents. This is not required for HDF5, as a utility is already provided. In Figure 16.6 (shown later in this section), we used the h5dump command-line utility to view the contents. You can avoid writing source code for many common operations by using existing HDF5 utilities.
4. HDF5 as a self-describing format for better data management
The HDF5 parallel source code is implemented using MPI-IO. Because it is built on MPI-IO, the structure of HDF5 is similar. Despite the similarities, the terminology and individual function calls differ enough to cause some difficulties. We will consider the functions needed to write a similar parallel file processing routine as was done for MPI-IO. The HDF5 library is divided into lower-level functional groups. These functional groups are conveniently distinguished by the prefixes of all calls in the group. The first group is the mandatory file manipulation operations (Table 16.4), which together handle the operations of opening and closing files.
Table 16.4 HDF5 Collective File Procedures
4. HDF5 as a self-describing format for better data management
Next, we need to define new memory types. These are used to specify the parts of the data to write and the schema. In HDF5, these memory types are called data spaces. The data space operations in Table 16.5 include ways to extract patterns from a multidimensional array. Information about many additional procedures is in the "Further Reading" section at the end of this chapter (16.7.1).
Table 16.5 HDF5 Data Space Procedures
4. HDF5 as a self-describing format for better data management
There are other operations in the data space, including point-based operations, that we haven't discussed here. Now we need to apply these data spaces to a set of multidimensional arrays (Table 16.6). In HDF5, a multidimensional array is called a data set, which typically represents a multidimensional array or some other form of data in the application.
There's only one remaining operation group we need. This group, called property lists, allows you to modify or provide hints for operations, as shown in Table 16.7. Property lists can be used to set attribute values to enable collective read or write operations. Property lists can also be used to pass hints to the MPI-IO reference library.
4. HDF5 as a self-describing format for better data management
Table 16.6 HDF5 Dataset Procedures
Table 16.7 HDF5 Property List Procedures
4. HDF5 as a self-describing format for better data management
Let's move on to an example. We start this HDF5 example with the source code to create the file and memory data spaces. This process is shown in the following listing. In this listing, all arguments to HDF5 are in bold.
4. HDF5 as a self-describing format for better data management
4. HDF5 as a self-describing format for better data management
4. HDF5 as a self-describing format for better data management
In Listing 16.5, we used the same pattern to create the two data spaces: create a data object, specify the dimensional data arguments, and then select a rectangular array region. First, we created the global array region by calling the H5Screate_simple procedure. For the file data space, we set the dimensions to the global array sizes nx_global and ny_global on line 23, then used these dimensions on line 25 to create the data space. We then selected a region of the file data space for each processor using calls to the H5Sselect _hyperslab procedure on lines 32 and 48. A similar process is then performed for the memory data space.
Now that we have data spaces, the process of writing data to a file is straightforward. We open the file, create a data set, and write it. If there are other data sets, we continue writing them, and when we are finished, we close the file. The following listing shows how this is done.
4. HDF5 as a self-describing format for better data management
4. HDF5 as a self-describing format for better data management
4. HDF5 as a self-describing format for better data management
4. HDF5 as a self-describing format for better data management
4. HDF5 as a self-describing format for better data management
In Listing 16.6, the main write_hdf5_file routine uses the filespace data space created in Listing 16.5. We then wrote the data set using the H5Dwrite routine on line 72, using both the memory space and the filespace. We also created and passed a property list to tell the HDF5 library to use MPI-IO collective routines. Finally, on line 82, we closed the file. We also closed the property list and the data set on the previous lines to avoid a memory leak. To perform the file creation routine, we finally call the H5Fcreate routine on line 106, but several lines are needed to set up the hints. We wrapped the property list setup for collective writing and the MPI-IO hints along with the call and placed them in a separate routine. We also used the same approach to calling HDF5 on line 121 to create the dataset, allowing us to detail the different property lists that can be used.
The HDF5 data file read procedure, shown in the following listing, follows the same basic pattern as the previous write operation. The biggest difference between this listing and Listing 16.6 is that fewer hints and attributes are required.
4. HDF5 as a self-describing format for better data management
4. HDF5 as a self-describing format for better data management
4. HDF5 as a self-describing format for better data management
4. HDF5 as a self-describing format for better data management
Because the file already exists, we use the open call on line 168 in Listing 16.7 to specify read-only mode. (Using read-only mode allows for additional optimizations.) The file being accessed already has several attributes that were specified when it was written. Some of these attributes do not need to be specified when reading. The HDF5 listings so far may have contained a general-purpose library within the application. The following listing shows the calls that would be placed at various points in the main application.
4. HDF5 as a self-describing format for better data management
4. HDF5 as a self-describing format for better data management
In Listing 16.8, the initialization operation to set up the data spaces on line 53 can be performed once at program startup. You can then periodically write data to your program for graphics and checkpoints. Reads are then typically performed when restarting from a checkpoint at the beginning of execution. Finally, the call to finalize should be performed at the end of the program before completing the computation. Now let's compile and run this example. The build is a standard CMake build. We run it on four processors:
mkdir build && cd build
cmake ..
make
mpirun -n 4 ./hdf5block2d
With a single installation, the HDF5 package can be installed as either a parallel or serial version, but not both. A common problem is linking the wrong version with your application. We added some special source code to the CMake build system to preferentially select the parallel version, as shown in the following listing. This causes the program to fail if the HDF5 version is not parallel, preventing us from getting this error during the build.
4. HDF5 as a self-describing format for better data management
The example source code performs a validation test to ensure that the data read from the file matches the data we started with. We can also use the h5dump utility to print the data to the file. You can use the following command to view your data file. Figure 16.6 shows the output of the command below.
h5dump -y example.hdf5
5. Other parallel file software packages
In this section, we'll briefly review two of the most common parallel file manipulation software packages: PnetCDF and ADIOS. PnetCDF, short for Parallel Network Common Data Form, is another self-describing data format popular in the Earth systems community and among National Science Foundation (NSF)-funded organizations. Originally a completely separate software package, its parallel version is built on top of HDF5 and MPI-IO. The decision whether to use PnetCDF or HDF5 is heavily influenced by your community. Because the files your application produces are often shared with other users, it's important to use the same data standard.
5. Other parallel file software packages
Fig. 16.6 Using the h5dump command line utility also shows the contents of an HDF5 file without having to write any source code
ADIOS, or Adaptable Input/Output System, is also a self-describing data format at Oak Ridge National Laboratory (ORNL). ADIOS has its own binary format, but it can also use HDF5, MPI-IO, and other storage software.
6. Parallel File System: Hardware Interface
As data needs grow, more complex file systems become necessary. In this section, we introduce these parallel file systems. A parallel file system can significantly speed up file writing and reading by distributing the operations across multiple hard drives using multiple file writers or readers. Although we now have some parallelism in the file system, it is still a complex situation. There is still a mismatch between the parallelism of applications and the parallelism provided by the file system. For this reason, managing parallel operations is complex and highly dependent on the hardware platform and application requirements. To cope with this complexity, many parallel file systems use an object-oriented file structure. Object-oriented file systems are naturally suited to solving these kinds of problems. However, the performance and robustness of a parallel file system are often limited by the metadata that describes the location of the file data.
DEFINITION An object-oriented file system is a system that is organized around objects rather than files in a folder. An object-oriented file system requires a database or metadata to store all the information that describes an object.
6. Parallel File System: Hardware Interface
Writing parallel file operations is closely tied to the parallel file system's software and information system. This requires knowledge of the parallel file system being used and the existing settings for that installation and file system. Fine-tuning the software and information system for parallel files can sometimes lead to significant performance improvements.
Everything You Wanted to Know About Parallel File Tuning but Didn't Know How to Ask
When delving into the interaction of parallel file operations with the file system, it's useful to have additional information about the parallel library's settings. These settings can be configured differently for each installation. You can also obtain some high-level statistics that can help debug performance issues.
6. Parallel File System: Hardware Interface
Most MPI-IO libraries are one of two implementations: ROMIO, which is distributed with MPICH and many vendor system implementations, or OMPIO, which is the default in newer versions of OpenMPI. Let's first cover how to retrieve information from the OpenMPI OMPIO plugin or how to revert to ROMIO. To retrieve OMPIO OpenMPI configuration information, use the following commands:
Specifies the I/O plugin, either OMPIO or ROMIO. Older versions use ROMIO as the default plugin, while newer versions use OMPIO by default.
Prints information about the local OpenMPI configuration for this plugin.
Shows tooltips parsed from calls to the program's MPI_Info_set command.
6. Parallel File System: Hardware Interface
First, you can get the names of I/O plugins using the ompi_info command. We only need I/O component plugins, so we filter the output for them:
ompi_info |grep "MCA io:"
MCA io: romio321 (MCA v2.1.0, API v2.0.0, Component v4.0.3)
MCA io: ompio (MCA v2.1.0, API v2.0.0, Component v4.0.3)
Then you can get the individual settings available for each plugin. Using the ompi_info command, we get the following abbreviated output:
6. Parallel File System: Hardware Interface
You can also verify how MPI_Info_set command calls are interpreted by the MPI-IO library using the following runtime option. This can be a useful tool for verifying the correctness of the source code for your file system and parallel file operations libraries.
mpirun --mca io_ompio_verbose_info_parsing 1 -n 4 ./mpi_io_block2d
File: example.data info: collective_buffering value true enforcing using individual fcoll component
< ... repeat three more times ... >
In the ROMIO parallel file information system, part of MPICH, we have various mechanisms for querying the information system installation. Cray adds several additional environment variables to its ROMIO implementations. We'll list some of them and then look at examples where they're used.
– ROMIO_PRINT_HINTS=1.
– MPICH_MPIIO_HINTS_DISPLAY=1;
– MPICH_MPIIO_STATS=1;
– MPICH_MPIIO_TIMERS=1.
6. Parallel File System: Hardware Interface
Below is the result when using ROMIO_PRINT_HINTS:
6. Parallel File System: Hardware Interface
export MPICH_MPIIO_HINTS_DISPLAY=1; srun -n 4 ./mpi_io_block2d
6. Parallel File System: Hardware Interface
MPIIO WARNING: DVS stripe width of 8 was requested but DVS set it to 1 See MPICH_MPIIO_DVS_MAXNODES in the intro_mpi man page.
PE 0: MPIIO hints for example.data:
6. Parallel File System: Hardware Interface
export MPICH_MPIIO_STATS=1; srun -n 4 ./mpi_io_block2d
6. Parallel File System: Hardware Interface
6. Parallel File System: Hardware Interface
General Hints Applicable to All File Systems
Sometimes it's useful to provide some hints about the type of file operations you'll be using in your application. You can modify parallel file settings using environment variables, a hint file, or at runtime using the MPI_Info_set command. This provides a convenient method for manipulating different scenarios if you don't have access to the program's source code to add the MPI_Info_set command. To set parallel file options in this case, use the following commands.
MPICH_MPIIO_HINTS="*:<ключ>=<значение>:<ключ>=<значение>.
Например:
export MPICH_MPIIO_HINTS=\
"*:striping_factor=8:striping_unit=4194304"
6. Parallel File System: Hardware Interface
ROMIO_HINTS=<filename>.
For example: ROMIO_HINTS=romio-hints, where the file romio-hints contains:
striping_factor 8 // the file is split into 8 parts and
// written in parallel to 8 disks
striping_unit 4194304 // the size in bytes of each
// written block
OMPI_MCA_<parameter_name> <value>
For example: export OMPI_MCA_io_ompio_verbose_info_parsing=1.
The OpenMPI mca runtime option as an argument to the mpirun command is:
mpirun --mca io_ompio_verbose_info_parsing 1 -n 4 <exec>
6. Parallel File System: Hardware Interface
The default OpenMPI file location is in $HOME/.openmpi/mca-params.conf, or you can specify it as follows:
--tune <filename>
mpirun --tune mca-params.conf -n 2 <exec>
The most important hint you can specify is whether to use collective operations or data sifting. We'll cover collective operations first, then data sifting operations.
Collective operations use MPI collective calls and a two-phase I/O approach that collects data for aggregators, which then write to or read from your file. Use the following commands for collective I/O:
– cb_buffer_size=integer specifies the buffer size in bytes for two-phase collective I/O. It must be a multiple of the page size.
– cb_nodes=integer specifies the maximum number of aggregators.
6. Parallel File System: Hardware Interface
– romio_cb_read=[enable|automatic|disable] specifies when to use shared buffering for read operations.
– romio_cb_write=[enable|automatic|disable] specifies when to use shared buffering for write operations.
– cb_config_list=*:<integer> specifies the number of aggregators per node.
– romio_no_indep_rw=[true|false] specifies whether to use any independent I/O. If these are not enabled, no file operations (including file opens) will be performed on nodes that are not aggregators.
– collective_buffering=[true|false] uses collective operations when writing from a parallel job to the file system.
6. Parallel File System: Hardware Interface
Data sieving performs a single read (or write) covering a file block and then passes the data to individual reader processes. This avoids a large number of small reads and the potential contention between file readers. Use the following commands to sieving data with ROMIO:
Filesystem-Specific Hints
Some hints apply only to a specific file system, such as Lustre or GPFS. We can detect the file system type from our program and set appropriate file system hints. The fs_detect.c program in the examples does just that. This program uses the statfs command, as shown in the following listing, and you can find it in the examples directory for this chapter.
6. Parallel File System: Hardware Interface
6. Parallel File System: Hardware Interface
This listing includes a magic number for some parallel file systems. When using it for other applications, replace the filename in line 18 with the appropriate filename based on the directory where your files are stored. Build fs_detect, then run the following command to get the file system type:
mkdir build && cd build cmake ..
make
grep `./fs_detect | cut -f 4 -d' '` /usr/include/linux/magic.h ../fs_detect.c
Now we're ready for file system-related hints. We won't list every possible hint. You can get a current list using the commands shown earlier.
Lustre File System: The Most Widely Used File System in HPC Centers
Lustre is the dominant file system in the largest HPC systems. Originating at Carnegie Mellon University, its primary development and ownership have been transferred to Intel, HP, Sun, Oracle, Intel, Whamcloud, and others. During this process, it has transitioned between commercial and open source. It is currently under the auspices of the Open Scalable File Systems (OpenSFS) and European Open File Systems (EOFS) organizations.
6. Parallel File System: Hardware Interface
The Lustre filesystem is built on the concept of object storage using Object Storage Servers (OSS) and Object Storage Targets (OST). When we specify the striping_factor 8 hint on line 56 of Listing 16.2 and line 96 of Listing 16.6, we tell ROMIO to use Lustre to subdivide writes (and reads) into eight pieces and send them to eight operating systems, effectively writing data with eight-way parallelism. The striping_unit hint tells ROMIO and Lustre to use 4 MB stripes. Lustre also has Metadata Servers (MDS) and Metadata Targets (MDT) to store important descriptions of where each piece of a file is stored. For striping operations, use the following.
– striping_unit=<integer> specifies the stripe size in bytes.
– striping_factor=<integer> specifies the number of stripes, where -1 indicates automatic selection.
– fs_lustre_stripe_size=<integer> specifies the stripe size in bytes.
– fs_lustre_stripe_width=<integer> specifies the number of stripes, where -1 indicates automatic selection.
6. Parallel File System: Hardware Interface
We can confirm the Lustre parameters for OpenMPI using a console prompt on the command line:
ompi_info --param fs lustre --level 9
MCA fs lustre: parameter "fs_lustre_priority" (current value: "20") …
MCA fs lustre: parameter "fs_lustre_stripe_size" (current value: "0") … MCA fs lustre: parameter "fs_lustre_stripe_width" (current value: "0") …
GPFs: IBM's File System
IBM systems have a General Parallel File System (GPFS), also part of their Spectrum Scale product, which offers striping and parallel file operations on their systems. GPFS is an enterprise data storage product with associated support infrastructure and services. By default, GPFS defines stripes across all existing devices. However, MPI hints don't have much of an impact on this filesystem. For MPI-CH (ROMIO), the following command should be used to assist with large memory read/write operations:
IBM_largeblock_io=true
6. Parallel File System: Hardware Interface
datawarP: a file system from Cray
Cray's DataWarp integrates burst buffer hardware on top of another parallel file system, such as their version of Lustre. However, leveraging the benefits of burst buffers is still in its infancy, but Cray remains a leader in these efforts.
Panasas®: a commercial file system that requires less user input
Panasas® is a commercial parallel file system consisting of object storage and metadata servers. Panasas also contributed to the extension of parallel support to the Network File System (NFS). Panasas was used in several of LANL's top ten computing systems, although it is not as prevalent there today. For MPICH (ROMIO), use the following commands to set the stripe size and number of stripes, respectively:
6. Parallel File System: Hardware Interface
OrangeFS (PvFs): The Most Popular Open Source File System
OrangeFS, formerly known as the Parallel Virtual File System (PVFS), is an open source parallel file system from Clemson University and Argonne National Laboratory. It is popular on Beowulf clusters. In addition to being a scalable parallel file system, OrangeFS has been integrated into the Linux kernel. The following commands can be used for MPICH (ROMIO) to set the stripe size (in bytes) and stripe numbering (where -1 is automatic):
BeeGFs: A New, Gaining Popularity, Open Source File System
BeeGFS, formerly FhGFS, was developed at the Fraunhofer Center for High Performance Computing and is freely available. It is popular due to its open source characteristics.
6. Parallel File System: Hardware Interface
Distributed Application Object Storage (DAOS):
Setting New Performance Benchmarks
Intel is developing its new open-source DAOS object storage technology as part of the Department of Energy (DOE) FastForward program. DAOS (Distributed Application Object Storage) ranks first in the 2020 ISC IO500 supercomputer file speed list (https://www.vi4io.org). It is scheduled to be deployed in 2021 on the Aurora supercomputer, Argonne National Laboratory's first exoscale computing system. DAOS is supported by the MPI-IO library in ROMIO, is accessible with MPICH, and is portable to other MPI libraries.
WekaIO: A Newcomer to the Big Data Community
WekaIO is a fully POSIX-compliant file system that provides a large shared namespace with highly optimized performance, low latency, and high throughput, and utilizes the latest solid-state hardware components. The WekaIO file system is attractive for applications requiring large volumes of high-performance data file operations and is popular in the big data community. WekaIO ranked first in the 2019 SC IO500 supercomputer file speed list.
6. Parallel File System: Hardware Interface
cePh File System: An Open-Source Distributed Storage System
Ceph originated at Lawrence Livermore National Laboratory. Development is currently led by RedHat for a consortium of industrial partners and is integrated into the Linux kernel.
Network File System (NFS): The most common network file system
NFS is the dominant cluster file system for local area networking. It is not recommended for highly parallel file operations, although it functions correctly when configured appropriately.
7. Exercises
1. Check for hints available on your system using the techniques described in Section 16.6.1.
2. Try the MPI-IO and HDF5 examples on your system with much larger datasets to see the performance you can achieve. Compare this to the IOR microbenchmark for extra credit.
3. Use the h5ls and h5dump utilities to perform reconnaissance analysis of the HDF5 data file created by the HDF5 example.