1 of 12

Linux Clusters Institute:�Storage Lab

J.D. Maloney | Lead HPC Storage Engineer

National Center for Supercomputing Applications (NCSA)

malone12@illinois.edu

1

Feb 5-9, 2024

This document is a result of work by volunteer LCI instructors and is licensed under CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/).

2 of 12

Goals of the Exercises

  • Demo/Get Experience with some common storage-related commands that are generic across a variety of solutions
    • We’ll walk through these commands together and discuss them
  • Combine local disks on head node with two different tools:
    • mdadm
    • ZFS
  • Briefly touch on some ZFS policy tools
  • NFS export your head-node volume to your compute nodes

2

Feb 5-9, 2024

3 of 12

Common/Handy Storage Commands

Commands: lsscsi & fdisk

  • Helpful for viewing disks attached/presented to a host
  • Useful for verifying sizes, counts, and other details about devices
  • Use these commands to:
    • See how many drives are attached to your instance
    • Check the capacity of these drives
    • What devices map to the data drives for this lab?
  • Note what you see here are examples; not from your VM

​

3

Feb 5-9, 2024

4 of 12

Common/Handy Storage Commands

Commands: iostat

  • Helpful for viewing disk activity, performance characteristics
  • Provided by the sysstat package in RHEL-based distros; you’ll likely need to run the below to install, not in minimal installs usually
    • dnf -y install sysstat
  • I usually invoke it with the following arguments: iostat -xkz 2
    • You can mess with other flags/intervals as you please
  • We don’t have an FS up yet but you can run the command either against your OS drive (to watch its activity)…or open a second SSH session into your login node and issue the below command:
    • dd if=/dev/urandom of=/dev/$your_dev bs=1M count=4500
    • Quickly on your original session run: iostat -xkz 2

4

Feb 5-9, 2024

5 of 12

Common/Handy Storage Commands

Commands: netstat & iotop

  • These commands are handy to view sources of storage traffic
    • iotop is only helpful if I/O is being done by a process on a host to its local disk (eg. /scratch.local or /tmp on a compute node)
    • netstat is handy to quickly view what clients are doing I/O to/from storage servers for network file systems
  • You’ll need to install iotop: dnf -y install iotop
  • Commands will look like:
    • iotop
    • netstat -i $interface_to_monitor

5

Feb 5-9, 2024

6 of 12

Common/Handy Storage Commands

Commands: namei -l

  • Very useful when debugging permissions issue on a file system of any type
  • New cluster/linux users often struggle in this area and this is a great visual way to help folks see things
  • Pick any file on your head node (try to find one at least 3-4 levels deep)

6

Feb 5-9, 2024

7 of 12

mdadm - Standard Linux Software RAID

  • Make sure mdadm is installed: dnf -y install mdadm
  • Let’s make a RAID6 across our four devices:
    • mdadm --zero-superblock --force /dev/$dev_regex
    • mdadm --create --verbose /dev/md0 --level=6 --raid-devices=4 /dev/$dev_regex
    • Running lsblk should show these all a part of md0
    • Also you can check the status by running: cat /proc/mdstat
    • View information about the array: mdadm -D /dev/md0
    • Format the volume: mkfs.ext4 /dev/md0
    • Mount the volume: mkdir /scratch && mount /dev/md0 /scratch

7

Feb 5-9, 2024

8 of 12

mdadm - Standard Linux Software RAID

  • If you want to experiment more with mdadm you can destroy the current pool and re-create it other ways (eg. using RAID1 or RAID5 for example); to reset things run:
    • umount /scratch
    • mdadm --stop /dev/md0
    • mdadm --remove /dev/md0
    • mdadm --zero-superblock --force /dev/$dev_regex

8

Feb 5-9, 2024

  • Now you can re-run create commands with other topologies of disks

9 of 12

ZFS

  • First we need to install the ZFS repository:
    • dnf -y install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm
  • Install dependencies:
    • dnf install -y epel-release kernel-devel
  • Install ZFS itself
    • dnf install -y zfs
  • Load the ZFS module
    • /sbin/modprobe zfs
  • You should now be be able to run the below and see there are currently no pools:
    • zpool status

9

Feb 5-9, 2024

10 of 12

ZFS

  • Let’s create a zpool, with two drive parity
    • zpool create scratch raidz2 $dev1 $dev2 $dev3 $dev4
  • Check for success with a:
    • zpool status
    • df -h
  • You should now have a functioning zpool, you can see some details about it by running:
    • zfs get all scratch
  • Leave this pool in place, do not destroy it, we’re using it momentarily

10

Feb 5-9, 2024

11 of 12

NFS Export

  • We’re going to export our ZFS scratch space out to our compute nodes so they can share it for scratch
  • Install the nfs-utils package
    • dnf -y install nfs-utils
  • Edit your exports file to configure the exports, a single line should look something like:
    • /scratch 192.168.0.5(rw,sync,no_root_squash)
    • You can regex the IP to look like 192.168.0.[5-8] and get all four compute nodes covered by one export
  • Start up nfs-server and make sure FS is exported
    • systemctl start nfs-server && exportfs -ra

​

11

Feb 5-9, 2024

12 of 12

NFS Export

  • SSH into a compute node from a head node, create the mount directory
    • mkdir -p /scratch
  • Configure fstab
    • Add line: 192.168.0.4:/scratch /scratch nfs defaults 0 0
  • You should be able to mount the file system
    • mount /scratch

12

Feb 5-9, 2024