Linux Clusters Institute:�Storage Scale Hands On Exercise

J.D. Maloney | Lead HPC Storage Engineer

Storage Enabling Technologies Group (SET)

National Center for Supercomputing Applications (NCSA)

malone12@illinois.edu

Mississippi State, August 21^st – 25^th 2023

Goal of Hands on Exercise

• Create Storage Scale cluster
• Create File System
• Create & Link Filesets
• Run policy on sample data
• LUN manipulation
• Rebalancing/data migration
• Failure groups
• Explore common commands
• mmdiag commands
• mmls* commands

2

August 21 – 25, 2023

Lay of the Land

• You should have 4 storage servers; 1 for metadata, 3 for data
• Each have 2 5GB volumes
• Root SSH between all nodes in cluster
• Time in sync on all nodes
• DNS and/or /etc/hosts to allow nodes to lookup each other’s IP addresses
• Storage Scale packages and dependencies should already be installed

3

August 21 – 25, 2023

Creating Storage Scale Cluster

4

August 21 – 25, 2023

• Create file with the storage nodes names in it that looks like this:

[root@storage1 ~]# cat ~/lci_storage_nodes

storage1:quorum

storage2:quorum

storage3:quorum

storage4

[root@storage1 ~]# mmcrcluster -N ~/lci_storage_nodes -r `which ssh` -R `which scp` -C lci

[root@storage1 ~]# mmchlicense server --accept -N ~/lci_storage_nodes

• Create your Storage Scale Cluster

• Get the licensing fixed

• Create files with your clients

[root@storage1 ~]# cat ~/lci_client_nodes

compute1

compute2

Creating Storage Scale Cluster

5

August 21 – 25, 2023

• Add clients to cluster & apply client license

[root@storage1 ~]# mmaddnode –N ~/lci_client_nodes�[root@storage1 ~]# mmchlicense client –accept -N ~/lci_client_nodes

• Check that all 6 nodes are in your cluster

[root@storage1 ~]# mmlscluster

• Create separate node classes for your storage nodes and your client nodes

[root@storage1 ~]# mmcrnodeclass nsdnodes –N storage1,storage2,storage3,storage4

[root@storage1 ~]# mmcrnodeclass clients –N compute1,compute2

Creating the NSDs

• Create your NSD File, name it “lci_nsds”
• Sample stanza below add a %nsd block for each NSD disk; make sure to change the following for each NSD: device_name, nsd name, servers list

6

August 21 – 25, 2023

• Once you have a file that has all 8 NSDs in it, run create command

%nsd:

device=/dev/$device_name

nsd=storage1_disk0

servers=storage1

usage=dataAndMetadata

failureGroup=1

[root@storage1 ~]# mmcrnsd –F ~/lci_nsds -

• Run ”mmlsnsd” and confirm you have NSDs, should look like output on the right

​

Startup the Cluster & Create FS

• Startup the cluster on your NSD servers

7

August 21 – 25, 2023

• Wait until they all are in active state, check via “mmgetstate

[root@storage1 ~]# mmstartup -N nsdservers

[root@storage1 ~]# mmgetstate -N nsdservers

• Create your file system using your NSDs

[root@storage1 ~]# mmcrfs lci –F ~/lci_nsds –B 1M –m 1 -r 1 –Q yes –T /gpfs/lci

• Mount the file system on your storage servers

[root@storage1 ~]# mmmount lci –N nsdnodes

• Startup clients and mount the FS there

[root@storage1 ~]# mmstartup –N clients && mmmount lci –N clients

Adding Some Filesets & Quotas

• Create filesets for home and scratch and junction them into place

8

August 21 – 25, 2023

• Set some quotas in Storage Scale

[root@storage1 ~]# mmcrfileset lci home --inode-space=new

[root@storage1 ~]# mmcrfileset lci scratch --inode-space=new

[root@storage1 ~]# mmlinkfileset lci home -J /gpfs/lci/home

[root@storage1 ~]# mmlinkfileset lci scratch -J /gpfs/lci/scratch

## Default quota of 5GB per user in the home fileset

[root@storage1 ~]# mmsetquota lci:home --default user –block 5G:5G

​

## Set a fileset quota on the scratch fileset of 10GB�[root@storage1 ~]# mmsetquota lci:scratch --block 10G:10G

Sample Policy Engine Run

9

August 21 – 25, 2023

• Copy over some sample data (password written on board):

• Sample policy run is provided, it is in lci_policy.sh

• Look at script to get an understanding of what it’s doing

• Go ahead and run the script

[root@storage1 ~]# rsync -avP lci@levi2.ncsa.illinois.edu:/gideon/scratch/lci_demo_data/* /gpfs/lci/scratch/

[root@storage1 ~]# /gpfs/lci/scratch/lci_policy.sh

LUN Manipulation

10

August 21 – 25, 2023

• Take a look at LUN capacities with mmdf (take screenshot)

• Suspend one of the data LUNs (doesn’t matter which)

[root@storage1 ~]# mmdf lci

[root@storage1 ~]# mmchdisk lci suspend –d “storage2_disk0”

• Run an mmdeldisk to drain the suspended NSD and remove it from the file system

[root@storage1 ~]# mmdeldisk lci “storage2_disk0” –r –N nsdnodes

• Run an mmdf again and compare with the “before” screenshot you captured

[root@storage1 ~]# mmdf lci

Running Useful Storage Scale Commands

11

August 21 – 25, 2023

• Take a look at the following Storage Scale commands, run them, as a team work to understand the output, ask questions if you have them

• mmlsconfig
• mmlscluster
• mmlsmgr
• mmlsnsd
• mmlsdisk
• mmlsfs
• mmlsfileset

mmdiag

Following Flags:

--config

--stats

--network

--waiters

--iohist

Wrap Up

12

August 21 – 25, 2023

• Further Exploration
• Other Storage Scale Commands you find interesting
• Run Through the delete steps
• mmdelfs
• mmdelnsd