This document is maintained by the departmental computing committee.
Cluster training video 2024. (2023) (slides).
Cluster Configuration and Resources
The login node for the CompBio cluster is at cluster.csb.pitt.edu, also referred to as the “head node”. To submit your jobs in a cluster, you should connect to the login node via ssh protocol. ssh is available as a command line program in Windows 10, Mac OS, and Linux. On Windows, you can either get Windows Subsystem for Linux (WSL), which comes with ssh, or use PuTTY which is a popular graphical user interface for ssh.
When connecting to the cluster from off-campus or the wireless network, VPN with GlobalProtect (https://services.pitt.edu/TDClient/33/Portal/KB/ArticleDet?ID=3427) will be required to connect. Request this VPN role from the department IT administrators.
The head node uses Slurm as workload manager software. In order to connect to the head node type: ssh <USERNAME>@cluster.csb.pitt.edu where <USERNAME> is your cluster userid. For example, a user abc123 would use: ssh abc123@cluster.csb.pitt.edu.
(The username is case sensitive and should be all lowercase)
Please change your password from the temporary one using the “passwd” command.
It is strongly recommended that you setup password-less authentication with ssh keys. This is more secure and more convenient. To do this first generate a key:
ssh-keygen -t ed25519
and then copy it over to the cluster head node:
ssh-copy-id <USERNAME>@cluster.csb.pitt.edu
The cluster is a shared resource that supports research by all members and affiliates of the department. We seek to maximize the conflicting goals of utilization and availability. If no one else is using the cluster, one person should be able to use every possible compute cycle. On the flip side, even if the cluster is in full use, someone with a single job to run should be able to expect their job to start running in a matter of hours, if not faster.
It is your responsibility to be cognizant of how your usage of the department cluster impacts other users and not prevent others from using this shared resource. See "Running Many Jobs" below.
As part of ensuring access to the cluster, never run computationally intensive tasks on the head node. Doing so could possibly disrupt the entire queuing system. If you need to compile software or debug a script, launch an interactive job to a cluster node. See "Interactive Jobs" below.
The cluster has more than 2600 compute cores. The department queue has the following classes of machines.
Additionally, the big_memory partition is available for departmental use and contains two nodes that each have 1TB of memory. Usage of these machines should be reserved for high-memory workloads.
There are currently 26 GPU nodes with a total of 163 GPU cards. The number of each class of GPU in the department queue is shown below.
You can get a summary of the current status of each GPU with the /opt/bin/gpus.py script, which can also be used to show the GPUs available in non-departmental partitions.
Nodes are assigned to one or more partitions. Jobs are submitted to one or more partitions and the SLURM scheduler automatically selects the best node to run on. The partitions available for use by anyone in the department are:
Other partitions available on the cluster are specific to individual research groups and should only be used with permission. The commands sinfo and snodes provide information on the availability of partitions and nodes. As an example:
sinfo --format="%.12P %.14F %C"
will print how many nodes and cores are allocated or idle on each partition.
The cluster nodes are currently running Ubuntu 22.04. Although several basic packages are already preinstalled, in most cases additional software is accessed as a module or installed locally to the user. If a package is likely to be used by multiple research groups, see the IT team about installing it cluster-wide.
Several software packages are available as modules accessible via the module tool. This tool creates/removes environment variables as needed to make installed software available. This allows users to select specific versions of a software package.
To see the available packages, use module avail
$ module avail
----------------------------------------------------- /usr/share/modules/modulefiles ------------------------------------------------------
amber/14 cuda/7.0 gromacs/5.1.4 namd/2.10/multicore namd/2.13/multicore
amber/16 cuda/7.5 gromacs/5.1.4-cuda namd/2.10/TCP namd/2.13/multicore-CUDA
amber/18(default) cuda/8.0(default) gromacs/2016.4 namd/2.11/cuda-7.5 namd/2.13/netlrts
anaconda/2 cuda/9.0 gromacs/2016.4-cuda(default) namd/2.11/multicore null
anaconda/3 cuda/9.1 gromacs/2019.3 namd/2.11b2/cuda-6.0 salmon/0.11.0
anaconda/3-cluster(default) cuda/9.2 gromacs/2019.3-cuda namd/2.11b2/multicore salmon/0.14.1
charmm/44b2(default) cuda/10.0 matlab/R2019b(default) namd/2.11b2/netlrts smina/smina
cuda/4.0 cuda/10.1 namd/2.7b4/multicore namd/2.11b2/TCP use.own
cuda/5.0 cuda/10.2 namd/2.8/multicore namd/2.12(default) westpa/2017.10
cuda/5.5 fastqc/0.11.9(default) namd/2.9/cuda-4.0 namd/2.12/cuda-6.5
cuda/6.0 gaussian/g09 namd/2.9/multicore namd/2.12/cuda-8.0(default)
cuda/6.5 gaussian/g16 namd/2.10/cuda-6.0 namd/2.12/multicore
To load a specific package use module load (and unload to remove it)
module list will show the currently loaded modules
module purge will remove all loaded modules
The Anaconda python distribution is designed for scientific computing in mind and comes with a lot of useful libraries pre-installed. If you are doing something scientific with python, you’ll likely want to use anaconda. You can load anaconda with:
$ module load anaconda # for python 3
$ which python # this should now point to anaconda
You can use conda (which is the package manager for Anaconda and related distributions) to create conda environments, which allow you to specify exactly what libraries python will have access to. This is very useful when you want to ensure you are using specific versions of a set of specified libraries.
The conda package manager is very useful for keeping package dependencies isolated and under control, especially through the use of environments as mentioned above. After loading the system-wide Anaconda module, you should then have access to conda features and be able to create your own environments where you can install the packages you need.
conda info --envs provides information about what environments are already available. These will include all the environments that anyone creates including your own.
These can be activated with the conda activate or source activate command followed by their name. To deactivate the currently active conda environment, you can use the conda deactivate command without any name.
To create a new environment, use the following command: conda create --name <ENV_NAME> where <ENV_NAME> is your desired name for the new environment. You can also add package names at the end of the command, which you know you need in your environment. For example, to create an environment named jk_prody (including your initials to help people, in this case JK for James Krieger) that needs scipy, numpy and matplotlib, you could type the following:
conda create --name jk_prody scipy numpy matplotlib
Once you’ve activated an environment, any conda or pip commands you run will act on that environment. This can be useful for adding more packages that aren’t accessible to the base conda environment or that you didn’t think of when creating the environment.
Another useful feature for conda is channels, which provide access to other packages not supported by the default channel from Anaconda, such as conda-forge. These can be listed with the command conda config --show channels and new ones can be added with the command conda config --add channels <CHANNEL_NAME>. Their priority order can also be controlled using --append or --prepend (same as --add) instead. It is best to manage channels in your own environment so as not to cause surprises for other users.
For whatever reason, conda does not act normally when run in a non-interactive shell (e.g., a slurm script). Include the following in your slurm script before activating your conda environment:
eval "$(conda shell.bash hook)"
or use conda run:
conda run -n ENV script.py
Python packages can also be installed using pip and the --user option which will install the package into the user's home directory in ~/.local
$ pip install --user MDAnalysis
Packages installed with pip will override the system installed versions and are generally much newer. Passing the -U option (for “upgrade”) will also upgrade any dependencies, and is often helpful in resolving issues. The -e option (for “editable”) can also be helpful for packages under development.
Note that if using system python (i.e., anaconda is not loaded) pip will install python2 packages and pip3 will install python3 packages.
The easiest way to transfer data to/from the cluster is with scp. To copy file into your home directory:
scp file USERID@cluster.csb.pitt.edu:.
To copy a directory dirname into a tmp subdirectory in your home directory (requires -r):
scp -r dirname USERID@cluster.csb.pitt.edu:tmp/
You can also mount your home directory as a remote file system using sshfs.
In order to submit a job, you put the commands you want to run in a shell script and provide this shell script to SLURM. SLURM will select a node to run your commands on. The shell script can communicate information to SLURM by embedding #SBATCH comments and SLURM communicates information to the shell script by setting environment variables.
$ cat test.sh
#!/bin/bash
#SBATCH -J test_job
#SBATCH -t 1-00:00:00
#SBATCH --partition=dept_cpu
#SBATCH --cpus-per-task=1
echo ${SLURM_JOB_NAME} allocated to ${SLURM_NODELIST}
The above script sets the job name to "test_job", sets a time limit of 1 day, requests scheduling to the dept_cpu partition, and requests a single CPU. When the job is scheduled it will print out the name of the job ("test_job") and the node it was scheduled on.
To run this job:
$ sbatch test.sh
Submitted batch job 1749905
$ cat slurm-1749905.out
test_job allocated to g011
Note that, as shown above, all the output of the job is put into a slurm-<JOBID>.out file in the current directory. This is the first place to look for errors when a job ends unexpectedly.
sbatch accepts a large number of options for specifying resource requirements and other features as described in the documentation. All command line options can be embedded as #SBATCH comments in the script, as shown above. The most common options are:
The status of jobs can be seen with the squeue command. The default output shows the JOBID, PARTITION, NAME, USER, ST (status), TIME (spend running), NODES (number of nodes used), and NODELIST(REASON) (the nodes the job is running on or the reason it is not currently running). Typical reasons for a job not running are Priority (there are other jobs that need to run first) and Resources (the resources needed to run the job are not available). A status of C means the job has completed and will shortly be removed from the queue.
For convenience, you may want to add the following aliases to your .bashrc
alias q='squeue -o "%10i %11P %12j %8u %2t %10M %9l %3D %3C %R"'
alias qg='squeue -o "%10i %11P %12j %8u %2t %10M %9l %3D %3C %R" -p dept_gpu,any_gpu'
These show additional information in a wide format (needed because our job ids are large numbers).
The sinfo and the scripts “sinfo.cpus” or “sinfo.gpus” can be used to see how many CPUs/GPUs are currently unused.
Once you have a job scheduled on a node you are allowed to ssh into that node. You should not use this ssh session to run additional jobs, but you can run utilities like top and nvidia-smi to make sure your job is running.
Another way to monitor the progress of your jobs is to inspect their output files. To avoid slow, over-the-network I/O, you should be primarily reading and writing files on the local /scr partition. These partitions are network mountable and accessible at /net/NODENAME/scr. This makes it easy to inspect files as you do not need to connect to the compute node. For example, to list the contents of the /scr partition on node n001:
ls /net/n001/scr
As a manner of good etiquette, you should delete your files from /scr once you are done with them (although if many jobs are using a shared data set you may leave it around temporarily). Files left on /scr will eventually be deleted. Do not leave important information there.
Jobs can be canceled with scancel.
$ sbatch test.sh
Submitted batch job 1750083
$ scancel 1750083
When cancelling an array job (see Running Many Jobs below), passing the JOBID will cancel the entire array job. To cancel a single element of the array, append the array id.
$ sbatch -a 1-10 test.sh submit an array of 10 jobs
Submitted batch job 1750105
$ scancel 1750105_1 cancel only the first job
$ scancel 1750105 cancel the remaining jobs
Your home directory is stored on a network file server. Accessing files in your home directory will always be slower than accessing files on the local hard drive of a node. The local hard drive of a node is accessible in /scr. You should copy files you are reading to a directory with /scr and write your outputs to /scr. When your job is done you can copy the outputs back to your home directory. The local /scr directories are accessible from any cluster node at /net/<NODENAME/scr. This lets you monitor your output without directly connecting to the node your job is running on.
A common pattern in a SLURM script is something like this:
#scratch drive folder to work in
SCRDIR=/scr/${SLURM_JOB_ID}
#if the scratch drive doesn't exist (it shouldn't) make it.
if [[ ! -e $SCRDIR ]]; then
mkdir $SCRDIR
fi
chmod +rX $SCRDIR
echo scratch drive ${SCRDIR}
cp $SLURM_SUBMIT_DIR/INPUTS ${SCRDIR}
cd ${SCRDIR}
#setup to copy files back to working dir on exit
trap "mv OUTPUTS $SLURM_SUBMIT_DIR" EXIT
If you are running many jobs that all require the same large input dataset, instead of creating a local directory based off of the JOBID, you should use a fixed directory name and rsync the data over. Since rsync only copies the data as needed, this means only the first job will have to copy the data over. If you are transferring a lot of small files constantly while your job runs, you should consider using tar to create an archive of the files before using rsync to reduce the number of files transferred over the network.
Data written to the /scr may get automatically deleted as the partition fills up and/or the files get older. When your jobs finish you should clean up your scratch space.
If you have many jobs to run you should submit your jobs as a single array job. Array jobs launch many jobs using the same run script, changing only the SLURM_ARRAY_TASK_ID variable. Array jobs are more efficient - since all the jobs in an array job have the same resource requirements, SLURM does not have to inspect everyone one of your individual jobs to determine if they can run. Array jobs also make it easier to dynamically manage how many resources you are using so you can avoid being a "cluster hog". An array job is submitted with --array (or -a for short) followed by a specification for which members of the array to run.
To run a script 1000 times:
sbatch --array 1-1000 script.slurm
To run a script 1000 times, but run a maximum of 24 jobs at a time (to avoid blocking access to the cluster for other people - the default is to run as many jobs as possible):
sbatch -a 1-1000%24 script.slurm
To increase the number of simultaneously allowed jobs to 50 (e.g., if there are plenty of nodes available):
scontrol update ArrayTaskThrottle=50 JobId=<JOBID>
The script called by the array job should change what it does based on the value of the SLURM_ARRAY_TASK_ID variable (see examples below).
When running many jobs it is your responsibility to ensure that you are not blocking others from using the queue. Jobs are scheduled first-in-first-out, so if you have a job that is Pending and ahead of other jobs (listed first in the squeue output) then it is blocking those jobs from running. To avoid blocking jobs:
When debugging issues or getting your jobs up and running for the first time it is useful to launch an interactive session on the node. This is done using srun with --pty to run bash:
srun --pty -p dept_cpu /bin/bash
Do not use interactive mode for long running jobs. This is a breach of cluster etiquette since the job will likely finish and the interactive session left idle for many hours, blocking others from using its resources.
A detailed writeup of how to run a docker container on the cluster (using singularity) can be found here.
You can launch graphical user interfaces (e.g. vmd, matlab) from cluster nodes by forwarding X over an ssh connection to a node you have a running job on. You will need to ssh to the head node with trusted X forwarding enabled (-X). First create an interactive session with X forwarding.
srun --nodes=1 -p dept_cpu --x11 --pty /bin/bash
You should now be able to launch graphical applications using your local X server. Note that to enable 3D graphics (e.g. pymol or vmd) you may have to enable indirect rendering (iglx).
* If you need a GPU, change the partition to dept_gpu and add to the line: –-gres=gpu:1
Below are some sample SLURM scripts for common applications
If you have a file your_cmds.txt of commands you want to run (all in parallel, one command per a line), you can run them all as an array job with this script. An example might be running many docking jobs where each line is docking a different ligand.
#!/bin/bash
#SBATCH --job I_forgot_to_name_my_job
#SBATCH --partition=dept_cpu
#SBATCH --nodes=1
##SBATCH --gres=gpu:1 #uncomment if you need a gpu
echo Running on `hostname`
echo workdir $SLURM_SUBMIT_DIR
echo ld_library_path $LD_LIBRARY_PATH
cd $SLURM_SUBMIT_DIR
module load cuda/11.5
#if necessary make a scr drive and copy files to it
cmd=`sed -n "${SLURM_ARRAY_TASK_ID}p" your_cmds.txt`
eval $cmd
VS Code is a popular IDE. Do not run VS Code on the head node. Do not use the Remote-SSH plugin. Instead, use the newer Remote Tunnel extension, which makes it easy to connect to a running connection from anywhere.
First, install VS Code on the cluster:
wget --content-disposition -E -c "https://code.visualstudio.com/sha/download?build=stable&os=linux-x64"
tar -xzf code-stable-x64-*.tar.gz
This will install VSCode into VSCode-linux-x64. You can then launch it inside of a slurm job:
#!/bin/bash
#SBATCH --job-name=vscode
#SBATCH -p dept_cpu
#SBATCH -t 8:00:00
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=4
#SBATCH --output sbatch.stdout
~/VSCode-linux-x64/bin/code tunnel --accept-server-license-terms --name "slurm-${SLURM_JOB_ID}"
Once running, inspect the slurm output file sbatch.out. Follow the instructions in the message for authenticating:
*
* Visual Studio Code Server
*
* By using the software, you agree to
* the Visual Studio Code Server License Terms (https://aka.ms/vscode-server-license) and
* the Microsoft Privacy Statement (https://privacy.microsoft.com/en-US/privacystatement).
*
[2026-07-07 19:59:03] info Using GitHub for authentication, run `code tunnel user login --provider <provider>` option to change this.
To grant access to the server, please log into https://github.com/login/device and use code XXXX-XXXX
On your client (e.g. laptop) you should now be able to see the tunnel you created in the remote explorer and can connect to it with "Remote-Tunnels: Connect to Tunnel...". Note the first time you do this it may take extra time connecting as it sets up.
Keep in mind that the VSCode server will continue to run on the cluster until the slurm job times out or you scancel it. Do not leave idle processes monopolizing resources.
You should never run tasks directly on the head node. It does not have the resources to support extensive computation. If you'd like to analyze data on the cluster interactively, you can launch a Jupyter Notebook using the script below. This script will print out exactly what commands are needed to tunnel a connection through the head node to the cluster node. When you are done, be sure to quit the Jupyter server or cancel the job.
This script requests a GPU. If you do not need a GPU, change the script to run on dept_cpu and not request a GPU or load cuda.
#!/bin/bash
#SBATCH --job-name=NAME
#SBATCH -p dept_gpu
#SBATCH -t 8:00:00
#SBATCH --gres=gpu:1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --output sbatch.stdout
## get tunneling info
XDG_RUNTIME_DIR=""
ipnport=$(shuf -i8000-9999 -n1)
ipnip=$(hostname -i)
token=$(xxd -l 32 -c 32 -p < /dev/random)
## print tunneling instructions to sbatch.stdout
echo -e "
Copy/Paste this in your local terminal to ssh tunnel with remote
-----------------------------------------------------------------
ssh -N -L $ipnport:$ipnip:$ipnport $USER@cluster.csb.pitt.edu
-----------------------------------------------------------------
Then open a browser on your local machine to the following address
------------------------------------------------------------------
http://localhost:$ipnport?token=$token
------------------------------------------------------------------
"
## start an ipcluster instance and launch jupyter server
module load anaconda
module load cuda
jupyter-notebook --NotebookApp.iopub_data_rate_limit=100000000000000 --port=$ipnport --ip=$ipnip --NotebookApp.password='' --NotebookApp.token="$token" --no-browser
The following script assumes a simulation was prepared using prepareamber.py conventions and the base name of the simulation is identical to the directory it is stored in.
#!/bin/bash
#SBATCH --job I_forgot_to_name_my_jobs
#SBATCH --nodes=1
#SBATCH --partition=dept_gpu
#SBATCH --gres=gpu:1
echo Running on `hostname`
echo workdir $SLURM_SUBMIT_DIR
echo ld_library_path $LD_LIBRARY_PATH
cd $SLURM_SUBMIT_DIR
#scratch drive folder to work in
SCRDIR=/scr/${SLURM_JOB_ID}.dcb.private.net
module load amber/20-cluster
#if the scratch drive doesn't exist (it shouldn't) make it.
if [[ ! -e $SCRDIR ]]; then
mkdir $SCRDIR
fi
chmod +rX $SCRDIR
echo scratch drive ${SCRDIR}
cp $SLURM_SUBMIT_DIR/*.in ${SCRDIR}
cp $SLURM_SUBMIT_DIR/*.prmtop ${SCRDIR}
cp $SLURM_SUBMIT_DIR/*_md2.rst ${SCRDIR}
cp $SLURM_SUBMIT_DIR/*.inpcrd ${SCRDIR}
cp $SLURM_SUBMIT_DIR/dist.RST ${SCRDIR}
cd ${SCRDIR}
#setup to copy files back to working dir on exit
trap "mv *md3.nc $SLURM_SUBMIT_DIR" EXIT
#run the MD, default to name of directory
prefix=${SLURM_SUBMIT_DIR##*/}
pmemd.cuda -O -i ${prefix}_md3.in -o $SLURM_SUBMIT_DIR/${prefix}_md3.out -p ${prefix}.prmtop -c ${prefix}_md2.rst -r ${prefix}_md3.rst -x ${prefix}_md3.nc -inf $SLURM_SUBMIT_DIR/mdinfo
It is common to want to run multiple simulations of the same system. This can be easily accomplished by using an array job (sbatch -a 1-3) and modifying the last line above to:
pmemd.cuda -O -i ${prefix}_md3.in -o $SLURM_SUBMIT_DIR/${prefix}_${SLURM_ARRAY_TASK_ID}_md3.out -p ${prefix}.prmtop -c ${prefix}_md2.rst -r ${prefix}_${SLURM_ARRAY_TASK_ID}_md3.rst -x ${prefix}_${SLURM_ARRAY_TASK_ID}_md3.nc -inf $SLURM_SUBMIT_DIR/mdinfo.${SLURM_ARRAY_TASK_ID}
Note this only has the desired result if the amber configuration sets initial velocities randomly (irest=0).
The following script runs a namd simulation. You will likely need to change the name of the files that are being copied. For optimal performance, the ratio of CPUs to GPUs should be optimized for the system under study. However, we find that on our cluster using14 cores with 1 GPU often provides good performance (28 SLURM CPUs are requested to ensure the 14 cores, not threading contexts, are reserved). Increasing the number of processors used can decrease the performance of the simulation (in some cases dramatically) so we do not recommend changing these settings without first benchmarking your system.
Note that NAMD 3.0, which is currently under development, will behave like AMBER and perform all calculations on the GPU.
#!/bin/bash
#SBATCH --job namd
#SBATCH --nodes=1
#SBATCH --partition=dept_gpu
#SBATCH --gres=gpu:1
#SBATCH -c 8
echo Running on `hostname`
echo workdir $PBS_O_WORKDIR
echo ld_library_path $LD_LIBRARY_PATH
cd $SLURM_SUBMIT_DIR
#scratch drive folder to work in
SCRDIR=/scr/${SLURM_JOB_ID}
#if the scratch drive doesn't exist (it shouldn't) make it.
if [[ ! -e $SCRDIR ]]; then
mkdir $SCRDIR
fi
chmod +rX $SCRDIR
echo scratch drive ${SCRDIR}
module load namd/2.13/multicore-CUDA
cp $SLURM_SUBMIT_DIR/sim.conf ${SCRDIR}
cp $SLURM_SUBMIT_DIR/*.psf ${SCRDIR}
cp $SLURM_SUBMIT_DIR/eq3.xsc ${SCRDIR}
cp $SLURM_SUBMIT_DIR/eq3.vel ${SCRDIR}
cp $SLURM_SUBMIT_DIR/eq3.coor ${SCRDIR}
cp $SLURM_SUBMIT_DIR/*prm ${SCRDIR}
cp $SLURM_SUBMIT_DIR/*inp ${SCRDIR}
cd ${SCRDIR}
#setup to copy files back to working dir on exit
trap "mv sim.dcd sim.xsc $SLURM_SUBMIT_DIR" EXIT
namd2 +p14 +devices 0 sim.conf
Interactive Matlab requires a user with a valid license, although all nodes can run compiled programs. Contact us if you need to use Matlab on the cluster.
To launch an interactive session:
srun --pty -p big_memory -w n201 /bin/bash -i
module load matlab
matlab