1 of 59

In-Person: Please sign in using the sign-in sheet provided

Remote: Please set your zoom name to the name that matches your registration

MSI X103 - Job Submission and Scheduling at MSI

Fall 2026

Minnesota Supercomputing Institute

2 of 59

Introduction to Job Submission and Scheduling

Ham Lam & Borgne Raasch

Recommended background

Basic UNIX commands and BASH scripting experience

Training Level

Beginner

Tutorial format

Lecture combined with hands-on

examples of submitting jobs

Fall 2026

Minnesota Supercomputing Institute

3 of 59

Introduction to Job Submission and Scheduling

Agenda

o Connect to MSI

o HPC hardware

o Job scheduling

o Submit Jobs

o Monitoring Jobs

o Troubleshooting Tips

o Hands-on demo

Minnesota Supercomputing Institute

4 of 59

Accessing MSI Compute Resources

Open OnDemand Portal

https://ondemand.msi.umn.edu

Interactive Web Interface

File Transfer: Manage your data easily
Remote Shell: Access Agate cluster
Data Visualization: Remote desktop access

Development & Apps

Notebooks: Jupyter & RStudio
GUI Apps: Desktop, MATLAB, COMSOL, etc.

IMPORTANT: UMN VPN or on-campus network connection is REQUIRED for access.

Minnesota Supercomputing Institute

UNIVERSITY OF MINNESOTA

Minnesota Supercomputing Institute

5 of 59

Accessing MSI Compute Resources

Command Line Interface

Terminal clients

terminal

Please note: To access Open OnDemand a UMN VPN or UMN network connection is required.

Minnesota Supercomputing Institute

UNIVERSITY OF MINNESOTA

Minnesota Supercomputing Institute

6 of 59

Overview of job scheduling

Job submission and scheduling

Minnesota Supercomputing Institute

7 of 59

Social Science

Physics

Other

Mathematical Sciences

Informatics

Health Sciences

Veterinary Medicine

Agriculture

Astronomy

Biochemistry

Computer

Science

Earth Sciences

Economics

Chemistry

Biology

Engineering

Genetics

1041 Groups

122 Departments

5409 Users

Why do we need job scheduling?

Ensure fair use of HPC
Automated allocation of limited resources
Tracking and Monitoring jobs

Minnesota Supercomputing Institute

8 of 59

Job scheduling and SLURM

MSI uses SLURM (Simple Linux Utility for Resource Management) to manage HPC resources.

Job scheduling is the process of arranging, controlling and optimizing work and workloads in a shared HPC environment.

SLURM is a highly-scalable scheduler for HPC clusters. It allocates resources, provides a framework for executing tasks, and arbitrates contention for resources by managing queues of pending work.

Minnesota Supercomputing Institute

9 of 59

Quick SLURM highlights

It schedules jobs to be executed on a cluster of machines based on priorities
Pending jobs are “queued” jobs waiting to be executed
Jobs are submitted by users via shell commands
It takes care of Input and Output (I/Os)
It launches jobs on assigned compute node(s) and clean up after each job finishes

Job submission and scheduling

Job scheduling and SLURM

Minnesota Supercomputing Institute

10 of 59

Agate HPC

High-speed network connections for CPU and GPU compute nodes

Over 80,000 AMD processor cores
448 Nvidia GPUs

High memory capacity

> 248GB - 2TB per cpu compute node
> 300GB - 1TB per gpu compute node

High performace storage

30 TB of storage

High Performance Cluster (HPC)

Minnesota Supercomputing Institute

11 of 59

Agate GPUs

Nvidia GPU	Memmory (GB)	Dedicated GPU partition
V100	32	msigpu
A100	40	msigpu
A40	48	preempt-gpu,interactive-gpu
L40S	48	interactive-gpu
H100	94	msigpu

CUDA Version: 13.0

Driver Version: 580.95.05

Minnesota Supercomputing Institute

12 of 59

A logical set(s) of compute nodes grouped together depending on their hardware characteristics or function.

A slurm job partition can been seen as an automated waiting list for use of a particular set of computational hardware.

Different job partitions have different resources and limitations.

Make sure to choose a job partition which has resources and limitations suitable to your jobs

https://msi.umn.edu/our-resources/slurm-scheduler/slurm-partitions

Slurm Partitions

Minnesota Supercomputing Institute

13 of 59

Slurm Partitions

Complete list of slurm partitions: msi.umn.edu/our-resources/slurm-scheduler/slurm-partitions

Minnesota Supercomputing Institute

14 of 59

What is a slurm job?

A small text file must be prepared by an user that says what program to run, where to get the input, and where to put the output.

The user then submit this job script to the SLURM scheduler which decides when and where it will run.

Once the job has finished, the user can retrieve the results of the calculation.

There is no interaction between the user and the program while the job is running.

#!/bin/bash

#SBATCH --job-name=demo1

echo "I ran on node: "

hostname; sleep 120

Minnesota Supercomputing Institute

15 of 59

What is a slurm job?

When you submit your job, Slurm interprets the #SBATCH directives and finds a time and a node(s) on the cluster to run your job. There are factors that impact WHEN your job will run:

The compute resources you request

Nodes, CPUs, GPUs, Memory, Time, etc

If you request a lot of resources, your job will wait longer depending on how many jobs are in the queue

The frequency that you submit jobs

If you submit a lot of jobs with a small amount of resources, they will likely be backfilled (run quickly) but your future jobs’ priority will take a hit

The other jobs in the queue
Maintenance windows

If you request a walltime that cuts into the maintenance window, your job won’t be scheduled until after the maintenance window.

Minnesota Supercomputing Institute

16 of 59

Job is added to the partition

Submission

Priority

WAIT

Resources allocation and

Execution!

Assign job priority value

Job waits until

(1) resources are available

(2) No job with higher priority in partition

Slurm will kill your job at runtime if your job exceeds the requested amount of resources.

!

Life cycle of a slurm job

Minnesota Supercomputing Institute

17 of 59

Software packages at MSI

MSI has hundreds of software modules. Software environment modules are used to make software available to you!

A module is a self-contained description of a software package - it contains the settings required to run a software package and, usually, encodes required dependencies on other software packages.

On a high-performance computing system, it is often the case that no software is loaded by default. If we want to use a software package, we will need to “load” the module ourselves.

MSI also maintains a searchable directory of available software at https://www.msi.umn.edu/software.

Software

Note: Some software titles require a different login than your MSI user account. Please email help@msi.umn.edu if the software you wish to use is password-protected. Depending on licensing, if the software is only available on select MSI machines, additional machines can be requested via the MSI Helpdesk.

Minnesota Supercomputing Institute

18 of 59

Working with software modules

Software

Description	Command	Example
See all available modules	module avail	module avail
Load a module	module load	module load matlab/2021a
Unload a module	module unload	module unload matlab/2021a
Unload all modules	module purge	module purge
See what a module does	module show	module show matlab/2021a
List currently loaded modules	module list	module list

The module command is used to interact with environment modules. An additional subcommand is usually added to the command to specify what you want to do.

Minnesota Supercomputing Institute

19 of 59

Run scripted and interactive jobs

To access compute nodes (CPUs and GPUs), you must either submit a job script or initiate an interactive session.

Job submission and scheduling

Minnesota Supercomputing Institute

20 of 59

sba- - “sbatch” command to submit a job script to Slurm for remote execution

Submit batch jobs with sbatch

sbatch

Request resource allocation through a job script or at the command lines using the sbatch command. If invoked at command lines with options, these options take precedence over the #SBATCH options in the job script.

%sbatch demo1.sh

%sbatch: Setting account: support

%Submitted batch job 9704508

#!/bin/bash

#SBATCH --job-name=demo1

echo "I ran on node: "

hostname; sleep 120

Simple job script

submit a job

The #SBATCH directives must appear at the top of the submission file, before any other line except for the very first line which should be the #!/bin/bash.

Minnesota Supercomputing Institute

21 of 59

Check status of jobs

%sbatch demo1.sh #submit demo1.sh

sbatch: Setting account: support

Submitted batch job 9704508

%squeue -u lamx0031

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)

9704508 msismall demo1.sb lamx0031 PD 0:00 1 (None)

Use command squeue --me (or -u <username>) to view jobs status.

(ST) JOB STATE CODES

PD PENDING Job is awaiting resource allocation

NODELIST (REASON)

CG COMPLETING Job is in the process of completing

CA CANCELLED Job was explicitly cancelled by the user/admin

R RUNNING Job currently has an allocation

Priority One or more higher priority jobs exist for this partition

Resources The job is waiting for resources to become available

ReqNodeNotAvail Some node specifically required by the job is not currently available.

Minnesota Supercomputing Institute

22 of 59

Capture outputs of jobs

#!/bin/bash

## My first job script

#SBATCH --job-name=demo1

#SBATCH --output demo1_%j.output # -o (shorthand)

#SBATCH --error demo1_%j.error # -e (shorthand)

echo "I ran on node: "

hostname

sleep 120

Simple job script

%cat demo1_9704508.output

I ran on node:

cn1140

By default, the output of a job is saved in a file called “slurm-<job id>.out”. The ‘#SBATCH --output’ can be used to specify a different name and location of the output file. The ‘#SBATCH --error <filename>’ is used to capture stderr of the job to a file.

Minnesota Supercomputing Institute

23 of 59

Job account

Each job has an associated account name that corresponds to a PI group. If you belong to multiple groups, you can specify an account that will be used by the job.

#!/bin/bash

#SBATCH --job-name=demo1

#SBATCH --account=support

echo "I ran on node: "

hostname

sleep 120

Your MSI user account is associated with the PI who created the account, known as your primary group. If you are working with another PI on a project, you can use the ‘--account=<groupname>’ option to have jobs run under a different group that allows you to have access to compute resources (quota space, files, etc) of that group.

You can use the “id” command to find out if you belong to more than one group:

%id <username>

Minnesota Supercomputing Institute

24 of 59

Interactive jobs

Job submission and scheduling

Minnesota Supercomputing Institute

25 of 59

Interactive Jobs with srun

Dedicated partitions for interactive workflow: interactive and interactive-gpu (User can have 1 interactive job and 1 interactive-gpu job concurrently)

Using srun is to launch an interactive session on a compute node with the “--pty” option.

e.g. srun -t 100 -N1 -n1 -c1 -p interactive --pty bash #request a ‘shell’

Use the srun command at a login node to create a new Real Time allocation of compute resources for interactive use. Think of it as requesting a ‘shell’ and interact with the compute node directly from the terminal.

Minnesota Supercomputing Institute

26 of 59

User-provided command can be “/bin/bash” (to launch an interactive shell) or a script with arguments.

lamx0031@ahl01 [~] % srun --nodes 1 --ntasks 1 --cpus-per-task 4 -p msismall ./omp_hw

srun: Setting account: support

srun: job 23800915 queued and waiting for resources

srun: job 23800915 has been allocated resources

Hello World from thread = 1

Hello World from thread = 0

Number of threads = 4

Hello World from thread = 3

Hello World from thread = 2

1 node, 1 task, 4 cores with 1 thread per core

Job submission and scheduling

Interactive Jobs with srun

Minnesota Supercomputing Institute

27 of 59

Job submission and scheduling

Graphical Interactive Jobs

Use the Open OnDeman web portal for interactive apps. Applications with GUI such as MATLAB and RStudio can be created directly from the browser. A terminal is also available through Open OnDemand.

Minnesota Supercomputing Institute

28 of 59

Batch HPC and scripted job examples

Job submission and scheduling

Minnesota Supercomputing Institute

29 of 59

Using batch scheduling is the primary way to make use of MSI’s shared supercomputing resources.
You will submit jobs to the cluster scheduler, and your analysis eventually runs on the computer without any further intervention.

Requires jobs to be written as shell scripts and submitted via the command line.

Jobs are queued and priority is based on a combination of the size of the request and how many jobs your group has run recently.
Jobs will run when resources are available.

Batch HPC

Job submission and scheduling

Minnesota Supercomputing Institute

30 of 59

Serial job

This is the simplest type of job which uses one core on a single compute node. It is also our default setting when no “#SBATCH” statement is provided in a job script.

#!/bin/bash

#SBATCH --job-name=demo1

#SBATCH --nodes=1 # specify one node

#SBATCH --ntasks=1 # specify one task per cpu-core

#SBATCH --mem=4g # request 4 gb (default is 1 gb)

#SBATCH -o demo1_%j.output

#SBATCH -e demo1_%j.error

echo "I ran on node: "

hostname

sleep 120

Minnesota Supercomputing Institute

31 of 59

Multithreaded job

#!/bin/bash

#SBATCH --job-name=multithreaded #

#SBATCH --nodes=1 # A single node count

#SBATCH --ntasks=1 # One task

#SBATCH --cpus-per-task=4 # Request 4 cores

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

./omp_helloworld

A multithreaded job launches one slurm task (process) which uses several CPUs.

This example script launches a single process with 4 CPU cores

Minnesota Supercomputing Institute

32 of 59

Job array

#!/bin/bash

#SBATCH --job-name=my_array_job

#SBATCH --nodes=1

#SBATCH --ntasks-per-node=1

#SBATCH --time=00:10:00

#SBATCH --array=1-5 # 5 jobs

python arraytest.py file-${SLURM_ARRAY_TASK_ID}.txt >output_${SLURM_ARRAY_TASK_ID}

Creating a job array provides an easy way to group related jobs together. A job array is a collection of jobs that differ from each other by only a single index parameter. All jobs in a job array must have the same resource requirements.

Minnesota Supercomputing Institute

33 of 59

More advanced scripted job examples

Job submission and scheduling

Minnesota Supercomputing Institute

34 of 59

Preempt jobs

#!/bin/bash

#SBATCH --nodes=1

#SBATCH --ntasks=10

#SBATCH --mem-per-cpu=1G

#SBATCH --time=01:00:00

$SBATCH --partition preempt

$SBATCH --requeue

module load parallel

# use --resume to deal with interruption

parallel -v --delay .3 -j $SLURM_NTASKS --joblog logs/testjob.log --resume

Why run preemptable jobs?

Same priority value as non-preempt jobs but with a smaller fairshare impact!

What happen to my job when it is preempted?

Cancel or Requeue options: When re-queue, your job must be able to pick up where it left off or can deal with interruption effectively when it restarts.

Partitions available: preempt and preempt-gpu

Note: Jobs submitted to preempt queue can be killed at any time to make room for interactive jobs.

Minnesota Supercomputing Institute

35 of 59

GPU jobs

A40, A100 Architecture: Ampere

H100 Architecture: Grace Hopper

L40S architecture: Ada Lovelace

V100 Architecture: Volta

Partition name	Node sharing	GPU type	GPU Memory	Cores per node	Walltime limit	Total node memory	Advised mem per core	Local scratch per node	Max. nodes per job
preempt-gpu	Yes	A40	46068 MiB	64	24:00:00	499 GB	8000 MB	850 GB	1
interactive-gpu	Yes	A40/L40S	46068 MiB	24	24:00:00	60 GB	2 GB	228 GB	2
msigpu	Yes	mixed	40-90 GB	24-128	24:00:00	40-499 GB	1900 MB	380-850 GB	1

Slurm GPU partitions

Access to a100-*-long partitions requires creating a request ticket with help@msi.umn.edu.

Minnesota Supercomputing Institute

36 of 59

GPU jobs

#!/bin/bash

#SBATCH -p msigpu

#SBATCH --gres=gpu:a100:1

Specify the type of GPU using ‘--gres=gpu:<type>:count

In addition to selecting a GPU partition, GPUs need to be requested for all GPU jobs. Available GPU types include v100, a100 and h100 for the msigpu partition, and a40 and l40s in the interactive-gpu and preempt-gpu partitions. You can request up to 2 v100s, 4 h100s, and up to 8 a100s

You need to specify the GRES Generic Resource Scheduling parameter in your job script

If you don’t have a GPU type preference, you can request any available GPU in the msigpu partition

#SBATCH -p msigpu�#SBATCH --gres=gpu:1

Minnesota Supercomputing Institute

37 of 59

Only codes that are written to run on a GPU can take advantage of a GPU.
Do not run CPU only code on a GPU node.
When you request memory for jobs on HPC, that is CPU memory.
If you request a GPU, the job will receive all of the GPU memory!
If the code is written to run on a single GPU, request 1 GPU only.

#SBATCH -j gpu_job

#SBATCH --mem=128g # host CPU memory, not GPU memory!

#SBATCH -p msigpu # Select a gpu partition

#SBATCH --gres=gpu:a100:1

#SBATCH -o output

python script.py

GPU jobs

Job submission and scheduling

GPU compute resources: Slurm partitions

The relationship between CPUs and GPUs

GPU: Streaming multiprocessor

GPU compute resources for AI/ML jobs

Try to get your job running with a single GPU before trying to use multiple GPUs. Jobs that request multiple GPUs will typically take longer to schedule because the cluster has a limited supply of GPUs

Minnesota Supercomputing Institute

38 of 59

Job submission and scheduling

When you request a GPU node, it will come with CPU cores, so what do we do with the CPU cores?

Request at least as much RAM as the GPU you select
If you request multiple GPUs, get as much RAM as the total number of GPUs
Do you need to preprocess data?

Use these CPU cores to preprocess data in parallel (NumPy auto. uses multiple cores)
If you use PyTorch’s DataLoader, set the num_workers parameter to match the CPU cores for efficient data loading.

Data, CPUs and GPUs

Task	CPU or GPU
Data Preprocessing	CPU but either is fine
Deep Learning model training	GPU (A100, V100, H100)
Deep Learning Inference	GPU (CPU for small toy models)

GPU compute resources for AI/ML jobs

Minnesota Supercomputing Institute

39 of 59

Locations for jobs

Minnesota Supercomputing Institute

40 of 59

Locations for jobs

/home/project/<MSI_Login_ID> directory

Most of the time you will stage and run your jobs here

/scratch.local

Temporary storage space from a compute node.
The job script must copy the result files back to home directory at the end of execution.

/scratch.global

My job produces many large intermediate files but I only need to keep a few.
My group quota is not large enough for all of the intermediate and persistent data.
It’s a shared space that is visible to all nodes (including login nodes)
Data in /scratch.global is not backed up and is deleted after 30 days.

Minnesota Supercomputing Institute

41 of 59

Monitoring scripted jobs

Job submission and scheduling

Minnesota Supercomputing Institute

42 of 59

Monitoring jobs

Current jobs

%squeue -u lamx0031 -l #-l gives more information about your job

squeue command gives all the jobs the scheduler is managing at the moment, use the “-u <your username>” option to list your jobs only.

lamx0031@cn1001 [14:03:43 ~] squeue -u lamx0031 -l

Thu Jan 06 14:03:47 2022

JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES NODELIST(REASON)

9780263 interacti bash lamx0031 RUNNING 40:38 4:00:00 1 cn1001

Minnesota Supercomputing Institute

43 of 59

Monitoring jobs

Current jobs

Insert slurm email notification statements in the job script

#!/bin/bash

#SBATCH --mail-type=begin # send email when job begins

#SBATCH --mail-type=end # send email when job ends

#SBATCH --mail-user=<id>@umn.edu

echo "I ran on node: "

hostname

sleep 120

Sender: MSI Slurm <msi_slurm@msi.umn.edu>

Subject: Slurm Job_id=8343769 Name=jobscript.sbatch Began, Queued time 00:05:00

Body: <empty>

--mail-type=begin

Minnesota Supercomputing Institute

44 of 59

Monitoring jobs

Current jobs

Attach to a running job using ‘srun’

%srun --jobid=<jobid> --pty bash #note that if you need to run any MPI tasks, this will not work.

%ssh <node id> #log into the compute node (node id returned by squeue command)

Log into the compute node directly using ‘ssh’.

Minnesota Supercomputing Institute

45 of 59

Monitoring jobs

Completed jobs

The seff <jobid> command will output a short summary of the CPU and Memory efficiency of a job.

%seff 8457631

Job ID: 8457631

Cluster: mesabi

User/Group: lamx0031/support

State: COMPLETED (exit code 0)

Nodes: 1

Cores per node: 24

CPU Utilized: 06:33:21

CPU Efficiency: 95.29% of 06:52:48 core-walltime

Job Wall-clock time: 00:17:12

Memory Utilized: 726.06 MB

Memory Efficiency: 1.48% of 48.00 GB

Minnesota Supercomputing Institute

46 of 59

Monitoring jobs

Completed jobs

The reportseff -u $USER command will output a summary of all jobs for a time period. (“reportseff --help” to see more options)

Minnesota Supercomputing Institute

47 of 59

Monitoring jobs

Canceling jobs

Use scancel with the job ID to cancel a job:

$ scancel <jobid>

You can cancel all your jobs, or all your pending jobs with scancel:

$ scancel -u $USER

$ scancel -t PENDING -u $USER

Minnesota Supercomputing Institute

48 of 59

Job submission and scheduling

Command	Example	Description
sbatch	>sbatch slurm_script.batch	Submit a batch job
scancel	>scancel --me (all jobs)	Cancel a job
	>scancel <jobID> (single job)	cancel a particular job
	>scancel -p <partition>	cancel all jobs on a partition
	>scancel <job_arrayid>	cancel specific job array IDs
sinfo	>sinfo (general)	Show details about the cluster partitions
squeue	>squeue --me -l	Monitor my jobs
sacct	>sacct	Show details of your recent jobs
srun	>srun -N 1 -n 10 --mem=24g -t 120 <other options>	Request interactive realtime session

Summary

Slurm Commands

Minnesota Supercomputing Institute

49 of 59

Troubleshooting Tips

Job submission and scheduling

Minnesota Supercomputing Institute

50 of 59

Estimating Compute resources

How large is the data I am working with?

Start by requesting memory about twice the size of a data file. If this isn’t enough, increase the request by at least half until no memory errors.

Does my script keep large amount of data in memory?

If yes, start by requesting more than twice the size of the data file.

Can my scripts take advantage of multiple cores?

If yes, then request more than one cores.

If no, request a single core only.

My script needs a GPU?

If yes, request a GPU specific partition and add “gres=gpu: <type>:<number>” option

How much walltime do I need to request for my job?

Don’t request too little! Try to factor in a buffer to account for scheduler and network issues.

Requesting too long will lengthen the queue time.

If my local desktop/laptop is able to run the code in N hours, Agate should at least match that!

Minnesota Supercomputing Institute

51 of 59

Estimating Compute resources

Host (CPU) Memory?

Request too little: job will terminate when it runs out of RAM

Request too much: Lots of memory will sit idle and no one else can use it

Best: Request slightly more RAM than you need.

CPUs?

Request too little: your job will take a performance hit as a result

Request too much: Lots of CPUs will sit idle and no one else can use them.

Best: request the number of CPUs that your job can use.

GPUs?

Request too little: your job will not have the “speed” due to overheads between CPUs and GPUs.

Request too much: Your code isn’t written to use multiple GPUs and others won’t be able to use them

Best: Request exactly the number of GPUs that your job can use.

Time?

Request too little: your job will not finish before the time limit runs out.

Request too much: your job will wait longer in queue. If a maintenance window is coming up, your job will not be scheduled until after the maintenance window.

Best: Request slightly more time than you need, but not too much.

Minnesota Supercomputing Institute

52 of 59

Factors to consider

Factor	Higher Priority	Example
Age	Longer time in queue has higher priority	Job in queue for 10 hours will start faster than 2 hours
Resources	Fewer requested	1 CPU starts faster than 4 CPUs
Partition	Lower resource partitions	Pre-empt partitions
Fairshare	Fewer jobs currently running	Having 1 job already running will start faster than having 10 running jobs

Minnesota Supercomputing Institute

53 of 59

Typical workflow

A typical batch job workflow:

You create a job script

Several key resource requests:

Node (-N), task(-n) or core(-c) How many nodes and tasks/cores does your job need?
--time, How long does your job need to run?
--mem, How much total memory does your job need?
-p <partition>, which partition fits your job best?

You submit job script with command: sbatch
You check job status with command: squeue
When job completes, check output or log file(s)
If job failed, modify job script and resubmit (back to step 1)

Check job information with command “sacct” or “seff”

If job succeeded, check job information with “sacct” or “seff”

Minnesota Supercomputing Institute

54 of 59

Submitting jobs

sbatch messages: incorrect resource configuration

% sbatch my_job_script

sbatch: error: Setting account: support

sbatch: error: Memory specification can not be satisfied

sbatch: error: Batch job submission failed: Requested node configuration is not available

For example, if you requested more memory than what a compute node actually has.

*Check our Slurm partitions specification webpage for proper configuration settings.

Minnesota Supercomputing Institute

55 of 59

Do a dry-run using the “--test-only” option with sbatch

%sbatch --test-only my_job_script

sbatch: Setting account: support

sbatch: Job 41274906 to start at 2025-09-16T12:06:46 using 1 processors on nodes acn45 in partition agsmall

A good way to validate the slurm script and return an estimate of when a job would be scheduled to run. No job is actually submitted.

Submitting jobs

Minnesota Supercomputing Institute

56 of 59

Using sacct

#!/bin/bash

#SBATCH --j OOM

#SBATCH --ntasks=1

#SBATCH --mem-per-cpu=10M

#SBATCH --time=00:05:00

stress --cpu 1 --io 1 --vm 1 --vm-bytes 128M --timeout 120s

%sacct -j 10662930 --format=JOBID,JOBNAME,averss,maxrss,START,END,NCPUS,NTASK,STATE --unit=M

JobID JobName AveRSS MaxRSS Start End NCPUS NTasks State

------------ ---------- ---------- ---------- ------------------- ------------------- ---------- -------- ----------

10662930 OOM 2022-01-28T15:07:29 2022-01-28T15:09:34 1 OUT_OF_ME+

10662930.ba+ batch 2.62M 2.62M 2022-01-28T15:07:29 2022-01-28T15:09:34 1 1 OUT_OF_ME+

10662930.ex+ extern 0.89M 0.89M 2022-01-28T15:07:29 2022-01-28T15:09:35 1 1 COMPLETED

After the job exits, run the below sacct command to check the status of the job.

Out of Memory Issues

Minnesota Supercomputing Institute

57 of 59

Exit code

Troubleshooting tips

Code	Meaning	Note
0	Success	Check output
1	General error	Check log files
2	Incorrect use of shell builtins	Check log files
3-124	Job error	check exit code of software
125	Out of memory
126	Command not executed
127	Command not found
128	Invalide argument
129 -192	Terminated via signal	Subtract 128 from the number and match to signal code

Minnesota Supercomputing Institute

58 of 59

Need more help?

Check our SLURM webpage

https://msi.umn.edu/our-resources/slurm-scheduler

Submit a ticket to the helpdesk

Email: help@msi.umn.edu

https://www.msi.umn.edu/helpdesk

SLURM cheat sheet

https://slurm.schedmd.com/pdfs/summary.pdf

Troubleshooting tips

Check our Job FAQ webpage

https://msi.umn.edu/our-resources/knowledge-base/jobs-faqs

Minnesota Supercomputing Institute

59 of 59

THANK YOU!

Questions?

Job submission and scheduling

Minnesota Supercomputing Institute