1 of 52

Data Storage & Workflows

David Porter & Angel Mancebo

Sept 25, 2025

Minnesota Supercomputing Institute

2 of 52

Tutorial Outline

Kinds of storage
Storage spaces at MSI
User Interfaces for managing data
Use cases

Minnesota Supercomputing Institute

3 of 52

POSIX vs. Object Storage

POSIX Compliant	Object Store
Directories (folders) / Files	Buckets / Objects (files)
Standard File system IO	URL or S3 Interface
Direct access on "local" host	Access by networked hosts
Random & fragmented file access	Put and get by object (file)
MSI: Tier-1 Home, Scratch & RAM	MSI: Tier-2, Tier-3 (tape)

Minnesota Supercomputing Institute

4 of 52

Data Hierarchy: Posix (P) & Object (O)

CPU GPU

RAM (P)

On Node SSD (P)

Parallel File System (P)

On Site S3 Object Store (O)

On Site Tape Backup (O)

Wide Area Net Repositories (SSH, URL or API)

Capacity

&

Granularity

Performance

&

Cost/Byte

Minnesota Supercomputing Institute

5 of 52

Storage at MSI

Old PanFS Home, Shared, & Public
Current User Directories
Global Scratch
Local scratch and RAM disk
2nd Tier Storage: CEPH with S3 interface
3rd Tier Storage: Tape
Storage quotas & limits

Minnesota Supercomputing Institute

6 of 52

Old PanFS Group Directories

Previous Primary Tier-1 Storage (POSIX)

Large /home/PROJECT/USER organization unmanageable

Large home & shared directories ⇒ poor or no data planning
Group members leave & strand data
Issues with large groups coordinating usage

Old PanFS directories are NOT going away, but are

old technology & nearing end of life
under provisioned for volume IO & number of clients & systems
slow and increasingly in degraded/restoring state

Minnesota Supercomputing Institute

7 of 52

Access Old Home, Shared, & Public

Same directory paths as before

/home/GROUP/USER USER = your user name
/home/GROUP/shared GROUP = PI or PROJECT name
/home/GROUP/public
Still on all LINUX systems
$HOME and ~ now point to your new home directory

Convenience Shell variables:

$OLDHOME Your old PanFS home directory
$MSIPROJECT Root of your groups project space
https://msi.umn.edu/our-resources/knowledge-base/new-home-directories

Minnesota Supercomputing Institute

8 of 52

New Home Directories

What it is

VAST: POSIX (new and performant file system)

Visible on all MSI systems

Persistence: duration of your account at MSI

How you access it:

Directories: home director: $HOME or ~/

Shell commands: cp, mv, rm, grep, …

Applications: all POSIX file IO

Minnesota Supercomputing Institute

9 of 52

New Home Directory

Locations & Uses

/user/<#>/<user>

Here <#> is number of volume

Home directory for your "small" files

Limits

200 GB Check your usage: userquota

1 million files

Limits apply to each user separately

Minnesota Supercomputing Institute

10 of 52

Global Scratch

What it is

VAST: POSIX (new and performant file system)

Visible on all MSI systems

Persistence: purge after 1 month

How you access it:

Global scratch visible as a directory

Shell commands: cp, mv, rm, grep, …

Applications: all POSIX file IO

Minnesota Supercomputing Institute

11 of 52

Global Scratch

Locations & Uses

/scratch.global/

Create your own directories under this

Limits

40 TB Check your group's usag: groupquota -r

13 million files

Limits apply to user groups

Minnesota Supercomputing Institute

12 of 52

Local Scratch

What it is

Non-RAIDed Disk or SSD: POSIX

Visible on host system only

Persistence: duration of SLURM job

How you access it:

Shell commands: cp, mv, …

Applications: all POSIX file IO

Minnesota Supercomputing Institute

13 of 52

Local Scratch

Locations & Uses

/scratch.local

[/<user>/<path>]/<file name>

Scales well to many hosts writing to their own files

⇒ Good place for your scratch/work directory

Limits

Scope: local host and life of batch job

relatively poor bandwidth, except for fragmented IO

Capacity: 32 - 420 GB – varies with node and job

Minnesota Supercomputing Institute

14 of 52

RAM Disk

What it is

Local system memory (POSIX)

Visible only on local host

Persistence: duration of SLURM job

How you access it:

Shell commands: cp, mv, …

Applications: all POSIX file IO

Minnesota Supercomputing Institute

15 of 52

RAM Disk

Locations & uses

/dev/shm

[/path]/<file name>

Scalable to many hosts reading their own files

High bandwidth and low latency

Efficient fragmented IO

Limits

About ½ system memory (256 GB on an Agate node)

Scope: local to node and only during SLURM job.

Minnesota Supercomputing Institute

16 of 52

Tier-2 Storage

What it is

CEPH: Object storage with S3 interface

Visible on all MSI systems and Web

Persistence: duration of allocation

How you access it:

By file only

Files organized in “buckets”

Shell: s3cmd

Web URL & GLOBUS

https://www.msi.umn.edu/content/second-tier-storage

Minnesota Supercomputing Institute

17 of 52

Tier-2: S3 interface

Locations & Uses

s3://<bucket name>/<file name>

s3cmd commands: mb; ls; get; put

Save & stage large volumes of data

Limits

120 TB per group Check usage: s3info

Group PI or designated admin must creatch root bucket

Minnesota Supercomputing Institute

18 of 52

3rd Tier Storage: Tape

What it is

Blackpearl: LTO-7 tape (6 - 15 TB per tape)

Visible: MSI HPC systems (s3 interface – Object store)

Persistence: ~5 years

This is a service: NOT just tapes

How you access it:

Purchase: $456 per “unit” (= 1 redundant pair of tapes)

Large files: 1-1000 GB (approx)

Latency: 1-7 days to recover data (approx)

For more info: send email to help@msi,umn.edu

Minnesota Supercomputing Institute

19 of 52

Storage Usage, Quotas, & Limits

Location	Command	Limit	Applies To
Old PanFS Project	groupquota	0.5 - 10 TB	Group
New VAST Home	userquota	200 GB	User
Global scratch	groupquota -r	40 TB	Group
Tier-2 Object Store	s3info	120 TB	Group
Local scratch	du -sh . or df -h	~ 1 TB	Job
RAM Disk	du -sh . or df -h	½ job memory	Job

Minnesota Supercomputing Institute

20 of 52

Data Applications & Utilities

scp can push to msi from external host

wget Pull from web servers

Git Pull from and push to a repository

s3cmd Pull from and push to Tier-2

Globus Web based control of copies between systems

Minnesota Supercomputing Institute

21 of 52

Access to MSI

Use within UofM domain (use UofM VPN)

Go through an MSI front end server

agate.msi.umn.edu
OnDemand

Minnesota Supercomputing Institute

22 of 52

Secure Copy (scp)

Login to MSI

ssh <msi_user>@agate.msi.umn.edu

Copy to MSI

scp <r_user>@<r_host>:<path>/<file> <path>

scp -r <r_user>@<r_host>:<path> <path>

Copy from MSI

scp <file> <r_user>@<r_host>:<path>

scp -r <path> <r_user>@<r_host>:<path>

Minnesota Supercomputing Institute

23 of 52

Get Files from web (wget)

Run client (wget) from MSI host
Get files, source code, data posted on web

Files must be posted on a server that support wget

You must have the URL

On an MSI host: get a file from the web:

wget <URL>

Minnesota Supercomputing Institute

24 of 52

Repositories (git)

Sharing data & source with others: Version control
Can run git locally or with a github
UofM github: https://github.umn.edu
Documentation: https://training.github.com

On MSI host: command prompt

git add

git commit

git merge

Minnesota Supercomputing Institute

25 of 52

Tier-2 Access (s3cmd)

What is it good for?

Move large volumes of data to and from Tier-2 storage
Stage and share data for processing
High bandwidth: up to 1,400 MB/s (c2018 – much higher now)

From MSI Linux shell (command prompt)

s3cmd mb s3://<bucket>

s3cmd put <file> s3://<bucket>

s3cmd get s3://<bucket>/<file> <directory>

s3cmd ls s3://<bucket>

https://www.msi.umn.edu/support/faq/how-do-i-use-second-tier-storage-command-line

Minnesota Supercomputing Institute

26 of 52

Globus

What is it good for?

Move data between sites across WAN, Tier-1, & Tier-2
Web GUI driven
Move LARGE directory trees with a few mouse clicks
Runs in background

How to use

Login to GLOBUS website w/ your UofM ID
Register your certificate ID with Globus endpoints
Use web GUI to drag and drop between endpoints

www.globus.org

Minnesota Supercomputing Institute

27 of 52

Globus Home Page: start here

globus.org

A few years ago:

>>254 PB

… and counting

Minnesota Supercomputing Institute

28 of 52

Globus Home Page: Log In

globus.org

Select:

Log in

Will use UofM Internet ID

Minnesota Supercomputing Institute

29 of 52

Select University of Minnesota

Select:

University

of

Minnesota

Minnesota Supercomputing Institute

30 of 52

Use Your UofM Account

NOTE:

Use your UofM ID here

Minnesota Supercomputing Institute

31 of 52

Manage Data

Select: 1st endpoint field

University

of

Minnesota

Minnesota Supercomputing Institute

32 of 52

Select Globus Endpoint

MSI Home Directories:

umnmsi#home

Minnesota Supercomputing Institute

33 of 52

Authenticate with MSI Account

Minnesota Supercomputing Institute

34 of 52

Folders & Files at MSI

Minnesota Supercomputing Institute

35 of 52

Etner 2nd Enpoint

Physics Endpoint

umnphys#data

Same UofM authentication

Minnesota Supercomputing Institute

36 of 52

Connected to Physics Server

Endpoint in phys. connected to a 200 TB disk system

This physics endpoint is in the same domain as MSI

⇒ did not need to authenticate again.

Minnesota Supercomputing Institute

37 of 52

Example: pipe directory tree

About 4 levels deep

Irregular

Hundreds of directories

Thousands of files

~0.6 GB

Minnesota Supercomputing Institute

38 of 52

Source, Destination, & GO!

Browse to source and destination

Source

Folder:

pipes

Could be a file or a directory.

Destination

path:

/data/uchu

Minnesota Supercomputing Institute

39 of 52

File transfer Requested

Temporary notice

Confirms submission of request

Minnesota Supercomputing Institute

40 of 52

View Request Status

Small transfer ~3 min

Minnesota Supercomputing Institute

41 of 52

Details

Click on request to see details.

7788 files

476 folders

598 MB

~3.5 MB/s

Minnesota Supercomputing Institute

42 of 52

Larger & Fewer Files

More efficient

From Physics

32 files

1 folder

200 GB

38 min.

88 MB/s

From NCSA

220 GB

300+ MB/s

Minnesota Supercomputing Institute

43 of 52

Email Confirmation

Sent when done

Includes stats

Minnesota Supercomputing Institute

44 of 52

Task Parallel Job Using Tier-2 Storage

Example of a data driven workflow

Tier-2 (object store) ⇒ Tier 1 (scratch) ⇒ Tier-1 (home)

SLURM Job Arrays:

https://slurm.schedmd.com/job_array.html

MSI Queues (Partitions):

https://www.msi.umn.edu/partitions#slurm

https://www.msi.umn.edu/content/choosing-job-partition#slurm

Minnesota Supercomputing Institute

45 of 52

Simple Task Array

SLURM script

1 task (core) on 1 node
Up to 3800 MB per task
Up to 10 min.

Submit task array
Output

logs/task_ouput.######_#

#!/bin/bash

#SBATCH -N 1

#SBATCH --ntasks-per-node=1

#SBATCH --mem-per-cpu=3800

#SBATCH -t 00:10:00

#SBATCH -p agsmall

#SBATCH -o logs/task_output.%A_%a

echo "Task " ${SLURM_ARRAY_TASK_ID} " on " $(hostname)

ahl03:task_arrays % sbatch --array=0-9 do_one_task.sh

ahl03:task_arrays % grep Task logs/*

logs/task_output.67134323_0:Task 0 on acn06

logs/task_output.67134323_1:Task 1 on acn06

logs/task_output.67134323_2:Task 2 on acn06

logs/task_output.67134323_3:Task 3 on acn06

logs/task_output.67134323_4:Task 4 on acn06

logs/task_output.67134323_5:Task 5 on acn06

logs/task_output.67134323_6:Task 6 on acn06

logs/task_output.67134323_7:Task 7 on acn06

logs/task_output.67134323_8:Task 8 on acn06

logs/task_output.67134323_9:Task 9 on acn06

do_one_task.sh

Minnesota Supercomputing Institute

46 of 52

Data Driven Workflow

Sentinel 2 data

Top of atmosphere
Focus on one tile: T14TSQ for 6 months (2020 May-Oct)
Variation in distribution of values in band 3 (Green)

Data in Tier-2 bucket

86 tiles

S2B_MSIL1C_20200502T171849_N0209_R012_T14TQS_20200502T204806.zip
S2A_MSIL1C_20200504T170901_N0209_R112_T14TQS_20200504T204849.zip
S2B_MSIL1C_20200509T170849_N0209_R112_T14TQS_20200509T203631.zip
S2A_MSIL1C_20200507T171901_N0209_R012_T14TQS_20200507T205829.zip

…

Minnesota Supercomputing Institute

47 of 52

Split list

Split full list of work items into 20 pieces

20 files: x000 - x019
4-5 items per file

Each job in array

Process all zip files lines in one of the x??? files

split -a 3 -d -n l/20 zip_list.txt

ls x*

x000 x001 x002 x003 x004 x005 x006 x007 x008 x009 x010 x011 x012 x013 x014 x015 x016 x017 x018 x019

cat x005

S2A_MSIL1C_20200626T171901_N0209_R012_T14TQS_20200626T222141.zip

S2B_MSIL1C_20200628T170849_N0209_R112_T14TQS_20200628T204257.zip

S2A_MSIL1C_20200703T170901_N0209_R112_T14TQS_20200703T204911.zip

S2B_MSIL1C_20200701T171859_N0209_R012_T14TQS_20200701T204911.zip

Minnesota Supercomputing Institute

48 of 52

SLURM Script

Resources

1 core on 1 node

Work directory

1 per worker

Conda environment

Supports rasterio

Loop over data

Uses split file

#!/bin/bash

#SBATCH -N 1

#SBATCH --ntasks-per-node=1

#SBATCH --mem-per-cpu=3800

#SBATCH -t 00:10:00

#SBATCH -p agsmall

#SBATCH -o logs/task_output.%A_%a

workdir=w${SLURM_ARRAY_TASK_ID}

mkdir -p $workdir

cd $workdir

source ../conda_init_script

zlist=$(printf "x%03d" ${SLURM_ARRAY_TASK_ID})

for z in $(cat ../$zlist)

do

bucket="s3://dhp-S2MSI1C"

s3cmd get $bucket/$z

unzip $z

python ../s2stats.py >> ../B03_stats_$zlist

rm -rf $(echo $z | cut -d'.' -f 1).*

done

conda deactivate

s2_stats.sh

Minnesota Supercomputing Institute

49 of 52

Python Script

Get Stats of an S2 Image:

Path to the image file
Date – from file name
Read – use rasterio
Calculate percentiles
Write

import rasterio

import glob

import numpy as np

# Get path to image file & extract date from name

jp2_path = glob.glob("*/GRANULE/*/IMG_DATA/*B03.jp2")[0]

jp2_file = jp2_path.split("/")[-1]

date = jp2_file[7:11]+'-'+jp2_file[11:13]+'-'+jp2_file[13:15]

# Read in image values as a numpy ndarray

with rasterio.open(jp2_path) as src:

a = src.read(1)

# Report percentile values (excluding nodata values)

b = np.extract(a > 0, a)

v01,v25,v50,v75,v99 = np.percentile(b, [1,25,50,75,99])

print(date, v01,v25,v50,v75,v99)

s2stats.py

Minnesota Supercomputing Institute

50 of 52

Run Task Array

Submission

sbatch –array=0-19 script

20 parallel workers
script – can do a lot

Monitor

squeue -l -u $USER

Long listing
Only your jobs

ahl03:ta1 % sbatch --array=0-19 s2_stats.sh

sbatch: Setting account: dhp

Submitted batch job 67134525

ahl03:ta1 % squeue -l -u $USER

JOBID PARTITION NAME USER STATE TIME TIME_LIMI NODES

67134525_0 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_1 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_2 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_3 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_4 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_5 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_6 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_7 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_8 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_9 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_10 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_11 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_12 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_13 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_14 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_15 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_16 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_17 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_18 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

67134525_19 agsmall s2_stats dhp RUNNING 0:09 10:00 1 acn06

Minnesota Supercomputing Institute

51 of 52

Output

Output files

B03_stats_x000 B03_stats_x001 B03_stats_x002

B03_stats_x003 B03_stats_x004 B03_stats_x005

B03_stats_x006 B03_stats_x007 B03_stats_x008

B03_stats_x009 B03_stats_x010 B03_stats_x011

B03_stats_x012 B03_stats_x013 B03_stats_x014

B03_stats_x015 B03_stats_x016 B03_stats_x017

B03_stats_x018 B03_stats_x019

cat B03* | sort > B03stats_full.txt

2020-04-02 4309 5792 6597 7596 8803 1.000

2020-04-04 671 918 1091 1376 5687 0.457

2020-04-07 608 864 1033 1255 3099 1.000

2020-04-09 805 1643 3017 5289 8839 0.364

2020-04-12 2831 4588 4993 5391 6235 1.000

2020-04-14 1201 3979 5088 5985 7670 0.463

2020-04-17 728 1422 2813 4907 7741 1.000

2020-04-19 542 955 1135 1324 1847 0.459

…

Minnesota Supercomputing Institute

52 of 52

Questions?

Minnesota Supercomputing Institute