1 of 15

Next Steps with Jupyterhub and ARM

ARM Summer School

Friday May 24, 2024

​

Michael Giansiracusa : ORNL

​

​

1

September 20, 2023

1

2 of 15

Summer School JupyterHub

  • Summer School Profile will remain active for some time

​

  • While the Summer School Profile will be gone one day, we will always have the memories to keep us warm during the winter

​

  • Long term support of elevated dask cluster is in progress

​

Built on brand new hardware to support computational science

2

3 of 15

Getting Access to ARM Data Workbench

​

The Atmospheric Radiation Measurement (ARM) Data Center (ADC) offers JupyterHub notebook environments for science users to stage data and access compute resources remotely. Documentation and access requests can be found at our ServiceNow Knowledge base

​

​

The improvements we’ve tested during this class will be made available to all ARM elevated access science users in the near future.

​

​

3

4 of 15

Service Levels and Applying for Elevated Access

4

5 of 15

ARM Data Center HPC resources

​

The Atmospheric Radiation Measurement (ARM) Data Center (ADC) offers high-performance computing (HPC) resources for users to conduct big-data analytics and machine learning for atmospheric and climate science research.

​

The ADC provides a co-located data and computing platform that enables users to work with large volumes of ARM data without the need to download them.

​

5

6 of 15

Computational resources

Cumulus Cluster

Primary computational cluster consisting of 119 nodes (16,512 cores).

Hostname: cumulus.ccs.ornl.gov

Authentication: ORNL UCAMS/XCAMS

Cumulus Cluster

  • Total Nodes: 129 (16, 512 cores)
  • Standard nodes [112]: 2 x AMD Milan 7713 processors 3Ghz (64 cores/processor) 128 cores per node; 256 GB memory;
  • High memory nodes [16]: 2 x AMD Milan 7713 processors 3Ghz (64 cores/processor) 128 cores per node; 512 GB memory;
  • GPU nodes [1]: 4 x NVIDIA A100 80GB GPUs; 2 x AMD Milan 7713 processors 3Ghz (64 cores/processor) 128 cores per node; 1 TB memory;

​

6

7 of 15

Computational resources - JupyterHub

JupyterHub

Interactive development environment

URL: https://jupyter-open.olcf.ornl.gov/

Authentication: ORNL UCAMS/XCAMS

  • Compute: 64 cores;
  • Memory: 196 GB memory;
  • Storage: Mounts same /home and /gpfs/wolf as Cumulus

Single User Allocation

  • JupyterLab 4
  • Compute: 8 CPU
  • Memory: 24 GB

​

7

8 of 15

Computational resources - JupyterHub

Example notebooks, slurm submit templates, and interactive nodes for job inspection:

Code repository: https://code.arm.gov/docs/cumulus-examples/

​

8

9 of 15

Storage on Cumulus cluster + JupyterHub

9

Filesystem

Mount Points

Backed Up?

Purged?

Quota

Notes

Home Directories (NFS)

/ccsopen/home/$USER

Yes

No

50 GB

User HOME for long term data

​

Project Space (NFS)

/ccsopen/proj/$PROJECT_ID

Yes

No

50 GB

Project space for long term data

Parallel Scratch (GPFS)

​

​

/gpfs/wolf/scratch/$USER/$PROJECT_ID

​

​

/gpfs/wolf/proj-shared/$PROJECT_ID

​

​

/gpfs/wolf/world-shared/$PROJECT_ID

No

No?

4 PB

User scratch space not accessible by other project members

​

​

Project shared space, with Read/Write access to all project members

​

​

Space for sharing data outside of project. Read/Write access to project members, read access for world.

​

​

10 of 15

Software management

The software environment is managed through the Environmental Module tool.

​

​

10

Command

Description

module list

Lists modules currently loaded in a user’s environment

module avail

Lists all available modules on a system in condensed format

module avail -l

Lists all available modules on a system in long format

module display

Shows environment changes that will be made by loading a given module

module load <module-name>

Loads a module

module swap

Swaps a currently loaded module for an unloaded module

Cumulus-2 software stack contains a large number of standard packages and new ones can be requested.

11 of 15

Job scheduling

  • Cumulus utilizes the Slurm batch scheduler
  • 8 Slurm queues are available to fit computational tasks from small to large in size requiring various resource types (standard/high. mem/GPU)
  • Your access, depending on project needs, may be restricted to a subset of queues

#!/bin/bash

#SBATCH -A atmXXX

#SBATCH -J my_job

#SBATCH -N 1

#SBATCH --ntasks-per-node=128

#SBATCH -t 1:00:00

#SBATCH --mem 0

#SBATCH -p batch_high_memory

#SBATCH --mail-type=FAIL

#SBATCH --mail-type=BEGIN

#SBATCH --mail-type=END

#SBATCH --mail-user=email@email.gov

#SBATCH --output=stdout.log

#SBATCH --error=stderr.log

11

Sample Slurm script:

12 of 15

Available Slurm queues

12

Queue name

Max. Nodes

Max. Cores

Priority

Max. Runtime

batch_all

112 standard + 16 high memory

16,384

PriorityTier=1

120:00:00

priority_long

112 standard

14,336

PriorityTier=2

120:00:00

batch_long

112 standard

14,336

PriorityTier=1

120:00:00

batch_short

56 standard

7,168

PriorityTier=1

120:00:00

batch_high_memory

16 high memory

2,048

PriorityTier=1

120:00:00

debug

112 standard

14,336

PriorityTier=1

2:00:00

debug_high_memory

16 high memory

2,048

PriorityTier=1

2:00:00

gpu

1 GPU node

128

PriorityTier=1

120:00:00

13 of 15

Getting Access

Science Users

  • Users applying for access as part of already approved projects should submit a ARM Research Account Request.

​

  • For requesting new projects, submit a ARM HPC Project Request. This request will be reviewed by the ARM Resource Utilization Council (RUC).

​

  • Once the ARM Research Application Request is approved, the next step is to apply for a system account via the OLCF application form [once per Project ID].

​

13

14 of 15

Documentation and resources:

Keep in touch with us:

  • Send any questions/queries to clustersupport@arm.gov

​

14

15 of 15

Thanks for your attention.

​

HPC Team: clustersupport@arm.gov

15