1 of 36

High-Performance Computing with Python/RS-DAT

OpenGeoHub Summer School, Wageningen, 03-09-2025

2 of 36

The Netherlands eScience Center

National centre / independent foundation since 2012 /

NWO & SURF

“Empowering researchers across all disciplines through advanced research software”

Meiert

Grootes

Francesco

Nattino

Ou

Ku

3 of 36

4 of 36

Let’s stay�in touch

www.eScienceCenter.nl

info@esciencecenter.nl

+31 20 460 4770

Check for our open calls

5 of 36

EO and RS as a (scientific) resource

6 of 36

EO and RS as a (scientific) resource

All images Credit: ESA/NASA CC-BY

Big data - Challenge of the The V’s: Volume, Velocity, Variety

7 of 36

EO and RS as a (scientific) resource

Big data - Challenge of the The V’s: Volume, Velocity, Variety

Credit: ESA/NASA, AHN, NCG, SkyGeo

8 of 36

From PC to HPC/HTC

Data deluge across many/most scientific domains.

Local storage and processing resources inadequate

Institutional or national resources at larger scale needed

In NL SURF offers storage and processing resources at PB scale

National labs or Compute centers comparable. US often institutional resources

9 of 36

Failure of the status quo

Remote data resources

Desktop / Workstation

!

Commercial GUI tools

!

10 of 36

What then?

Scalable Compute

Mass storage

Remote data resources

User

Scalable software stack

?

11 of 36

What then?

Scalable Compute

Mass storage

Remote data resources

User

Scalable software stack

?

12 of 36

HPC/HTC vs Cloud vs Platform

User

Provider

Responsibility

13 of 36

HPC/HTC vs Cloud vs Platform

"No cost", merit based Full control of workflow flexible and ‘easily’ configurable (Hardware and environment) Well established filesystem support Highly optimized I/O infrastructure High level of support No dependency on commercial providers	Available on demand at requested level Professional support (if paid for) Configuration via presets (VMs) Possibly no/little need to download data	Available on demand (at requested level if paid for) High level of support (if paid for) Easy configuration via presets (VMs) No need to download data (for some data sets) available high-level convenience functions (e.g. mosaicing) Shallower learning curve
Must retrieve data from remote Storage can be problematic (not so for SURF) Workflow must be implemented at fairly low level Delayed (initial) access Moderately steep learning curve	Commercial provider; paid access, limited access (on free research allotments) Intricate config for non-standard requirements Support (only) at cost Scaling can lead to high costs Moderately steep learning curve	Semi-commercial provider; paid access, limited access (on free research allotments) Limited config Scaling can lead to high costs Limited data variety; require import Intricate remote system

14 of 36

HPC/HTC vs Cloud

Pangeo link

Pythia project link (tutorials and educatioal materal

"No cost", merit based
Highly flexible and configurable (Hardware)
Simple environment customization and dissemination
Well established filesystem support
Highly optimized I/O infrastructure
High level of support
No dependency on commercial providers

Must retrieve data from remote
Storage can be problematic (not so for SURF)
Workflow must be implemented at fairly low level
Possibility of resource competition
Delay between first request and access to resources

No/little need to download data
Easy configuration via (VM) presets
Available on demand at requested level
Possibly available high-level convenience functions (e.g. mosaicing)
Professional support (if paid for)

Commerical provider; paid access
Limited access (on free research allotments)
Intricate config for non-standard requirements
Support (only) at cost
Scaling can lead to high costs

15 of 36

Software stack

Scalable Compute

Mass storage

Remote data resources

User

?

16 of 36

The PANGEO community

Pangeo link

Pythia project link (tutorials and educatioal materal

Core:

Interoperable, scalable, Python data science stack with geospatial focus

Scalable computation

Rich data model w/ out-of-core support

Interactive analysis and execution

17 of 36

RS-DAT – facilitating HPC/HTC platforms

Scalable Compute

Mass storage

Remote data resources

User

?

HPC/HTC

18 of 36

RS-DAT – facilitating HPC/HTC platforms

Scalable Compute

Mass storage

Remote data resources

User

?

HPC/HTC

19 of 36

What is RS-DAT ?

Snellius

SPIDER

SRC

dCache

Remote data resources

User

Community/ Third party software stack

Adoption Barrier

20 of 36

Jupyter at scale

Scale analysis: accustomed interactive workflows, but backed by HPC/Cloud
Combine Jupyter server with Dask cluster on HPC/HTC/Cluster system
Simple, customizable environment deployment

21 of 36

RS-DAT Components

Scalable analysis with Jupyter

Using legacy (Docker) containers for HPC

Python interface to dCache

https://github.com/RS-DAT/DockerToSingularity

https://github.com/RS-DAT/JupyterDaskOnSLURM

https://github.com/NLeSC-GO-common-infrastructure/stac2dcache

https://github.com/NLeSC-GO-common-infrastructure/dcachefs

utility functions to manage STAC catalogs (and the underlying data) on dCache.

Today

22 of 36

Excursion: HPC/HTC basics

Clusters of compute servers (100s –1000s nodes), with 10000s – 100000s cores.

High bandwidth internal network between nodes

Shared RAM per node; configurable

Shared storage/mounted FS

Generally Linux based (command line, SSH keys)

deic

23 of 36

Excursion: HPC/HTC basics

Clusters of compute servers (100s –1000s nodes), with 10000s – 100000s cores.

High bandwidth internal network between nodes

Shared RAM per node; configurable

Shared storage/mounted FS

Generally Linux based (command line, SSH keys)

Designed for non-interactive, large jobs or batch jobs

Either VERY computationally expensive (classic HPC) and/or involving massive data volumes to process (HTC)

HPC systems are a SHARED resource. (login, etiquette)
Resources must be allocated and jobs scheduled deic

24 of 36

scheduling

HPC is shared system running multiple jobs from users.

Need system/software to handle resource allocation

The batch scheduler

25 of 36

scheduling

HPC is shared system running multiple jobs from users.

Need system/software to handle resource allocation

The batch scheduler

Elements of scheduling:

Job script: declaration of job and required resources

Job queue: ordered registry of submitted job(script)s

Scheduler: Software running scheduling algorithm over job queue and allocating resources and priority

26 of 36

scheduling

HPC is shared system running multiple jobs from users.

Need system/software to handle resource allocation

The batch scheduler

Elements of scheduling:

Job script: declaration of job and required resources

Job queue: ordered registry of submitted job(script)s

Scheduler: Software running scheduling algorithm over job queue and allocating resources and priorities

Scheduling algorithms:

First come, first served

Everything scheduled, but inefficient

Shortest job first

Long/resource heavy may never be scheduled

27 of 36

scheduling

HPC is shared system running multiple jobs from users.

Need system/software to handle resource allocation

The batch scheduler

Elements of scheduling:

Job script: declaration of job and required resources

Job queue: ordered registry of submitted job(script)s

Scheduler: Software running scheduling algorithm over job queue and allocating resources and priorities

Scheduling algorithms:

First come, first served

Everything scheduled, but inefficient, long wait

Shortest job first

Long/resource heavy may never be scheduled

Backfilling

Combine and interleave, all scheduled

time

28 of 36

Jupyter Dask on Slurm

SURF H*C Infrastructure: Spider (HTC), Snellius (HPC); SLURM scheduler

JupyterLab instance with Dask and Git extensions, scalable Dask cluster, user defined environment, running on SLURM managed HTC/HPC system
Basic idea

Launch JupyterLab server and Dask scheduler as long-running batch job
SSH port-forwarding to connect to JupyterLab server
Launch workers as short-lived batch jobs (fast thru queue)

29 of 36

Focus for today

Providing you with experience/expertise to leverage SURF H*C Infrastructure: Spider (HTC), Snellius (HPC); SLURM scheduler

Define and deploy an analysis environment to SPIDER using JupyterDaskOnSlurm
Basics of scalable (EO) computation in the Python/Dask ecosystem
Example EO use case (notebooks)

Required materials and point of departure for the interactive session is https://edu.nl/4jc6u

30 of 36

scheduling

SLURM scheduler on SURF systems (Spider, Snellius)

SLURM at a glance

https://slurm.schedmd.com/quickstart.html

31 of 36

Solution dichotomy

Pangeo link

Pythia project link (tutorials and educatioal materal

"No cost", merit based
Highly flexible and configurable (Hardware)
Simple environment customization and dissemination
Well established filesystem support
Highly optimized I/O infrastructure
High level of support
No dependency on commercial providers

Must retrieve data from remote
Storage can be problematic (not so for SURF)
Workflow must be implemented at fairly low level
Possibility of resource competition
Delay between first request and access to resources

No/little need to download data
Easy configuration via (VM) presets
Available on demand at requested level
Possibly available high-level convenience functions (e.g. mosaicing)
Professional support (if paid for)

Commerical provider; paid access
Limited access (on free research allotments)
Intricate config for non-standard requirements
Support (only) at cost
Scaling can lead to high costs

32 of 36

RS-DAT core in brief

33 of 36

Flexible library for parallel and distributed computing on "big (larger than memory) data".

Extends pandas and NumPy APIs to larger-

than-memory data (chunks)

Fully customizable

Scales from laptop to cluster (and back)

Designed for interactive use

Native python

Almost like writing serial code (some

idiosyncracies)

34 of 36

What next?

Snellius

SPIDER

SRC

dCache

Remote data resources

User

Scalable software stack

?

35 of 36

What is RS-DAT ?

Snellius

SPIDER

SRC

dCache

Remote data resources

User

Community/ Third party software stack

Adoption Barrier

36 of 36

Democratizing Big GeoData

EO data essential in many applications
Satellite as unique resource offering large area w/ repeat
EO data central and essential to many applications (cryosphere monitoring, agriculture, ecology
Broad range of missions (imaging, radar, lidar,…)
Well adopted and embedded in scientific communitty