1 of 43

Parallel Geospatial �Data Analysis & Visualization at WHOI

Katy Abbott & Anthony Meza

2 of 43

The Big Data “Problem”

“Three Vs” of big data: Volume, velocity and variety

  • Volume: What is the total amount of data?
  • Velocity: How quickly is that data being produced?
  • Variety: What different types of data can we work with?

​

3 of 43

Parallel Computing/High Performance Computing

  • We have more observational/model data than we know what to do with�
  • Parallelization allows us to efficiently work with large datasets

​

  • Efficient computing means faster results�
  • The Poseidon Cluster allow you to execute parallelization easily and asynchronously (its free real estate!)

Size of Neural Network

Kan et. al. 2018

4 of 43

What is Parallel Computing? �Serial Computing

Traditionally, software has been written for serial computation�

  • A problem (or data) is broken into discrete chunks
  • Each chunk is operated on sequentially
  • Each operation is done by a single processor

Source: https://hpc.llnl.gov/documentation/tutorials/introduction-parallel-computing-tutorial##Overview

Problem

5 of 43

What is Parallel Computing? �Serial Computing

Traditionally, software has been written for serial computation�

  • A problem (or data) is broken into discrete chunks
  • Each chunk is operated on sequentially
  • Each operation is done by a single processor

Source: https://hpc.llnl.gov/documentation/tutorials/introduction-parallel-computing-tutorial##Overview

Problem

Problem Chunks

{

6 of 43

What is Parallel Computing? �Serial Computing

Traditionally, software has been written for serial computation�

  • A problem (or data) is broken into discrete chunks
  • Each chunk is operated on sequentially
  • Each operation is done by a single processor

Source: https://hpc.llnl.gov/documentation/tutorials/introduction-parallel-computing-tutorial##Overview

Problem

Problem Chunks

+

Processor

Operation

{

{

7 of 43

What is Parallel Computing? �Serial Computing

Traditionally, software has been written for serial computation�

  • A problem (or data) is broken into discrete chunks
  • Each chunk is operated on sequentially
  • Each operation is done by a single processor

Source: https://hpc.llnl.gov/documentation/tutorials/introduction-parallel-computing-tutorial##Overview

Problem

Problem Chunks

+

Processor

Operation

Modified Chunk

{

{

{

8 of 43

What is Parallel Computing? �Serial Computing

Source: https://hpc.llnl.gov/documentation/tutorials/introduction-parallel-computing-tutorial

Problem: Moving Apartments

9 of 43

What is Parallel Computing? �Serial Computing

Source: https://hpc.llnl.gov/documentation/tutorials/introduction-parallel-computing-tutorial

Problem: Moving Apartments

Problem Chunks: Moving Boxes

{

10 of 43

What is Parallel Computing? �Serial Computing

Source: https://hpc.llnl.gov/documentation/tutorials/introduction-parallel-computing-tutorial

Problem: Moving Apartments

Problem Chunks: Moving Boxes

{

+

Processor:�You

Operation: �Loading Boxes

{

{

11 of 43

What is Parallel Computing? �Serial Computing

Source: https://hpc.llnl.gov/documentation/tutorials/introduction-parallel-computing-tutorial

Problem: Moving Apartments

Problem Chunks: Moving Boxes

{

+

Processor:�You

Operation: �Loading Boxes

{

{

Modified Chunk: �Boxes in Truck

{

12 of 43

What is Parallel Computing? �Parallel Computing

Source: https://hpc.llnl.gov/documentation/tutorials/introduction-parallel-computing-tutorial##Overview

Parallel computing is using multiple processors to simultaneously operate on your problems/data

Problem

{

Problem Chunks

13 of 43

What is Parallel Computing? �Parallel Computing

Parallel computing is using multiple processors to simultaneously operate on your problems/data

Problem

Problem Chunks

{

+

+

+

+

Source: https://hpc.llnl.gov/documentation/tutorials/introduction-parallel-computing-tutorial##Overview

Parallel�Operations

{

14 of 43

What is Parallel Computing? �Parallel Computing

Parallel computing is using multiple processors to simultaneously operate on your problems/data

Problem

Problem Chunks

{

+

+

+

+

Modified Chunks

{

Source: https://hpc.llnl.gov/documentation/tutorials/introduction-parallel-computing-tutorial##Overview

Parallel�Operations

{

15 of 43

Toy Oceanographic Problem

time

SST

precip

8:00 AM

27.8 °C

9:00 AM

27.9 °C

0.06 mm

. . .

. . .

. . .

Oceanography Data

0.01 mm

16 of 43

Toy Oceanographic Problem

time

SST

precip

8:00 AM

27.8 °C

9:00 AM

27.9 °C

0.06 mm

. . .

. . .

. . .

Operation: Time mean

Oceanography Data

Mean�SST

Mean

precip

27.8 °C

0.03 mm

0.01 mm

17 of 43

Toy Oceanographic Problem

time

SST

precip

8:00 AM

27.8 °C

9:00 AM

27.9 °C

0.06 mm

. . .

. . .

. . .

Oceanography Data

Q1: How many processors would you use?

Q2: What would your chunks look like?

27.8 °C

0.03 mm

0.01 mm

Operation: Time mean

Mean�SST

Mean

precip

18 of 43

Why Python? (And not MATLAB/Julia/R etc.)

19 of 43

Why Python? (And not MATLAB/Julia/R etc.)

  • Language of choice for many computational scientists — so a lot of packages have been written for Python!
    • Pandas, scikit-learn, SciPy, xarray, xESMF
  • Big online community and tons of StackOverflow posts for any issue you come up with

20 of 43

Introduction to xarray and Dask

21 of 43

Why use xarray?

  • xarray makes it so easy to open, write and save multi-dimensional labeled arrays (e.g. netCDFs, Zarr) and perform basic manipulations
  • Built off of and very similar to Pandas
  • Data structure: Dimensions, coordinates, data variables and attributes

22 of 43

Look Ma, one line climatology!

  • xarray has many functions that take advantage of the data structure to compute things efficiently

23 of 43

Look Ma, one line climatology!

  • xarray has many functions that take advantage of the data structure to compute things efficiently

24 of 43

What is Dask?

  • Open-source Python library for parallel computing
  • Very easy to use (albeit harder to use it well!)

25 of 43

What is Dask?

  • Open-source Python library for parallel computing
  • Very easy to use (albeit harder to use it well!)
  • Dask lets you compute in parallel on larger-than-memory datasets

26 of 43

What is Dask?

  • Open-source Python library for parallel computing
  • Very easy to use (albeit harder to use it well!)
  • Dask lets you compute in parallel on larger-than-memory datasets
  • “Lazy loading”: Will only load data chunks as needed to perform an operation

27 of 43

Chunking overview

time

x

y

28 of 43

Chunking overview

time

x

y

A

B

29 of 43

30 of 43

BREAK

BREAK

BREAK

Come back in 15 minutes for some hands-on practice with Poseidon!

​

(Please see Katy and Anthony during the break if you are having any issues with your Poseidon set-up)

31 of 43

Using Miniforge as Scientific Package Manager

  • Benefits of using virtual environments + Miniconda include:
  • Semi-automatic conflict management
  • Scientific code organization + shareability
  • Self-contained messes

32 of 43

Using Miniforge as Scientific Package Manager

  • Benefits of using Miniforge include:
  • Semi-automatic virtual environment management
  • Scientific code organization + shareability
  • Self-contained messes

33 of 43

HPC terminology

Cluster

34 of 43

HPC terminology

Cluster

node

35 of 43

HPC terminology

Cluster

login

compute

compute

compute

compute

gpu

node

36 of 43

HPC terminology

node:

  • Like a physical computer, with RAM and a central processor unit (CPU)
  • A CPU contains multiple CPU cores

37 of 43

HPC terminology

What if I want to submit a job with 1 step and 6 tasks that will be executed in parallel?

38 of 43

HPC terminology

What if I want to submit a job with 1 step and 6 tasks that will be executed in parallel?

39 of 43

HPC terminology

  • Slurm is a job scheduler that matches your job specifications to the resources available
  • Note: on Slurm, cpu == core

​

​

40 of 43

Poseidon Specifics

node

  • Poseidon has 85 compute nodes
  • Each compute node has �36 cores

41 of 43

References

https://ar5iv.labs.arxiv.org/html/1908.03356

42 of 43

Tmux/screen

  • Tmux == terminal multiplexer
  • Allows you to switch between different “pseudoterminals” and detach from them
  • tmux new -s session_name - create a new tmux session
  • Control-b d - detach from a session while letting it run in the background!!
  • tmux ls - see all sessions you have currently running
  • tmux a -t session_name - reattach to a previous session

​

Within a session, you can split terminal windows, toggle between them and then detach from everything

43 of 43

Thanks for coming!

🚨🚨🚨 DON’T RUN ANYTHING ON THE LOG-IN NODE🚨🚨🚨