1 of 15

2025 Zarr Summit – Rome, Italy

how to make zarr go �zoom zoom

2 of 15

Some context

I tend to be most interested in…

  • making Zarr fast in the cloud
  • using Xarray on top of Zarr-Python
  • working with climate/weather/geospatial data

​

This talk is not meant to be an exhaustive list of benchmarks of Zarr…

  • I’ll give that talk next year 😉
  • but this is meant to show how one specific workflow has been dramatically accelerated over the past 2 years

​

​

2

3 of 15

Example workflow: Xarray + Zarr

# 1 - open a zarr store using Xarray

ds = xr.open_zarr(“s3://bucket/dataset.zarr”)

​

# 2 - load all arrays into memory

ds.load()

​

# 3 - write the dataset to a s3 bucket

ds.to_zarr(“s3://bucket/dataset2.zarr”)

3

4 of 15

Simple benchmark

  • Read / write a typical ~1GB dataset to/from S3
  • Chunk size ~1MB with standard compression
  • Zarr Format == 2
    • (allows us to test w/ zarr 2.x )
  • No Dask or other parallel array framework
  • Measure wall time for:
    • xr.open_zarr
    • ds.load()
    • ds.to_zarr()
  • Run for an environment snapshot on the 1st of each month going back 2 years

4

5 of 15

Big gains in the past two years!

# 1 - open a zarr store using Xarray

ds = xr.open_zarr(“s3://bucket/dataset.zarr”)

​

# 2 - load all arrays into memory

ds.load()

​

# 3 - write the dataset to a s3 bucket

ds.to_zarr(“s3://bucket/dataset2.zarr”)

5

~3x faster

>3x faster

~6x faster

6 of 15

6

Zarr 3.0

Zarr 3.0.8�(Obstore)

Bugfix 🤦

7 of 15

7

Zarr 3.0

Zarr 3.0.8�Obstore

8 of 15

how did we do it?

8

9 of 15

First review: why do we expect Zarr to be “fast”?

1. Chunked, streaming access - Data are split into independently readable chunks, enabling selective, efficient I/O.

2. Compression saves on both storage and bandwidth - Per-chunk compression reduces data transfer and storage costs, critical for networked workloads.

3. Designed for parallel I/O - Independent chunks can be read or written concurrently across threads or processes.

4. High-throughput storage backends - Backend-agnostic design lets S3, GCS, and others optimize access with range and batch operations.

9

10 of 15

How did we do it?

  1. Aggressively limit any “extra” I/O calls

→ Both Xarray and Zarr have I/O counters in their test suite now

​

Zarr Python 3

​

  • Asynchronous I/O everywhere

→ Stores interact exclusively with backends (e.g. S3) via AsyncIO

  • Concurrently do I/O and codec execution�→ Encode/decode chunks while other chunks are being read/written
  • Offload codec pipeline execution to separate threads

→ Codecs use AsyncIO’s to_thread

  • Use fast cloud backends

→ e.g. Use Obstore S3 backend instead of s3fs

​

10

11 of 15

pip install xarray zarr

# Ready to use today 🚀

11

12 of 15

Still lots to do!

12

13 of 15

What next?

  1. Move from ad-hoc benchmark suite to something automated
  2. Codec pipeline optimizations
    1. Avoid memory copies whenever possible
  3. Sharding
    • Write performance (avoid incrementally building the
    • Read performance (request coalescing)
  4. GPU performance

13

14 of 15

Questions?

joe@earthmover.io

14

v

15 of 15

A look under the hood

xarray.open_zarr()

​

  • Open the root group
  • List all children
  • Open all child arrays
  • Parse CF-metadata (domain specific, used to determine coordinate arrays)
  • Load coordinate arrays
  • Construct Dataset

​

Dataset.load()

​

  • For each data variable, load all values into NumPy arrays

15

https://github.com/pydata/xarray/blob/main/xarray/backends/zarr.py