1 of 73

Towards AI-based Data Assimilation for Weather Forecasting

Ryan Keisler

Daniel Rothenberg

Gideon Dresdner

Jacob Bieker

Hans Mohrmann

NCAR / CISL Seminar

March 20, 2025

2 of 73

Making AI weather forecasting tools available to all, to help humanity adapt to increasingly extreme weather

�

Building an end-to-end probabilistic forecasting system�

Open-sourcing benchmark datasets, models and metrics to spur innovation in AI weather forecasting�

Providing AI weather forecasting tools to all

3 of 73

NNJA-AI - �An AI-Ready,�Cloud-Optimized�Observational Dataset

for Machine Learning Weather Prediction

4 of 73

Observation-Driven MLWP is here…

Aardvark Weather (Vaughn et al, 2024)

Heavily re-processes data to common grids

GraphDOP (Alexe et al, 2024)

Leverages Level 1 data in native spatial/temporal resolution

* Hear more about Brightband’s observation-driven DA and forecasting system on Tuesday @ 9AM in Rm 339 - 5A.3 - Keisler et al, “AI Data Assimilation Directly from Observations”

Anyways, the story I want to share with you all today begins at the very frontier of AI weather forecast model development.

Over the past year, a new wave of models that we like to call “observation-driven MLWP” (or machine learning weather prediction) have emerged from the research community.

Unlike the first wave of data-driven AI or MLWP models, which are initalized from the same sort of assimilated, homogenized 3D fields that traditional NWP models are initialized from, observation-driven MLWP models directly ingest large collections of raw or lightly processed observational data.

The very first examples of this technique, including the Aardvark Weather paper by Vaughan et al, performed some degree of regridding and distillation to map disparate observational datasets onto common grids. This is also a common practice in the AI-mediated nowcasting world. More recently, works such as ECMWF’s GraphDOP eschew any sort of analysis or processing and directly consume raw, Level 1 datasets in their native spatial and temporal resolution. No regridding, no reanalysis - just the same sort of raw datasets that you’d feed into a traditional data assimilation routine.

Again, as a quick plug - the team at Brightband has also developed similar approaches. We are also able to run these operationally - so check out Ryan’s talk tomorrow if you’re interested in learning more.

5 of 73

Data Challenges for Observation-Driven MLWP

Observational data is multi-modal and heterogeneous

Timeseries at fixed points (surface stations, buoys)
Lagrangian trajectories (weather balloons, gliders, drones)
Gridded / Raster-like with multiple channels (LEO / GEO satellites)
Volumetric (doppler radar)

Multiplicity of providers disseminate data using different

distribution technologies (APIs, FTP/Cloud Buckets…),
formats (BUFR, HDF5/NetCDF, ASCII, other binary…) and
standards (ECMWF ParamDB, CF-Conventions, DX BUFR, …)
licenses (mixture of open and restricted commercial)

Curating observational data for AI/ML tasks is a significantly complex engineering task and impediment to rapid research progress

Now that groups are actively pursuing observation-driven MLWP, we can begin to understand some the unique requirements and challenges involved in this sort of modeling.

First, observational data vary tremendeously from source to source. From surface stations or buoys which are fixed in space and measure timeseries to polar-orbiting satellites, which retrieve data along swaths that lay over the Earth’s surface like a ticker tape, varying in both time and space, to more exotic datasets like Lagrangian trajectories captured by gliders, observational data takes on many modalities.

Furthermore, and partly because of this heterogeneous nature, providers disseminate observations in a wide variety of manners. These involves different file formats - running the gamut to highly-specialized binary formats to very generic ones with good software support - different data conventions or standards, as well as different licenses. Observational data can be something of a “wild west”, where everywhere you go you find something interesting and valuable, but it’s a new showdown every day if you want to do something useful with what you find.

As a result, curating these data for AI/ML tasks is daunting. The sheer complexity and variety of data out there imposes a burden to developers, and slows down research progress and applications.

6 of 73

NOAA NASA Joint Archive (NNJA) of Observations for Earth System Reanalysis

Curated archive of observational data from a large swath of sensors/platforms

Implements FAIR principles:

Single, consistent archive structure + file format (BUFR)
Clear documentation; consistent metadata + self-documentation
Published openly on AWS S3 using a permissive license (CC-BY 4.0)
Prepared and preprocessed in a consistent fashion intended for NWP modeling

https://psl.noaa.gov/data/nnja_obs/

However, there are some efforts to solve some of these problems, at least for narrow tasks. For example, a joint collaboration between NOAA and NASA has recently developed the “NNJA” dataset, which is an archive of data curated for work aimed at improving reanalysis modeling techniques.

NNJA is purpose-built for its scientific task, and it has been developed with FAIR principles in mind.

Most notably, the NNJA dataset is well-documented with a nice overview website, and it’s distributed freely under a permissive license, on a popular cloud storage platform, AWS S3. The data has been consistently processed into a single, legacy format called BUFR which is widely used by the data assimilation and numerical modeling communities.

—

This bullet list is effectively the FAIR data standards from the BPC / ESIP Data Readiness Cluster - https://www.esipfed.org/checklist-ai-ready-data/

The NOAA NASA Joint Archive (NNJA) of Observations for Earth System Reanalysis dataset is a comprehensive, open dataset of observations intended to be used in comparing reanalysis methodologies
Reproduce figures from NNJA site on dataset availability, sensors, etc.
Note distribution of data on S3, but in BUFR format (foreshadowing)

7 of 73

Example: NNJA AMSU-A vs Reference

Left: AMSU-A L1C data from NCEI catalog / AIRS

Right: AMSU-A Tb sampled directly from NNJA

Just as a quick demo of what NNJA data looks like, I’ve plotted here a sample of AMSU-A microwave brightness temperatures from two sources. On the left is the “authoritative” reference from the NOAA climate data record archive of these data and on the right is the equivalent data as found in the NNJA archive. Aside from a slightly smaller data volume in the NNJA archive, the two datasets are effectively identical.

—

(break up flow by providing a slide which is just a large visualization to discuss)
Note that contents of NNJA very closely match other archives containing specific subsets of the dataset (e.g. the NOAA CDR AMSUA, which is an authoritative record of just this dataset from NOAA’s satellites).
With visual, note the distribution of retrievals - follows geometric patterns along the satellite’s trajectory over the Earth’s surface; each pixel is a different “measurement”

8 of 73

Challenges with NNJA BUFR

Wikipedia

Schema-based encoding of heterogeneous weather data to a binary format.�
Record-based / tabular format.�
Data is still geospatial/temporal but fully unstructured. �
Each “subset” is one observation - e.g. a pixel or a single sounding.�
Fractured and limited software / tooling - some difficult-to-use Fortran libraries

BUFR file

Message 1

Message 2

…

Message 1

Obs/Subset 1

Obs/Subset 2

…

NC021045 | MSG TYPE 021-045 PROC. GOES-16 ALL SKY RADIANCES (ASR)

├──SIDENSEQ | SATELLITE IDENTIFICATION

│ ├──SIDGRSEQ | SATELLITE IDENTIFIER/GENERATING RESOLUTION

│ │ ├──SAID | SATELLITE IDENTIFIER

│ │ ├──GCLONG | ORIGINATING/GENERATING CENTER

│ │ ├──SCLF | SATELLITE CLASSIFICATION

│ │ ├──SSNX | SEGMENT SIZE AT NADIR IN X DIRECTION

│ │ └──SSNY | SEGMENT SIZE AT NADIR IN Y DIRECTION

│ ├──YYMMDD | DATE -- YEAR, MONTH, DAY

│ │ ├──YEAR | YEAR

│ │ ├──MNTH | MONTH

│ │ └──DAYS | DAY

│ ├──HHMMSS | TIME -- HOUR, MINUTE, SECOND

│ │ ├──HOUR | HOUR

│ │ ├──MINU | MINUTE

│ │ └──SECO | SECOND

│ └──LTLONH | HIGH ACCURACY LATITUDE/LONGITUDE POSITION

│ ├──CLATH | LATITUDE (HIGH ACCURACY)

│ └──CLONH | LONGITUDE (HIGH ACCURACY)

├──NPPR | NUMBER OF PIXELS PER ROW

├──NPPC | NUMBER OF PIXELS PER COLUMN

├──SAZA | SATELLITE ZENITH ANGLE

├──BEARAZ | BEARING OR AZIMUTH

├──SOZA | SOLAR ZENITH ANGLE

├──SOLAZI | SOLAR AZIMUTH

├──HITE | GEOPOTENTIAL HEIGHT

├──CLOUDCOV | CLOUD COVERAGE

│ ├──NCLDMNT | AMOUNT SEGMENT CLOUD FREE

│ ├──LSQL | LAND/SEA QUALIFIER

│ ├──NCLDMNT | AMOUNT SEGMENT CLOUD FREE

│ ├──LSQL | LAND/SEA QUALIFIER

│ ├──CLDMNT | CLOUD AMOUNT IN SEGMENT

| ...

│ ├──VSAT | VERTICAL SIGNIFICANCE (SATELLITE OBSERVATIONS)

│ ├──CLDMNT | CLOUD AMOUNT IN SEGMENT

│ └──VSAT | VERTICAL SIGNIFICANCE (SATELLITE OBSERVATIONS)

├──SIDP | SATELLITE INSTRUMENT DATA USED IN PROCESSING

├──RDCM | RADIANCE COMPUTATIONAL METHOD

├──"ALLSKYRC"10 | ALL SKY RADIANCE DATA

│ ├──SCCF | SATELLITE CHANNEL CENTER FREQUENCY

│ ├──SCBW | SATELLITE CHANNEL BAND WIDTH

│ ├──TMBRST | BRIGHTNESS TEMPERATURE

│ ├──METFET | METEOROLOGICAL FEATURE

| ...

│ ├──VSAT | VERTICAL SIGNIFICANCE (SATELLITE OBSERVATIONS)

│ └──TMBRST | BRIGHTNESS TEMPERATURE

Despite these virtues, NNJA is a challenging dataset to work with. Or rather, one of its virtues makes it challenging - specifically, the fact that it’s archived in BUFR format.

Most of you in the audience probably haven’t worked directly with BUFR data unless you regularly run data assimilation systems. Even then, I don’t think it’s terribly likely that you manually unpacked BUFR datasets.

BUFR as a format is quite old, but despite its age it has some conscientious design principles. It heavily uses schemas and supports building complex nested structures with lists. The schemas mean the dataset is self-documenting, although these schemas are packaged separately from the BUFR file itself. It’s also a record-based format, meaning that each observation or “subset”, which could be anything from one pixel from a satellite retrieval or one timestamp from a surface station, are individually and sequentally packed into a file.

But in practice BUFR is difficult to work with. There’s a fractured and highly limited software ecosystem supporting it, dominated by legay Fortran libraries. Worse off, there is a divide in the BUFR standard that is used operationally at NOAA and at ECMWF, meaning that it can be difficult to use the tools from each community of practice to decode the others’ data.

—

BUFR

Advantages: WMO standard; consistent, schema-structured dataset applied to the entire data record; binary format w/ small file sizes; “battle-tested” (old and reliable)
Disadvantages: limited libraries or tools for reading/decoding - virtually all written in Fortran; incompatibilities between ECMWF and NOAA “flavors”; reading/decoding libraries are not performant and require the user to manually write driver code (can’t just “read” the entire file and create a data structure in memory representing entire contents); likely no path to efficiently reading directly off the cloud due to internal structure of files (message offsets not saved in headers)

Nit: archive structure

Output files contain +/- 3 hours of data based on the timestamp embedded in their filename, e.g. a file labeled 2022-01-13 06Z would have data from 3Z-9Z on 2022-01-13, which can make it a bit cumbersome to fetch observations restricted to a specific observation time window.

9 of 73

NNJA-AI Approach and High-Level Design

Directly re-process entire archive into an intermediate format, preserving original data schemes, but optimized for large-scale cloud-based data engineering.�
Convert to a structured, cloud-native archive with an intermediate / consolidated data schema and cataloging system to enable both SQL-like and direct access to the underlying data.�
Provide scalable and flexible pipelines to allow users to easily explore, extract, post-process, and otherwise coerce data from the archive to meet their own requirements and use cases.

10 of 73

De-BUFR’ing

.------------------------------------------------------------------------------.

| ------------ USER DEFINITIONS FOR TABLE-A TABLE-B TABLE D -------------- |

|------------------------------------------------------------------------------|

| MNEMONIC | NUMBER | DESCRIPTION |

|----------|--------|----------------------------------------------------------|

| | | |

| NC021241 | A61207 | MTYP 021-241 IASI 1C RADIANCES (VARIABLE CHNS) (METOP) |

| | | |

| YYMMDD | 301011 | DATE -- YEAR, MONTH, DAY |

| HHMM | 301012 | TIME -- HOUR, MINUTE |

| HHMMSS | 301013 | TIME -- HOUR, MINUTE, SECOND |

| LTLONH | 301021 | HIGH ACCURACY LATITUDE/LONGITUDE POSITION |

| SIDGRSEQ | 301071 | SATELLITE IDENTIFIER/GENERATING RESOLUTION |

| SIDENSEQ | 301072 | SATELLITE IDENTIFICATION |

| . . . . | . . . | . . . . . . . . . . . . . . . . . . . . . . . . . . . . |

| | | |

|------------------------------------------------------------------------------|

| MNEMONIC | SEQUENCE |

|----------|-------------------------------------------------------------------|

| | |

| NC021241 | SAID GCLONG SIID SCLF YEAR MNTH DAYS HOUR MINU 202131 |

| NC021241 | 201138 SECO 201000 202000 CLATH CLONH SAZA BEARAZ SOZA |

| NC021241 | SOLAZI FOVN ORBN 201133 SLNM 201000 201132 MJFC 201000 |

| NC021241 | 202126 SELV 202000 QGFQ QGQI QGQIL QGQIR QGQIS QGSSQ |

| NC021241 | "IASIL1CB"10 (IASICHN) SIID AVHCST "IASIL1CS"7 |

| | |

| IASICHN | 201136 CHNM 201000 SCRA |

| | |

|------------------------------------------------------------------------------|

| MNEMONIC | SCAL | REFERENCE | BIT | UNITS |-------------|

|----------|------|-------------|-----|--------------------------|-------------|

| | | | | |-------------|

| . . . . | . . | . . . . . . | . . | . . . . . . . . . . . . |-------------|

| | | | | |-------------|

`------------------------------------------------------------------------------'

NOAA-produced BUFR files include an ASCII-based “DX BUFR Table” which define mnemonics.

Mnemonics are the building blocks of the schema used to encode a single observation written to a BUFR file.

Notable special format strs:

<xxx>: 1-bit delayed replication; an “optional” field

{xxx}: 8-bit delayed replication;

Expect 0-255 repetitions

(xxx): 16-bit delayed replication; Expect 0-65536 repetitions

“xxx”N: non-delayed replication; exactly N repetitions

11 of 73

De-BUFR’ing

NC021241 | MTYP 021-241 IASI 1C RADIANCES (VARIABLE CHNS) (METOP)

├──SAID | SATELLITE IDENTIFIER

├──GCLONG | ORIGINATING/GENERATING CENTER

├──SIID | SATELLITE INSTRUMENTS

├──SCLF | SATELLITE CLASSIFICATION

├──YEAR | YEAR

├──MNTH | MONTH

├──DAYS | DAY

├──HOUR | HOUR

├──MINU | MINUTE

├──SECO | SECOND

├──CLATH | LATITUDE (HIGH ACCURACY)

├──CLONH | LONGITUDE (HIGH ACCURACY)

| * * * * * * * * |

├──QGSSQ | OUTPUT OF TEC FUNCTION

├──"IASIL1CB"10 | IASI LEVEL 1C BAND DESCRIPTION SEQUENCE

│ ├──STCH | START CHANNEL

│ ├──ENCH | END CHANNEL

│ └──CHSF | CHANNEL SCALE FACTOR

├──(IASICHN) | IASI LEVEL 1C SCALED RADIANCE SEQUENCE

│ ├──CHNM | CHANNEL NUMBER

│ └──SCRA | SCALED IASI RADIANCE

├──SIID | SATELLITE INSTRUMENTS

├──AVHCST | AVHRR CHANNEL COMBINATION

└──"IASIL1CS"7 | IASI LEVEL 1C AVHRR SINGLE SCENE SEQUENCE

├──YAPCG | Y ANGULAR POSITION OF CENTER OF GRAVITY

├──ZAPCG | Z ANGULAR POSITION OF CENTER OF GRAVITY

├──FCPH | FRACTION OF CLEAR PIXELS IN HIRS FOV

└──"AVHRCHN"6 | IASI LEVEL 1C MEAN AND STANDARD DEVIATION RADIANCE SEQ

├──CHNM | CHANNEL NUMBER

├──CHSF | CHANNEL SCALE FACTOR

├──SMRA | SCALED MEAN AVHRR RADIANCE

├──CHSF | CHANNEL SCALE FACTOR

└──SSDR | SCALED STANDARD DEVIATION OF AVHRR RADIANCE

For any given mnemonic, we can construct a graph representing the structure of a given message type.

This graph is a sort of schematic representation of the contents of a given message, and we can use it to encode or decode a message to pass between different serialization libraries

12 of 73

De-BUFR’ing

syntax = "proto3";

message NC021241 {

enum SAID_enum {

ERS_1 = 0;

ERS_2 = 1;

...

NOAA_21 = 226;

...

}

optional SAID_enum SAID = 1;

optional int32 GCLONG = 2; /* or just encode as int */

...

optional float QGSSQ = 27;

message IASIL1CB_entry {

optional int32 STCH = 1;

optional int32 ENCH = 2;

optional float CHSF = 3;

}

repeated IASI1LCB_entry IASI1LCB = 28;

...

message IASIL1CS_entry {

optional float YAPCG = 1;

...

message AVHRCHN_entry {

optional int32 CHNM = 1;

optional float CHSF01 = 2;

optional float SMRA = 3;

optional float CHSF02 = 4;

optional float SSDR = 5;

}

repeated AVHRCHN_entry AVHRCHN = 4;

}

repeated IASIL1CS_entry IASIL1CS = 32;

}

Example: sketch of NC021241 (IASI) message as a Protocol Buffer (protobuf)

Protobufs require a pre-compiled schema, so we could generate sets of them from published DX BUFR tables and Code/Flag tables.

Not ideal format for data serialization:

Require loading entire message into memory to parse
Messages are not compressed
Not a formal standard (developed and maintained by Google)
Cannot (easily) embed metadata (use reflexive .proto file definitions)

13 of 73

De-BUFR’ing

{

"doc": "MTYP 021-241 IASI 1C RADIANCES (VARIABLE CHNS) (METOP)",

"fields": [

{"doc": "SATELLITE IDENTIFIER", "name": "SAID", "type": "int"},

{"doc": "ORIGINATING/GENERATING CENTER", "name": "GCLONG", type": "int"},

...

{"doc": "OUTPUT OF TEC FUNCTION", "name": "QGSSQ", "type": "double"},

{"doc": "IASI LEVEL 1C BAND DESCRIPTION SEQUENCE DATA",

"name": "IASIL1CB",

"type": {

"items": {

"doc": "IASI LEVEL 1C BAND DESCRIPTION SEQUENCE",

"fields": [

{"doc": "START CHANNEL", "name": "STCH", "type": "double"},

{"doc": "END CHANNEL", "name": "ENCH", "type": "double"},

{"doc": "CHANNEL SCALE FACTOR", "name": "CHSF01", "type": "double"}

],

"name": "IASIL1CB",

"type": "record"

},

"type": "array"

}

},

{"doc": "IASI LEVEL 1C SCALED RADIANCE SEQUENCE DATA",

"name": "IASICHN",

"type": {

"items": {

"doc": "IASI LEVEL 1C SCALED RADIANCE SEQUENCE",

"fields": [

{"doc": "CHANNEL NUMBER", "name": "CHNM01", "type": "double"},

{"doc": "SCALED IASI RADIANCE", "name": "SCRA","type": "double"}

],

"name": "IASICHN",

"type": "record"

},

"type": "array"

}

},

...

Example: sketch of NC021241 (IASI) message as Apache Avro

Avro is a data serialization system which provides both rich, dynamic data structure specification and a compact binary file format.

Schemas are embedded within an Avro file.

Primary downside to Avro is that it is a record-oriented format (like BUFR) - when processing large amounts of data, each record must be processed sequentially in its entirety (potentially by a pool of workers), even if only a subset of the data is needed.

14 of 73

De-BUFR’ing

SAID: string

-- field metadata --

NUMBER: '001007'

DESCRIPTION: 'SATELLITE IDENTIFIER'

SCAL: '0'

REFERENCE: '0'

BIT: '10'

UNITS: 'CODE TABLE'

GCLONG: string

...

IASIL1CB: fixed_size_list<element: struct<STCH: double, ENCH: double, CHSF01: double>>[10]

child 0, element: struct<STCH: double, ENCH: double, CHSF01: double>

child 0, STCH: double

child 1, ENCH: double

child 2, CHSF01: double

-- field metadata --

NUMBER: '340002'

DESCRIPTION: 'IASI LEVEL 1C BAND DESCRIPTION SEQUENCE'

...

IASIL1CS: fixed_size_list<element: struct<YAPCG: double, ZAPCG: double, FCPH: double, AVHRCHN: fixed_size_list<element: struct<CHNM02: double, CHSF02: double, SMRA: double, CHSF03: double, SSDR: double>>[6]>>[7]

child 0, element: struct<YAPCG: double, ZAPCG: double, FCPH: double, AVHRCHN: fixed_size_list<element: struct<CHNM02: double, CHSF02: double, SMRA: double, CHSF03: double, SSDR: double>>[6]>

child 0, YAPCG: double

child 1, ZAPCG: double

child 2, FCPH: double

child 3, AVHRCHN: fixed_size_list<element: struct<CHNM02: double, CHSF02: double, SMRA: double, CHSF03: double, SSDR: double>>[6]

child 0, element: struct<CHNM02: double, CHSF02: double, SMRA: double, CHSF03: double, SSDR: double>

child 0, CHNM02: double

child 1, CHSF02: double

child 2, SMRA: double

child 3, CHSF03: double

child 4, SSDR: double

-- field metadata --

NUMBER: '361139'

DESCRIPTION: 'IASI LEVEL 1C MEAN AND STANDARD DEVIATION RADIANCE SE' + 1

-- field metadata --

NUMBER: '340004'

DESCRIPTION: 'IASI LEVEL 1C AVHRR SINGLE SCENE SEQUENCE'

-- schema metadata --

NUMBER: 'A61207'

DESCRIPTION: 'MTYP 021-241 IASI 1C RADIANCES (VARIABLE CHNS) (METOP)'

Example: sketch of NC021241 (IASI) message as Apache Parquet

Parquet is a “column-oriented data file format designed for efficient data storage and retrieval,” based on the infamous Dremel paper from Google.

Incorporates a “semi-structured” encoding scheme which allows for highly-efficient data reads - can trivially read from a single column (arbitrarily nested).

Very similar to internal data structures used to back peta-scale analytics tools like Google Cloud BigQuery.

15 of 73

NNJA-AI Data Pipeline - High-Level Overview

NNJA Archive

BUFR File

Phase 1) De-BUFR’ing

Phase 2) AI-Ready Archive

Phase 3) ML Dataset Gen

debufr

Avro Shards

Consolidation

Hive-Partitioned Parquet Shards

“AI-Ready” Archive

ML Prep�(Homogenization)

ML Train/�Inference�Dataset

Per �application

16 of 73

Cloud-Native Structure and Design

Column-oriented, flat data format

“Flattened” BUFR messages written to Parquet

Hierarchical / Hive-partitioned layout

Logically structures archive and provides explicit indices

┌───────────────┬─────────────┐

│ column_name │ column_type │

├───────────────┼─────────────┤

│ CLAT │ DOUBLE │

│ CLON │ DOUBLE │

│ SAID │ INTEGER │

│ SIID │ INTEGER │

│ FOVN │ DOUBLE │

│ LSQL │ INTEGER │

│ SAT_ZA │ DOUBLE │

│ SOL_ZA │ DOUBLE │

│ HOLS │ DOUBLE │

│ HMSL │ DOUBLE │

│ SOL_AZ │ DOUBLE │

│ BEAR_AZ │ DOUBLE │

│ OBS_TIMESTAMP │ TIMESTAMP │

│ CHNM_00001 │ UTINYINT │

│ TMBR_00001 │ DOUBLE │

│ CSTC_00001 │ DOUBLE │

| - - - - - | - - - │

│ CHNM_00015 │ UTINYINT │

│ TMBR_00015 │ DOUBLE │

│ CSTC_00015 │ DOUBLE │

├───────────────┴─────────────┤

“List of Structs” unpacked to separate columns

gs://nnja-ai/

├── README.md

├── LICENSE

├── ...

└── v1/

├── .pmetadata[.json]

├── SOURCE=amsua/

│ └── DATASET=1bamua/

│ └── MSG_TYPE=NC021023/

│ ├── .pmetadata[.json]

│ ├── OBS_DATE=2021-01-01/

│ │ └── data.pq

│ ├── ...

│ └── OBS_DATE=2024-12-31/

└── SOURCE=geo/

├── DATASET=gsrcsr/

│ └── ...

└── DATASET=gsrasr/

└── ...

Partition names embedded in URIs

Single data file per partition

So what exactly are we doing to the NNJA dataset?

First, we’re pulling it entirely out of its original BUFR format. For those of you who love record-oriented data, fear not - we have a intermediate product which preserves the BUFR schemas but re-writes them in a modern wire format, Avro, and that data is available on request. For the majority of users, though, we’re publishing a re-formatted version where we have “flattened” the nested structures in BUFR into a column-oriented format which we archive to Parquet. We found that although Parquet supported complex structured data, the majority of analytics tools that users would likely want to use, from dask to pandas and polars to BigQuery, Spark, Beam and beyond all were most performant when data was optimized for single-column reads.

Second, we’re restructuring the format of the archive using a hierarchical, hive-partitioned format. This provides implicit indexing mechanisms that exposes useful information for analytics tools, including things like temporal or possible spatial chunking. This makes it trivial to interact with an entire archive of data rather than assemblages of user-filtered filenames.

—

Key Points

Column-oriented, flattened data format

Greatly accelerates user querying / filtering through column scanning
Works seamlessly and performantly with a wide cross-section of common data analytics tools (polars, pandas, dask, Spark, BigQuery, and more)

Hierarchical / Hive-partitioned layout

Integrates natively with many high-performance analytics tools to create structured indices on top of the data, to accelerate a wide variety of queries and retrievals

Key Points:

Data extracted from BUFR, re-formatted to a “flat” structure, and written to a hierarchical, Hive-partitioned collection of Parquet files

Works well with a wide cross-section of analytics tools; allows us to optimize performance for a variety of query patterns and use cases
Column-oriented data allows for easy retrieval and subsetting of data

Save intermediate files which retain the original BUFR message/subset structure, but which may be cumbersome for many users to work with
Easily extensible as data is added to NNJA or historical data is re-processed and improved over time
…

17 of 73

Why…

… not write the data as Zarr?

For most of the datasets in NNJA, “time” is the only real index; geospatial information is a type of metadata and is non-gridded.
Data is fundamentally a “record” format - although we could preserve the array-like structure of data for each channel.
Access patterns are expected to primarily involve subsetting on time and channel dimension
Zarr doesn’t provide us any benefits over standard tabular data formats

… flatten all the columns?

We initially wanted to preserve the nested structure – tools like BigQuery can exploit it very easily
We found that most off-the-shelf analytics tools could not optimize for subsetting arrays (we had to retrieve it entirely) or nested columns (retrieve entire struct)
This design is still subject to review based on early user feedback

18 of 73

Why…

… tabular instead of raster data?

Adapted from Table 1, Alexe et al (2024)

373+ channels as a starting point

Figure 2, Keisler (2022)

Flexibility beyond “simple” vision-based encoders

… flexible - defer decisions which might impact model design / application to end users.�

… support myriad applications beyond just MLWP or AI-DA development.��… already a standard / convention for NWP / data assimilation applications.

19 of 73

SDK Example - Proposed API

import nnja

catalog = nnja.DataCatalog(version=nnja.V1)

amsu_ds = catalog.search('amsua/1bamua')

tmbr_fld = nnja.field(

'BRITCSTC.TMBR', channels='all')

amsu_df: pl.DataFrame = amsu_ds.sel(

time=slice('2021-01-13', '2021-01-14'),

bbox=(-120, -70, 20, 60),

fields=[

'OBS_TIMESTAMP', 'MSG_TIMESTAMP',

'LAT', 'LON',

'SAZA', 'SOZA', 'SOLAZI', 'BEARAZ',

tmbr_fld,

],

SAID=('NOAA-18', 'NOAA-19')

).load_dataset(engine='polars')

Fetch a published catalog

Search / retrieve a particular dataset

Helper to retrieve a subset of channels for a given field

Subset and retrieve data

Slice by observation timestamp
Slice by bounding box (lat/lon)
Select fields�Timestamps for obs + publishing�Geospatial�Satellite viewing geometry�Data field (helper from earlier)�
Slice by an actual data field (implicit)
Load via Polars

To go along with this structure, we’re building a Python-based SDK that will allow users to easily subset and manipulate data from the NNJA-AI archive without ever needing to manually deal with Parquet files themselves.

In this example, we read a pre-packaged DataCatalog from the public cloud storage bucket where NNJA-AI is published. With that catalog, we can search for a dataset or sensor of interest and fuzzily match against the contents or tags in the catalog.

Once we have a dataset of interest, we can read it and load its specific manifest. The manifest provides a great deal of interest, including breakdowns of all the key metadata or data columns in the dataset. It will also provide metadata on those columns themselves, as well as expose interfaces which allow the users to sub-select specific channel regions or data field subsets without needing to understand in detail how those data are laid out in the archive files.

Finally, using simple syntax familiar to any pandas or xarray user, a developer can quickly slice the dataset and load it using an analytics engine like pandas.

—

Key Points

Column-oriented, flattened data format

Greatly accelerates user querying / filtering through column scanning
Works seamlessly and performantly with a wide cross-section of common data analytics tools (polars, pandas, dask, Spark, BigQuery, and more)

Hierarchical / Hive-partitioned layout

Integrates natively with many high-performance analytics tools to create structured indices on top of the data, to accelerate a wide variety of queries and retrievals

Key Points:

Data extracted from BUFR, re-formatted to a “flat” structure, and written to a hierarchical, Hive-partitioned collection of Parquet files

Works well with a wide cross-section of analytics tools; allows us to optimize performance for a variety of query patterns and use cases
Column-oriented data allows for easy retrieval and subsetting of data

Save intermediate files which retain the original BUFR message/subset structure, but which may be cumbersome for many users to work with
Easily extensible as data is added to NNJA or historical data is re-processed and improved over time
…

20 of 73

Example: Subset of data available in NNJA-AI

January 13, 2021 - 00Z - 03Z

21 of 73

NNJA-AI v1 - Preview Release

Complete NNJA-AI v1 release planned for end of Q1, 2025 on AWS and GCS

	Sensor	Platforms	Years
Microwave �(LEO)	AMSU-A	NOAA 15/18/19, Metop B/C	2021-2024
	ATMS	NOAA 20, Suomi-NPP	2021-2024
	MHS	NOAA 18/19, Metop B/C	2021-2024
Infrared (LEO)	CrIS	NOAA 20, Suomi-NPP	2021-2024
Infrared (LEO)	IASI	Metop B/C	2021-2024
Infrared (GEO)	SEVIRI (all-sky)	Meteosat 10/11	2022*-2024
	ABI (all-sky)	Himawari, GOES-*	2021-2024
	ABI (clear-sky)	GOES-*	2021-2024
In Situ	Upper Air	SYNOP Weather Balloons	2021-2024
In Situ	Surface	SYNOP / METAR	2021-2024

2021-2024

Preview release available today on Google Cloud Storage (gs://nnja-ai; DOI 10.5281/zenodo.14633509) under CC-BY 4.0 license�
Preview Python-based SDK available on GitHub at �github.com/brightbandtech/nnja-ai�
We want your feedback!��Our goal is to create a highly useful resource for the broader AI/weather communities. We can – and will – improve archive structure and tooling based on your wants/needs.

We’re proud to announce that right now, you can play with the data and tools I’ve just shown you. We’ve launched what we’re calling a “preview release” of NNJA-AI.

This includes two things. First, we’re launching the cloud-optimized file archive I described on Google Cloud Storage. In this archive we’ve highlighted a subset of polar-orbiting, LEO satellites. I do want to emphasize that we are providing _all_ the data for these satellites - it’s not just one or two channels, it’s everything available in the source archive, which includes hundreds of channels for instruments like CrIS and IASI. We’ll shortly be adding some geostationary and upper air data to this sample - check back in the next few weeks to see that.

Second, the initial version of our Python SDK is available now under a permissive open source license on GitHub. We’re really interested in hearing feedback on our API and design - there are a lot of features we already have planned, but we want to know about what features you would like to see. Please, download it and give it a spin.

The original NNJA dataset is released under a permissive creative commons license, and we’ve opted to carry this forward to ensure that our own data release is just as accessible and useful for everyone.

22 of 73

AI Data Assimilation

Directly from Observations

23 of 73

The Future of AI Weather is Observations

24 of 73

The Future of AI Weather is Observations

Observations

Current State

Forecast

Traditional Numerical Weather Prediction

Physics

Data Assimilation

Forecast

25 of 73

The Future of AI Weather is Observations

Observations

Current State

Forecast

Observations

Current State

Forecast

Traditional Numerical Weather Prediction

Partial AI �Weather Prediction

Physics

Recent breakthroughs �(GraphCast, etc.)

Opportunity

Physics

AI Model

Data Assimilation

Forecast

26 of 73

The Future of AI Weather is Observations

Observations

Current State

Forecast

Observations

Current State

Forecast

Traditional Numerical Weather Prediction

Partial AI �Weather Prediction

Physics

Recent breakthroughs �(GraphCast, etc.)

Opportunity

Physics

AI Model

Performance: more data, better forecasts

Data Assimilation

Forecast

27 of 73

The Future of AI Weather is Observations

Observations

Current State

Forecast

Observations

Current State

Forecast

Traditional Numerical Weather Prediction

Partial AI �Weather Prediction

Physics

Recent breakthroughs �(GraphCast, etc.)

Opportunity

Physics

AI Model

Performance: more data, better forecasts�
Structural: more organizations can create forecasts

Data Assimilation

Forecast

28 of 73

The Future of AI Weather is Observations

Observations

Current State

Forecast

Observations

Current State

Forecast

Observations

Forecast

Traditional Numerical Weather Prediction

Partial AI �Weather Prediction

Full AI �Weather Prediction

Physics

Opportunity

Recent breakthroughs �(GraphCast, etc.)

Physics

AI Model

Data Assimilation

Forecast

29 of 73

The Future of AI Weather is Observations

Observations

Current State

Forecast

Traditional Numerical Weather Prediction

Partial AI �Weather Prediction

Full AI �Weather Prediction

Observations

Current State

Forecast

Current State

Forecast

Physics

AI Model

AIDA

Recent breakthroughs �(GraphCast, etc.)

Opportunity

Physics

AI Model

Data Assimilation

Forecast

30 of 73

“AIDA”

AI Data Assimilation

31 of 73

AIDA Overview

AI System for Global Data Assimilation�
Directly estimates atmospheric state from Level 1 observations

No background state�
Can be used to feed AI Forecast models

Running operationally�

32 of 73

Guided Diffusion

33 of 73

Guided Diffusion

We use a Guided Diffusion approach, building on previous work, including:

Score-based Data Assimilation, Rozet and Louppe 2023�
DiffDA: a Diffusion model for weather-scale �Data Assimilation, Huang et al 2024�
Deep Generative Data Assimilation in Multimodal Setting, �Qu et al 2024�
Generative Data Assimilation of Sparse Weather Station Observations at Kilometer Scales, Manshausen et al 2024

34 of 73

Guided Diffusion

35 of 73

Guided Diffusion

0h

3h

6h

9h

0h

3h

6h

9h

Diffusion Model

Generate a realistic trajectory of the atmosphere,

36 of 73

Guided Diffusion

0h

3h

6h

9h

0h

3h

6h

9h

Diffusion Model

True�Observations

Generate a realistic trajectory of the atmosphere, �that is also consistent with observations.

37 of 73

Guided Diffusion

0h

3h

6h

9h

0h

3h

6h

9h

Diffusion Model

Observation Operator

Predicted Observations

True�Observations

Generate a realistic trajectory of the atmosphere, �that is also consistent with observations.

38 of 73

Guided Diffusion

0h

3h

6h

9h

Diffusion Model

Observation Operator

0h

3h

6h

9h

ML

39 of 73

Guided Diffusion

0h

3h

6h

9h

Diffusion Model

Observation Operator

1B parameter U-Net

1-degree resolution

Trained on ERA5

0h

3h

6h

9h

ML

40 of 73

Unguided Diffusion

41 of 73

Unguided Diffusion

42 of 73

Guided Diffusion

0h

3h

6h

9h

Diffusion Model

Observation Operator

1B parameter U-Net

1-degree resolution

Trained on ERA5

Linear interpolation + MLP

Trained on ERA5 + 2021-2023 Observations

0h

3h

6h

9h

ML

43 of 73

Observation Operators are Modular

AMSU-A

ATMS

Your Sensor Here

Each observation operator is trained independently.

54 of 73

AIDA in Action: January 9, 2024

AIDA is Operational: February 14, 2025

61 of 73

AIDA is Operational: February 14, 2025

Category 5

Cyclone Zelia

62 of 73

AIDA is Operational: March 20, 2025

63 of 73

AIDA + Forecast

64 of 73

AIDA + Forecast

Observations

Current State

Forecast

AI Model

AIDA

AI Model

Data Assimilation

Forecast

65 of 73

AIDA + Forecast

Observations

Current State

Forecast

AI Model

AIDA

AI Model

Data Assimilation

Forecast

5 days

ΔT (K)

ERA5

GFS

AIDA + GraphCast

72 of 73

7 days

ΔT (K)

ERA5

GFS

AIDA + GraphCast

73 of 73

Conclusion

AIDA: a new AI system for Global Data Assimilation

Directly estimates atmospheric state from Level 1 observations

Can be used to feed AI Forecast models�
Running operationally

We’d love to talk!

hello@brightband.com