1 of 21

Python Embedding with METplus

Dan Adriaansen on behalf of the METplus Team

NCAR/RAL�NOAA/GSL

and�Developmental Testbed Center

METplus Advanced Training Series: Session 5

15 November 2023

2 of 21

What is Python embedding?

A mechanism for passing data to MET tools, that the MET tools cannot directly read on their own, or novel derived fields not available in the input.

What types of data can MET receive via Python embedding?

Two-dimensional arrays of gridded data are supported�
Point observations are supported in a nested Python list following MET’s 11-column format for observation data�
Matched-pair data are supported conforming to MET’s “Matched Pair” (MPR) line-type definition

3 of 21

The power of Python embedding

Opens the possibility to support endless file formats with MET, provided a user has a way to read the data with Python

Allows a user to perform data manipulation prior to calling MET tools

Data cleaning
Data formatting
Derivations/calculations

Reduces/eliminates intermediate data files on disk�
Connects METplus to the Python ecosystem including popular packages like Xarray and Pandas

4 of 21

Python Embedding with MET Tools (v11.1.0+)

5 of 21

Some Python embedding requirements

It starts with compiling the MET software!

Python Installation (v3.10.4+)

Use

--enable-python compilation flag

This enables access to any Python modules installed in the Python Installation linked against during MET software compilation!

Must include:

NumPy
Pandas
Xarray
netcdf4

6 of 21

Python embedding for gridded data

Use keyword [ PYTHON_NUMPY | PYTHON_XARRAY ] in place of gridded data file on the command line

Tells MET tool to obtain data from Python, rather than a data file��

�

Replace field name with full path to your Python script and any command line arguments in the MET config file

Tells MET which Python script to call instead of which field name to get from a data file

7 of 21

Python embedding for gridded data

��

METplus Wrappers Equivalent:

FCST_GRID_STAT_INPUT_TEMPLATE=PYTHON_NUMPY�
FCST_VAR1_NAME=/path/to/fcst/python/script.py python_arg1 python_arg2

8 of 21

Python embedding for gridded data

Checklist for gridded data Python embedding script:

2D Arrays Only
Python variable must be named met_data
Python variable can be NumPy N-D Array (PYTHON_NUMPY), or Xarray DataArray (PYTHON_XARRAY) object type
Provide metadata Python dictionary named attrs
Remove existing attributes from Xarray DataArray

For more information about the required attributes, please see the user documentation here:

https://met.readthedocs.io/en/latest/Users_Guide/appendixF.html#attributes-for-2d-gridded-dataplanes

9 of 21

Python embedding for gridded data

Test your script before proceeding!

Prior to running your workflow, it is recommended to use plot_data_plane to verify your Python script is working as intended.

Python Keyword

Forecast Field Name Entry

10 of 21

Python embedding for point data

Replace input point observation file with the Python keyword PYTHON_NUMPY*, an “=”, and then the full path to your Python script and any command line arguments, enclosed in double quotations.

Tells MET tool to obtain data from Python, rather than a data file
point_stat, plot_point_obs, point2grid, ensemble_stat��

For ASCII2NC, use the “-format python” command line argument, followed by a string containing the full path to your Python script and any command line arguments in double quotations.

*NumPy is not actually used in this context but the keyword is reused from the gridded data instance to signal MET to use Python.

11 of 21

Python embedding for point data

METplus Wrappers Equivalent:

OBS_POINT_STAT_INPUT_TEMPLATE = \�PYTHON_NUMPY = /path/to/script.py arg1 arg2

12 of 21

Python embedding for point data

Checklist for point data Python embedding script:

Python variable must be named point_data
Python variable must be a nested Python list
Must contain each of the 11 columns in the MET 11-column format for point observations
The lists must be nested in the correct order of the 11-column format
Pandas makes ordering and manipulating data easier, but is not required

13 of 21

Python embedding for point data

Test your script before proceeding!

Prior to running your workflow, it is recommended to use plot_point_obs to verify your Python script is working as intended.

Python Keyword + Script & Args

14 of 21

Gridded vs. point data differences

Python keyword + script & args replaces point observation file

Python keyword only replaces gridded file

One script & args per field (2d array)

15 of 21

Gridded vs. point data differences

METplus Wrappers Equivalent:

FCST_GRID_STAT_INPUT_TEMPLATE=PYTHON_NUMPY�
FCST_VAR1_NAME=/path/to/fcst/python/script.py python_arg1 python_arg2

METplus Wrappers Equivalent:

OBS_POINT_STAT_INPUT_TEMPLATE = \�PYTHON_NUMPY = /path/to/script.py arg1 arg2

16 of 21

Python embedding for MPR data

stat_analysis tool only, MPR output only
Use the “-lookin python” command line argument, followed by the full path to your Python script and any command line arguments

METplus Wrappers Equivalent:

MODEL<n>_STAT_ANALYSIS_LOOKIN_DIR = \�python scripts/python/examples/read_ascii_mpr.py \�point_stat_mpr.txt

17 of 21

Getting started with Python embedding

Review Appendix F of the MET User’s Guide:

MET User's Guide Appendix F: Python Embedding�

Explore simple Python embedding example scripts included with MET:

In the MET installation directory, share/met/python/examples/read*.py�

Review Python embedding use cases for METplus Wrappers

METplus Wrappers Python embedding use cases

18 of 21

Special case: custom Python environment

The recommended use of Python embedding is to rely on the Python that was used during compilation to run your scripts. This is most efficient, as it is able to pass data directly to the MET tool in memory.�
We realize this is not always possible; especially for users of shared MET installations that have no control over the Python environment used for compiling MET.�
If a user has custom Python module needs, they can specify the MET_PYTHON_EXE environment variable to point to the python3 executable they wish.

However, this will force the writing and reading of the data (write out from Python, read in by MET). For large datasets or long workflows, this can dramatically increase the run time.

19 of 21

Other Python capabilities within METplus

UserScript

Run a user-supplied Python script as part of a METplus Wrappers workflow
METplus Wrappers only
Script does NOT perform Python embedding
Examples:

File management prior to running MET tools (unzipping, organizing)
Data conversion into MET-compatible formats (e.g. “poly” files for gen-vx-mask)
Performing a diagnostic computation

See more: METplus UserScript Use Cases�

Python Embedding Ingest (PyEmbedIngest)

METplus Wrappers only
Intended to reduce Python embedding script calls by writing MET-compatible data to disk to be available for multiple iterations of a METplus workflow
See more: METplus PyEmbedIngest Use Cases

20 of 21

Summary: Python Embedding with METplus

Python Embedding creates a unique, customizable entry point for users into the MET tools, and is supported in METplus via METplus wrappers and also directly in MET tools.�
We encourage users to leverage this capability for their workflows and datasets!

Please reach out via GitHub discussions if you need any assistance with Python Embedding.

21 of 21

Thank you!

Questions?

E-mail:

dadriaan @ ucar . edu