1 of 15

ecPoint-Calibrate

Anirudha Bose

Software Engineer

Ledger, Paris

Fatima Pillosu

Scientist�ECMWF, Reading

Timothy Hewson

Principal Scientist�ECMWF, Reading

Interactive weather-calibration, post-processing, and conditional-verification software in Python

​

Reading, UK

September 20, 2019

European Centre for Medium-Range

Weather Forecasts

2 of 15

An overview of ecPoint-Calibrate

Calibration

  • Determine relationships between old forecasts and verifying observations.

​

​

​

​

Statistical postprocessing

  • adjustment of meteorological forecasts, based on a prior calibration activity.
  • delivering a better match to observations than is given by the unadjusted forecast.

​

​

​

​

Conditional Verification

  • analogous to calibration
  • identify how relationships between predictors and observations are affected by supplementary variable(s).

3 of 15

Prelude - pain points in v1

  • Data loader for Geopoint files was slow and unreliable.�
  • Non-native mathematical operations, using Python.
    • Very slow

​

  • Computation of predictors took days to finish for even 1 month data.

​

  • Written in Python 2
    • Incompatible with new Pandas features, required by us.
    • Blocker for using Metview-Python bindings.

​

  • Software couldn’t handle large amounts of input data.

​

  • Rendering of Decision Tree was sluggish.

4 of 15

Performance optimizations in v2

  • Complete rewrite of the software stack to interface with Metview.�
  • Ported the software core to Python 3.�
  • Replaced ecCodes with loaders from Metview-Python framework.�
  • Efficient low-level data-structures.�
  • Non-blocking streaming of logs to the GUI.�
  • Lazy evaluation of Decision Tree.

5 of 15

Splash screen

6 of 15

Input Parameters for Calibration

7 of 15

Input Parameters for Calibration

8 of 15

Predictor Computations

9 of 15

Creation of Point Data Table

10 of 15

Live Demo!

11 of 15

Software Architecture

12 of 15

13 of 15

What I learned

  • Insights into calibration and analysis of weather forecasts in the real-world.�
  • Python is a highly expressive, mature, and performant software for scientific computing.
    • Don’t use Python 2.
    • Don’t use MATLAB.
    • ...it’s 2019.�
  • Building fluid User Interfaces is hard.
    • But Electron is your best bet!

​

  • What the mind can conceive, it can achieve.
    • If you use React!

​

  • Thinking from First Principles.
    • Even if it means you need to trash 75% of your codebase!

14 of 15

Collaborators

15 of 15

Thank you!

github.com/onyb