1 of 15

Software at UW-SSEC

ANL-CELS / UW-SSEC joint workshop

Ray Garcia <rkgarcia@wisc.edu> 2020-03-160

2 of 15

Introduction

A sample section of other software efforts at SSEC

- My background: comp sci & instrumentation; ECE ’96

​

- Satellite science and algorithm development lifecycle

- Aircraft and ground-based science and engineering

- Infrastructure and bridge-work and technical shepherding

​

This is just a tiny sampler pack of SSEC projects and efforts;

some intersect my desk, many do not.

3 of 15

SSEC computing facilities and support

In support of the scientists and the “programmer pool”

  • Technical Computing
    • Network, workstation, server management
    • Research clusters (2), Lustre SANs, virtual machine provisioning
    • Software tool support
  • Data Center
    • Petabytes of historical satellite meteorology archive
    • Data acquisition, distribution, processing

4 of 15

Software-focused groups

Bringing the science to community and customers

  • McIDAS / McIDAS-V: Interactive meteorological data analysis software
  • RealEarth: satellite product web mapping site
  • Atmosphere SIPS: integration & production of NASA investigator algorithms
  • Satellite Data Ingestor: turnkey antenna data production appliances
  • Rooftop Instrumentation: met station and cameras site and archive
  • AWIPS dev: collaboration with NOAA/NWS on AWIPS enhancements

5 of 15

Community Satellite Processing Package

CSPP-GEO, CSPP-LEO

  • Integrate turn-key science software packages for satellite meteorology
  • NOAA-funded effort
  • GOES-R ABI, Himawari AHI, Suomi NPP / NOAA-20 CrIS/ATMS/VIIRS, more
  • Follows from IMAPP: MODIS-AIRS processing package
  • https://cimss.ssec.wisc.edu/csppgeo/
  • https://cimss.ssec.wisc.edu/cspp/
  • Knowledge gained: portable software bundling and automation, e.g. ShellB3

6 of 15

CSPP-GEO AXI-Tools: ABI / AHI AWIPS bridge for NWS

7 of 15

CSPP GeoSphere

Cloud Pilot Project

  • Integrate CSPP-GEO software in a cloud setting as a low-latency demonstration
  • OpenLayers, CSPP Geo2Grid, CSPP GOES Rebroadcast reception, AMQP, MapServer
  • In-house development Kubernetes; GCP demo upcoming

DHoese, CSuplinski, NBearson, TJasmin, GMartin, KStrabala

8 of 15

GOES-R AWG-AIT

Algorithm integration, delivery and verification

  • Delivery support of CIMSS algorithms to GOES-R project office / operations
  • Collaboration with NOAA STAR on software integration
  • Tools, infrastructure, research and data support, e.g. libHimawari
  • Long-term goal of reducing costs of Research-to-Operations
  • Upcoming: further cloud-compatible containerization and orchestration of science production systems

9 of 15

CrIS L1B + ULL

CrIS radiance processing and ultra-low-latency demo

  • Collaboration with NASA Atmosphere SIPS at UW-SSEC to demonstrate processing of direct broadcast CrIS radiance data in <30s latency
  • ULL inspired in part by 2011-2014 Hurricane Sentinel 3 experiment with S-HIS: <60s atmospheric state retrieval latency to NASA Mission Tools Suite, <10s for preliminary radiance products
  • Modified NASA CrIS L1B climate product software (MATLAB-python) authored by SSEC-UMBC team
  • Anticipated: Integration of retrieval software (E. Weisz, W. Smith) descended from S-HIS retrieval system

10 of 15

Scanning HIS Upgrade Path Study

Toward another 20 years of aircraft interferometer data

  • Evaluate replacement hardware and software upgrade options for Scanning HIS, including imaging interferometer enhancements
  • Focus on incremental options with spaceflight-compatible components where reasonable
  • Electronics replacement (move toward FPGA-hybrid computing) and options for onboard processing and software interferometry
  • Anticipate testing needs of potential future orbital sounders

Scanning HIS Dashboard Displays in Python/Tk, 2003

11 of 15

Scanning HIS Upgrade Path Study

Toward another 20 years of aircraft interferometer data

  • Evaluate replacement hardware and software upgrade options for Scanning HIS, including imaging interferometer enhancements
  • Focus on incremental options with spaceflight-compatible components where reasonable
  • Electronics replacement (move toward FPGA-hybrid computing) and options for onboard processing and software interferometry
  • Anticipate testing needs of potential future orbital sounders

poster excerpt, JTaylor

12 of 15

Recent AERI and S-HIS software work

(one foot still firmly planted in the ‘90s?)

  • Containerized radiance processing (complete for S-HIS, in progress for AERI)
  • FIREX radiance and retrieval improvements for Scanning HIS
  • AERIOE retrieval containerization for Windows deployment
  • AERI Windows 10 transition

S-HIS FIREX Atmospheric State Retrieval Derived Values (EWeisz, JTaylor, WSmith)

13 of 15

Cross-cut software projects

when projects collide

  • amqpfind / amqpsend: easy-to-integrate AMQP notifications with JSON payloads for pipeline prototyping across SAN and network:
  • ShellB3: bespoke, configuration-controlled, trivially-relocatable scientific python stacks for “big binary bundle” software including CSPP, HSRL, CrIS
  • libHimawari: AHI HSD and Himawari HCast reader for Fortran, C, Python
  • uwglance: Dataset comparison utility for verification & validation during algorithm development and delivery lifecycle

14 of 15

Trends of the last 3 decades

the inexorable acquisition of layers of indirection

  • 1990s: AERI
    • mainframe ➡ networked PCs
    • CLI (DOS) ➡ GUI (OS/2)
    • 16 bits ➡ 32 bits
  • 2000s: S-HIS
    • OS/2 & AIX ➡ Linux
    • rise of “Beowulf clusters”
    • Python for automation/web
    • 32 bits ➡ 64 bits
  • 2010s: CrIS, ABI
    • Web & mobile
    • bundles and VMs
    • Python ascendant
  • 2020s: S-HIS Mk2? GEO-XO?
    • orchestrated containers
    • hybrid-architecture computing?
    • python GIL finally gone?

15 of 15

</slides>

Thank you!