1 of 8

2021 Cloud Hackathon

Final Project Presentations Project 1

Tom Farrar, Kyla Drushka, Bia Villas Boas., Matthew Archer, Kathleen Dohan, Severine Fournier, Eli Hunter, John Wilkin

​

​

Helpers: Jinbo Wang, Jack McNelis, Ed Armstrong

2 of 8

Project Goals

Project 1

Extract and visualize multiple data sets that can be used to give context to field campaigns or other regional events (e.g., the "Warm Blob" or the recent atmospheric river event on the West Coast). For example, choose a target region and time period, cycle through all available high-resolution sea surface temperature data, identify clear images, catalog them. Extract wind, wave, sea surface height, salinity data.

​

    • Identify a region and time period
    • Ingest/subset several datasets
    • Create specific functions to operate on the datasets, specifically regrid to a common space/time

​

3 of 8

Project Use Case/Workflow outline

Example Project: Find clear scenes in high-resolution SST

Goal: Choose a target region and time period, cycle through all available high-resolution sea surface temperature data, identify clear images, catalog them

PO DAAC Catalog exploration using the CMR AP

L3 VIIRS SST "VIIRS_NPP-OSPO-L3U-v2.61"

['2021-10-01T10:00:00Z','2021-11-01T00:00:00Z']

in S-MODE region

Method:

Plot all SST images available during the S-MODE windows and pick good days

​

Load in the data with xarray.open_mfdataset(). Use list comprehension to modify the list of granule URLs into a list of open s3 files before passing to xr.open_mfdataset().

Make a metric to select times with clear skies in region of interest: Choose the box defining the region of interest and use quality flag or NaN mask to count bad/good pixels

Choose the times with “good” data. Plot to visually check.

​

4 of 8

Methods or services used

  • CMR API, xarray in previous project, harmony, Zarr
  • Note that all tools and methods were tested out and worked with by all members of the group throughout the week, as most of us were unfamiliar with the tools presented
  • Data: VIIRS, SWOT, SMAP, MODIS, SMODE in-situ platforms

5 of 8

Demo of notebook created for project (optional - if desired/if time allows; alternatively, can walk through project goals, workflow and tools used while showing a notebook - just some suggestions)

  • Project repo or notebook:

6 of 8

Finding Level-2 SST imagery

7 of 8

Finding Level-2 SST imagery

8 of 8

Pain points / Lessons Learned / Meeting Hackathon Goals

  • Were there aspects of what you learned this week that you see as being helpful to your work?
    • Yes! Knowing where to start with working in the cloud, knowing what libraries to use for searching, subsetting, loading, and manipulating data.
  • Where did you have the most issues?
    • There were many data access methods presented - an overview of the various approaches and a “cheat sheet” of the advantages and typical use scenarios for each would have put the presentations in context (maybe presenting as Q&A, e.g. “Q: How do I subset the global data to my region? A: You have 3 methods to try, (1).....”
    • Learning how to sift through the CMR to get to specific details to access the desired dataset took more time than expected
  • How do you see NASA EO in the cloud contributing/enabling/improving your research/applications work? E.g. working more easily with big data? More collaboration with colleagues, practicing open science and open data? Machine learning? Etc etc
    • Subsetting, visualizing, downloading pieces of big data
    • Using Jupyter notebooks for collaboration
  • Did you accomplish what you had hoped to when you first started the hackweek?
    • Yes. Details to be ironed out but mostly accomplished
    • We are *so* grateful for all of the effort that the organizers put into this event, for the tutorials, and for all of the help provided throughout. Thank you!!!