Drafting User Guides for

working with NASA EDC data

Diagrams, Cheat Sheets, and Terminology

Cheetsheets are developed as single-slide decks so they can be published in multiple places (e.g. Cookbook and PO.DAAC)

To publish a Google Slide (Doc, Sheet, etc): File > Publish to the Web > Publish!

Link > copied here for easy-access Auto-publishes every 5 minutes

Embed > iframe code copied into Cookbook or elsewhere

​

This is the crumb trail!!

Notes:

• 7/6: Catalina: after review of slides 8 and 9, we can have the links show slides 4-10 as pngs in our Cookbook
• 5/19: work in progress
• 5/19: mentor co-working discussion: range of learning preferences/users range where in the learning process they are; daacs are at varying points in cloud migration/services. → Could offer: 1) High level diagram as a Getting Started Roadmap, 2) Cheatsheet for those who already took some training and are trying things on their own. Keep and leverage existing Cloud Paradigm and Cloud pathways diagrams
• Please provide edits, comments, input on any of these diagrams, cheat sheets
• Web and tutorial icons have links to existing resources - ID which tutorials need updating (to generalize)? Are there other tutorials out there that should be linked in the cheat sheet?
• Feel free to duplicate a diagram and make it your own/provide input e.g. Amy’s version
• Which elements should and should not be in the cross-daac cheat sheet? E.g. harmony on the common cheat sheet?
• How do we phrase/reference certain steps or tools?

Last updated: Apr 15, 2024

Main and DAAC specific

Making it your own while staying consistent across DAACs

The following set of guides contain “everything”. To make a version more suitable for your DAAC, make a copy of the slide in this google slides doc and edit at will.

One suggestion would be to add your DAAC’s logo in one of the corners to signify the guide version.

1. Cloud Paradigm diagram
1. conceptual explanation for NASA Earthdata cloud paradigm
2. Includes general concepts only, no specific services or their functionality
2. Cloud Ecosystem Access Pathways diagram
• conceptual explanation for how to work with cloud-archived data
• Includes specific ESDIS tools and services options and tech setup options (ec2, jupyterlab, local machine)
3. High level diagram - as Getting Started roadmap - - needs development
• conceptual roadmap for user getting started with cloud-archived data
• Outlines the steps user would take to get set up using EDC data (e.g. search, set up jupyter hub (optional), set up github (optional)
4. Tools & Services Roadmap
• Practical guide for learning and selecting the right tool or service for a given use case
5. Working with Earthdata Cloud Tools Cheat Sheet - - under development
• Practical reference guide as user begins taking the conceptual pieces to explore and implement in their own workflows
• Guide to selecting from available tools to enable and implement the Access Pathway(s)
• Links concepts and tool resources (tutorials)

What is the NASA

Earthdata Cloud (paradigm)?

Access pathways for cloud-archived data

Tools & Services Roadmap

Search & Access

Cheat Sheet

1

2

4

5

For internal organization/brainstorming

Getting started with cloud-archived data

3

Working with NASA Earthdata Cloud data :: What is the NASA Earthdata Cloud?

1.

To edit this slide, go HERE

The published link is THIS

(links updated Apr 15 2024)

Working with NASA Earthdata Cloud data :: Access Pathways

2.

To edit this slide, go HERE

The published link is THIS

(links updated Apr 15 2024)

Working with NASA Earthdata Cloud data :: Getting Started Roadmap

​

CLOUD Workflow

3.

To edit this slide, go HERE

The published link is THIS

(links updated Apr 15 2024)

Working with NASA Earthdata Cloud data :: Getting Started Roadmap

Last update: 19 May 2022

LOCAL workflow

4.

To edit this slide, go HERE

The published link is THIS

(links updated Apr 15 2024)

Working with NASA Earthdata Cloud data :: Tools & Services Roadmap

Last update: 8 July 2022

5.

To edit this slide, go HERE

The published link is THIS

(links updated Apr 15 2024)

Working with NASA Earthdata Cloud data :: Cloud Terminology 101

6.

To edit this slide, go HERE

The published link is THIS

(links updated Apr 15 2024)

Working with NASA Earthdata Cloud data :: Workflow Cheatsheet

Last update: 6 Jul 2022

7.

To edit this slide, go HERE

The published link is THIS

(links updated Apr 15 2024)

Working with NASA Earthdata Cloud data :: Cheatsheet Terminology

Last released update: 18 May 2022

8.

To edit this slide, go HERE

The published link is THIS

(links updated Apr 15 2024)

Other/ older/ existing versions

Choose your own cloud adventure diagram - version 0

This could be updated to be more general and not PODAAC specific

Choose your own cloud adventure diagram - version 1A

local

cloud

Legend - environment

Terminology Cheat Sheet

Choose your own cloud adventure diagram

- version 1B

local

cloud

Legend - environment

• Set up EDL authentication ( programmatic)

→ needed for data download and cloud access

Terminology Cheat Sheet

with map-based search for spatiotemporal queries

→ provides data download and cloud access links based on query, and access to transformation tools like OPenDAP

Working with NASA Earthdata Cloud data :: Cheat Sheet for Tools & Services

Last released update: 18 May 2022

• Data in the Cloud - Offers the opportunity to work with the data without the need for access to large storage or computing environments.
• Traditional mechanism such as download and analyze will continue to be enabled and supported, even with the data being archived in the Earthdata Cloud. Data download will continue to be free to the user.

To Cloud or not to Cloud

• Cloud workflows - bring your code and analysis to the date; no data download required
• Consider in-cloud workflow if:
• working with large data volumes
• wishing to explore new ways to do scientific research and applications, such as machine learning
• do not have access to large storage spaces or powerful computing e.g. through a home institution or organization, but do have access to internet and relatively small funding to set up a virtual machine in the AWS cloud for analysing and working with Earthdata and other data (AWS cloud is a pay-as-you-go cost model, only pay for what you need and use. Note: access to NASA Earthdata data and services that are cloud-based are free to the user. The cloud computing and analysis with the data is user responsibility.)
• Local workflows - download data from the Earthdata Cloud and work locally (e.g. laptop, cluster)
• Consider a local workflow if:
• working with relatively small data volumes
• have access to sufficient data storage and computing resources

Terminology Cheat Sheet

NASA Earthdata Terminology

• Collection
• Collection concept ID
• Collection Shortname
• Granule
• Granule ID

NASA Earthdata Tools & Services

• CMR
• CMR API
• CMR STAC API
• data-subscriber
• data-downloader
• Earthdata Search
• Earthdata login (EDL)
• earthdata library
• Harmony API
• harmonyPy library
• L2ss-py
• netCDF2zarr
• OPeNDAP
• OPeNDAP in the Cloud
• zarr-eosdis-store library

Cloud & Open-source Science Terminology

• Cloud optimized format (COF)
• Zarr
• Kerchunk
• xarray

Open Software Science Tools i.e. Interfaces & Workflow facilitators

• JupyterHub
• JupyterLab
• Jupyter Notebook
• Git
• GitHub
• BinderHub

​

GUI or API?

• GUI (graphic user interface) - enables quick, easy search, discovery and exploration of datasets, as well as data access | e.g. NASA Earthdata Search Portal
• API (application programming interface) - enables programmatic (scripted) workflows for reproducible open science workflows, easier co-development and collaboration | e.g. CMR API, Harmony API

NASA Earthdata Terminology

Collection

Collection concept ID

Collection Shortname

Granule

Granule ID

​

​

Place to capture the definitions

NASA Earthdata Tools & Services

CMR - NASA's Common Metadata Repository (CMR) is a high-performance, high-quality, continuously evolving metadata system that catalogs all data and service metadata records for NASA's Earth Observing System Data and Information System (EOSDIS) system and will be the authoritative management system for all EOSDIS metadata. These metadata records are registered, modified, discovered, and accessed through programmatic interfaces leveraging standard protocols and APIs.

CMR API - programmatic access to using the NASA CMR

CMR STAC API

data-subscriber

data-downloader

​

​

NASA Earthdata Tools & Services

earthdata library - a Python package to search, preview and access NASA datasets (on-prem or in the cloud) with a few lines of code. Github | Overview

OPeNDAP - a standard interface for transferring, transforming and subsetting Earth science data. It’s capable of subsetting by time, space and variable, and is a great tool for those looking to reduce the amount of gridded data required for research or analysis to a specific region, time, or set of variables.

“OPeNDAP in the cloud” - a hosted, Earthdata enterprise service that provides a single interface to all Earthdata Cloud data enabled for OPeNDAP. Guide

• Prerequisites - to use OPeNDAP-in-the-cloud, one needs to know the collection for which you want to subset data. The PO.DAAC Cloud Data Listing is a great place to find collections of interest.
• How-To: Virtual Browse User Interface Access, Earthdata Search Access, API Access. See the Guide for more details.

NASA Earthdata Tools & Services

NASA Earthdata Terminology

Collection Concept ID - a unique identifier for a collection, often used with CMR search

OTHER BRAINSTORMING

Cookbook thinking - leverage cooking idea

Menus (Tutorials)

Test Kitchen (in dev)

And Menus chapter will have content from nasa-openscapes.io page. – to be improved with more description, maybe with the schedule pages. But maybe not copies of tutorials? Or maybe just some - “Taster Menu?” Thinking for clarity and long-term maintenance (but we can reprioritize as we go too!)

Cookbook

Cookbook

- this page would just be a preamble to the cookbook with link (or could be a direct link to the cookbook without a page?)

Hi! Proposed changes; any feedback welcome - read from left to right: