1 of 15

NASA Earthdata Cloud &

The Cloud Paradigm

An Introduction

2 of 15

NASA Earthdata is moving to the Cloud

Era of Big Data → NASA EOSDIS cloud evolution
NASA DAAC data is/will be hosted in the Amazon Web Services (cloud), or AWS, in the new Earthdata Cloud
NASA DAACs are starting to migrate their data holdings into this cloud-based environment
In anticipation and support of this, DAACs are gearing up to support the data end-users e.g., this Cloud Workshop!

Big Data volume from upcoming missions are the main reason for the move of data to the cloud. There are challenges and concerns regarding the data storage, network, and data movement, and in response to these concerns NASA EOSDIS is using AWS or Amazon Web Services which is a commercial cloud to ingest, archive, process, distribute, and manage data.
So data from new missions will become available directly in the cloud and There is also a process for moving datasets from on-premises to the cloud, so that users can take advantage of the storage, network, and compute resources that are available in this environment.
Having the EOSDIS entire archive in the cloud will improve management and accessibility of these data. It also expedites the transformation of data into knowledge and information for data users.
And as data is being migrated to the cloud, ALl DAACs including LP DAAC are continuing the effort to support data end users in their workflow transition to the cloud

3 of 15

Transition to the Cloud

4 of 15

Cloud 101

NASA Earthdata Cloud

AWS - Amazon Web Services (Cloud)

AWS Cloud Data Storage - S3 buckets - Amazon S3 buckets are similar to file folders, store objects, which consist of data and its descriptive metadata.

AWS Regions and Zones

NASA data is in us-west-2 (U.S. Oregon)

Elastic Compute Cloud (Amazon EC2) : compute platform with choice of processor, storage, networking, operating system, and purchase model.

AWS S3 Access keys:

Access keys to work programmatically

See the Earthdata Cloud Primer downloadable PDF titled 'Glossary and Acronyms Explained'

Cloud Optimized Format (COF) e.g. Zarr

A format for the storage of chunked, compressed, N-dimensional arrays
“chunked” = dataset can be split into pieces and your computer’s memory (RAM) only needs to load the ones necessary for a calculation
Zarr specifically designed to overcome netCDF4 cloud issues

Metadata stored in single JSON object
Each data chunk is separate object; Parallel data reads and writes (increased efficiency)

Cloud Costs

Compute
Storage
Egress
HTTP requests (get and post)

There’s some new terminology here that you’ll start to see right away as you explore cloud access options.
The AWS regions and zones are separate geographic locations that aws uses to host data and resources. so the AWS region where all the NASA assets are stored is us-west-2 and this is also the region you need to be to work with data in a same region. �
AWS cloud data storage or the s3 buckets are where the data and its descriptive metadata are stored. s3 links are what we would use to access the data directly in-region (us-west-2) within the AWS cloud.
AWS S3 Access Keys: this is credentials that you need to pass to show you are the valid user – this is not EDL information – you should use the EDL to generate your keys
Elastic compute cloud or amazon ec2 is the computer platform that enables the process in the cloud
cloud optimized data formats enable us to work in the cloud more efficiently. Zarr and cog are examples of cloud optimized formats.
Cloud costs is not something you should not be worried about as user here because NASA’s openscapes 2i2c provide cloud-hosted JupyterHubs to us that is for research and education perpouses. so we can work in the cloud free of any charge but if you are willing to set up your own AWS account and work in the cloud you should know that you will be charged for data storage, data movement – egress – for the computation and HTTP request that you will use.

5 of 15

Paradigm Shift

The transition of data and workflows to the cloud is creating a new paradigm to wirk with data. Working with large volume of remote sensing data make the traditional download and analyze data workflow challenging.

In the new paradigm you can access data form a same starrting point rather than everyone having a copy of data locally, and use it into your analysis without downloading data to your local computer environment or local storage.

Accessing data in the cloud via cloud-based tools and services which are “next to the data” such as the CMR API enables users to only access the data they need.

This will offer new methods of access that was not possible with on-premises platforms and it provides an opportunity to research community to effectively expand the scale of their work.

6 of 15

2i2c Managed Cloud Environment

AWS us-west-2

Openscape 2i2c space

Highlights

Managed Environment
Access list based on Github UN
Python Pangeo framework
RStudio and Matlab environments
Git
Persistent user profile

1 of 15

2 of 15

3 of 15

4 of 15

5 of 15

6 of 15

7 of 15

8 of 15

9 of 15

10 of 15

11 of 15

12 of 15

13 of 15

14 of 15

15 of 15