1 of 22

Open-innovation program

A series of websites acting as the agency-wide collecting place for NASA’s public data

including:

API.nasa.gov
Code.nasa.gov
Data.nasa.gov
github.com/nasa

2 of 22

A Tour of NASA’s Data Universe

for a Space-Apps Audience

Justin Gosses

S.A.I.C. senior data scientist supporting

Office of the Chief Information Officer

Transformation & Data Division

3 of 22

The premise of this talk is that by telling you a little about why different open-data sites exist and how they relate to one another, you’ll be better prepared to find datasets.

Introduction to open-innovation data sites (nasa-wide data aggregators)
The NASA Data Universe:

Government Mandates
Harvesting Relationships
The range of sites: Open-innovation program, Science Archives, Others

Tips
Examples of finding datasets

Contents of this talk

4 of 22

Open-Innovation Program

Run by Office of the Chief Information Officer (OCIO),

Open-Government Mandate Driven,

& Agency-wide

4

5 of 22

API + Data + Code

Open-innovation program

API.nasa.gov

Data.nasa.gov

Code.nasa.gov

6 of 22

API.nasa.gov (a passthrough service with tracking by api.data.gov)

A.P.I. = Application Programming Interface (write code to get back data)

There are many other APIs available not listed on this site! This page serves as a central easy to find location for NASA’s easier to use A.P.I.

STATISTICS:

17 APIs: patents, exoplanets, satellite imagery, hand camera imagery, Mars, etc.
55,000 API key owners since 2015.
9 Million hits in May, 2019

https://www.freecodecamp.org/news/learn-how-to-build-astronomy-picture-of-the-day-app-with-nasa-api-and-react-redux-part-ii-83f15970d0e3/

https://www.twilio.com/blog/2017/04/texting-robots-on-mars-using-python-flask-nasa-apis-and-twilio-mms.html

7 of 22

CODE.nasa.gov

555+ open-source projects

Fed from software that has gone through Software Release System run by Office of Chief Engineer

Most but not all code is also on github.com/nasa

https://observablehq.com/d/d2f8833d92e1ea66

Table shows the open-source projects with the most interaction on GitHub using GSA’s pre-built scripts

8 of 22

DATA.nasa.gov

Started in July 2014
20K unique users
40K active datasets
23K page views this month (so far)

The largest number of datasets. Harvests data from other sites. Get’s harvested into data.gov.

9 of 22

Who Puts All This Together?

1000s of NASA & contractor staff who contribute code projects, APIs, and datasets

2.5 developers who maintain the open-innovation sites

You! a lot of our code for these sites is on public github.com repositories and we accept pull requests.

10 of 22

The NASA Data Universe

Why so many places to find data?

10

11 of 22

Congress mandates:

Mandate agency-wide open-innovation sites (code,api,nasa) that harvest data other sites.

Scientific grants:

Specify certain scientific domain archives.

Data consumed by other IT systems with requirements:

Websites, tools, etc. sometimes required being stored in specific systems (databases, APIs).

Examples:

Congressional mandate: Federal Data Strategy
Science data mandate for specific domain: Earth Science
Data consumed by other IT systems: 3D moon rocks with volumetric
scans without an open-source rendering engine.

Mandates & Requirements Drive Dataset Storage Diversity

This is the “why so many sites” slide. It is very short but the reasons are many. Here I’m summarizing them down to three competing forces that tend to drive things.

First, The U.S. congress and the White House play a role by creating mandates that specify data and code must be released to the public in certain ways and harvestable by other sites.
Second, a lot of NASA data is act.ually create by, analyzed by, or in some way connected to grants to university scientists and labs. It is very common for those science grants to come with strings attached that say the dataset creators must place the data in specific science archives for that specific science domain.
Third, a lot of this data doesn’t just exist as isolated data files but is actually created and consumed by other programmatic things, which means they tend to be pub in places and in forms that make those requirements easier to fulfill.

12 of 22

US Government-wide

Harvesting Relationships

data.gov

data.nasa.gov

api.nasa.gov

APOD website

NASA-wide

Project Specific

Domain Specific Archives

earthdata.nasa.gov

Small unique datasets

csv file on plants

13 of 22

Domain-specific NASA Data Sites:

Planetary Data System (data from other planets)
Earth Observations (mostly satellite-based Earth data)
Astromaterials (samples of: dust, comets, meteorites, moon-rocks + 3d models + impacts)
Solar Power datasets & APIs
Global Near Earth Observations (global maps of solar flux, rain, etc.)
Soil Moisture
NODIS (NASA Online Directives Information Service)
Scientific and Technical Information or STI (papers, reports,etc.)
NASA Technology Transfer Office (software, patents, tools, small business)
USGS & NOAA also have Earth Imagery
USGS & NOAA also have Climate data!

FIND LONGER LIST HERE: https://www.nasa.gov/open/data.html

14 of 22

Suggestions for Finding NASA datasets

14

15 of 22

Most used datasets are easiest to work with!

Most reused Code Clones

openmct 281 clones last 2 weeks
earthdata-search 212 clone last 2 weeks

API Downloads in May

APOD (astronomy picture of the day) 9,520,397
neo (near earth objects) 508,207
mars-photos/ 195,953

Datasets Total Downloads

Meteorite Landings: 49,000
NASA Facilities: 34,000
Global Landslide Catalog Export: 29,000
Surface Meteorology and Solar Energy: 22,000

16 of 22

Find Starter Code! saves time on dataset finding & prepping

Datanauts is a program where members of the public work with NASA open-data. The datanauts github org is a great place to find starter code.

Searching for the terms on github will often provide some open-source licensed code you can reuse! ‘NASA’ returns 10,000 results!

Observable Notebooks are like Jupyter notebooks but JavaScript, live, editable, & forkable on the web! Search for terms or check out this NASA collection.

Search through 5000 past SpaceApps projects using this app by datanaut Alexandre Belloni Alves

Bl.ocks is a site that collects live d3 visualizations. You can put ‘nasa’ into the search and get back things that use NASA data!

Live JavaScript Code Collections

Github

Starter code is a great way to get things going quickly. A lot of you will have to figure out how to download data, filter it, or process it. If you can find someone’s code how has already down the first few steps of your project, that can potentially save you a lot of time. Just searching through github.com with nasa plus another term will give you a lot of results. NASA along gets you 10,000 code repositories. There is also a datanauts org on github that works just with NASA data. The box in the lower left corner has a site that lets you search through past SpaceApps challenges that was build by a member of the public. Additionally, if you’re planning to write code in JavaScript, blocks and Observable are some great places to find live and editable code that sometimes uses NASA data.

17 of 22

Consider whether you’re finding or discovering datasets?

Dataset Finding = “You know the dataset exists and what it is called”

Dataset Discovery = “You don’t know the name, whether it exists, what it looks like until you see it, or how you’ll use it until you see it.”

Use data.nasa.gov or other sites where you can search via titles, names, and other things that work well with string matching.

Look at previous code projects or websites that only hold specific types of data. These are more likely to have visual representations of data that help you determine what exists and how you might use it.

18 of 22

Consider Discoverability vs. Data Site Type

data.nasa.gov

pds.nasa.gov

Insight Mars Weather API

Sites Type Example Meta-data Interfaces Built-for Discoverability Type

generic with links to more metadata

science field specific metadata

dataset specific metadata

General public & search engines

Scientists, engineers & developers who need authoritative files

dataset users

String matching in descriptions or titles.

Filter content types & location & format filtering

See example use-cases

Harvest generated site

Domain

specific site

Dataset specific site

19 of 22

Open-Innovation Site Specific:

API.NASA.GOV

API documentation may not be on the first page you find about a dataset.

Be sure to check out all links. API documentation often has some example code but not always easy to find.

This site holds easy to use APIs. It does not hold all APIs!
Data.nasa.gov datasets sometimes have their own APIs.

DATA.NASA.GOV

Data.nasa.gov does not have a good ability to filter by data format as most people do not upload datasets but merely link to other data sites, so do not think you have all the CSVs if you search for things in CSV format!
The ‘tags’ and ‘types’ are human-populated and should not be thought of as ‘complete’.

CODE.NASA.GOV & Github.com/nasa

Not everything on code.nasa.gov is on github. Some projects on gitlab.com or other.
Some projects under project specific org accounts, not ‘nasa’.
All data used to build code.nasa.gov is in the code.json and can be accessed via API at code.gov.

To go over a few pointers, specific to open-innovation sites.
API.nasa.gov does not hold all the NASA APIs. Another site with a large number of APIs is data.nasa.gov. If your project depends on an API, it is worthwhile going there and including API in your search.
Github.com has a lot of NASA code projects but code.nasa.gov has more. Additionally, not all of the NASA code projects on github are in the NASA org. Some are in other user accounts.
For data.nasa.gov, a critical thing to know is that the file formats, tags, and types you see in the side bar of the search pare are not populated for every datasets. This means if you search for a certain tag, there are probably many more datasets of a similar nature that don’t have that tag or are CSV file format but don’t show up when you search for CSVs. The nature of data.nasa.gov where a lot of the data is harvested from other sites, mean that those methods of searching don’t give super good results.

20 of 22

Example

Of Finding Data

20

21 of 22

Example 1: Lunar Sample Evaluation

THE CHALLENGE

You are the astronaut/robotic mission lead tasked with bringing valuable specimens from the Moon back to Earth for further study. How will you evaluate lunar samples quickly and effectively before or while still on the mission? How will you differentiate samples of potential scientific value from less interesting material?

Suggestion 1: Look for org pages of NASA groups that do related work:

Johnson Space Center Astromaterials Group: They have a database of all the Apollo samples.

Suggestion 2: Search for papers/descriptions of past NASA work in data.nasa.gov & sti.nasa.gov:

STI has a paper on this challenge: GeoLab Concept: The Importance of Sample Selection During Long Duration Human Exploration Mission
Data.nasa.gov has a report about a study similar to this task: SAND-E: Semi-Autonomous Navigation for Detrital Environments - Integrating robotic terrain analysis, geochemistry, and sedimentology

For this example, we’ll look at one of the moon challenges. It asks you to create something that will help decide which samples to bring back and which to leave on the moon. There are some datasets provided with this challenge that are good places to find data.

Additionally, it is worth while to look at NASA organization pages that work on what is fundamentally a challenge about moon rocks! Googling will tell you the group that does that at Johnson Space Center is called the Astromaterials group. And if you look around their page, you’ll find out that they have a nice dataset of every single moon rock brought back from the moon. They also have nice 3D images of several of them.
You can also rephase the challenge into a short sentence and see if anyone at NASA has published a paper or report on that task. We can do that both on data.nasa.gov and sti.nasa.gov, which holds scientific reports. When we do that, we find in fact that there are studies published on this very task that you can refer to for ideas on how to think about the challenge.

22 of 22

Any Questions?

Best of Luck with SpaceApps!