Embracing
Open Source
for NASA's
Earth Science
Data Systems
Katie Baynes, Dan Pilone, Ryan Boller, David Meyer, NASA Goddard Space Flight Center, Greenbelt, MD
Kevin Murphy, NASA HQ, Washington, DC
First of all, thanks for
having me. I am so excited to be here!
Look guys! I am on stage!
First, an EOSDIS*
Overview
(so where do you work at NASA?)
*Earth Observing System Data and Information System (i say “ee-oh-ess-diss”)
Putting EOSDIS in Context
EOSDIS
Applications
Capture and Clean
Education
Research
*Subset, reformat, reproject
Transform*
Distribute
Archive
Process
SIPS
DAAC
Distributed Active Archive Centers (DAACs), archive and distribute standard data products produced by Science Investigator-led Processing Systems (SIPS)
ASF DAAC �SAR Products, Sea Ice, Polar Processes
PO.DAAC �Ocean Circulation�Air-Sea Interactions
NSIDC DAAC �Cryosphere, Polar Processes
LPDAAC �Land Processes and Features
GHRC �Hydrological Cycle and Severe Weather
ASDC�Radiation Budget, Clouds, Aerosols, Tropo Composition
LAADS/MODAPS �Atmosphere
OB.DAAC �Ocean Biology and Biogeochemistry
SEDAC �Human Interactions in Global Change
CDDIS�Crustal Dynamics Solid Earth
NCAR, U. of Co.
MOPITT
JPL�MLS, TES, SNPP Sounder �
U. of Wisc.
SNPP Atmosphere
GHRC
AMSR-U, LIS
GSFC
SNPP, MODIS, OMI, OBPG
OPEN SOURCE
OPEN SOURCE
OPEN SOURCE
OPEN SOURCE
6
7
8
9
And we are poised for some really big missions in the early 2020s*, so we’ve got exciting things on the horizon!
* NISAR and SWOT are going to increase EOSDIS daily total ingest from about 6 TB/day to over 110 TB/day
Second, a Quick How To for
Open Sourcing at NASA
(ok, how does this work?)
“
Enhanced reuse of custom-developed code across the Federal Government can have significant benefits for American taxpayers, including decreasing duplicative costs for the same code and reducing Federal vendor lock-in.5�
This policy also establishes a pilot program that requires agencies, when commissioning new custom software, to release at least 20 percent of new custom-developed code as Open Source Software (OSS) for three years, and collect additional data concerning new custom software to inform metrics to gauge the performance of this pilot.6
I am not sure how many “enlightenment Buddhas” NASA gets. But I think we are doing this for the right reasons.
Free open data for all, easy to find, easy to use, easy to give back.
Here is the high-level process
N.B. This process ensures we are inclusive in our practices and abide by federal law
NASA’s Open Sourcing Process in More Detail
(a brief sub-presentation)
17
NASA releases tons of software!
18
What Forms Will I Need?
New Technology Report - NF1679 (online or via doc template)
(You will need the assigned NTR number (e.g. “GSC-17610”) for subsequent forms)
Export Control Form
Global Concerns Statement
508 Compliance Statement
GSFC Software Developer Form �(aka Software Release Request Authorization or SRRA)
Open Source Questionnaire
https://wiki.earthdata.nasa.gov/display/ESDSWG/Software+Release+Process+-+GSFC+Specific
19
Things to Be Prepared to Gather
List of prior publications related to software (conferences, etc)
Software Classification http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7150_002B_&page_name=AppendixD
Distributed Dependencies (a bit more on that later)
508 Compliance Information (especially for Web Interfaces)
Contractors must be prepared to release copyright claims
(more information available at https://software.nasa.gov/ )
20
21
Relevant NPRs
NPR 2210.1C
http://nodis3.gsfc.nasa.gov/displayDir.cfm?t=NPR&c=2210&s=1C
NPR 7150.2B
http://nodis3.gsfc.nasa.gov/displayDir.cfm?Internal_ID=N_PR_7150_002B_&page_name=main
(more information available at https://software.nasa.gov/ ).
22
Licensing
23
With Regards to Licensing
You will almost certainly need to provide a list of distributed dependencies.
https://en.wikipedia.org/wiki/Viral_license
Any GPL’d or “viral-licensed” software (or in distributed dependencies) will prevent the Office of Patent Counsel (OPC) from being able to release your work.
24
With Regards to Licensing
Releases are done under NASA Open Source Agreement (NOSA)
Apache 2.0 on a case-by-case basis*
25
With Regards to Licensing
26
NASA Open Source Agreement (#NASA) The NASA Open Source Agreement, version 1.3, is not a free software license because it includes a provision requiring changes to be your “original creation”. Free software development depends on combining code from third parties, and the NASA license doesn't permit this. We urge you not to use this license. In addition, if you are a United States citizen, please write to NASA and call for the use of a truly free software license. |
With Regards to Licensing
“
We are seeking the Apache 2.0 licenses based on the current statement from the gnu.org regarding its advice for adoption, explained below.
We feel that this would hinder contributors and give pause to anyone considering augmenting and extending our code with other existing code bases (e.g. ‘mash-ups’)
https://www.gnu.org/licenses/license-list.html#NASA
27
1022
number of projects released at software.nasa.gov
28
(End Sub-Presentation)
29
Recent Open Source
Efforts
(only talking about EOSDIS, actively “working” repos)
Some of our more active code bases....
🐙 - more on these later
OnEarth | |
Worldview🐙 | |
Earthdata Search | |
Common Metadata Repository | |
Metadata Management Tool | |
Cumulus🐙 | Approved for open source, not public yet |
pyCMR |
This is great!
Can’t wait to really start giving back!
Looking at Worldview:
How can we help?
University Students Working on NASA FOSS
University of Pennsylvania held a FOSS class in Fall of 2016.
A pair of students decided to tackle working on Worldview to get their feet wet in contributing.
The blogged about it!
University Students Working on NASA FOSS Great Idea!
Off to a good, but rocky start!
We’ve made great strides since then!
Project Roadmap
https://github.com/nasa-gibs/worldview/wiki/Worldview-Roadmap
Specific README Sections on Installation and Contribution
More transparent testing:
https://travis-ci.org/nasa-gibs/worldview
More transparent issue tracking:
https://waffle.io/nasa-gibs/worldview
Talk to Ryan Boller for more info!
What else could go wrong?
NASA is full of process and procedure.
How can we improve upon this one?
Starting with a clear plan.
Embracing
Open Source
(another case study in which we are starting on the ground floor)
Prototyping DAACs in the Cloud
What is Cumulus?
Lightweight cloud-native framework for data ingest, archive, distribution and management
Goals
46
47
48
49
We have long been a system of stovepipes and “not invented here” types. And that is changing as we evolve
We need to streamline, and create unified, interoperable system that can grow with us. Something we can claim group ownership of.
So ... how do we do this “right”
Drafting a contribution guide
Mimic the Apache Way, but for a simpler use case and narrower audience
Establishing Project Roles and Responsibilities
Project Management Board
Project Leadership Team
Technical Leadership Team
Committers
Contributors
As we scale (up or down) we can adjust/combine/tailor these roles.
We can adapt to other projects/systems.
Form a Working Group!
give them homework
set a release timeline
Understand your Obligations and Limitations
denouement
denouement
Denouement
(so, what’s next?)
We are abiding by these principles in Cumulus now.
By October 2017 (like, tomorrow in government time),
we will be rolling out this policy and making it “official”.
There is still work to do. We are just starting this journey.
https://pixabay.com/en/road-asphalt-space-sky-clouds-220058/ (licensed CC0)
Questions?