This url is part of this larger one : http://www.archivers.space/urls/B59D4BAC-EE63-4E01-8F3B-8689B220ABD2 (7 datasets)

GPM_3IMERGHHL: GPM L3 IMERG Late Half Hourly 0.1 degree x 0.1 degree Precipitation V03
Relevance/High Level Description: Satellite collected precipitation data that feeds into a climate prediction model and cloud classification model

Specific Description: These are provided to both the Climate Prediction Center (CPC) Morphing-Kalman Filter (CMORPH-KF) Lagrangian time interpolation scheme and the Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks Cloud Classification System (PERSIANN-CCS) re-calibration scheme.

You can find the ftp server address in the first xml file (click on online archive on the right). From then wget can download everything

Please make a pdf with all information from the landing page and download associated documents :
-Requires specific citation listed on landing page. Remember to save this along with each data file.
-Documentation also located on landing page. Save this as well (PDF format)
-Product Summary also contains important information regarding spatial coverage and relevant time range.

Please ping me on slack (@tek) if you need help

https://disc.sci.gsfc.nasa.gov/uui/datasets/GPM_3IMERGDL_V03/summary

GPM_3IMERGDL: GPM (IMERG) Late Precipitation L3 1 day 0.1 degree x 0.1 degree V03

Data was scraped using slightly modified [instructions](https://disc.sci.gsfc.nasa.gov/recipes/?q=recipes/How-to-Download-Data-Files-from-HTTP-Service-with-wget) offered by Goddard Earth Sciences Data and Information Services Center. * Create ~/.netrc and content like this: `machine urs.earthdata.nasa.gov login SOMELOGIN password SOMEPASS` * Creating a directory to handle cookies as ~/.urs_cookies The actual recursive fetch tries to flatten the directories. I think researchers will find the hierarchy more helpful, so maintained it with the --force-directories switch `wget --load-cookies ~/.urs_cookies --save-cookies ~/.urs_cookies --keep-session-cookies -r -c -nH -nd -np -A nc4,xml "https://gpm1.gesdisc.eosdis.nasa.gov/data/GPM_L3/GPM_3IMERGDL.03/" --force-directories`

https://s3.amazonaws.com/drp-upload/remote/B2034437-B4FE-4B9F-A117-FEC4EDE41084_1.zip

https://tidesandcurrents.noaa.gov/inundation

https://triticeaetoolbox.org

http://cdiac.ornl.gov/data_catalog.html

http://climate.nasa.gov/vital-signs/arctic-sea-ice

https://epa.gov/cleanups/cleanups-my-community

https://epa.gov/ejscreen/download-ejscreen-data

http:///%22dir%22%20:%20%22../data/howdo_weknow/National%20Science%20Foundation/To%20What%20Degree%20-%20How%20Do%20We%20Know/%22,

85803555-ABA8-4590-992D-DB9C9DF7D883

The purpose of the PPS is to process, analyze and archive data from the Global Precipitation Measurement (GPM) mission, partner satellites and the TRMM mission. The PPS also supports TRMM by providing validation products from TRMM ground radar sites. All GPM, TRMM and Partner public data products are available to the science community and the general public from the TRMM/GPM FTP Data Archive

Registration required for access to data archive. Once registered, access is made available to a basic FTP with direct downloads to data.

note: tried registering an email (registration immediate) for ftp on Feb 18, 2017 and was unable to connect to ftp://arthurhou.pps.eosdis.nasa.gov

2/19: Able to register and access FTP server. Many of the files/directories are actually shortcuts to other directories.

HDF.tar, .png, .html, .xml, .txt

https://pps.gsfc.nasa.gov

NASA Precipitation Processing System

https://epa.gov/caa-permitting/electronic-permit-submittal-system-region-9

ftp://arthurhou.pps.eosdis.nasa.gov

https://acf.hhs.gov/ocs/resource-library/search?area[2106]=2106&sort=recent

http://ready.noaa.gov/READYmetdata.php

http:///%22url%22%20:%20%22https:/www.youtube.com/playlist?list=PL0ujJTaPsv3erTG6jVPLIhYZ3TsYPDKGR",

- Coal Production and number of mines by state, number of employees by mine, etc.
- High significance

http://eia.gov/coal/annual

Annual Coal Report

https://drp-upload.s3.amazonaws.com/remote/29C5D29E-FB79-4AA6-B39E-0EBCDEF2DDD3.zip

https://drp-upload-bagger.s3.amazonaws.com/remote/29C5D29E-FB79-4AA6-B39E-0EBCDEF2DDD3.zip

https://www.datarefuge.org/dataset/annual-coal-report

http:///%22url%22%20:%20%22https:/www.youtube.com/user/NASAClimate/videos%22,

https://waterwatch.usgs.gov

Power generation and greenhouse gas and pollutant emissions data across the US

.zip containing .txt, .xls and .xlsx, .pdf, .jpg,

https://epa.gov/energy/emissions-generation-resource-integrated-database-egrid

Emissions & Generation Resource Integrated Database (eGRID)

clicking "download" button

https://drp-upload.s3.amazonaws.com/remote/5A98CDB1-51F9-482E-9EF2-815F9E535613.zip

https://drp-upload-bagger.s3.amazonaws.com/remote/5A98CDB1-51F9-482E-9EF2-815F9E535613 2.zip

https://www.datarefuge.org/dataset/https-epa-gov-energy-emissions-generation-resource-integrated-database-egrid

https://ncdc.noaa.gov/data-access/model-data

https://epa.gov/risk/risk-tools-and-databases

https://dropbox.com/sh/e5is592zafvovwf/AABVLBCyQEM0FQ3BadWwOfFka/CRU-TS%20v3.10.01?dl=0

https://epa.gov/research/human-health-risk-assessment-research-methods-models-tools-and-databases

https://ncdc.noaa.gov/cdr

nc, nas, iso, zip, pdf, sgp, csv, xls, doc, pptx, doc

ftp://ftp.cmdl.noaa.gov

https://tidesandcurrents.noaa.gov/astronomical.html

http://afdc.energy.gov/data

https://drp-upload.s3.amazonaws.com/remote/607D3252-2664-417E-928B-45DAC02D885B.tar

"The Data Pool is an on-line data cache that provides FTP access to select ASDC data products."

ftp download (note: symlinks involved are confusing wget. Surprisingly can't find a standard tool that will mirror ftp following symlinks?)

Unclear to me if the "OPeNDAP Access" links are the same data as the FTP links or different.

https://eosweb.larc.nasa.gov/datapool

https://nasa.gov/content/space-station-view-of-us-national-parks

Space Station View of U.S. National Parks

Datum is a reference needed to interpret local tide data. This info is critical for interpreting long time series data

https://tidesandcurrents.noaa.gov/api/. Note, the API will only scrape current epoch data (post 1970). A custom scraper is needed to get superseded (1968-1978) data.

https://tidesandcurrents.noaa.gov/stations.html?type=Datums

Datums - Station Selection

API, BeautifulSoup

https://drp-upload.s3.amazonaws.com/remote/8393A9FE-61D2-4AD6-87DD-EAC897794226 2_1.zip

Scan, Snotel, and water forcasting

Crawl Daily! Updated live weather station data (Not in the GHCN) and snow and water data

https://www.wcc.nrcs.usda.gov/ftpref/data/

National Resource Conservation Service - National Water and Climate Center

https://iaspub.epa.gov/apex/waters/f?p=ASKWATERS:MAIN_MENU

This server contains research data from NASA's Gravity Recovery And Climate Experiment (GRACE).

Any recursive FTP approach will work, wget or other FTP tree script. Note this is an e

.tar.gz, .gz, .pdf, .txt

ftp://podaac-ftp.jpl.nasa.gov/allData/grace

NASA GRACE Satellite Data

wget -m ftp://podaac-ftp.jpl.nasa.gov/allData/grace

https://eosweb.larc.nasa.gov/project/misr/cmare_table

https://ozoneaq.gsfc.nasa.gov/data/reflectivity

https://minerals.usgs.gov/minerals/pubs

https://eosweb.larc.nasa.gov/project/mopitt/mopitt_table

https://epa.maps.arcgis.com/apps/webappviewer/index.html?extent=-146.2334,13.1913,-46.3896,56.5319&id=5f239fd3e72f424f98ef3d5def547eb5

https://ozoneaq.gsfc.nasa.gov/data/omps

https://gispub.epa.gov/arcgis/rest/services

https://federalregister.gov/documents/search

https://data.giss.nasa.gov

Submitted to Data Refuge site by Project ClimateScienceSave as potentially vulnerable videos that are on the subject of climate change.

xml, json, mkv, mp4, .description

http:///%22dir%22%20:%20%22../data/nasa_visualizations/NASA%20Scientific%20Visualization%20Studio/Earth/%22,

https://cfpub.epa.gov/dmr/adv_search.cfm

EPA's Report on the Environment (ROE) shows how the condition of the U.S. environment and human health is changing over time. The ROE presents the best available indicators of national trends in five theme areas: Air, Water, Land, Human Exposure and Health, and Ecological Condition.

https://cfpub.epa.gov/roe

http:///%22agency%22%20:%20%22Interior%22,

https://minerals.usgs.gov/products/index.html

This url is part of this larger one : http://www.archivers.space/urls/B59D4BAC-EE63-4E01-8F3B-8689B220ABD2 (7 datasets)

GPM_3IMERGHHL: GPM L3 IMERG Late Half Hourly 0.1 degree x 0.1 degree Precipitation V03
Relevance/High Level Description: Satellite collected precipitation data that feeds into a climate prediction model and cloud classification model

Specific Description: These are provided to both the Climate Prediction Center (CPC) Morphing-Kalman Filter (CMORPH-KF) Lagrangian time interpolation scheme and the Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks Cloud Classification System (PERSIANN-CCS) re-calibration scheme.

You can find the ftp server address in the first xml file (click on online archive on the right). From then wget can download everything

Please make a pdf with all information from the landing page and download associated documents :
-Requires specific citation listed on landing page. Remember to save this along with each data file.
-Documentation also located on landing page. Save this as well (PDF format)
-Product Summary also contains important information regarding spatial coverage and relevant time range.

Please ping me (@tek¼ on slack if you need help)

https://disc.sci.gsfc.nasa.gov/uui/datasets/GPM_3IMERGM_V03/summary

GPM_3IMERGM: GPM L3 IMERG Final 1 month 0.1 degree x 0.1 degree precipitation V03

Used the JS script (in /tools) to generate lists of URLs, then downloaded them with wget

B56AA4A6-2065-404B-9114-C102811D063D

http://ndbc.noaa.gov/station_page.php?station=46071

https://lib.noaa.gov/collections/imgdocmaps/daily_weather_maps.html

/!\ I am working on this one, please take another one (tek - 11/02 12h15)

This url is part of this larger one : http://www.archivers.space/urls/B59D4BAC-EE63-4E01-8F3B-8689B220ABD2 (7 datasets)

GPM_3IMERGHHL: GPM L3 IMERG Late Half Hourly 0.1 degree x 0.1 degree Precipitation V03
Relevance/High Level Description: Satellite collected precipitation data that feeds into a climate prediction model and cloud classification model

Specific Description: These are provided to both the Climate Prediction Center (CPC) Morphing-Kalman Filter (CMORPH-KF) Lagrangian time interpolation scheme and the Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks Cloud Classification System (PERSIANN-CCS) re-calibration scheme.

You can find the ftp server address in the first xml file (click on online archive on the right). From then wget can download everything

Please make a pdf with all information from the landing page and download associated documents :
-Requires specific citation listed on landing page. Remember to save this along with each data file.
-Documentation also located on landing page. Save this as well (PDF format)
-Product Summary also contains important information regarding spatial coverage and relevant time range.

https://disc.sci.gsfc.nasa.gov/uui/datasets/GPM_3IMERGHHE_V03/summary

GPM_3IMERGHHE: GPM L3 IMERG Early Half Hourly 0.1 degree x 0.1 degree Precipitation V03

wget on the FTP server here : ftp://gpm1.gesdisc.eosdis.nasa.gov/data/s4pa/GPM_L3/GPM_3IMERGHHE.03/

https://science.nasa.gov/earth-science/decadal-surveys

Earth Decadal Survey

FDD2C5E8-5DBD-4FD4-8C91-C6FD5D9C3C5C

Link is acting strangely, just click the Data tab on top, to the right of the Home tab.

https://omg.jpl.nasa.gov/portal/browse

Oceans Melting Greenland

http://ndbc.noaa.gov/station_page.php?station=welm1

Air Quality Data - Raw data files, zips, notes,entire folder of ftp site

Pudasaini zip containing BP_ .csv files

ftp://newftp.epa.gov/AIR_QUALITY_DATA

newftp.epa.gov/AIR_QUALITY_DATA/

Use API/REST as for https://www2.usgs.gov/water/

https://waterdata.usgs.gov/nwis

historical data of ti

Not sure if there are historical versions of this or it is just the up-to-the-moment predictions

https://tidesandcurrents.noaa.gov/tide_predictions.html

NOAA Tide Predictions

Python- Beautiful Soup to get station IDs, requests to get txt files

http://nrel.gov/aim/npc.html

http:///%22dir%22%20:%20%22../data/nsf_watercycle/National%20Science%20Foundation/To%20What%20Degree%20-%20The%20Water%20Cycle/%22,

This url is part of this larger one : http://www.archivers.space/urls/B59D4BAC-EE63-4E01-8F3B-8689B220ABD2 (7 datasets)

GPM_3IMERGHHL: GPM L3 IMERG Late Half Hourly 0.1 degree x 0.1 degree Precipitation V03
Relevance/High Level Description: Satellite collected precipitation data that feeds into a climate prediction model and cloud classification model

Specific Description: These are provided to both the Climate Prediction Center (CPC) Morphing-Kalman Filter (CMORPH-KF) Lagrangian time interpolation scheme and the Precipitation Estimation from Remotely Sensed Information using Artificial Neural Networks Cloud Classification System (PERSIANN-CCS) re-calibration scheme.

You can find the ftp server address in the first xml file (click on online archive on the right). From then wget can download everything

Please make a pdf with all information from the landing page and download associated documents :
-Requires specific citation listed on landing page. Remember to save this along with each data file.
-Documentation also located on landing page. Save this as well (PDF format)
-Product Summary also contains important information regarding spatial coverage and relevant time range.

Please ping me on slack (@tek) if you need help

https://disc.sci.gsfc.nasa.gov/uui/datasets/GPM_3IMERGDE_V03/summary

GPM_3IMERGDE: GPM (IMERG) Early Precipitation L3 1 day 0.1 degree x 0.1 degree V03

The Soil Climate Analysis Network (SCAN) began as a soil moisture/soil temperature pilot project of the Natural Resources Conservation Service in 1991. The system is designed to provide data to support natural resource assessments and conservation activities. The SCAN system focuses on agricultural areas of the U.S. and is composed of over 200 stations. A typical SCAN site monitors soil moisture content at several depths, air temperature, relative humidity, solar radiation, wind speed and direction, liquid precipitation, and barometric pressure.

See https://github.com/antonpaquin/DataRescue-NRCS for a python script that does it
The main bottleneck is on their end.

https://www.wcc.nrcs.usda.gov/scan/

Soil Climate Analysis Network (SCAN) Data & Products

https://iaspub.epa.gov/sor_internet/registry/substreg/automatedservices/index.jsp

http://ftp.epa.gov

https://drought.gov/drought/search/data

https://nomads.ncdc.noaa.gov/data/cfsr

Field Campaign, Land Validation, Global and Regional Data.

Field Campaign Data:
ABOVE
AirMOSS
BOREAS and BOREAS Follow-On
Carbon Monitoring System
CARVE
FIFE and FIFE Follow-On
LBA
NACP
OTTER
SAFARI 2000
Superior National Forest (SNF)



Land Validation Data:
ACCP
BigFoot
EOS Land Validation
FLUXNET Data Sets
FLUXNET Web Site
MODIS Land Subsets
PROVE


Regional and Global Data:
Climate Collections
Daymet
Hydroclimatology Collections
ISLSCP II Project
Net Primary Productivity (NPP)
River Discharge (RIVDIS)
Russian Land Cover (RLC)
Soil Collections
TransCom 3
Vegetation Collections
VEMAP

Crawl this data periodically. Data is not updated everyday. If you crawl this ftp site 4 times a year that could be sufficient. This ftp site is not widely known.

all kinds of text, vector, raster, and db data. Lots of netcdf

https://thredds.daac.ornl.gov/thredds/catalog.html

Oakridge National Labs Distributed Active Archive Center (ORNL DAAC)

Not harvested. Too much data

https://java.epa.gov/castnet/datatypepage.do?reportTypeId=REP_001&reportTypeLabel=Measurement (Raw Data)

https://java.epa.gov/castnet/datatypepage.do?reportTypeId=REP_006&reportTypeLabel=Cloud Deposition Data

https://iaspub.epa.gov/sor_internet/registry/substreg/searchandretrieve/searchbylist/search.do

Monthly updated estimates of global surface temperature change, using current data from NOAA GHCN v3 (meteorological stations), ERSST v4 (ocean areas), and SCAR (Antarctic stations).

https://data.giss.nasa.gov/gistemp

GISS Surface Temperature Analysis (GISTEMP)

Python web scraper

https://drp-upload.s3.amazonaws.com/remote/9A2D2BEA-65AA-4FB8-98FE-305CA9CEF54C_2.zip

https://drp-upload-bagger.s3.amazonaws.com/remote/9A2D2BEA-65AA-4FB8-98FE-305CA9CEF54C_2.zip

https://www.datarefuge.org/dataset/giss-surface-temperature-analysis-gistemp

Information about the EPA center for Computational Toxicology and some of their research is here - https://www.epa.gov/aboutepa/about-national-center-computational-toxicology-ncct

ftp://newftp.epa.gov/COMPTOX

(FTP repository) - data relating to Computational Toxicology

http://cloudsat.cira.colostate.edu/resources/interactive-plots

http://fec.gov/data

Search page for chemical assessments in the IRIS program. The broader IRIS website also includes other documents such as drafts for peer review and public comment. See parent item for full IRIS website: https://epa.gov/iris, UUID: 14BDABB8-F78D-41BC-A160-C6E631E39081

Check status of parent: https://epa.gov/iris, UUID: 14BDABB8-F78D-41BC-A160-C6E631E39081 -- content of all assessments accessible through this search tool will be harvested there.

https://cfpub.epa.gov/ncea/iris/search/index.cfm

Integrated Risk Information System - Advanced Search

http://ndbc.noaa.gov/station_page.php?station=21414

http://echo.epa.gov

Download everything that be accessed through the APIs listed here

http://earthdata.nasa.gov/api

http://svs.gsfc.nasa.gov/cgi-bin/search.cgi?contentType=a

https://st.nmfs.noaa.gov/sisPortal

This is covered by
http://www.eia.gov/opendata/
1EA13C4E-B1E3-4B53-9E20-EAF33BFB2CCA

https://eia.gov/consumption/residential

https://watersgeo.epa.gov/beacon2

https://energy.gov/eere/femp/federal-energy-management-program

https://nsrdb.nrel.gov

https://epa.gov/air-emissions-inventories/air-pollutant-emissions-trends-data

https://maps.bts.dot.gov/Transit

https://epa.gov/waterdata/waters-web-services

This site serves USGS water data (https://waterdata.usgs.gov/nwis) via automated means using web services External Link and extensible markup language (XML) External Link, as well as other popular media types. Services are invoked with the REST External Link protocol. These services designed for high fault tolerance and very high availability.

Use API/REST: https://waterservices.usgs.gov/rest/
There are six API services to access different data each with their own documentation.
Read the FAQ for info about request limits and why your IP might get blocked.
Identical data to: https://waterdata.usgs.gov

https://waterservices.usgs.gov

USGS Water Services

Daily weather station data for the globe but mostly for the USA.

CRAWL DAILY IF POSSIBLE! Updated live data everyday.

https://www1.ncdc.noaa.gov/pub/data/ghcn/daily

Global Historical Climatology Network - Daily (GHCN-Daily) (NOAA ftp site)

https://nodc.noaa.gov/cgi-bin/OC5/woa13/woa13.pl

Complete collection of meteorological and solar irradiance data sets for the United States and a growing list of international locations. Link was pasted incorrectly but just use up until the first comma (https://nsrdb.nrel.gov/). Parent url (with links to 6 NREL databases is C837F320-C5B4-47B4-B8B4-F33DA519BD51.

https://nsrdb.nrel.gov/,%20http:/www.nrel.gov/rredc/,%20https:/maps.nrel.gov/nsrdb-viewer

National Solar Radiation Database

https://opendap.co-ops.nos.noaa.gov

Contains tables for projected outlooks (2015 and beyond) for total energy supply, production and disposition.

- Direct download from "download" buttons.
- "Publications and Tables" on the upper right also provides links for outlooks of other types and earlier years.

http://eia.gov/outlooks/aeo/data/browser

Annual Energy Outlook 2017

http:///%22url%22%20:%20%22https:/www.youtube.com/user/statevideo/search?query=climate",

https://beta.fec.gov/data/advanced

https://ozoneaq.gsfc.nasa.gov/data/trace-gases

http:///%22dir%22%20:%20%22../data/nsf_carboncycle/National%20Science%20Foundation/To%20What%20Degree%20-%20The%20Carbon%20Cycle/%22,

Contains data on the features of all products that currently qualify for EPA's ENERGY STAR label.

https://data.energystar.gov

The BLM compiles a large amount of statistical information relating to oil and gas leasing on Federal lands. Below are links to tables and spreadsheets with data that include the numbers of BLM-administered oil and gas leases, applications for permit to drill, and oil and gas wells. Because the Federal Onshore Oil and Gas Leasing Reform Act if 1987 set the competitive lease requirement for public lands, 1988 is the first year for which some of this data is available.

16 tables and a chart. Directly downloadable after expanding the section.

https://blm.gov/programs/energy-and-minerals/oil-and-gas/oil-and-gas-statistics

Oil and Gas Statistics

http://ospo.noaa.gov/Products/index.html