1 of 86

Import, export & render

Chris Herwig, Google

2018 Earth Engine User Summit

bit.ly/eeus-data

2 of 86

Agenda

  1. Key concepts
  2. Types of data in Earth Engine
  3. Working with data in public data catalog
  4. Analyzing your own data in EE
  5. Taking your results out of EE
  6. Creating new EE assets

3 of 86

CONFIDENTIAL

Organize the world's Earth observation information and make it universally accessible and useful

4 of 86

​

​

​

​

​

​

Chart

Image

Earth Engine Asset

Table

Video

Map

Google Confidential and Proprietary

5 of 86

Key

Concepts

6 of 86

Key Concepts

Pyramiding Policy

​

How each pixel at a given level of the pyramid is computed from the aggregation of a 4x4 block of pixels at the next lower level.

​

Continuous valued images: mean

​

Discrete valued images: sample (usually the top left pixel) of pixels at the next lower level.

​

​

​

.

7 of 86

Key Concepts

Access Control Lists (ACL)

​

scale

​

dimensions

By default, assets users upload to their Earth Engine account are private -- accessible only to that user. Users can change access control settings on individual assets or collections. Update ACLs using Asset Manager in Code Editor or via command line earthengine acl command

​

​

​

​

​

​

(in meters) Ex. 30

​

​

​

dimensions (in pixels): width and height for output image, Ex: '1024x768'

​

​

8 of 86

Key Concepts

crs

​

crsTransform

​

region

Coordinate Reference System. Ex: 'EPSG:4326'

​

​

​

A list of affine transformation parameters in row-major order [xScale, xShearing, xTranslation, yShearing, yScale, yTranslation]

If not explicitly specified, the CRS of the output will be taken from the first band of the image to be exported.

crsTransform, scale, and dimensions are mutually exclusive.

​

​

​

(ee.Geometry) region of image to export

​

​

9 of 86

maxPixels

​

​

​

​

​

The maxPixels parameter is intended to prevent very large exports from inadvertently being created.

​

​

​

​

​

​

Increase maxPixels if the default value is too low for your intended output image.

​

​

​

​

​

If the output image file is large, it will be exported as multiple tiles, with each named baseFilename-yMin-xMin.

maxPixels default is 1e8.

​

​

​

10 of 86

Command-line utility

install

​

authenticate

​

use

​

After installing, you'll need to authenticate via the earthengine authenticate command

​

​

Once authenticated, CLI supports: Create/manage assets and collections, ACL setting

moving/deleting/copying assets, uploading raster data

​

And more...

​

11 of 86

Data in Earth Engine

12 of 86

Vector

  • Geometries
    • Point
    • LineString
    • LinearRing
    • Polygon
    • MultiPoint
    • MultiLineString
    • MultiPolygon
  • Geometry Collections
    • Collection of geometries

13 of 86

Vector

  • Features
    • Geometry + attributes
  • Feature Collections
    • Collection of features

14 of 86

Raster

Image

  • Arbitrary number of bands + band designations
  • Metadata properties

​

Image Collection

  • Collection of images
  • Can have collection-level metadata attributes

​

​

15 of 86

Public

data catalog

16 of 86

>200 public datasets

17 of 86

Earth Engine

Public Data Catalog of 200+ datasets (>6 PB)

800+ analysis functions

Landsat and Sentinel

40+ year archive of the entire planet

MODIS

2 global images every day.

Terrain and Landcover

Weather and Climate

18 of 86

Asset management

19 of 86

Asset Manager

  • Code Editor assets tab
  • Upload rasters
  • Manage uploaded sources
  • Edit asset metadata
  • Create/edit image collections and folders
  • Change sharing settings on images or collections
  • View quota usage

20 of 86

Manage asset properties

  • Add property to define a property name and value.
  • Values can be a string or a number.
    • Enter strings without quotes.
    • To format a number as a string, enter a single quote (') before the number.
  • Click the delete icon to remove a property.

21 of 86

Command-line asset property management

# update/set date of an asset�$ earthengine asset set --time_start 2016-05-01T12:34:56 \� users/username/asset_id;

# add properties to asset�$ earthengine asset set -p '(number)clouds=42' \� users/username/asset_id

# add string property to asset�$ earthengine asset set -p '(string)station=Alaska' \� users/username/asset_id

22 of 86

Click the Share button to configure access to your private assets. You can also share by hovering over the asset and clicking the share icon.

The sharing dialog (Figure 4) lets you configure read or write access for individuals or groups (specified by a domain).

​

​

To make an asset public, check the 'Anyone can read' box. Any user with write access to a folder will be able to delete assets from it.

​

​

Proprietary + Confidential

Share

Proprietary + Confidential

Proprietary + Confidential

23 of 86

Earth Engine accounts come with 250GB of storage quota, which is independent of any Drive quota you might have.

Quota usage is measured by the total number and the compressed size of stored assets.

You can view your usage by clicking the data usage icon next to users/username in the Asset Manager.

Proprietary + Confidential

Quota

Proprietary + Confidential

Proprietary + Confidential

24 of 86

Move images and collections into folders with drag and drop.

To remove images from the collection, click on the collection to open dialog.

Proprietary + Confidential

Create image collections

Proprietary + Confidential

Proprietary + Confidential

25 of 86

Command-line create image collections

​

​

# create new image collection�$ earthengine create collection \� users/herwig/eeus/collection1�# add images to the collection�# can use mv or cp command�$ earthengine mv \� users/herwig/eeus/2_8_10x_2 \� users/herwig/eeus/collection1/2_8_10x_2

26 of 86

Questions

27 of 86

Upload vector

28 of 86

Upload shapefile with sidecar files or zip file

If prj file is omitted, WGS84 is assumed.

​

​

Code Editor: Upload table

Proprietary + Confidential

Proprietary + Confidential

29 of 86

30 of 86

31 of 86

32 of 86

Upload large vector datasets to Google Fusion Tables

​

​

Import in Earth Engine script

CSV or KML

​

​

Code Editor: Import table from Fusion Tables

var fusionTable = ee.FeatureCollection(�'ft:1kSWksPYW7NM6QsC_wnCuuXO7giU-5ycxJb2EUt8')

Proprietary + Confidential

Proprietary + Confidential

33 of 86

var region = ee.Geometry.Point(

[151.213, -33.87]);�

var vis = {� min:500,� max:3000,� gamma:1.4,� bands: ['B4', 'B3', 'B2']�}

�var collection = ee.ImageCollection('COPERNICUS/S2')� .filterBounds(region)� .sort('system:time_start')� // Need to make the data 8-bit.� .map(function(image) {� return image.visualize(vis);� });

Export.video.toDrive({� collection: collection,� description: 'sydney-5fps',� dimensions: 1080,� framesPerSecond: 5,� region: region�});

34 of 86

Hand-drawn geometries in code editor

  • Points (+Multi)
  • Polygons (+Multi)
  • LineStrings (+Multi)
  • Geometry
  • Feature
  • FeatureCollection

35 of 86

Copy/paste GeoJSON

var feats = ee.FeatureCollection(json.features);

​

36 of 86

Questions

37 of 86

Upload Raster

38 of 86

Upload raster

  • Geotiff
  • 10GB per-file maximum
  • Can set properties
  • Choose pyramiding policy
  • Set optional no data value
  • See Advanced image upload options
  • Tiled uploads supported, see docs

39 of 86

Command-line: Upload raster

  • Geotiff(s) on Google Cloud Storage
  • 10GB per-file maximum
  • (optional) Set pyramiding policy
  • (optional) Set no data value
  • Can also set metadata properties at time of upload.
  • See upload section of Command Line Usage docs

assetid="users/herwig/eeus/mytestupload";�earthengine upload image \� --asset_id="${assetid}" \� --pyramiding_policy=sample \� --nodata_value=0 \� gs://bucket/image.tif \� gs://bucket/image2.tif�

​

40 of 86

Command-line: Tiled Raster Upload

  • 10cm High Resolution orthoimagery of San Juan, Puerto Rico
  • 4 GeoTIFF tiles on Google Cloud Storage bucket uploaded into single EE asset
  • Provider: USGS

earthengine upload image \� --asset_id=users/herwig/eeus/pr_hro_tiled \� --property="(string)provider=usgs" \� --time_start=2013-03-13T00:00:00 \� gs://herwig-export-tests/pr-hro/7_12.tif \� gs://herwig-export-tests/pr-hro/7_13.tif \� gs://herwig-export-tests/pr-hro/8_12.tif \� gs://herwig-export-tests/pr-hro/8_13.tif�

​

41 of 86

Try it out

var importedAsset =

ee.Image('users/herwig/eeus/pr_hro_tiled');�Map.setCenter(-66.090923, 18.375517, 16);�Map.addLayer(importedAsset);

42 of 86

Questions

43 of 86

Export table

44 of 86

Export table

You can export a FeatureCollection as CSV, SHP (shapefile), GeoJSON, KML, or KMZ using Export.table.

Format-specific constraints:

  • KML: output FeatureCollection will have all the geometries transformed to unprojected (WGS84) coordinates.

​

  • SHP: output FeatureCollection must contain features with the same geometry type and projection and must fit within the Shapefile size limits (2Gb). Column names are truncated to 10 characters or fewer, and this must not create duplicate column names.

​

45 of 86

Export image

46 of 86

Export image

  • Specify region to export or use native image bounds
  • Scale (can only specify one): nominal scale of desired bands
  • All bands must be of same pixelType - so cast to best fit before export.
  • Ex: Cast image of bands [int16 and int32] to int32.

47 of 86

Cloud-optimized GeoTIFF

You can specify image output format (if the destination is not toAsset()) with the fileFormat parameter (currently only the default, 'GeoTIFF', is enabled).

Other configuration options are set with the formatOptions parameter, which should be a dictionary keyed by other format options (currently only cloudOptimized, to export a cloud-optimized GeoTIFF, is enabled and is false by default).

// Export a cloud-optimized GeoTIFF.�Export.image.toDrive({� image: landsat,� description: 'imageToCOGeoTiffExample',� scale: 30,� region: geometry,� fileFormat: 'GeoTIFF',� formatOptions: {� cloudOptimized: true� }�});

​

48 of 86

Exporting images as they appear in the Code Editor

  • Map.addLayer(image, visParams);
  • The image.visualize() method converts an image into an 8-bit RGB image for display or export.
  • Image displayed in code editor is EPSG:3857.
  • See visualization parameters in docs.

​

49 of 86

RGB export in QGIS

50 of 86

Setting no-data = 0 in QGIS

51 of 86

Add image mask as fourth band for use in QGIS/other

​

var mask = denmark.mask()� .reduce(ee.Reducer.min())� .multiply(255).toByte();�denmark = denmark.addBands(mask);

52 of 86

RGBA export in QGIS

53 of 86

Export Image to Cloud Storage

To use Google Cloud Storage, you'll need to set up a project, enable billing for the project, and create a storage bucket.

See the Cloud Storage Quickstart page for instructions.

Data exported to a Cloud Storage bucket will have the bucket's default object Access Control List (ACL).

You must have write permission for the specified bucket.

// Export a cloud-optimized GeoTIFF.�Export.image.toDrive({� image: landsat,� description: 'imageToCOGeoTiffExample',� scale: 30,� region: geometry,� fileFormat: 'GeoTIFF',� formatOptions: {� cloudOptimized: true� }�});

​

54 of 86

Export Image to Earth Engine Asset

  • To export an image to an asset in your Earth Engine assets folder, use Export.image.toAsset().
  • Use the Asset Manager in the code editor to monitor your storage quota, and manage your Earth Engine assets.
  • New parameter: assetId
  • pyramidingPolicy

55 of 86

Test it out

var newAsset = ee.Image('users/herwig/eeus/denmark_landsat8')�Map.addLayer(newAsset);

​

56 of 86

Questions

57 of 86

Export video

58 of 86

Export video

59 of 86

Export Video to Drive

  • Export ordered image collections as video, where frames are defined by images in the collection
  • Specify frame rate, scale (or) dimensions.
  • MP4.
  • Images in ImageCollection must be 8-bit RGB images.
  • Can take a significant amount of time to complete.

60 of 86

var region = ee.Geometry.Point(

[151.213, -33.87]);�

var vis = {� min:500,� max:3000,� gamma:1.4,� bands: ['B4', 'B3', 'B2']�}

�var collection = ee.ImageCollection('COPERNICUS/S2')� .filterBounds(region)� .sort('system:time_start')� // Need to make the data 8-bit.� .map(function(image) {� return image.visualize(vis);� });

Export.video.toDrive({� collection: collection,� description: 'sydney-5fps',� dimensions: 1080,� framesPerSecond: 5,� region: region�});

61 of 86

Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Proprietary + Confidential

62 of 86

ffmpeg \

-i input.mp4 \

output.gif

63 of 86

ffmpeg \

-i timelapse.mp4 \

-f image2 \

-start_number 1984 \

image-%03d.jpg

64 of 86

65 of 86

Surface water occurrence - Pekel et al., JRC

66 of 86

Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem

Proprietary + Confidential

67 of 86

68 of 86

Google Confidential and Proprietary

69 of 86

Google Confidential and Proprietary

70 of 86

Google Confidential and Proprietary

71 of 86

Google Confidential and Proprietary

72 of 86

Export Video to Cloud Storage

  • To use Google Cloud Storage, you'll need to set up a project, enable billing for the project, and create a storage bucket.
  • See the Cloud Storage Quickstart page for instructions.

Export.video.toCloudStorage({� collection: collection,� description:'videoToGCS',� bucket: bucketName,� dimensions: 1080,� framesPerSecond: 1,� fileNamePrefix: 'myvideos/',� region: region�});

73 of 86

Export Map Tiles

74 of 86

Export Map Tiles

  • Export.map.toCloudStorage() exports map tiles suitable for display using the Google Maps API and within Google Earth
  • Exported image must have either 1 or 3-bands.
  • Output Map is
    • publicly viewable*
    • can be embedded on other pages
    • does not require the viewer to be a registered Earth Engine user.
  • User exporting must be writer or owner of bucket.

​

*Unless you choose not to write public tiles (writePublicTiles=false), in which case the tiles inherit the bucket’s default object ACL

75 of 86

Export Map Tiles

  • fileFormat: map tiles' format jpg/png/auto ["auto"]
  • path: The string used as the output tiles prefix[task description].
  • writePublicTiles: write public tiles or use bucket's default ACL. [true]
  • maxZoom: maximum zoom level tiles to export
  • scale: The max image resolution in meters per pixel, as an alternative to maxZoom
  • minZoom: min map tile level to export. [zero]
  • See scale and zoom docs

76 of 86

Export Map Tiles

77 of 86

Export Map Tiles

15/10367/14681

15/10367/14682

15/10368/14681

15/10368/14682

78 of 86

Export Map Tiles

var image = ee.Image("users/herwig/eeus/rocket_tiled");�Map.addLayer(image);��var southBounds = ee.Geometry.Polygon(� [-180, 85, 0, 85,� 179.998, 85, 179.998, -85,� 0, -85, -180, -85� ], null, false);��Export.map.toCloudStorage({� image: image,� bucket: 'herwig-export-tests',� description: 'export-rocket',� path: 'maps/20180118/rocket',� minZoom: 0,� maxZoom: 6,� region: southBounds,�});

79 of 86

Export Map Tiles

80 of 86

Export Map Tiles

81 of 86

It's a map!

82 of 86

Export Map Tiles

83 of 86

84 of 86

Share Earth Link

85 of 86

Cloud + Maps Platform Credits Programs

Cloud: Geo for Good Cloud Credits Program: g.co/earth/cloud-credits

Maps: Eligible nonprofits, startups, crisis response, and news media organizations may apply for Google Maps Platform credits through the following programs:

​

86 of 86