1 of 27

Cloud Native Geospatial Workflows with QGIS

Ujaval Gandhi

ujaval@spatialthoughts.com

Learn the modern approach to working with large datasets in QGIS.

​

CC BY 4.0

2 of 27

Spatial Thoughts

  • A learning platform for modern geospatial technologies.
  • Free and open learning content for QGIS, Python, GDAL and Google Earth Engine.
    • Learning materials used by 1 million+ users every year.
    • All material is licensed under CC-BY
  • Online academy for cohort-based instructor-led classes.
    • Trained participants from over 150 countries.

✅ QGIS.org certified training provider

✅ QGIS.org sustaining member

3 of 27

Introduction

  • Background in GIS and Remote Sensing
    • Intern at Indian Institute of Remote Sensing (IIRS), Dehradun, India
    • MS in Geospatial Information Engineering from University of Wisconsin - Madison, USA
  • 15 years of professional experience
    • One of the early employees at Google Inc.
      • Moved to India in 2006 and established the maps team
      • Migrated internal teams to QGIS
      • Led the GIS and Aerial Imagery team in India from 2007-2019
    • Developed expertise in Google Earth Engine and trained 1000+ scientists and researchers across India
  • Presently
    • Left Google in 2020 to work on startup Spatial Thoughts.
    • Building open-source content and helping people master advanced geospatial skills.

Ujaval Gandhi

4 of 27

Cloud Native Geospatial Workflows with QGIS

This workshop will teach you the modern approach to working with large datasets in QGIS.

  • Introduction to Cloud Native Geospatial
    • Cloud Native Geospatial Formats
    • Viewing a Cloud-optimized GeoTIFF (COG)
    • Viewing a FlatGeoBuf (FGB) layer
    • Introduction to STAC
  • Application 1: Calculate Zonal Statistics
  • Application 2: Calculate LandCover Change
  • Supplement: Create a Sentinel-2 RGB Composite

bit.ly/cloud-native-qgis

This slide-deck

5 of 27

What is Cloud Native Geospatial?

  • A new way of working with spatial data that is centered around a set of new data standards for cloud-based access.
  • Use of Cloud-optimized Data Formats that allow querying, viewing and processing data without downloading entire files.
  • Streaming data from files without need of any specialized server.
  • Leveraging ecosystem of open-source tools to share and use large data sets in a standardized way.
  • Community-driven bottom-up standards.

6 of 27

Cloud Native Geospatial Formats*

  • Raster
    • Cloud Optimized GeoTIFF (COG)
    • Zarr
  • Vector
    • FlatGeoBuf (FGB)
    • GeoParquet **
    • PMTiles
  • Point Clouds
    • COPC: Cloud Optimized Point Cloud (COPC)

* Cloud-Optimized Geospatial Formats Guide

** Why GeoParquet Is Not A Cloud Native Format

​

7 of 27

Cloud Optimized GeoTIFF (COG)

8 of 27

Cloud Optimized GeoTIFF (COG)

  • A new format that makes it very efficient to access raster data over the internet.
    • Instead of downloading the entire image locally, you can access portions of imagery hosted on a cloud server.
    • Use and analyze data - without downloading large files.
  • The format is backwards compatible. All COG files work with any software that understands GeoTIFF files.
  • COG format is widely supported by many software - including QGIS.
  • Highly recommended format for distributing raster data.

9 of 27

FlatGeoBuf (FGB)

10 of 27

FlatGeoBuf (FGB)

  • A vector data format that uses the same streaming mechanism used by COGs.
  • Allows fast filtering of features using a bounding box.
  • Fully supported by QGIS.
  • Lossless format that keeps the original geometries as it is.
    • Does not contain simplified versions of features.
    • Requesting features from a large bounding box can be slow as it will load all the features at full resolution.
  • Recommended for distributing vector data in a cloud-native format.

11 of 27

Let’s get started

  1. Get access to this slide-deck. Go to the bitly link shown on the right.
  2. Visit the Workshop Page.

>> courses.spatialthoughts.com/qgis-cloud-native-geospatial.html <<

  • Download the Data Package.

​

​

bit.ly/cloud-native-qgis

This slide-deck

12 of 27

SpatialTemporal Asset Catalogs (STAC)

13 of 27

What is STAC?

  • STAC is a standardized way to expose collections of spatial temporal data.
  • Allows data providers to have a simple way to describe their data
    • Supports all types of geospatial data (satellite imagery, gridded rasters, vector ,point clouds etc.)
    • Encouraged to use cloud-native data formats
  • A rich ecosystem of open-source tools to work with data available through STAC.
  • Users can query and find the data from any STAC catalog using a standard query language.

14 of 27

STAC Catalogs

  • Anyone can publish their data as a STAC Catalog.
  • There are two main types of catalogs
    • Static Catalogs
    • API
  • Static catalogs are very simple to create.
    • Add a JSON file next to your data and your data is discoverage and usable via a wide array to tools
  • STAC API Catalogs need server and a database.
    • Suited for large dynamic catalogs
    • Allows one to query for matching items

15 of 27

Let’s look at some STAC Catalogs

https://stacindex.org/

16 of 27

Creating Cloud-Optimized Data

17 of 27

Creating COGs

  • GDAL makes it very easy to create COG files by specifying the -of COG

gdal_translate -of COG viirs_ntl_2021_global.tif viirs_ntl_2021_global_cog.tif

  • COG Driver has following options by default
    • Has internal tiling (i.e. -co TILED=YES)
    • LZW Compression (i.e. -co COMPRESS=LZW)
    • Automatic Selection of Predictor (i.e. -co PREDICTOR=YES chooses appropriate predictor for data type)
  • Specifying the -co NUM_THREADS=ALL_CPUS helps speed up the creation process by using all available CPUs for compression and creating internal overviews.

​

18 of 27

Creating FGB

  • OGR makes it very easy to create COG files by specifying -f FlatGeobuf

ogr2ogr -f FlatGeobuf hydrorivers.fgb hydrorivers.gpkg

  • You can also specify the subset of columns using -select

ogr2ogr -f FlatGeobuf admin2.fgb admin2.gpkg \

-select "adm2_name,adm1_name,adm0_name,iso_2,iso_3"

19 of 27

Hosting files on Cloud Storage

20 of 27

What do we mean by ‘Cloud Storage’?

  • Files that are accessible via internet via HTTP.
  • Internet-based storage system that allows direct access to files via a URL.
  • Also known as ‘dumb storage’
    • Cheap to store and serve large files
    • Typically read-only
    • Supports ‘HTTP Range requests’ - ask for a portion of the file
    • Can be read by many users in parallel

21 of 27

What do we mean by ‘Cloud Storage’?

✅ Cloud Object Stores

  • Google Cloud Storage
  • Microsoft Azure Blob Storage
  • Amazon S3
  • Cloudflare R2
  • …

✅ GitHub

  • Github Releases (Unlimited free storage for files upto 2GB)

✅ Any Simple HTTP Server

​

​

❌Consumer Storage Systems

They do not allow direct HTTP access to files

​

  • Google Drive
  • Microsoft OneDrive
  • Dropbox

❌ Databases

​

​

22 of 27

Quiz

23 of 27

Cloud-Native Geospatial Quiz

24 of 27

Resources

25 of 27

Cloud Native Formats Support in QGIS

  • Cloud data provider in QGIS Browser [Coming in QGIS 3.40]
  • STAC client in QGIS core [QGIS Enhancement Proposal]
  • Geoparquet Support
    • Available in GDAL >= 3.5
    • Supported on QGIS Windows and Linux. Mac support via Conda.
  • PMTiles
    • Available in GDAL >= 3.8
    • https://r2-public.protomaps.com/protomaps-sample-datasets/cb_2018_us_zcta510_500k.pmtiles

​

​

26 of 27

Resources

Cloud Native Geospatial Foundation

Postholer.com

  • Cloud Native Maps

​

​

​

27 of 27

https://spatialthoughts.com/2024/02/26/gdal-google-cloud-storage/