Geo for Good Summit Sept 25th- What’s Happening Now?

Embedding Fields: Accelerate Your Mapping & Monitoring Workflows with Geospatial AI

Room: Experiment

Operational field data collection with Ground

Room: Carina Nebula

Profiling Scripts and Advanced Debugging

Room: Birr Castle

1:45-2:45pm

2:45-3pm

Short break - time to walk to your next session!

3-4pm

Embedding Fields Classification with Open Architectures

Room: Experiment

Monitoring Land Use Land Cover Changes with Dynamic World

Room: Carina Nebula

Demystifying Data Export and Extraction

Room: Birr Castle

4-4:30pm

Coffee break outside Experiment

4:30-5pm

Hackathon Kickoff in Experiment

Office Hours - meet in Hubble

​

By appointment only

You are here

Embedding Fields Classification on

Open Architecture

​

​

Sean Wohltman, Kel Markert, Jeremy Malczyk, Gennadii Donchyts

September 2024 | #GeoForGood24

Repo

http://tiny.cc/efcoa

Embedding Fields Classification on

Open Architecture

​

Sean Wohltman, Kel Markert

September 2024 | #GeoForGood24

Repo

http://tiny.cc/efcoa

Agenda

01

​

​

​

​

​

02

​

​

​

03

EFM on Earth Engine

What it is and why we think it changes everything

How EE makes classification “easy” on top of a dataset like this

​

Going outside the sandbox

Model- as-Data for use cases beyond Earth Engine

​

Exporting EFM data

How to get EFM data into storage you can use outside of EE

A Desktop / non-scaled Approach

Frameworks / tools you can use in place of EE to run classifications on compute you manage

​

​

Running Classification at Scale

How you can leverage Kubernetes and Dask to classify at scale

​

​

04

​

​

​

​

​

05

​

​

​

​

​

#GeoForGood24

Embedding Fields Model on Earth Engine

Optional eyebrow

#3: Using FM embeddings directly

Multi-purpose remote sensing embeddings without specialization

General Patterns for Geospatial Foundation Models (GeoFMs); across a range of users, task types and domains

Foundational Model

Semantic Segmentation

Object detection

Change detection

Task type X

Agri landscape

Buildings & roads

Tree species

Oil, Gas, Infra

Disaster impact

Urban features

Task A

Task B

#2: Train open vocabulary models�Build models that specialize in a type of task, but for any class

#1: Fine tune task specific models�Models that Specialize in very specific task and classes.

Proprietary + Confidential

How does fine tuning work in practice?

Training a typical supervised remote sensing model

Large set of labeled images* (~10K-1M)

Model training with a supervised objective

A task specific model

* In this example we use images as the basic training data for simplicity. In practice many other data sources like weather, DSMs, structured data could be used and labeled data might be at the pixel level (for segmentation) or language (for VQA etc.).

Proprietary + Confidential

Fine tuning an FM for a specific task

First train (once) a foundational model…

Small set of labeled images* (10-1K)

… then fine tune it to (many) specific tasks

Task specific layers

Fine tuned layers

Task specific model

FM Fine tuning

* In this example we use images as the basic training data for simplicity. In practice many other data sources like weather, DSMs, structured data could be used and labeled data might be at the pixel level (for segmentation) or language (for VQA etc.).

Proprietary + Confidential

130+ enterprise-ready foundation models in Vertex AI Model Garden

• Choice and flexibility with Google, open source, and third-party foundation models
• Multiple modalities to match every use case
• Multiple model sizes to match cost and efficacy needs
• Domain-specific models for specialized industries
• Enterprise ready with safety, security, and responsibility
• Decrease time to value with fully integrated platform

Google Foundation Models

Google Task Specific Models

Google Domain Specific Models

Partner & Open Ecosystem

Imagen 2

PaLM 2

Speech-to-Text

Text-to-Speech

Natural Language

Translation

Vision

Video Intelligence

Doc AI OCR

Occupancy analytics

MedLM

Life Science and Healthcare

Sec-PaLM

Cybersecurity

Llama 2�Code Llama

Falcon

Claude 2

Pre-announce

Vertex AI Model Garden

Codey

Chirp

Embeddings

Gemini Foundation Models

A Vision Transformer (ViT)

Proprietary + Confidential

segment-geospatial

Proprietary + Confidential

Earth Engine Timelapse

sam2 Segmented

Proprietary + Confidential

Proprietary + Confidential

Pros to the model checkpoint approach

Optional eyebrow

Generalization:

Model checkpoints typically reflect an inflection point in training before you reach diminishing returns, across a wide field of intended tasks. Making them a good foundation for zero-shot attempts at those tasks, and also a good starting point for fine-tuning.

​

Portability:

Typically these can be single files of a few gigabytes - that can fit in modern GPU RAM.

​

Flexibility:

Can be easily spread across numerous workers / GPUs to support high-volume inference and classification.

Cons to the model checkpoint approach

Optional eyebrow

Everyone does the same thing with it:

You can imagine many users classifying the same exact Sentinel 2 scene for instance, using the same exact model checkpoint, getting the same result. This is pretty wasteful!

​

You need to bring it data:

The checkpoint is just one component, you will need to feed it input data (e.g. a new Sentinel 2 granule) to produce anything. Do you have a library of all the data you want to classify offline with you?

​

Temporal Context: Very hard to capture both spatial and temporal context in a checkpoint, especially globally.

​

Cost:

When you scale an inference or classification workload to many workers / GPUs, each one of them has to load the checkpoint into RAM, which can get very costly, very quickly!

​

Embedding Fields

​

Serving a model as data with the power of deep learning packed into every pixel

Bringing an EO paradigm shift to EE that only Alphabet can do

​

​

RGB

R

G

B

B34

B05

B06

64-dimensional embedding space

Embedding Fields

​

​

The portability of Model as Data

Optional eyebrow

Exporting EFM

Optional eyebrow

Why use Zarr?

Zarr is a relatively new cloud-based, self-describing data format specifically for N-Dimensional arrays with predefined chunks

​

“Use the right data model appropriate for your task”

Xarray

Earth Engine

Xee

Data Extraction

Wraps computePixels within Xarray paradigm to take advantage of automatic chunking, retries, and merging of data

Plays nicely with other GCP compute services like Dataflow and Dataproc

Exporting data to Zarr format

Used xarray-beam and xee with Dataflow to handle parallel request to EE and writing in parallel to Zarr

​

A Desktop or non-scaled Approach

Optional eyebrow

k-NN Regression / Classification

Offline with

Open Source Software

jupyter / ipyleaflet / scikit-learn

Notebook

Create a Slippy Map with ipyleaflet and a TileLayer

Digitize the first Class

Class 3 …

Class 2 …

Create a GeoDataFrame

Show the 3 Classes on the Map

Select a subset of the Zarr

Spatial Join Training Points to the Xarray DataSet

Train a KNN Classifier from the training points and their intersected Embedding Field values and check the accuracy.

Now use that model to predict the classes of all of the points

and plot the classified map

But, this is really points...

To do this on the DataSet, it needs to be turned into an array and transposed

Then, the transposed array needs to be sliced along the time dimension

Next, create predictions using the previously trained model (knn)

The array with predicted classes can now be plotted

To export as a GeoTIFF, reshape the array

Then use rioxarray to convert array to raster and write as GeoTIFF

Running Classification at Scale

Optional eyebrow

Borg → Kubernetes

​

“So let me get this straight. You want to build an external version of the Borg task scheduler. One of our most important competitive advantages. The one we don’t even talk about externally. And, on top of that, you want to open source it?”

– Urs Hölzle

Kubernetes is winning container orchestration

Google Search Trends keyword search 25-Oct-15 to 25-Oct-18

Dask on Google Kubernetes Engine (GKE)

GKE, Kubernetes-as-a-service

kubectl helm

gcloud

​

• Spin up a Kubernetes cluster on GKE with Dask scheduler and worker pods
• Use kubectl command to control the cluster
• Use helm command to instal workloads on the cluster
• Execute parallelized python code on that cluster

​

This notebook installs Dask and will multithread locally

It will create a local cluster and Dask client

Call dask_ml KMeans to create 10 classes with unsupervised classification

When the predictions are completed, add them to the DataArray and unstack

Using the Embedding Fields with unsupervised classification

gives us 10 very compelling classes!

Scaling Up and Out with Dask on GKE

https://kubernetes.io/

​

https://cloud.google.com/kubernetes-engine

https://docs.dask.org/en/stable/

Geo for Good Summit Sept 25th- What’s Happening Now?

Embedding Fields: Accelerate Your Mapping & Monitoring Workflows with Geospatial AI

Room: Experiment

Operational field data collection with Ground

Room: Carina Nebula

Profiling Scripts and Advanced Debugging

Room: Birr Castle

1:45-2:45pm

2:45-3pm

Short break - time to walk to your next session!

3-4pm

Embedding Fields Classification with Open Architectures

Room: Experiment

Monitoring Land Use Land Cover Changes with Dynamic World

Room: Carina Nebula

Demystifying Data Export and Extraction

Room: Birr Castle

4-4:30pm

Coffee break outside Experiment

4:30-5pm

Hackathon Kickoff in Experiment

Office Hours - meet in Hubble

​

By appointment only

Thank you!

Image set-up:

​

• Set slide background color to black
• Add image
• Right-click image
• Select ‘format options’
• In side panel, adjust transparency slider in adjustments section to 50%

​

#GeoForGood24