Machine Learning with Satellite Data

​

A blisteringly-fast crash course

Sean Foley

What is Machine Learning?

Linear Algebra

• The language used by the other fields
• Vectors, matrices, spaces
• Numerical linear algebra: how to go fast!

Signal Processing

• Signal vs. noise
• Mutual information / entropy
• Bits!

High-Dimensional Probability & Stats.

• If I throw one trillion twenty-sided dice…
• Surprisingly geometrical
• Some intuitive concepts break down in higher dimensions!

Optimization

• Make number go up!
• Iterative techniques
• Gradient-based methods
• Leverage hardware

Data Science

• Garbage in, garbage out
• 90% of the work is getting good data
• The most important part (in my opinion)

​

​

​

“The unreasonable effectiveness of data” - Peter Norvig

Inputs

Predictions

Targets / Labels

Supervised Learning

Unsupervised Learning

Reinforcement Learning

Loss Function

Model

Data

Method

Classes / clusters

Agent

Environment

Actions

• Observations

​

• Rewards

Supervised Learning

​

• Decision Trees
• Random Forests
• Support Vector Machines (SVM)

Unsupervised Learning

​

• K-Means Clustering
• Principal Component Analysis (PCA)
• Gaussian Mixture Models

Reinforcement Learning

• Neural Networks + Backpropagation

• Self-Supervised Learning

• Monte Carlo RL
• Deep Q-Learning
• Asynchronous Advantage Actor-Critic

Variants

• Semi-supervised learning
• Transfer learning
• Limited supervision
• Low-shot / few-shot
• Weakly supervised

Regression vs. Classification

P(x, y)

Inputs

Predictions

Targets / Labels

Supervised Learning

Loss Function

Model

Inputs

Predictions

Targets / Labels

Supervised Learning

Loss Function

Model

Regress chlorophyll-a

Slide credit: Patrick Gray

x

y

Loss

Classify phytoplankton types

phycoerythrin

chlorophyll-a

Prochlorococcus

Synechococcus

Prochlorococcus

Synechococcus

phycoerythrin

chlorophyll-a

Slide credit: Patrick Gray

Neural Network / Multi-Layer Perceptron (MLP)

Neural Network / MLP

Adding more layers?

Uh oh!!

Activation Functions

• Non-linear
• Historically:
• Sigmoid
• Tanh
• Modern:
• ReLU
• Leaky ReLU

​

Et voila…

Loss Function

• Differentiable
• Aligns with evaluation metrics
• e.g.
• Mean Squared Error (L2 loss) <- regression

​

​

• Cross Entropy Loss <- classification

W₁

W₂

Backpropagation

Chain rule!

Loss Function

Gradient Descent

W₁

W₂

Gradient Descent

• Step size
• Stochastic Gradient Descent (SGD)
• Other Gradient-based methods
• Adam: most commonly-used

Evaluation Metrics (Classification)

Evaluation Metrics (Regression)

• Mean Absolute Error (MAE) aka L1 Loss
• Mean Squared Error (MSE) aka L2 Loss

Instances, Batches, Epochs

• Instance: one sample
• Batch: bundle of instances
• Batch size!!
• Stochasticity
• Convergence speed
• Epochs
• Early stopping

Training, Validation, Testing

• Generalizability!
• Train: parameters
• Validation: hyperparameters
• Testing: nothing!

Training / validation script overview

Over/under fitting, revisited

underfitting

overfitting

Graphics Processing Units (GPUs)

Optimized for large matrix operations

High-level code

NVIDIA Graphics Card

CUDA

Satellite

Imagery

​

Spatial context!

​

Translational symmetry!

Convolution

1D, continuous convolution shown here

flip g!

2D discrete convolution

Image Cubes

RGB

OCI

Height

Width

3

Height

Width

~280

A Convolutional Layer: Basics

• Fully-connected in the channel dimension
• Sliding window in the spatial dimensions

A Convolutional Layer: More Details

• bias
• activation function
• differentiable

​

# parameters:

​

​

doesn’t depend on image size!!

A Convolutional Layer: Padding

Receptive Field

An Example CNN

Very cool interactive demo:

https://adamharley.com/nn_vis/cnn/2d.html

​

Data Augmentation

CNNs + Satellite Data: Caveats

Most CNNs are NOT designed for satellite data.

Especially not for Earth Science data!

• vast scale
• not multi-scale
• # channels

Resources / Credit

• Cheat sheets on Stanford AI / ML courses (many figures in this presentation)
• Figures from pytorch docs
• Amazing animations from this article
• Interactive CNN demo
• Many figures from Wikipedia