1 of 35

Landsat at 3 m resolution?!? Applying a super resolution model to surface water detection

Ethan D. Kyzivat and Laurence C. Smith

ethan_kyzivat@brown.edu

@EthanKyzivat

2 of 35

3 of 35

Generative Adversarial Networks (GANs)

Adam Harley https://www.cs.cmu.edu/~aharley/vis/conv/

https://thiscatdoesnotexist.com/

https://developers.google.com/machine-learning/gan/gan_structure

Composed of two models in opposition: generator (generates images) and discriminator (detects if images are real).

GANs better than CNN’s for image sharpness and “perceptual” quality (Wang et al. 2022)
SR for remote sensing: cross-sensor training

ethan_kyzivat@brown.edu 3

@EthanKyzivat

BACKGROUND

4 of 35

Here, we apply an SR model trained entirely on Planet data to Landsat:

Can super resolution improve detection of small lakes and ponds?

How well can historical Landsat scenes be super-resolved?

Research Questions

@EthanKyzivat

QUESTIONS

ethan_kyzivat@brown.edu 4

In recent years, thanks to the efforts of companies like Planet, there has been a huge increase in the availability of high resolution daily (~3 m) imagery. Research in northern Canada, in particular, has shown that using high resolution imagery is integral for capturing fine-scale, short-term fluctuations in water bodies, especially along shorelines.

However, the availability of this high-resolution data is really a recent phenomenon. There is a question of, what can we do when this high-resolution data isn’t available? How can we make use of the coarser resolution imagery, such as Sentinel or Landsat, that is really integral to earth science?

This brings us to the idea of super resolution, which means essentially upsampling coarse imagery to a higher resolution. Recent papers have shown that super resolution can improve object identification in satellite imagery. But, we are interested in water.

So, the goal for this project was to determine whether we could use satellite super resolution to improve landcover classification in satellite data.

Notes on lit review.

5 of 35

Study domain chosen from 2017 airborne dataset (Kyzivat et al. 2019) from NASA ABoVE
Historical domain based on 1985 Fairbanks lake map (Walter Anthony and Lindgren 2021)

Spatial domain

@EthanKyzivat

METHODS

ethan_kyzivat@brown.edu 5

6 of 35

Pre-processing and classification

@EthanKyzivat

METHODS

16-bit integer, scaled to surface reflectance

14 Landsat-8, one Landsat-5 scenes from Collection 2

Run SR model ESRGAN (Wang et al. 2018)

Reconstructed SR scenes

Simple thresholding water classifier, remove rivers

Stretch images, convert to 8-bit integer, tile

Image chips

Polygonize, compute overlaps and accuracy metrics

ethan_kyzivat@brown.edu 6

Model training: 189 Planet scenes with spatially degraded pairs

Lezine, Kyzivat & Smith, 2021

7 of 35

Smoothing seamlines via overlapping tiles

@EthanKyzivat

Overlapping tiles

Tiled

Problem of edge effects

METHODS

(NASA DELTA package)

ethan_kyzivat@brown.edu 7

8 of 35

@EthanKyzivat

Use radial Gaussian kernel to weight contributions of each pixel

Weighted images are combined and normalized by sum of kernels (above)

Tiled

Overlapping tiles

Smoothing seamlines via overlapping tiles

METHODS

ethan_kyzivat@brown.edu 8

In recent years, thanks to the efforts of companies like Planet, there has been a huge increase in the availability of high resolution daily (~3 m) imagery. Research in northern Canada, in particular, has shown that using high resolution imagery is integral for capturing fine-scale, short-term fluctuations in water bodies, especially along shorelines.

However, the availability of this high-resolution data is really a recent phenomenon. There is a question of, what can we do when this high-resolution data isn’t available? How can we make use of the coarser resolution imagery, such as Sentinel or Landsat, that is really integral to earth science?

This brings us to the idea of super resolution, which means essentially upsampling coarse imagery to a higher resolution. Recent papers have shown that super resolution can improve object identification in satellite imagery. But, we are interested in water.

So, the goal for this project was to determine whether we could use satellite super resolution to improve landcover classification in satellite data.

Notes on lit review.

9 of 35

Results

10 of 35

Landsat 5

Kyzivat and Smith, submitted

19 of 35

Lake location

@EthanKyzivat

RESULTS

Airborne

LR

SR

Shorelines

Kyzivat and Smith, submitted

ethan_kyzivat@brown.edu 19

20 of 35

Lake location

@EthanKyzivat

RESULTS

Airborne

LR

SR

Shorelines

Kyzivat and Smith, submitted

ethan_kyzivat@brown.edu 20

21 of 35

Some errors from changing cloud conditions (0-5% cloud cover)

Lake placement

@EthanKyzivat

RESULTS

Airborne

LR

SR

Shorelines

Kyzivat and Smith, submitted

ethan_kyzivat@brown.edu 21

22 of 35

We propose an SR Minimum Mapping Unit (MMU) of ~500 m²
SR lakes: lesser precision (more Type I error), but greater recall and F-1 score

Lake object detection

RESULTS

Kyzivat and Smith, submitted

Mean F-1 score:

0.75

vs.

0.73

23 of 35

Lake object detection

RESULTS

SR lakes have a remarkably accurate size distribution (including false positives)
Resolution improvements do not improve power law modeling

25,990

vs.

25,281

Landsat 5

62

vs.

242

Kyzivat and Smith, submitted

24 of 35

Can super resolution improve detection of small lakes and ponds?
Slightly improves the detection limit for lakes to ~2/3 Landsat pixel
Spurious lakes and “realistic-looking” errors in shape cause ethical challenges
How well can historical Landsat scenes be super-resolved?
SR “works” for Landsat 5, but not all lakes can be detected
Challenged by coarser radiometric resolution

Conclusions

@EthanKyzivat

CONCLUSIONS

ethan_kyzivat@brown.edu 24

In recent years, thanks to the efforts of companies like Planet, there has been a huge increase in the availability of high resolution daily (~3 m) imagery. Research in northern Canada, in particular, has shown that using high resolution imagery is integral for capturing fine-scale, short-term fluctuations in water bodies, especially along shorelines.

However, the availability of this high-resolution data is really a recent phenomenon. There is a question of, what can we do when this high-resolution data isn’t available? How can we make use of the coarser resolution imagery, such as Sentinel or Landsat, that is really integral to earth science?

This brings us to the idea of super resolution, which means essentially upsampling coarse imagery to a higher resolution. Recent papers have shown that super resolution can improve object identification in satellite imagery. But, we are interested in water.

So, the goal for this project was to determine whether we could use satellite super resolution to improve landcover classification in satellite data.

Notes on lit review.

25 of 35

Questions?

ethan_kyzivat@brown.edu

@ethankyzivat

Super resolution model code and example input/output satellite data are available at Zenodo.

Download this presentation at

ethan.kyziv.at/AGU22

Thank you to Ekaterina Lezine for assistance with slides; Planet-NASA CSDA partnership for data access.

Funding: NASA FINESST fellowship, Google Cloud Platform and Planet Education and Research Program.

26 of 35

Extra slides

27 of 35

28 of 35

289 training Planet scenes

29 of 35

bullet

Title

ethan_kyzivat@brown.edu

@EthanKyzivat

30 of 35

Pre-processing Planet images

Mean histogram for all training images with original digital number values

Example image after quantizing and converting to 8-bit. As a result, bright land values are saturated. (Band order in plot: R/G/NIR)

ethan_kyzivat@brown.edu

@EthanKyzivat

31 of 35

Enhanced Super Resolution Generative Adversarial Network (ESRGAN)

(Wang, 2018; https://github.com/xinntao/ESRGAN)

In brief: two models (a generator and a discriminator) compete against each other, learning how to generate the most accurate images.

24

28

32

PSNR

Iterations (thousands)

10x

4x

Plot from wandb.ai

1

2

3

4

32 of 35

Canadian Shield study area

Islands in the wrong location, wrong shape

33 of 35

In press, Canadian J. of Remote Sensing

Low-Res (30m)

High-Res (3m)

Cubic (3m)

Super-Res (3m)

Image Comparison

High-Res (orig. image, 3m)

Low-Res (30m)

Super-Res (3m)

Cubic Resampling (3m)

ethan_kyzivat@brown.edu

@EthanKyzivat

34 of 35

Evolution of one image during training

70,000

2,500

137,500

Number of iterations

Original

ethan_kyzivat@brown.edu

@EthanKyzivat

35 of 35

Image resampling

Adam Harley https://www.cs.cmu.edu/~aharley/vis/conv/

Wikimedia commons / Cmglee

High-resolution

Low-resolution

Cubic resampling

In recent years, thanks to the efforts of companies like Planet, there has been a huge increase in the availability of high resolution daily (~3 m) imagery. Research in northern Canada, in particular, has shown that using high resolution imagery is integral for capturing fine-scale, short-term fluctuations in water bodies, especially along shorelines.

However, the availability of this high-resolution data is really a recent phenomenon. There is a question of, what can we do when this high-resolution data isn’t available? How can we make use of the coarser resolution imagery, such as Sentinel or Landsat, that is really integral to earth science?

This brings us to the idea of super resolution, which means essentially upsampling coarse imagery to a higher resolution. Recent papers have shown that super resolution can improve object identification in satellite imagery. But, we are interested in water.

So, the goal for this project was to determine whether we could use satellite super resolution to improve landcover classification in satellite data.

Notes on lit review.