1 of 1

Cloudy with a Chance of Labels

Understanding Satellite Images with Advanced Deep Learning

Norman Bukingolts

nbukingolts@ufl.edu

Introduction

Research Question

Datasets

Approach 1: CNN & CycleGAN

Approach 2: CLIP

Results & Evaluation

Key Takeaways

References

This research explores two complementary approaches to satellite image understanding in scenarios like disaster response, climate monitoring, and land use classification:

  • Multi-label classification using CNN and �CycleGAN architectures,
  • Image captioning and semantic search using CLIP-based models that bridge visual and textual representations via multimodal learning.

A VGG-style CNN architecture [3, 4] was implemented for multi-label classification of satellite imagery, featuring repeating blocks of convolutional layers with small 3×3 filters followed by max pooling.

To address class imbalance, a CycleGAN�was developed to augment the dataset by generating synthetic images through unpaired image-to-image translation [5], particularly for the "cloudy" label.

The CNN achieved strong performance with a validation accuracy of 0.95 and an F2 score of 0.81 after 15 epochs. Saliency maps revealed that the model correctly focused on relevant image features (e.g., edges of clouds).

  1. Deep learning models can be beneficial in efforts to automate satellite imagery analysis.
  2. Generative and multimodal approaches can help address data imbalance and search vast datasets.
  3. Future work could explore training with multi-spectral satellite imagery (4+ bands instead of only RGB) to enhance model performance.

A contrastive image-language pre-training (CLIP) architecture [6] was implemented for image captioning, initially fine-tuning OpenAI's pre-trained encoders. This foundation was enhanced by replacing the original encoders with domain-specific alternatives [7, 8, 9] along with a learnable temperature parameter to optimize contrast between image-text pairs, improving captioning and enabling semantic search through a vector database.

Scan the QR code to see the full list of the references used in this poster and links to associated reports.

The CLIP model initially achieved modest validation metrics of BLEU: 0.40 and CIDEr: 2.70, while the enhanced architecture more than doubled performance to CIDEr: 5.79 and BLEU: 0.60, with strong test results confirming better generalization.

The "Planet: Understanding the Amazon from Space" dataset [1, left] contains 40,478 Amazon rainforest satellite images with 17 labels. The RSICD dataset [2, right] includes 10,921 satellite images, each with five descriptive sentences.

How effectively can deep learning algorithms interpret satellite imagery, and how can their validity for automated remote sensing analysis be demonstrated?