1 of 15

Comprehensive Transformer-based Model Architecture for Real-World Storm Prediction

¹University of Delaware, ²University of Louisiana at Lafayette

³Intel Corporation, ⁴Tulane University

Fudong Lin¹, Xu Yuan¹, Yihe Zhang², Purushottam Sigdel³,

Li Chen², Lu Peng⁴, Nian-Feng Tzeng²

2 of 15

Significance of Storm Predictions

Timely and precise storm prediction can provide an early alert for preparation, avoiding potential damage to property and human safety.

Unexpected storm events (e.g., hurricanes) cause significant damage to human property and health every year.

Hurricane Katrina resulted in 1,392 fatalities and caused property damage estimated between $97.4 billion to $145.5 billion in late August 2005.

Hurricane Katrina (August 2005)

3 of 15

Existing Solutions for Storm Predictions

Conventional Physical Models

In this paper, our goal is to develop a DL-based model, specifically using Transformers, to enhance storm prediction performance while minimizing computational resource requirements.

Superb performance but incurring excessive computational overhead

Deep Learning (DL)-based Models

Computationally efficient but achieving unsatisfactory performance

4 of 15

Background: Vision Transformers (ViT)

Advantage

Scalability to large model and data size
Global representation
Connection of visual and textual data

Limitation

Computational Overhead
Overfitting

Figure Credit: Alexey, Dosovitskiy, et. al. “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, ICCV 2021

ViT Model Architecture

To achieve our goal, we develop a comprehensive Transformer-based model for storm predictions, which is built on top of Vision Transformers (ViT) and Masked Auto-Encoders (MAE).

Here, we introduce the background information of Vision Transformers.

ViT starts by dividing an image into multiple image patches.

Then, the linear projection layer is used to project image patches into patch embeddings.

Next, ViT appends a learnable classification token in the input sequence.

After adding the positional embedding, the input sequence is fed into a stack of Transformer Encoders, with the head of the output sequence fed into an MLP for classification.

The right figure shows the architecture within the Transformer Encoder, which includes a Multi-head Self-Attention (MSA) block, an MLP block, and two layer normalization layers.

ViT is a powerful powerful surrogate for CNNs due to its scalability to large model sizes and datasets, providing global representation, and better connection of visual and textual data.

However, ViT is also infamous for its memory and computational overhead as well as its overfitting issue.

5 of 15

Background: Masked Auto-Encoder (MAE)

MAE is effective for learning visual representation without human-supervision by reconstructing images from the masked inputs.

Figure Credit: Kaiming, He, et. al. “Masked Autoencoders Are Scalable Vision Learners”, CVPR 2022

Encoder:

Consider a small subset of visible image patches
No positional embeddings required

Decoder:

Consider a full set of image patches, including both visible patches and masked tokens
Require positional embeddings

MAE Model Architecture

Masked Auto-Encoder (MAE) has lately been developed for addressing the overfitting issue of ViT by developing a novel self-supervised learning method.

Its key idea is to learn visual representation via reconstructing images from the masked inputs.

Here, we present the details of Masked Auto-Encoder (MAE), where its encoder and decoder use the ViT as the backbone networks.

Given an image, MAE randomly masks a large subset (e.g., 75%) of image patches.

Next, the Encoder only takes the small subset of visible patches as the input.

After the encoding, a full set of image patches, including both visible patches and masked tokens, are fed into the Decoder for reconstruction.

Note that the positional embeddings are used only on the input sequence for the Decoder. No positional embeddings are required on the Encoder.

As such, MAE is effective for learning visual representation without human supervision.

6 of 15

Background: SEVIR Dataset

Details of SEVIR Dataset

It contains a collection of sensor images captured by satellite and radar, characterizing weather events during 2017-2019.
It consists of 10180 normal events and 2559 storm events.
Four types of images with different resolutions are considered.

Illustration of Four Types of Sensor Data

Description of the SEVIR Dataset

Mark Veillette, et. al. “SEVIR : A Storm Event Imagery Dataset for Deep Learning Applications in Radar and Satellite Meteorology”, NeurIPS 2020

7 of 15

Challenge

Three challenges prevent researchers from using the SEVIR for storm predictions:

Limited Observational Samples: The storm events only account for 20% of total events (2559 storm events v.s. 10180 normal events).
Intangible Pattern: Weather images usually include erratic and intangible shapes.
Multi-scale Data: The resolutions of weather images vary from 192x192 pixels to 768x768 pixels.

8 of 15

Our Design: Model Overview

Three MAE encoders with different scales to learn high-quality visual representations from multi-scale solutions
Temporal representation for incorporating domain knowledge into deep learning model for storm predictions
Representation concatenation for addressing the limited observational storm events by considering visual and temporal representation simultaneously

Our Model Architecture

Insights Underlying Our Design

To tackle the three challenges, we proposed a comprehensive Transformer-based model, using ViT and MAE, for storm predictions.

The Figure illustrates the overview of our proposed model architecture for storm predictions, consisting of two stages, i.e., representation learning and prediction.

The representation learning stage involves three different scales of transformer encoders trained by MAE encoders for extracting image representation from four types of sensor images.

The Word2vec tool is used to extract the temporal representation of weather events.

The pooling layer and the linear projection layer serve to bridge the two stages by matching the input sequence length and the hidden vector dimension, respectively.

This design decouples the two stages, enhancing our model’s ability to scale up for larger model sizes and datasets.

The prediction stage derives the input sequence from a weather event by concatenating its comprehensive image representation extracted from four types of sensor images and its temporal representation extracted from the time description of that event.

A learnable classification token is fed to the multi-layer perceptron (MLP) for storm predictions.

Next, we summarize the insights underlying our design in the right.

First, the three MAE encoders with different scales empower our work to learn high-quality visual representations from multi-scale solutions.

Second, the Temporal representation allows our work to incorporate domain knowledge into deep learning models for storm predictions.

Third, the representation concatenation is used for addressing the limited observational storm events by considering visual and temporal representation simultaneously.

9 of 15

Our Design: Content Embedding

Our Model Architecture

Content Embedding

Intuitions Underlying Our Content Embedding

Devised for differentiating the membership of data sources
MAE Encoders: Only the small-scale MAE with the content embedding
ViT Encoder: Including Five types of content embeddings (4 for sensor data and 1 for temporal data)

10 of 15

Experiment: Overall Performance

Settings: Binary Classification, i.e., normal events or storm events
Baselines: ResNet-50 and ViT-Base
Metrics: Precision, Recall, F1-Score, and Accuracy

Our approach outperforms two baselines under all scenarios, with an overall accuracy of 94.4 % and an F1-Score of 85.0% on storm events.

Overall Performance for Storm Predictions

Here, we present the experimental results on the SEVIR dataset.

We frame the storm prediction as the binary classification problem, that is, given the sensor and temporal data, the model predicts weather the storm event will occur.

We compare our proposed model with the ResNet-50 and the ViT-Base model, representing the CNN and the Vision Transformer counterparts, respectively.

The prediction results are measured by four metrics, i.e., Precision, Recall, F1-Score, and Accuracy.

The Table lists the overall comparative results.

Since the two baselines cannot take input with multiple resolutions, we consider two cases for comparison: 1) each type of sensor image is regarded as a single dataset, and 2) scaling up or cropping four types of sensor images to the same size (denoted as “All”).

We discovered that our approach can always beat all baselines in predicting both normal and storm events, with an overall accuracy of 94.4 % and an F1-Score of 85.0% on storm events.

These results demonstrate that our approach is effectively for real-world storm predictions.

11 of 15

Experiment: Ablation Studies

Temporal Representation w/ Different Time Intervals

Removal of Different Components on Our Model

Beyond the overall performance, we also conducted ablation studies to justify the importance of novel designs in our work.

We start by respectively removing the MAE Encoder, the Temporal Representation, and the Content Embedding in the ViT Encoder.

The Table lists the experimental results.

We can see that the removal of each component results in a non-trivial performance degradation.

These results validate that all design components are necessary and important in contributing to our model architecture’s prediction performance.

Next, we explore how different time intervals of temporal representations affect the performance results.

In particular, if the time interval is set to T days, we construct its temporal representation by embedding T days before its beginning date, its beginning date, and T days after its beginning date.

In the Figure, we observed that when T is set to 14 days, our proposed model architecture achieves the best performance under all metrics.

These results support our design choice of temporal representations in the paper.

12 of 15

Experiment: Ablation Studies

Content Embeddings on the MAE Encoder

Positional Embeddings on the ViT Encoder

This slide presents additional ablation studies in our work.

The Figure illustrates the experimental results by using the MAE Encoder with/without the Content Embedding.

We discovered that removing the Content Embedding on the MAE Encoder (i.e., the grey column) results in a noticeable performance degradation.

This validates that our Content Embedding can help the MAE Decoder differentiate the membership of data sources.

Next, we explore the necessity of positional embeddings in our ViT Encoder, with the Table listing the results by using the ViT Encoder with/without the positional embedding.

We discovered that the positional embedding in the ViT Encoder hurts the prediction performance.

The reason is that no valuable positional information exists among the input sequences of our ViT encoder as they come from various sources of representations.

13 of 15

Experiment: A Failure Scenario

Settings: Multi-class Classification, i.e., predicting specific event types
Baselines: ResNet-50 and ViT-Base
Metrics: Precision, Recall, F1-Score, Accuracy

Our approach still outperforms two baseline models but all methods achieve a very poor performance. This is because the number of training data for some storm types is extremely limited.

Overall Performance for Storm Type Predictions

We frankly present a failure scenario by using our model and the baseline models for the storm type prediction.

Given a weather event, we need to predict its specific storm type here.

We frame the storm type prediction as the multi-class classification problem.

Other experimental settings are the same as the previous storm prediction.

The Table shows the experimental results, where the first row points out the specific storm type and the second row indicates the number of training examples.

We discovered that our approach still outperforms two baseline models but all methods achieve a very poor performance, especially on storm types with a very limited number of training examples.

This is because the number of training data for some storm types is extremely limited, e.g., <= 50 training examples for the last two storm types.

14 of 15

Conclusion and Future Work

Conclusion

This work has developed a comprehensive Transformer-based model architecture for real-world storm prediction, utilizing both ViT and MAE as key components.
With a collection of novel designs (such as image representation concatenation, temporal representation, and content embedding), our approach can achieve state-of-the-art storm prediction performance.
Although we conduct experiments on the curated SEVIR dataset, our model architecture can be generalized to effectively handle any type of real-world satellite and radar image data.

Future Work

Both our approach and the baseline models exhibit poor performance in accurately predicting storm types due to the extremely limited number of events for certain storm types, calling for the creation of multi-modal storm datasets to provide a more comprehensive and holistic view of storm events.

We have three conclusions.

First, this work has developed a comprehensive Transformer-based model architecture for real-world storm prediction, utilizing both ViT and MAE as key components.

Second, with a collection of novel designs (such as image representation concatenation, temporal representation, and content embedding), our approach can achieve state-of-the-art storm prediction performance.

Third, although we conduct experiments on the curated SEVIR dataset, our model architecture can be generalized to effectively handle any type of real-world satellite and radar image data.

On the other hand, our empirical results demonstrate that both our approach and the baseline models exhibit poor performance in accurately predicting storm types due to the extremely limited number of events for certain storm types.

This motivates us to create multi-modal storm datasets, including sensor data and textual descriptions, to provide a more comprehensive and holistic view of storm events in our future work.

15 of 15

Q & A

Thank you!