1 of 1

​

​

​

​

​

​

​

​

​

​

​

​

​

​

​

Loss Functions (GT image I(x) and restored image R(x)):

  1. L2 Loss between I(x) and R(x)
  2. Perceptual loss: L2 Loss between pre-trained VGG features of I(x) and R(x)

Feature Engineering

​

​

​

​

​

​

​

Dark Channel Prior

  • Relevant weather types: fog, snow
  • .

​

Local Contrast

  • Relevant weather types: fog
  • Variance over local regions

​

Local Saturation

  • Relevant weather types: fog, rain
  • Extracted from HSV color space

​

Histogram of Gradients (HoG)

  • Relevant weather types: rain streaks
  • Gradients counted over local regions
    • format to binned histogram

​

​

Multi-task Classification

Each detection head serves a unique purpose and encodes pertinent scene information

​

  1. Weather type encodes general scene-level atmospheric weather condition (rain vs fog vs snow)
  2. Time of day encodes (day vs dawn/dusk vs night)
    1. nighttime vision is a separate problem
  3. Road condition encodes road state independently from atmospheric condition
    • (slick vs snowy vs dry)

Methodology

Abstract & Motivation

​

Improving Perception for Autonomous Driving in Adverse Weather Conditions

Lulu Ricketts, Roshini Rajesh Kannan

Advised by Prof. Srinivasa Narasimhan

​

​

The autonomous driving industry has boomed in the last couple of years, having the potential to have a widespread impact on how transportation is conducted all over the world. One current obstacle to these advancements is the problem of understanding a vehicle’s surroundings in bad weather conditions, such as rain, snow, and fog. Poor weather often results in a limited field of view of objects within the scene and cause artifacts on captured images such as reflections or dispersion of light that is not present in clear weather images. This combined with insufficient training data causes many perception algorithms to perform significantly worse when faced with these conditions.

​

​

​

​

​

To tackle this problem, we propose a method to restore bad weather images in a way such that downstream perception tasks can improve. We combine several datasets together to create a generalized, scene-free multi-class weather classifier. Additionally, we generate synthetic fog images consistent with physical properties of fog to train an image restoration model.

Future Work

Restoration

Data collection

​

Many datasets suffer from limited or lack of bad weather data. We combat this in 3 distinct ways:

References

Classification

Depth map

(LiDAR/MonoDepth)

Increase contrast

Gaussian blur

Atmospheric

scattering model

Synthetic Fog Generation Pipeline

[1] Jeya Maria Jose Valanarasu, Rajeev Yasarla, and Vishal M. Patel. Transweather: Transformer-based restoration of images degraded by adverse weather conditions. CoRR, abs/2111.14813, 2021.

[2] S.G. Narasimhan and S.K. Nayar. Contrast restoration of weather degraded images. IEEE Transactions on Pattern Analysis and Machine Intelligence,25(6):713–724, 2003.

[3] Kaiming He, Jian Sun and Xiaoou Tang, "Single image haze removal using dark channel prior," 2009 IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, 2009, pp. 1956-1963, doi: 10.1109/CVPR.2009.5206515.

[4] Zheng Zhang, Huadong Ma, Huiyuan Fu, Cheng Zhang,

Scene-free multi-class weather classification on single images, Neurocomputing, Volume 207, 2016, Pages 365-373, ISSN 0925-2312,https://doi.org/10.1016/j.neucom.2016.05.015.

​

  1. More quantitative evaluation of downstream perception tasks on real world data (segmentation)
  2. Incorporating more real world data into training of restoration model
  3. Explore different approaches for handling vision in the dark

Foggy image

Collected datasets from multiple sources

1

Bad weather images

Weather Classification

2

Extract image features

Encoder

Classification heads

Road segmentation

3

Road condition label

Weather label

Time of day label

Synthetic Fog Generation

Clear weather images

TransWeather Model

Image Restoration

Atmospheric Scattering Model

Depth map

4

Input Image

train time: synthetic

test time: real

Conv

Restored Image

5

1

Aggregating existing datasets

​

BDD100k

DENSE

Dawn

MWI

Rain

5808

118

219

2097

Fog

143

1939

330

5000

Snow

6318

4618

224

2100

Pittsburgh bus webcams

Buses drive the same routes every day throughout Pittsburgh with GPS location data. We use this to query the National Weather Service API for bad weather days and extract data from those days.

Capturing our own (video) data while driving

Light source

Certain weather types hold consistent physical properties for how light travels through the atmosphere and hits a camera sensor.

We can model these properties of weather by engineering features on the resulting images.

Clear

Rain

Snow

Fog

2

Optimal β value: [1,5]

​

3

Optimal atmospheric light value: [180,240]

Test accuracy

without using engineered features: 77.5% / with engineered features: 87.7%

4

​

I(x) = J(x)t(x) + A(1-t(x))

t(x) = exp(-𝛽d(x))

​

I(x) = foggy image

J(x) = clear image

t(x) = transmission map

A = atmospheric light (airlight)

β = scattering coefficient

d(x) = depth map

​

​

Foggy image

Depth map

Gaussian blur

Daytime

Nighttime

Image Restoration

5

Generated fog images

Input

Restored

Dataset: Dawn

Input

Restored

Dataset: BDD100k

Input

Restored

Dataset: Dashcam

2D Object Detection

mAP pre-restoration

mAP post-restoration

26.01

31.33

GT

​

​

Before restoration

​

​

After restoration