1 of 18

DeepMAO: Deep Multi-scale Aware Overcomplete Network for Building Segmentation in Satellite Imagery

1Robert Bosch Centre for Cyber-Physical Systems, Indian Institute of Science, Bengaluru, India.

2Department of Aerospace Engineering, Indian Institute of Science, Bengaluru, India.

Aniruddh Sikdar*1, Sumanth Udupa*2, Prajwal Gurunath*2, Suresh Sundaram2

Artificial Intelligence and Robotics Lab

*

1

2 of 18

Background

  • Remote sensing used for…

​

​

​

Changes in urban footprint over one area of Philadelphia city between 2004 and 2014.

https://www.iceye.com/satellite-data

*

2

3 of 18

Background

  • Semantic Segmentation for urban environments

​

​

​

Ground truth

  • Many missed small buildings

​

​

​

Unet Prediction

Input image

  • Low pixel-representation

​

​

​

*

3

4 of 18

Background

  • Large scale of aerial images

*

4

5 of 18

Background - Convolutions

  • Feature extraction

​

​

​

  • Encoder-decoder architectures
  • Increasing receptive field in UNet

​

​

​

​

Layer 1

Layer 2

Layer 3

Receptive field in typical convolutions

*

5

6 of 18

Background - Overcompleteness

  • Decreasing receptive field

​

​

​

  • Overcomplete architectures

​

​

​

​

Layer 1

Layer 2

Layer 3

Receptive field in overcomplete convolutions

Overcomplete representation in a standard neural network

*

6

7 of 18

Related works

  • Valanarasu, Jeya Maria Jose, et al. "Kiu-net: Towards accurate segmentation of biomedical images using over-complete representations." Medical Image Computing and Computer Assisted Intervention–MICCAI 2020.
  • Overall network
  • Overcomplete encoder-decoder branch
  • Undercomplete encoder-decoder branch

Overcompleteness

  • Small image size
  • Curtailed memory constraints
  • Lewicki, Michael S., and Terrence J. Sejnowski. "Learning overcomplete representations." Neural computation 12.2 (2000): 337-365.

Ki-Unet

*

7

8 of 18

Proposed method – Deep Multi-scale Aware Overcomplete Network (DeepMAO)

*

8

9 of 18

Proposed method – Training Loss

  • Representation learning through Segmentation Loss: Lseg
  • Combination of Dice Loss and Focal Loss: �yi : pixelwise ground truth; pi : predicted probability values
  • αi ∈ [0,1] ; ϒi > 1

*

9

10 of 18

Proposed method - Loss-Mix

  • Increase representation of small and complicated building segments

L1 Loss

Tiling, HxW patches

Max loss patch selected

Loss-Mix augmentation

  • Increases instances of harder samples
  • Self-regulative learning scheme: Loss-Mix

*

10

11 of 18

Visual analysis

Focus of branches

GRADCAM visualisations

  • Larger features
  • Smaller features
  • Small building footprints
  • Intricate geometries of larger buildings

Receptive Field

Receptive Field

*

11

12 of 18

Datasets

  • SpaceNet6 Multi-sensor All Weather Mapping:
    • Electro-Optical
    • Synthetic Aperture Radar: Speckle noise
  • INRIA Aerial Image Labeling Dataset

​

SpaceNet 6 SAR

SpaceNet 6 EO

INRIA

*

12

13 of 18

Results on SpaceNet6 MSAW dataset

Evaluation metrics (%) of building segmentation networks and DeepMAO (EO Images)

Evaluation metrics (%) of building segmentation networks and DeepMAO (SAR Images)

*

13

14 of 18

Results

  • INRIA dataset

Performance comparison of overall F1 score vs building footprint size on SpaceNet 6 optical images

Evaluation metrics (%) of building segmentation networks and DeepMAO (EO Images)

  • SpaceNet6 dataset

*

14

15 of 18

Results

Ablation study of Loss-Mix strategy on DeepMAO and state-of-the-art models on SpaceNet 6 dataset - EO Images

*

15

16 of 18

Results

Selective crops of SAR input image has been used for visualization purposes. Green box indicates a case where DeepMAO performance is better than Unet-BE. Blue box indicates a case wherein both models fail to detect a building

First row consists of Spacenet 6 Challenge dataset. Second row consists of INRIA dataset.

*

16

17 of 18

Conclusion

  • First work to specifically focus on improvement in small building segmentation on the SpaceNet6 dataset
  • A novel Deep Multi-scale Aware Overcomplete network (DeepMAO) for building segmentation tasks in satellite imagery for SAR and EO modalities.
  • Two parallel branch network that focuses on high level semantic information and smaller finer structural features
  • Self-regulative learning scheme:Loss-Mix; hardest to segment patches used as an augmentation
  • ~2.5% increase and ~1% increase in overall segmentation performance of EO and SAR modalities
  • ~4% increase in small building segmentation

​

*

17

18 of 18

Thank you�

*

18