Fully Complex-valued Fully Convolutional Multi-feature Fusion Network (FC²MFN) for Building Segmentation of InSAR images

Aniruddh Sikdar¹, Sumanth Udupa², Suresh Sundaram², Narasimhan Sundararajan²�

¹Robert Bosch Centre for Cyber-Physical Systems, Indian Institute of Science, Bengaluru, India.�²Department of Aerospace Engineering, Indian Institute of Science, Bengaluru, India.

*

1

Outline.

• Motivation.
• Previous Works.
• Main contributions.
• Fully Complex-valued Deep learning model for analysis of SAR images.
• InSAR building detection dataset.
• Fully Complex-valued Fully Convolutional Multi-feature Fusion Network(FC²MFN).
• Experimental results.
• Conclusions.

​

*

2

Motivation

​

​

Changes in urban footprint over one area of Philadelphia city between 2004 and 2014.

https://www.iceye.com/satellite-data

This series of radar satellite images show oil tanks in the Port of Rotterdam, Netherlands. These daily images were between March 6-31, 2021.

https://www.iceye.com/technology/sar-imagery

​

Synthetic Aperture Radar(SAR) images can be used,

• Retains information in all weather and low light conditions.

Change detection of buildings on the terrain,

• 3-D modeling,
• Map updating,
• Urban change monitoring.

*

3

Previous Works

• SAR Building detection.
• Yu, Lingjuan, et al, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, (2022)[1].
• Proposed a lightweight complex-valued DeepLabv3+ for the small PolSAR data to avoid overfitting.
• Showed empirically that complex-valued (CV) network performs better compared to real-valued networks on complex-valued SAR data.
• Chen, Jiankun, et al. IEEE Transactions on Geoscience and Remote Sensing 60 (2021)[2].
• Building segmentation on high-resolution InSAR images.
• Proposed CVCMFF-Net learns multi-scale features and performs multi-feature fusion.
• Performed experiments on synthetic InSAR as well as on Airborne InSAR images.
• Showed empirically that the complex-valued network outperforms the real-valued counterpart.

​

• Gaps in literature :
• These networks project complex-valued data to the real domain losing valuable phase information.
• Pooling operation does not take phase information into consideration.

​

[1]Yu, Lingjuan, et al. "A Lightweight Complex-Valued DeepLabv3+ for Semantic Segmentation of PolSAR Image." IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 15 (2022): 930-943.

[2] Chen, Jiankun, et al. "CVCMFF Net: Complex-valued convolutional and multifeature fusion network for building semantic segmentation of InSAR images." IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-14.

*

4

Main contributions

• Proposed Fully Complex-valued Fully Convolutional Multi-feature Fusion Network(FC²MFN),
• learns multi-scale features useful for high-resolution InSAR images,
• performs multi-feature fusion.
• Novel fully complex-valued learning scheme proposed for FC²MFN using orthogonal decision boundary theory [3],
• to operate and learn entirely in the complex domain,
• to avoid the loss of phase information.
• New complex-valued pooling layer proposed to compare two complex numbers using the magnitude and the phase information.
• FC²MFN network achieves state-of-the-art performance, on the simulated InSAR dataset [4].
• Computationally efficient compared to other networks in teams of floating-point operations(FLOPs).

[3]Nitta, "Orthogonality of decision boundaries in complex-valued neural networks." Neural computation,2004.

[4] Jiankun Chen, October 21, 2020, "Simulated InSAR building dataset for CVCMFF Net", IEEE Dataport, doi: https://dx.doi.org/10.21227/2csm.

​

*

5

Fully Complex-valued Deep learning for analysis of SAR images.

• Description of InSAR building dataset.
• The simulated InSAR building dataset [4] contains
• 312 simulated SAR image pairs generated from 39 different building models.
• Each building model is simulated at 8 viewing angles.

CONFIGURATION OF THE GROUND TRUTH LABEL AND THE CORRESPONDING COLOR MARK.

Interferometric SAR (InSAR).

Simulated InSAR building dataset.

Interferometric phase angle.

[4] Jiankun Chen, October 21, 2020, "Simulated InSAR building dataset for CVCMFF Net", IEEE Dataport, doi: https://dx.doi.org/10.21227/2csm.

*

6

Network architecture

• Proposed Fully Complex-valued Fully Convolutional Multi-feature Fusion Network(FC²MFN):

Fig. (a) Network architecture of FC²MFN with complex-valued (CV) pooling layers. (b) CBR blocks- Complex-valued convolution layer followed by batch normalization and CRelu. The filter size in CBR blocks is 3x3. (c) Residual block 1. (d) Residual block 2. The output of FC²MFN is a complex-valued feature map with real and imaginary channels.

*

7

Prerequisites of Complex-valued Deep learning (existing).

• Complex-valued convolution: Convolving a complex matrix with the kernel W = A +iB, the output corresponding to the input patch h =X +iY is given by,

​

​

​

• Complex Differentiability: In order to perform complex-valued backpropagation,
• Cost functions need to be real-valued.
• Activation functions (e.g CReLU) should be differentiable with respect to the real and imaginary parts of each complex parameter in the network. (Wirtinger calculus[5].)

​

​

​

[5] Ken Kreutz-Delgado, “The complex gradient operator and the cr-calculus,” arXiv preprint arXiv:0906.4835, 2009.

CRelu activation function.

*

8

Complex-valued Deep learning (cont…)

• Pooling operation:
• Max or average pooling operation performed only on the amplitude of the complex numbers.
• Phase information is not taken into account.
• Proposed complex-valued pooling operation to compare complex numbers taking phase into consideration.
• Complex-valued pooling operation is as follows:

​

Complex-valued pooling operation.

*

9

Proposed Fully Complex-valued learning.

• Novel fully complex-valued learning using Orthogonal Decision Boundary theory to train the network in complex-domain and preserve phase information.
• Complex-valued one-hot encoding of labels:
• For the label c^t of the t^th sample, one-hot encoding y^t_k ={y^t₁, y^t₂, .. y^t_k... y^t_n} is as follows:

Real and imaginary decision boundaries of fully complex-valued deep learning models.

[3] Nitta, "Orthogonality of decision boundaries in complex-valued neural networks." Neural computation,2004.

Complex-valued loss e is defined as,

Real valued loss E is defined as,

Block diagram of fully complex-valued learning.

Orthogonal decision boundary theory[3].

*

10

Experimental results.

• Segmentation results of Fully Complex-valued network on InSAR dataset.
• Comparison of class-wise segmentation performance.

​

​

​

​

• Ablation study of fully complex-valued FC²MFN with real-valued and complex-valued (CV) networks.

​

​

​

�

*

11

Segmentation results of Fully Complex-valued network on InSAR dataset.

• Performance curves of FC²MFN on test data.
• (a)Intersection over Union(IoU) of class 0 (b)Overall accuracy.
• FC²MFN achieves high accuracy from the beginning without using any specific kind of pre-training or complex-valued weight initialization and has a smooth convergence.

*

12

Conclusions

• Fully Complex-valued Fully Convolutional Multi-feature Fusion Network (FC²MFN) on the InSAR dataset:
• Proposed novel fully complex-valued learning for FC²MFN using Orthogonal Decision Boundary theory to train the network in complex-domain.
• Proposed complex-valued loss function.
• Update network parameters using phase information.
• Proposed new complex-valued pooling operation.
• Achieved state-of-the-art results on the complex-valued simulated InSAR dataset [4].
• Without the use of a complex-valued weight initialization scheme or complex-valued batch norm.
• Emphirically validated the idea that fully complex networks outperform real and complex-valued networks on complex-valued data.

​

​

[4] Jiankun Chen, October 21, 2020, "Simulated InSAR building dataset for CVCMFF Net", IEEE Dataport, doi: https://dx.doi.org/10.21227/2csm.

*

13

Thank you!!!�

*

14

Appendix

• Complex-valued pooling derivation

*

15

Appendix

• Image reconstructions of FC²MFN are shown. The reconstructions are very similar to ground truth labels.

​

*

16