1 of 23

NeRT: Implicit Neural Representations for General Unsupervised Turbulence Mitigation

Weiyun Jiang, Vivek Boominathan, Ashok Veeraraghavan

Electrical and Computer Engineering

Rice University

​

1

2 of 23

2

What is turbulence?

3 of 23

3

Atmospheric (air) turbulence

Atmospheric turbulence

  • Distance
  • Temperature
  • Humidity

Forward modelling

  • Spatially and temporally-varying tiltings
  • Spatially and temporally-varying blurings

4 of 23

4

Existing methods

W. Jiang, V. Boominathan and A. Veeraraghavan, "NeRT”, IEEE/CVF CVPR-W (2023), Arxiv (2024)

5 of 23

5

Modeling

"all models are wrong, but some models are useful.“

�- George Box �Statistician

6 of 23

6

Background – Phase-over-aperture model

Full model:

  • Multiple step per output pixel
    • Split-step method
  • Computationally super-expensive

Collapsed phase screen model:

  • Single step per output pixel
  • Computationally less expensive
  • Adequately models

Popularized by Stanley Chan, Purdue

7 of 23

7

Background – Phase-over-aperture model

Phase screen model:

  • Single step per output pixel
  • Computationally less expensive
  • Adequately models

Each sub field-of-view:

  • Locally convolutional
  • Each parameterized by Zernike polynomials
  • Can be split into tilt and blur components
  • Zernike coeffs can be smoothly interpolated�over the full FoV.

Popularized by Stanley Chan, Purdue

8 of 23

8

Background – Tilt then blur or Blur then tilt

Comparison: Tilt then blur or Blur then tilt

more accurate than

Popularized by Stanley Chan, Purdue

9 of 23

9

Contemporary recontruction approaches

more accurate than

Measurement

Un - tilt

De - blur

Image formation assumption:

This is less accurate

10 of 23

10

The error Blur-then-tilt vs Tilt-then-blur

Error shows up in the gradients or the �high-frequency regions.

11 of 23

11

What is new?

  • The first unsupervised physically grounded deep learning method.

​

  • Highly generalizable.

W. Jiang, V. Boominathan and A. Veeraraghavan, "NeRT”, IEEE/CVF CVPR-W (2023), Arxiv (2024)

12 of 23

12

Pipeline of NeRT – Optimization with Implicit Neural Representation

The more accurate forward model

W. Jiang, V. Boominathan and A. Veeraraghavan, "NeRT”, IEEE/CVF CVPR-W (2023), Arxiv (2024)

Zernike

param

13 of 23

13

Pipeline of NeRT - Inference

W. Jiang, V. Boominathan and A. Veeraraghavan, "NeRT”, IEEE/CVF CVPR-W (2023), Arxiv (2024)

14 of 23

14

Pipeline of NeRT - Inference

  • No training needed
  • Have great generalization ability over different types of distortions: atmospheric, water, and ripple reflection.
  • First to follow the correct tilt-then-blur forward model.

W. Jiang, V. Boominathan and A. Veeraraghavan, "NeRT”, IEEE/CVF CVPR-W (2023), Arxiv (2024)

15 of 23

15

Results - air turbulence (static)

NeRT is able to achieve high spatial resolution, recover high-contrast text, and reconstruct fine details, such as wire fences.

W. Jiang, V. Boominathan and A. Veeraraghavan, "NeRT”, IEEE/CVF CVPR-W (2023), Arxiv (2024)

16 of 23

16

Results - air turbulence (dynamic)

  • NeRT is able to recover high-contrast and fine details of the license plate while other methods show blurry and low-contrast license plate numbers.

W. Jiang, V. Boominathan and A. Veeraraghavan, "NeRT”, IEEE/CVF CVPR-W (2023), Arxiv (2024)

17 of 23

17

Results - air turbulence (synthetic)

  • Synthetic dataset generated using atmospheric turbulence simulator.

​

  • NeRT outperforms other state-of-the-art air turbulence mitigation methods.

W. Jiang, V. Boominathan and A. Veeraraghavan, "NeRT”, IEEE/CVF CVPR-W (2023), Arxiv (2024)

18 of 23

18

Other distortions: Water turbulence

19 of 23

19

Results - water turbulence

Results obtained from NeRT, are on par with the performance of current state-of-the-art NDIR.

W. Jiang, V. Boominathan and A. Veeraraghavan, "NeRT”, IEEE/CVF CVPR-W (2023), Arxiv (2024)

20 of 23

20

NeRT in the wild?

21 of 23

21

Results - ripple reflection turbulence in the wild

  • NeRT demonstrates the ability to effectively eliminate water ripple turbulence from reflections and produce clean images.

W. Jiang, V. Boominathan and A. Veeraraghavan, "NeRT”, IEEE/CVF CVPR-W (2023), Arxiv (2024)

22 of 23

22

Results - ripple reflection turbulence in the wild … more

  • NeRT demonstrates the ability to effectively eliminate water ripple turbulence from reflections and produce clean images.

W. Jiang, V. Boominathan and A. Veeraraghavan, "NeRT”, IEEE/CVF CVPR-W (2023), Arxiv (2024)

23 of 23

23

Summary of NeRT

  • The first unsupervised physically grounded deep learning method.

​

  • Highly generalizable.

​

  • The state-of-art performance on atmospheric turbulence and comparable performance on water turbulence.