1 of 35

Bridging Physics-based ML and Polarization

1

2 of 35

Topics Covered

2

1. Physics-based Learning (PBL)

2. Applying PBL to SfP to create Deep SfP

3. Overview of benefits and limitations

3 of 35

Physics + Machine Learning

3

Machine learning?

Not robust

Physics?

Not accurate

4 of 35

An example about trajectory estimation

4

Bull's-Eye? Miss Short? Miss Long?

5 of 35

Solution from high school physics

5

High School Physics Approach

​

​

​

​

​

Will Physics Alone Solve Grad Student Paper Toss ?

Predict Next 15 frames?

6 of 35

Solution from high school physics

6

High School Physics Approach

​

​

​

​

​

Will Physics Alone Solve Grad Student Paper Toss ?

Predict Next 15 frames?

Observed

Trajectory

Physics

Prediction

7 of 35

Deep Learning Alone is Challenging

High School Physics Approach

​

​

​

​

Will Physics Alone Solve Grad Student Paper Toss?

​

… Probably Not, Air Resistance, Etc.

7

Predict Next 15 frames?

Neural Network Approach

ResNet18

Initial Trajectory (3 frames)

Rest of Trajectory (15 frames)

8 of 35

Naïve Deep Learning Suboptimal - Limitations in Data

8

  • Limited Data
  • Only 32 Training Pairs (“Tosses”)
  • Predicts an Average Trajectory
  • Scene Agnostic!

9 of 35

Physics-based residual learning

9

ResNet18

Initial Trajectory (3 frames)

Rest of Trajectory (15 frames)

Neural Network Approach

​

Physical solutions

10 of 35

Combine Physics with Learning

10

Knowledge from Physics

  • Environment: Ideal environment
  • Data Amount: Small
  • Generalizability: Good

Knowledge from Data

  • Environment: Real environment
  • Data Amount: Large
  • Generalizability: Poor

Physics + Data

​

  • Environment: Real environment
  • Data Amount: Small
  • Generalizability: Good

11 of 35

Machines Discovering High School Physics

11

12 of 35

Machines Discovering High School Physics

12

13 of 35

Example Discovery Pipeline

13

Input Video

Positions

Discovered Equations

Latent Physics

Position Detection

Equation Discovery

An overview of the proposed Visual Physics framework. The pipeline discovers underlying physics features of the input data that are fed to a symbolic regression algorithm for equation discovery

14 of 35

Discovered Equations (Ball Toss)

14

Input Video Embedding Trends

Equations

  • v0y and v0x are still embedded into independent latent features

High correlation

High correlation

15 of 35

Applications to Imaging

15

Kepler’s Law

Rayleigh Scattering

DOCI

Imaging*

* Cheng, Harrison, et al. "Dynamic optical contrast imaging (DOCI): system theory for rapid, wide-field, multispectral optical imaging using fluorescence lifetime contrast mechanism." (2019).

16 of 35

What is Shape from Polarization (SfP)?

16

Reflected light has a polarization state that corresponds to object shape

[1]

[2]

17 of 35

Advantages and Applications of SfP

17

  • Advantages:
    • Completely passive capture
    • Single shot capture
    • Minimal constraints on capture conditions

​

  • Potential Applications:
    • Autonomous Vehicles
    • Augmented/Virtual Reality
    • Medical Imaging
    • Industrial Applications

18 of 35

What makes SfP difficult?

18

  • Multiple reflection models: diffuse, specular, and a combination of both
  • Fresnel equations governing polarization ambiguous up to a constant factor

[3]

19 of 35

Intro to Fresnel’s Equation and SfP

19

Fresnel's equations for transmission relate electric field amplitude (intensity in image) to surface geometry

[4]

20 of 35

SfP, the TRS, and the Ambiguity Problem

20

[4]

Degree of Polarization

Transmitted Radiance Sinusoid

Measured Intensity

Polarization Angle

Phase Angle

21 of 35

Need to Handle Multiple Reflection Models

21

[5]

Specular Reflection

Diffuse Reflection

22 of 35

Two Surface Reconstruction Models

22

Specular Reflection

Diffuse Reflection

Where θ and Φ are the zenith and azimuth angles of the surface normal, and 𝜑 represents the phase of polarization

23 of 35

Paper’s Method vs. Previous Methods

23

  • Previous SfP methods use purely physics-based methods, have many assumed values for unknown physical parameters
    • Intensity, refractive index, lighting conditions, specular / diffuse reflections, etc.

​

  • Additional problem in shape estimation because of inherent ambiguities in physical equations

​

  • Paper proposes SfP solution: fuse polarization physics into the deep learning pipeline to overcome limitations of pure physics

24 of 35

Summarizing the Paper’s Contributions

24

  • Initial attempt to use deep learning for SfP
  • Fuse physics with deep learning to improve SfP
  • State-of-the-art results compared to physics-based methods
  • An SfP dataset with ground truth shape

25 of 35

DeepSfP Dataset

25

Novel dataset with polarization images and ground truth shape

26 of 35

Data Acquisition Setup

26

  • Scanner used to obtain 3D geometry of object

​

  • Single-shot polarization camera captures 4 polarization images simultaneously (0, 45,

90, 135 degrees)

​

  • 2D-3D registration to produce polarization image and surface normal pair

27 of 35

Variation in the DeepSfP Dataset

27

Material, Shape, and Texture

Changing Lighting, Capture Conditions

28 of 35

28

Paper Technical Contributions

  • Initial deep-learning based SfP approach

​

  • Incorporating polarization physics into pipeline (presenter to explain)

Qst: What are the physics-based priors?

Ans: The data structures that are returned after applying the Fresnel equations to input images. These data structures represent object shape.

29 of 35

Robustness to Illumination Conditions

29

  • Our method exhibits minimal variation across illumination conditions
  • Physics based methods vary significantly when lighting changes

30 of 35

Importance of Priors: Image-Related Artifacts

30

  • Naive learning methods suffer from ‘Texture Copy’ without guidance from physics

​

​

  • The numbers show the mean angular error, with and without prior fusion (lower is better)

31 of 35

Importance of Physics: Ablation Experiments

31

  • Both quantitative and qualitative performance suffer without

physics-based priors

32 of 35

Important Results

32

Polarization Images

Ground Truth Shape

Miyazaki '18

Mahmoud '12

Miyazaki '03

Deep SfP [Ba 2020]

33 of 35

Important Results

33

Deep SfP [Ba 2020]

Polarization Images

Ground Truth Shape

Miyazaki '18

Mahmoud '12

Miyazaki '03

34 of 35

Benefits / Limitations of a Physics-based Learning Approach

34

  • New metrics beyond mean angular error that penalizes high-frequency errors
  • Fusing depth estimation with Deep SfP
  • Making prior estimation a differentiable component of the network

35 of 35

Works Cited

35

  1. : https://commons.wikimedia.org/wiki/File:EM-Wave.gif
  2. : http://homepages.inf.ed.ac.uk/rbf/CVonline/LOCAL_COPIES/AV0405/HOWARD/prac2.html
  3. : http://vacuumcoating.info/obtaining-good-transmission-t-and-reflection-r-measurements
  4. : https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=1632218
  5. : https://www.physicsclassroom.com/class/refln/Lesson-1/Specular-vs-Diffuse-Reflection