Lighting Estimation using GANs

Bhushan Sonawane

Agenda

• Timeline
• Ideas on GANs
• Stepping into light estimation
• Dataset
• LDAN
• AutoLighting

Timeline

Ideas on GANs - 1. Co-Operative GANs

• Train multiple GANs and copy weights of one performing well
• Experimented with same architecture and same hyperparameters
• TODOs-
• Same architecture and different hyperparameters
• Different architecture ??
• Source - https://github.com/bhushan23/GAN/tree/master/Co-Operative-GAN

​

Ideas on GANs - 2. Designing NN using GANs

• Design NN for any deep learning task
• Design Discriminator which is Task specific
• One approach for Generator -
• Separate Generator for feature of Network
• Number of layers
• Activation functions
• Hidden layer size
• …….

Lighting Estimation

• Problem - Lack of Ground Truth

Dataset

• Synthetic Dataset used in LDAN
• Synthetic Dataset used in SfSNet
• Noisy SH, Normal generated using SIRFS
• SIRFS implementation of Jon Barron with some bug fixes
• CelebA
• SfSNet Synthetic dataset
• Source - https://github.com/bhushan23/SIRFS

Label Denoising Adversarial Network (LDAN)

• Fills the gap of Synthetic Images and Real Images
• Use existing methods to gather SH of real images and train the network
• Here, Author used SIRFS
• Maps Real Images into Synthetic Images to get accurate SH

LDAN - Architecture

Loss functions

• Training Feature net and Lighting Net using

• GAN loss

Experiment 1 - CelebA images as Real images

• Synthetic dataset - LDAN
• Two frontal pose images having same SH
• Real Images - CelebA
• SH and Normal generated using SIRFS

​

Experiment 1 - CelebA dataset used as Real Images

Experiment 1 results - SIRFS vs LDAN Shading

Experiment 1 - mean MSE loss of SH on Validation set duing GAN training

LDAN Results - Experiment 2

• Synthetic dataset - LDAN dataset
• Two frontal pose images having same SH
• Real image dataset - Synthetic images of SfSNet
• Have ground truth and noisy SIRFS SH
• Used Synthetic data from SfSNet paper as Real Images
• Why? - We wanted to verify against ground truth.

Experiment 2 Dataset - Synthetic Images used for training FeatureNet

Experiment 2 Dataset - Synthetic Images used for training as Real Images

Experiment 2 Results - Expected vs Predicted Shading

Experiment 2 Results - Ground Truth SH vs Predicted SH Shading

Experiment 2 Results - SIRFS shading vs LDAN shading

Experiment 2 Results - GAN training to adapt Synthentic Lighting

Mean MSE of SH

Experiment 2 Results - Comparison with SIRFS

LDAN Original SIRFS

AutoLighting - Using AE to denoise SH

• LDAN needs to map real images into Synthetic space
• Can we use AutoEncoder to denoise SH of real images?
• Design doc - https://docs.google.com/document/d/1-1uKpTmdFloRBMK2mk1IRWrlxgafH22VnTaSvZtnbq4/edit

​

​

AutoLighting Architecture

AutoLighting - Experiment

1. Train Synth_Net with Synthetic Images
1. Training Image-SH network with Synthetic Images and True SH
2. Train AutoEncoder to denoise SH using Synthetic Image
• Input: Lighting Features from Step 1 and SIRFS SH
• Output: Lighting Features from Step 1 and True SH
• Goal: Denoise SH given Lighting features
3. Denoise SH of Real images using AE trained in step 2
• We already have generated SIRFS SH for Real Images
4. Train Real_Net with Real Images
• Training Image-SH network with Real Images and Denoised SH (step 3)

AutoLighting Results - SIRFS vs Predicted Shading

AutoLighting Results - MSE loss training Network with Real Images

Reference

• LDAN
• https://arxiv.org/abs/1709.01993
• SIRFS
• https://ieeexplore.ieee.org/document/6975182/
• GAN
• https://arxiv.org/abs/1406.2661

​

New findings

2. Training GAN with SIRFS SH

2. True Normal

2. Without GAN

2. Expected Shading

2. Predicted SH

2. SIRFS shading

2. Validation MSE

3. Training GAN with ground truth SH

3. True Normal

3. Without GAN

3. True Shading

3. Predicted SH

3. SIRFS

3. Validation plot