Sensor-independent illumination estimation for DNN models

Mahmoud Afifi¹ and Michael S. Brown^1,2

¹York University ²Samsung AI Center - Toronto

Onboard camera processing

When we capture a photograph,

a number of steps are applied�onboard the camera to produce�the final sRGB output image.

A key routine in the pipeline is white balance

When we capture a photograph, a number of steps are applied�onboard the camera to produce�the final sRGB output image.

White balance is called computational colour constancy�in computer vision

White balance is applied to remove the colour cast caused by the scene illumination.

Illumination estimation

raw-RGB image

Illumination estimation

Illumination�estimation algorithm

raw-RGB image

White balance is applied to raw-RGB

When we capture a photograph.

A number of steps are applied�onboard the camera to produce�the final sRGB output image.

raw-RGB

sRGB

Illumination estimation methods

Two main approaches to illumination estimation

​

DNNs

[2015 - foreseeable future]

​

DNN-based approaches

Sensor X raw-RGB image

White-balanced image

DNN model

for sensor X

DNN-based approaches

Our proposed method

Sensor X raw-RGB image

Sensor Y raw-RGB image

White-balanced image

White-balanced image

Sensor-independent framework

(1) Sensor-mapping�network

(2) Illumination estimation

network

Overall framework and network architecture

Starting point: RGB–uv histogram block

We build on the RGB-uv histogram feature from [1,2] by adding two learnable parameters to control the contribution of each color channel in the generated histogram and the smoothness of histogram bins.

[1] J. Barron, Convolutional Color Constancy, ICCV, 2015.

[2] M. Afifi et al., When Color Constancy Goes Wrong, CVPR, 2019.

Sensor space mapping network

Input image histogram

Illumination estimation network

Mapped image histogram

Mapped raw-RGB image

conv/ReLU

​

128 5x5 with

stride = 2

fc

3

conv/ReLU

​

256 3x3 with

stride = 2

conv/ReLU

512 2x2 with

stride = 1

Training and loss function

R

G

B

Estimated

GT

Training and loss function

Mapping to original

raw space

M^-1

Angular �Error Loss

Angular error Loss =

Ground Truth

Training and loss function

Mapping to original

raw space

M^-1

Angular �Error Loss

Angular error Loss =

Ground Truth

Test time example #1

Testing raw-RGB image

(Fujifilm XM1)

​

Test time example #2

Testing raw-RGB image

(Canon 1Ds Mk III)

​

Experimental results

• All camera models in:
• NUS 8-Camera dataset (8 camera models)
• Gehler-Shi dataset (2 camera models)
• Cube/Cube+ dataset (1 camera model)

​

​

​

NUS-8 dataset

Gehler-Shi dataset

Cube/Cube+ datasets

Results

Angular errors

Sensor-independent methods

- J. Van De Weijer, et al., Edge-based color constancy, TIP, 2007

- S. Bianco and C. Cusano, Quasi-unsupervised color constancy, �In CVPR’19

- Y. Qian, et al., On finding gray pixels, In CVPR’19

​

Sensor-dependent methods

- W. Shi, et al., Deep specialized network for illuminant estimation, In ECCV’16

- . J. T. Barron and Y.-T. Tsai, Fast Fourier color constancy,

In CVPR’17

- Y. Hu, et al., FC4: Fully convolutional color constancy with confidence-weighted pooling, In CVPR’17

Results

Sensor-independent methods

- J. Van De Weijer, et al., Edge-based color constancy, TIP, 2007

- S. Bianco and C. Cusano, Quasi-unsupervised color constancy, �In CVPR’19

- Y. Qian, et al., On finding gray pixels, In CVPR’19

​

Sensor-dependent methods

- W. Shi, et al., Deep specialized network for illuminant estimation, In ECCV’16

- . J. T. Barron and Y.-T. Tsai, Fast Fourier color constancy,

In CVPR’17

- Y. Hu, et al., FC4: Fully convolutional color constancy with confidence-weighted pooling, In CVPR’17

Angular errors

Results

Summary

• Presented a deep learning framework for sensor-independent illumination estimation

​

• Our method learns an image-specific mapping to transform different camera raw-RGB images into a canonical "working space"

​

• Our framework is on par with sensor-dependent DNNs, but requires only a single DNN.

Thank you

Mapping input image from original raw

space to the learned working space

Questions?