1 of 16

Semi-Supervised Raw-to-Raw Mapping

Mahmoud Afifi

Lassonde School of Engineering, York University, Canada

Project page

Abdullah Abuolaim

2 of 16

Introduction

Semi-Supervised Raw-to-Raw Mapping

2

Images are from our proposed dataset

Project page

3 of 16

Introduction

Semi-Supervised Raw-to-Raw Mapping

3

Canon1D Mark III

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

400

450

500

550

600

650

700

750

Wavelength (nm)

Normalized sensitivity

Canon 60D

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

400

450

500

550

600

650

700

750

Wavelength (nm)

Normalized sensitivity

Canon 300D

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

400

450

500

550

600

650

700

750

Wavelength (nm)

Normalized sensitivity

Sony Nex5N

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

400

450

500

550

600

650

700

750

Wavelength (nm)

Normalized sensitivity

Olympus EPL2

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

400

450

500

550

600

650

700

750

Wavelength (nm)

Normalized sensitivity

Nokia N900

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

400

450

500

550

600

650

700

750

Wavelength (nm)

Normalized sensitivity

Nikon D5100

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

400

450

500

550

600

650

700

750

Wavelength (nm)

Normalized sensitivity

Nikon D40

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

400

450

500

550

600

650

700

750

Wavelength (nm)

Normalized sensitivity

Data source: Jiang et al., What is the space of spectral sensitivity functions for digital color cameras? In WACV, 2013

Project page

4 of 16

Introduction

Semi-Supervised Raw-to-Raw Mapping

4

Raw image captured by camera A

Raw image captured by camera B

Raw image mapped to camera B

Images are from our proposed dataset

Project page

5 of 16

Prior work

Semi-Supervised Raw-to-Raw Mapping

5

Nguyen, et al., Raw-to-raw: Mapping between image sensor color responses. In CVPR, 2014.

Daylight

Daylight

Camera A

Camera B

Fluorescent

Fluorescent

Camera A

Camera B

Project page

6 of 16

Dataset

Semi-Supervised Raw-to-Raw Mapping

6

Unpaired set

Small paired set

Project page

7 of 16

Dataset

Semi-Supervised Raw-to-Raw Mapping

7

Unpaired set

Images are from our proposed dataset

Project page

8 of 16

Dataset

Semi-Supervised Raw-to-Raw Mapping

8

Paired set (anchor set)

Images are from our proposed dataset

Project page

9 of 16

Method

Semi-Supervised Raw-to-Raw Mapping

9

Project page

10 of 16

Method

Semi-Supervised Raw-to-Raw Mapping

10

Project page

11 of 16

Method

Semi-Supervised Raw-to-Raw Mapping

11

Project page

12 of 16

Results

Semi-Supervised Raw-to-Raw Mapping

12

NUS dataset

Our dataset

NUS dataset: Cheng et al., Illuminant estimation for color constancy: why spatial-domain methods work and the role of the color distribution. JOSA A, 2014

Project page

13 of 16

Results

Semi-Supervised Raw-to-Raw Mapping

13

Project page

14 of 16

Failure cases

Semi-Supervised Raw-to-Raw Mapping

14

iPhone X

Samsung Galaxy S9

Source camera

Ours

Target camera

Source camera

Ours

Target camera

Samsung Galaxy S9

iPhone X

Project page

15 of 16

Summary

  • We have presented a semi-supervised raw-to-raw mapping method
  • It is a practical way to achieve this mapping with a limited set of paired images required to train the model.
  • Under this practical scenario, we demonstrated SOTA results on two different datasets of DSLR and our proposed smartphone camera dataset.
  • Our method is the first step towards having practical and accurate raw-to-raw mapping to assist camera ISP manufacturing.

Semi-Supervised Raw-to-Raw Mapping

15

Project page

16 of 16

Thank you!

Semi-Supervised Raw-to-Raw Mapping

Project page