1 of 67

Learning a self-supervised tone mapping operator via feature contrast masking loss

Chao Wang1, Bin Chen1, Hans-Peter Seidel1

Karol Myszkowski1, Ana Serrano2

1Max-Planck-Institut für Informatik

2University of Zaragoza

Reims 2022

2 of 67

Luminance Mapping

2

​

​

​

​

​

​

​

10-6

10-4

10-2

100

102

104

106

108

Lumiance [cd/m2]

Cannot cover the real world luminance.

Reims 2022

3 of 67

High Dynamic Range Imaging

3

LDR

​

HDR

Up to 500 cd/m^2

peak brightness

2000 – 1000cd/m^2

50 dB

camera dynamic range

120 dB

1:1000

display contrast

1:1000 000

from 8 to 16 bit

quantization

Floating point or variable

display-referred

image compression

scene-referred

display-limited

fidelity

As good as the eye can see

Reims 2022

4 of 67

High Dynamic Range Imaging

4

Display-referred

​

Scene-referred

Up to 500 cd/m^2

peak brightness

2000 – 1000cd/m^2

50 dB

camera dynamic range

120 dB

1:1000

display contrast

1:1000 000

from 8 to 16 bit

quantization

Floating point or variable

display-limited

fidelity

As good as the eye can see

Reims 2022

5 of 67

Tone Mapping

5

Real World

Display

Reims 2022

6 of 67

Challenge in Tone Mapping

6

Linear HDR

Tone Mapping Result

Reims 2022

7 of 67

Review

7

TMO

Traditional Methods

(Model Driven)

Global Methods

Local Methods

Learning-based Methods

(Data Driven)

[Jack 93], [Larson 97], [Drago 03], [Mantiuk 08]

[Fattal 02], [Reinhard 02], [Durand 02], [Mantiuk 06]

Reims 2022

8 of 67

Review

8

TMO

Traditional Methods

(Model Driven)

Global Methods

Local Methods

Learning-based Methods

(Data Driven)

Not user-friendly

Reims 2022

9 of 67

Review

9

TMO

Traditional Methods

(Model Driven)

Global Methods

Local Methods

Learning-based Methods

(Data Driven)

[Zhang 19], [Rana 19], [Panetta 21], [Zhang 21], [Guo 21]

Reims 2022

10 of 67

Review

10

TMO

Traditional Methods

(Model Driven)

Global Methods

Local Methods

Learning-based Methods

(Data Driven)

The HDR-LDR training pairs?

Not user-friendly

Reims 2022

11 of 67

Learning-based Methods

11

TMO 1

TMO 2

TMO 3

.....…

TMO N

TMQI 1

TMQI 2

TMQI 3

…….

TMQI N

The Best TMQI [1]

HDR

LDR Label

[1] Objective quality assessment of tone-mapped images. IEEE Transactions on Image processing, 2012

Reims 2022

12 of 67

Learning-based Methods

12

HDR inputs

LDR labels

Outputs

Training:

Reims 2022

13 of 67

Learning-based Methods

13

HDR inputs

LDR Prediction

Testing:

Reims 2022

14 of 67

Learning-based Methods

14

TMO

Traditional Methods

(Model Driven)

Global Methods

Local Methods

Learning-based Methods

(Data Driven)

Not user-friendly

No perfect TMO labels ;

Limited HDR image datasets

Reims 2022

15 of 67

Rethinking Tone Mapping

15

TM: Simpler task than image restoration

Reims 2022

16 of 67

Rethinking Tone Mapping

16

TM: Simpler task than image restoration

Key cues of Perception: Contrast [1]

[1] Vision science: Photons to phenomenology. MIT Press, 1999.

Reims 2022

17 of 67

Rethinking Tone Mapping

17

TM: Simpler task than image restoration

Key cues of Perception: Contrast [1]

[1] Vision science: Photons to phenomenology. MIT Press, 1999.

Self-supervised network & Label free (perfect image content adaptation )

Reims 2022

18 of 67

Rethinking Tone Mapping

18

TM: Simpler task than image restoration

Key cues of Perception: Contrast [1]

[1] Vision science: Photons to phenomenology. MIT Press, 1999.

Self-supervised network & Label free (perfect image content adaptation )

Problems to solve:

​

  • Linear HDR images feature different intensity histograms than LDR images. How to use an HDR image as a label for a tone mapped image?

​

  • How to model and reproduce the contrast appearance in HDR images?

​

Reims 2022

19 of 67

HDR Distribution

19

HDR image (linear)

HDR Histogram

Reims 2022

20 of 67

LDR Distribution

20

LDR image

LDR Histogram

Reims 2022

21 of 67

μ-Law

21

 

[1] µ-law based hdr coding and its error analysis. IEICE Trans. (2011)

Reims 2022

22 of 67

μ-Law

22

X axis: Pixel value of HDR image

​

Y axis: Pixel value after μ law compression

 

[1] µ-law based hdr coding and its error analysis. IEICE Trans. (2011)

Reims 2022

23 of 67

Issues of μ-law

23

µ = 10

µ = 100

µ = 1000

A larger µ is better

Reims 2022

24 of 67

Issues of μ-law

24

A smaller µ is better

µ = 10

µ = 100

µ = 1000

Reims 2022

25 of 67

Issues of μ-law

25

A smaller µ is better

µ = 10

µ = 100

µ = 1000

A fixed µ cannot handle all situations!

Reims 2022

26 of 67

Adaptive μ-law

26

X axis: Median Value of HDR image

​

Y axis: Corresponding µ value

Reims 2022

27 of 67

Adaptive μ-law

27

X axis: Median Value of HDR image

​

Y axis: Corresponding µ value

 

 

Reims 2022

28 of 67

Distribution re-arrangement

28

HDR image (linear)

HDR image (Ada-µ)

Reims 2022

29 of 67

Loss Space

29

Feature Space

Image Space

Better in modeling human perception [1]

[1] Perceptual losses for real-time style transfer and super-resolution. ECCV 2016

Reims 2022

30 of 67

Perceptual Loss

30

 

 

 

Loss

 

 

Reims 2022

31 of 67

Perceptual Loss

31

Over exposed

Dark

Over exposed

Dark

Reims 2022

32 of 67

Feature Contrast Masking Loss

32

Feature Contrast

Self Feature Contrast Masking

Neighborhood Feature Contrast Masking

All contrast processing in feature space

Reims 2022

33 of 67

Feature Contrast

33

Convolution

Gaussian Kernel

P

Reims 2022

34 of 67

Self Feature Contrast Masking

34

 

 

** α promote low contrast and suppress high contrast like human visual system (HVS).

typical alpha = 0.5

 

Reims 2022

35 of 67

Neighborhood Feature Contrast Masking

35

 

 

: local standard deviation of feature maps

: local mean of feature maps

Reims 2022

36 of 67

Feature Contrast Masking Loss

36

 

 

 

Loss

 

FCM

FCM

 

 

Reims 2022

37 of 67

Visualization of FCM

37

Feature contrast masking visualization of VGG feature maps (1st layer, 18th channel).

Image

Feature map

Reims 2022

38 of 67

Visualization of FCM

38

Feature contrast masking visualization of VGG feature maps (1st layer, 18th channel).

Image

Feature map

FCM feature map

FCM

Self feature contrast masking map

Neighborhood feature contrast masking map

Reims 2022

39 of 67

FCM vs. Standard VGG Loss

39

Reims 2022

40 of 67

FCM vs. Standard VGG Loss

40

Reims 2022

41 of 67

FCM vs. Standard VGG Loss

41

Reims 2022

42 of 67

FCM V.S. Standard VGG Loss

42

arrow to emphasize

Reims 2022

43 of 67

Network Input

43

Logarithms

Darker Brightness

log2(luminance)

HDR image

-6

2

Reims 2022

44 of 67

Network Input

44

HDR generation

Multiple exposure sampling?

Certain number of exposures can keep all the HDR information [1]

[1] ꟻLIP: A Tool for Visualizing and Communicating Errors in Rendered Images. EG 2022.

Reims 2022

45 of 67

Network Input

45

 

 

 

 

 

 

 

Floating values;

Only the highlight region clipped;

Darker pixels are still there.

Reims 2022

46 of 67

Full Pipeline

46

Reims 2022

47 of 67

Network Structure

47

Reims 2022

48 of 67

Results Comparison

48

Reims 2022

49 of 67

Results Comparison

49

Reims 2022

50 of 67

Results Comparison

50

Reims 2022

51 of 67

Visualization Results

51

Ours

Guo

Zhang

Liang

Color shift

Blurry

Blurry & dark

Reims 2022

52 of 67

Visualization Results

52

Ours

Guo

Zhang

Liang

Artifacts

Blurry

Dark

Reims 2022

53 of 67

Visualization Results

53

Ours

Guo

Zhang

Liang

Artifacts

Blurry

Dark

Reims 2022

54 of 67

DeepTMO Comparison

54

Ours

DeepTMO [1]

Blurry

Saturated

[1] Deep Tone Mapping Operator for High Dynamic Range Images. IEEE Trans Image Processing (2019).

Reims 2022

55 of 67

DeepTMO Comparison

55

Ours

DeepTMO

Blurry

Saturated

Reims 2022

56 of 67

TMO-Net Comparison

56

Ours

TMO-Net [1]

Over-exposed

[1] TMO-Net: A Parameter-Free Tone Mapping Operator Using Generative Adversarial Network, and Performance Benchmarking on Large Scale HDR Dataset. IEEE Access 2021.

Reims 2022

57 of 67

TMO-Net Comparison

57

Ours

TMO-Net

Over-exposed

Reims 2022

58 of 67

Ablation Study: Adaptive μ-law

58

Adaptive μ-law

w/o Adaptive μ-law

Dark

Reims 2022

59 of 67

Ablation Study: Adaptive μ-law

59

Adaptive μ-law

w/o Adaptive μ-law

Artifacts

Saturated

Reims 2022

60 of 67

Ablation Study: FCM loss V.S VGG loss

60

FCM loss

VGG loss

Over exposed

Dark

Reims 2022

61 of 67

Ablation Study: FCM loss V.S VGG loss

61

FCM loss

VGG loss

Over exposed

Dark

Reims 2022

62 of 67

Ablation Study: MEI V.S Log inputs

62

Multiple Exposure Inputs

Log inputs

Reims 2022

63 of 67

Ablation Study: MEI V.S Log inputs

63

Multiple Exposure Inputs

Log inputs

Reims 2022

64 of 67

Limitation and Discussion

64

720 x 480: 126 s

Train image specific model

Time consuming

Offline Training

720 x 480: 0.12 s

Performance decrease a bit

Reims 2022

65 of 67

More contents

65

For more results and code:

​

https://self-supervisedTMO.mpi-inf.mpg.de/

Reims 2022

66 of 67

Conclusion

66

  • Design adaptive μ-law to re-arrange the HDR distribution;

​

  • Propose feature contrast masking loss to constrain the tone mapping results have a good perceptual reproduction with the HDR image;

​

  • Introduce self-supervised learning for the first time to deep tone mapping;

​

  • Achieve remarkable stable performance for various kinds of HDR images, and have better perceptual matching.

​

​

​

Reims 2022

67 of 67

67

Thanks

Reims 2022