1 of 16

Line Art Correlation Matching Feature Transfer Network �for Automatic Animation Colorization

Qian Zhang, Bo Wang, Wei Wen, Hai Li, Jun Hui Liu

iQIYI Inc

arXiv:2004.06718, 14 Apr. 2020

2 of 16

ABSTRACT

  • GOAL : Original animation key frame are sketched by lead artists and in- between frames are sketched by inexperienced artists. To reduce workload, this network try to colorized in-between frame automatically

Input line art

This method result

Original result

↑based color frame

3 of 16

WORKFLOW

U-Net

4 of 16

Correlation Matching Feature Transfer Model(CMFT)

5 of 16

 

  • f is kernel func. computes similarity of scalars(gaussian func. in this paper).

​

​

​

  • Pixels with similar semantic contents are similar in features,

so correlation can be represented as similarity.

6 of 16

 

  •  

7 of 16

Network Structure(LCMFTN)

  • With 4 encoder and 1 decoder

8 of 16

NETWORK

  •  

9 of 16

DATASET

  • Only 10 cartoon films
  • Divided into many shots, get training pairs in same shot
  • To extend more training data, sliding window in a shot
  • Using LeNet convert colored frames to line arts

->get 60k pairs data

10 of 16

Average Time for colorize a frame

  • With single Tesla P40 GPU

​

​

​

​

​

11 of 16

LCMFTN RESULT-stride 1

Input line art

LCMFTN result

LCMFTN (no CMFT)

Ground truth

12 of 16

LCMFTN RESULT-stride 5

Input line art

LCMFTN result

LCMFTN (no CMFT)

Ground truth

13 of 16

LCMFTN RESULT-stride 10

Input line art

LCMFTN result

LCMFTN (no CMFT)

Ground truth

14 of 16

COMPARISON

Input line art

LCMFTN

LCMFTN

(no CMFT)

TCVC

(our loss)

TCVC

Pix2Pix

(ref/our loss)

Pix2Pix

(ref loss)

DeepAnalogy

Ground truth

15 of 16

CONCLUSION

  • Design CMFT model to maintain spatial and time consistency, especially when big motion occurs.
  • Strategy of extending dataset

​

​

​

​

16 of 16

END�