Line Art Correlation Matching Network for Automatic Animation Colorization
share
Automatic animation line art colorization is a challenging computer vision problem, since the information of the line art is highly sparse and abstracted and there exists a strict requirement for the color and style consistency between frames. Recently, a lot of Generative Adversarial Network (GAN) based image-to-image translation methods for single line art colorization have emerged. They can generate perceptually appealing results conditioned on line art images. However, these methods can not be adopted for the purpose of animation colorization because there is a lack of consideration of the in-between frame consistency. Existing methods simply input the previous colored frame as a reference to color the next line art, which will mislead the colorization due to the spatial misalignment of the previous colored frame and the next line art especially at positions where apparent changes happen. To address these challenges, we design a kind of correlation matching model (called CM) to align the colored reference in a learnable way and integrate the model into an U-Net structure generator in a coarse-to-fine manner. This enables the generator to reconstruct the layer-wise synchronized features from the deep semantic code to the content progressively. Extension evaluations show that CM model can effectively improve the in-between consistency and the quality of colored frames especially when the motion is intense and diverse.
READ FULL TEXT