Eliminating Gradient Conflict in Reference-based Line-Art Colorization (2022-07-13T00:00:00.000000Z)

TL;DR

This work proposes a novel attention mechanism using this training strategy, Stop-Gradient Attention (SGA), outperforming the attention baseline by a large margin with better training stability and demonstrates significant improvements in Fr\'echet Inception Distance and structural similarity index measure on several benchmarks.

Abstract

Reference-based line-art colorization is a challenging task in computer vision. The color, texture, and shading are rendered based on an abstract sketch, which heavily relies on the precise long-range dependency modeling between the sketch and reference. Popular techniques to bridge the cross-modal information and model the long-range dependency employ the attention mechanism. However, in the context of reference-based line-art colorization, several techniques would intensify the existing training difficulty of attention, for instance, self-supervised training protocol and GAN-based losses. To understand the instability in training, we detect the gradient flow of attention and observe gradient conflict among attention branches. This phenomenon motivates us to alleviate the gradient issue by preserving the dominant gradient branch while removing the conflict ones. We propose a novel attention mechanism using this training strategy, Stop-Gradient Attention (SGA), outperforming the attention baseline by a large margin with better training stability. Compared with state-of-the-art modules in line-art colorization, our approach demonstrates significant improvements in Fr\'echet Inception Distance (FID, up to 27.21%) and structural similarity index measure (SSIM, up to 25.67%) on several benchmarks. The code of SGA is available at https://github.com/kunkun0w0/SGA .

Authors

Zhao Kang

2 papers

Zhengyang Geng

1 papers

Wenyu Chen

1 papers

TL;DR

Abstract

Authors

References61 items

On Training Implicit Models

Is Attention Better Than Matrix Decomposition?

The Animation Transformer: Visual Correspondence via Segment Matching

Dual Color Space Guided Sketch Colorization

When Vision Transformers Outperform ResNets without Pretraining or Strong Data Augmentations

Unbalanced Feature Transport for Exemplar-based Image Translation

CoCosNet v2: Full-Resolution Correspondence Learning for Image Translation

User-Guided Line Art Flat Filling with Split Filling Mechanism

Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks

An Empirical Study of Training Self-Supervised Vision Transformers

ViViT: A Video Vision Transformer

CvT: Introducing Convolutions to Vision Transformers

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Video Transformer Network

Training data-efficient image transformers & distillation through attention

PCT: Point cloud transformer

Exploring Simple Siamese Representation Learning

Point Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Modeling Artistic Workflows for Image Generation and Editing

Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Linformer: Self-Attention with Linear Complexity

Reference-Based Sketch Image Colorization Using Augmented-Self Reference and Dense Semantic Correspondence

Line Art Correlation Matching Feature Transfer Network for Automatic Animation Colorization

Cross-Domain Correspondence Learning for Exemplar-Based Image Translation

Efficient Content-Based Sparse Attention with Routing Transformers

Gradient Surgery for Multi-Task Learning

StarGAN v2: Diverse Image Synthesis for Multiple Domains

SEAN: Image Synthesis With Semantic Region-Adaptive Normalization

Adversarial Colorization of Icons Based on Contour and Color Conditions

Tag2Pix: Line Art Colorization Using Text Tag With SECat and Changing Loss

Expectation-Maximization Attention Networks for Semantic Segmentation

Graph matching based anime colorization with multiple references

LatentGNN: Learning Efficient Non-local Relations for Visual Recognition

Learning Combinatorial Embedding Networks for Deep Graph Matching

Semantic Image Synthesis With Spatially-Adaptive Normalization

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context

Graph-Based Global Reasoning Networks

A2-Nets: Double Attention Networks

Self-Attention Generative Adversarial Networks

Non-local Neural Networks

Neural Discrete Representation Learning

Graph Attention Networks

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Attention is All you Need

Style Transfer for Anime Sketches with Enhanced Residual U-net and Auxiliary Classifier GAN

Real-time user-guided image colorization with learned deep priors

Image-to-Image Translation with Conditional Adversarial Networks

Least Squares Generative Adversarial Networks

Semi-Supervised Classification with Graph Convolutional Networks

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Identity Mappings in Deep Residual Networks

LogDet Rank Minimization with Application to Subspace Clustering

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Adam: A Method for Stochastic Optimization

Sinkhorn Distances: Lightspeed Computation of Optimal Transport

XDoG: An eXtended difference-of-Gaussians compendium including advanced image stylization

Gauge Equivariant Transformer

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

GENERATIVE ADVERSARIAL NETS

Eliminating Gradient Conflict in Reference-based Line-art

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names