Collaborative Neural Rendering using Anime Character Sheets (2022-07-12T00:00:00.000000Z)

TL;DR

This paper presents the Collaborative Neural Rendering (CoNR) method, which creates new images for specified poses from a few reference images (AKA Character Sheets), and collects a character sheet dataset containing over 700,000 hand-drawn and synthesized images of diverse poses to facilitate research in this area.

Abstract

Drawing images of characters with desired poses is an essential but laborious task in anime production. Assisting artists to create is a research hotspot in recent years. In this paper, we present the Collaborative Neural Rendering (CoNR) method, which creates new images for specified poses from a few reference images (AKA Character Sheets). In general, the diverse hairstyles and garments of anime characters defies the employment of universal body models like SMPL, which fits in most nude human shapes. To overcome this, CoNR uses a compact and easy-to-obtain landmark encoding to avoid creating a unified UV mapping in the pipeline. In addition, the performance of CoNR can be significantly improved when referring to multiple reference images, thanks to feature space cross-view warping in a carefully designed neural network. Moreover, we have collected a character sheet dataset containing over 700,000 hand-drawn and synthesized images of diverse poses to facilitate research in this area. The code and dataset is available at https://github.com/megvii-research/IJCAI2023-CoNR.

Authors

Zuzeng Lin

1 papers

Ailin Huang

1 papers

Zhewei Huang

1 papers

TL;DR

Abstract

Authors

References75 items

SMPL: A Skinned Multi-Person Linear Model

A Dynamic Multi-Scale Voxel Flow Network for Video Prediction

Improving the Perceptual Quality of 2D Animation Interpolation

Neural-GIF: Neural Generalized Implicit Functions for Animating People in Clothing

Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies

EigenGAN: Layer-Wise Eigen-Learning for GANs

Motion Representations for Articulated Animation

Surrogate Gradient Field for Latent Space Manipulation

Deep Permutation Equivariant Structure from Motion

Deep Animation Video Interpolation in the Wild

Learning Transferable Visual Models From Natural Language Supervision

Template-Free Try-On Image Synthesis via Semantic-Guided Optimization

Neural Re-rendering of Humans from a Single Image

ANR: Articulated Neural Rendering for Virtual Avatars

PBNS: Physically Based Neural Simulator for Unsupervised Garment Pose Space Deformation

Portrait Neural Radiance Fields from a Single Image

Pose-Guided Human Animation from a Single Image in the Wild

Single-Shot Freestyle Dance Reenactment

One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing

D-NeRF: Neural Radiance Fields for Dynamic Scenes

Continuous Surface Embeddings

Liquid Warping GAN With Attention: A Unified Framework for Human Image Synthesis

Modeling Artistic Workflows for Image Generation and Editing

Closed-Form Factorization of Latent Semantics in GANs

Swapping Autoencoder for Deep Image Manipulation

Learning to Cartoonize Using White-Box Cartoon Representations

First Order Motion Model for Image Animation

Learning to Shadow Hand-Drawn Sketches

AnimeGAN: A Novel Lightweight GAN for Photo Animation

Learning an intrinsic garment space for interactive authoring of garment animation

Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis

Tex2Shape: Detailed Full Human Body Geometry From a Single Image

DenseBody: Directly Regressing Dense 3D Human Pose and Shape From a Single Color Image

Learning to Paint With Model-Based Deep Reinforcement Learning

OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

Everybody Dance Now

Improving Shape Deformation in Unsupervised Image-to-Image Translation

Non-Photorealistic Rendering

Human Appearance Transfer

Single Image Reflection Separation with Perceptual Losses

Interactive Sketch-Based Normal Map Generation with Deep Neural Networks

Synthesizing Images of Humans in Unseen Poses

DensePose: Dense Human Pose Estimation in the Wild

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Decoupled Weight Decay Regularization

Towards the Automatic Anime Characters Creation with Generative Adversarial Networks

Deformable Convolutional Networks

Video Frame Synthesis Using Deep Voxel Flow

Pose estimation of anime/manga characters: a case for synthetic data

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network

Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image

View Synthesis by Appearance Flow

Deep Residual Learning for Image Recognition

U-Net: Convolutional Networks for Biomedical Image Segmentation

FlowNet: Learning Optical Flow with Convolutional Networks

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Very Deep Convolutional Networks for Large-Scale Image Recognition

LabelMe: Online Image Annotation and Applications

Vectorizing Cartoon Animations

Et al

Markerless garment capture

Nonphotorealistic rendering

A Morphable Model For The Synthesis Of 3D Faces

Large steps in cloth simulation

Vectorization of Raster Manga by Deep Reinforcement Learning

: This anime does not exist

Nerf: Representing scenes as neural radiance fields for view synthesis

Danbooru2020: A large-scale crowdsourced and tagged anime illustration dataset

Danbooru community

Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos

Automatic Illu-mination Effects for 2D Characters

Pixiv dataset

In NIPS Workshop on Machine

In 25th annual conference on Computer graphics and interactive techniques

Field of Study