Deep Spatial Transformation for Pose-Guided Person Image Generation and Animation (2020-08-27T00:00:00.000000Z)

TL;DR

A differentiable global-flow local-attention framework to reassemble the inputs at the feature level and show that this framework can spatially transform the inputs in an efficient manner.

Abstract

Pose-guided person image generation and animation aim to transform a source person image to target poses. These tasks require spatial manipulation of source data. However, Convolutional Neural Networks are limited by the lack of ability to spatially transform the inputs. In this article, we propose a differentiable global-flow local-attention framework to reassemble the inputs at the feature level. This framework first estimates global flow fields between sources and targets. Then, corresponding local source feature patches are sampled with content-aware local attention coefficients. We show that our framework can spatially transform the inputs in an efficient manner. Meanwhile, we further model the temporal consistency for the person image animation task to generate coherent videos. The experiment results of both image generation and animation tasks demonstrate the superiority of our model. Besides, additional results of novel view synthesis and face image animation show that our model is applicable to other tasks requiring spatial transformation. The source code of our project is available at https://github.com/RenYurui/Global-Flow-Local-Attention.

Authors

Ge Li

5 papers

Shan Liu

6 papers

Yurui Ren

1 papers

TL;DR

Abstract

Authors

References66 items

Generative Adversarial Networks

Cross-Domain Correspondence Learning for Exemplar-Based Image Translation

Deep Image Spatial Transformation for Person Image Generation

First Order Motion Model for Image Animation

Your Local GAN: Designing Two Dimensional Local Attention Mechanisms for Generative Models

Few-shot Video-to-Video Synthesis

ClothFlow: A Flow-Based Model for Clothed Person Generation

DwNet: Dense warp-based network for pose-guided human video generation

Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis

PSGAN: Pose and Expression Robust Spatial-Aware GAN for Customizable Makeup Transfer

StructureFlow: Image Inpainting via Structure-Aware Appearance Flow

Bio-inspired synthesis of xishacorenes A, B, and C, and a new congener from fuscol

Local Relation Networks for Image Recognition

Generating Long Sequences with Sparse Transformers

Progressive Pose Attention Transfer for Person Image Generation

Unsupervised Person Image Generation With Semantic Parsing Transformation

Dense Intrinsic Appearance Flow for Human Pose Transfer

Linearized Multi-Sampling for Differentiable Image Transformation

Animating Arbitrary Objects via Deep Motion Transfer

OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

CrowdPose: Efficient Crowded Scenes Pose Estimation and a New Benchmark

3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Everybody Dance Now

Deep Video‐Based Performance Cloning

Video-to-Video Synthesis

Self-Attention Generative Adversarial Networks

A Variational U-Net for Conditional Appearance and Shape Generation

FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces

Spectral Normalization for Generative Adversarial Networks

DensePose: Dense Human Pose Estimation in the Wild

Generative Image Inpainting with Contextual Attention

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Deformable GANs for Pose-Based Human Image Generation

Non-local Neural Networks

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Attention is All you Need

Pose Guided Person Image Generation

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

Mask R-CNN

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields

Optical Flow Estimation Using a Spatial Pyramid Network

Instance Normalization: The Missing Ingredient for Fast Stylization

Layer Normalization

DeepFashion: Powering Robust Clothes Recognition and Retrieval with Rich Annotations

View Synthesis by Appearance Flow

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Deep Residual Learning for Image Recognition

ShapeNet: An Information-Rich 3D Model Repository

Scalable Person Re-identification: A Benchmark

Spatial Transformer Networks

FlowNet: Learning Optical Flow with Convolutional Networks

Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

Two-Stream Convolutional Networks for Action Recognition in Videos

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

I and i

A and V

Book Reviews

and s

Deep Learning

Quo vadis?

I and J

A. and Q

Quo Vadis

AND T

Field of Study

Journal Information

Name

Page

Volume

Venue Information

Name

Type

URL

Alternate Names