MotionVideoGAN: A Novel Video Generator Based on the Motion Space Learned From Image Pairs (2023-03-06T00:00:00.000000Z)

TL;DR

This work designs a novel framework to build the motion space, aiming to achieve content consistency and fast convergence for video generation, and proposes an image pair generator named MotionStyleGAN to generate image pairs sharing the same contents and producing various motions.

Abstract

Video generation has achieved rapid progress benefiting from high-quality renderings provided by powerful image generators. We regard the video synthesis task as generating a sequence of images sharing the same contents but varying in motions. However, most previous video synthesis frameworks based on pre-trained image generators treat content and motion generation separately, leading to unrealistic generated videos. Therefore, we design a novel framework to build the motion space, aiming to achieve content consistency and fast convergence for video generation. We present MotionVideoGAN, a novel video generator synthesizing videos based on the motion space learned by pre-trained image pair generators. Firstly, we propose an image pair generator named MotionStyleGAN to generate image pairs sharing the same contents and producing various motions. Then we manage to acquire motion codes to edit one image in the generated image pairs and keep the other unchanged. The motion codes help us edit images within the motion space since the edited image shares the same contents with the other unchanged one in image pairs. Finally, we introduce a latent code generator to produce latent code sequences using motion codes for video generation. Our approach achieves state-of-the-art performance on the most complex video dataset ever used for unconditional video generation evaluation, UCF101.

Authors

Jin Zhu

1 papers

Huimin Ma

1 papers

Jiansheng Chen

1 papers

TL;DR

Abstract

Authors

References73 items

UAV-Satellite View Synthesis for Cross-View Geo-Localization

Video Captioning based on Image Captioning as Subsidiary Content

Scenario-Aware Recurrent Transformer for Goal-Directed Video Captioning

Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks

StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2

Efficient Geometry-aware 3D Generative Adversarial Networks

EditGAN: High-Precision Semantic Image Editing

StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN

Alias-Free Generative Adversarial Networks

Low-Rank Subspaces in GANs

Self-Supervised Video GANs: Learning for Appearance Consistency and Motion Coherency

A Good Image Generator Is What You Need for High-Resolution Video Synthesis

VideoGPT: Video Generation using VQ-VAE and Transformers

DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization

Image-to-Image Translation: Methods and Applications

You Only Need Adversarial Supervision for Semantic Image Synthesis

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation

Closed-Form Factorization of Latent Semantics in GANs

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

Training Generative Adversarial Networks with Limited Data

InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs

Temporal Shift GAN for Large Scale Video Generation

Non-Adversarial Video Synthesis with Learned Priors

Unsupervised Discovery of Interpretable Directions in the GAN Latent Space

Controlling generative models with continuous factors of variations

Lower Dimensional Kernels for Video Discriminators

Analyzing and Improving the Image Quality of StyleGAN

Image2StyleGAN++: How to Edit the Embedded Images?

Markov Decision Process for Video Generation

Adversarial Video Generation on Complex Datasets

AutoGAN: Neural Architecture Search for Generative Adversarial Networks

Interpreting the Latent Space of GANs for Semantic Face Editing

On the "steerability" of generative adversarial networks

Efficient Video Generation on Complex Datasets

Large Scale Adversarial Representation Learning

GANalyze: Toward Visual Definitions of Cognitive Image Properties

Adversarial Video Captioning

Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?

A Layer-Based Sequential Framework for Scene Generation with GANs

A Style-Based Generator Architecture for Generative Adversarial Networks

Towards Accurate Generative Models of Video: A New Metric & Challenges

Train Sparsely, Generate Densely: Memory-Efficient Unsupervised Training of High-Resolution Temporal GAN

Towards High Resolution Video Generation with Progressive Growing of Sliced Wasserstein GANs

Large Scale GAN Training for High Fidelity Natural Image Synthesis

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

Diverse Image-to-Image Translation via Disentangled Representations

Exemplar Guided Unsupervised Image-to-Image Translation

Self-Attention Generative Adversarial Networks

FaceForensics: A Large-scale Video Dataset for Forgery Detection in Human Faces

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Learning to Generate Time-Lapse Videos Using Multi-stage Dynamic Generative Adversarial Networks

MoCoGAN: Decomposing Motion and Content for Video Generation

Wasserstein Generative Adversarial Networks

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Improved Training of Wasserstein GANs

GP-GAN: Towards Realistic High-Resolution Image Blending

AdaGAN: Boosting Generative Models

Temporal Generative Adversarial Nets with Singular Value Clipping

Image-to-Image Translation with Conditional Adversarial Networks

Generating Videos with Scene Dynamics

Improved Techniques for Training GANs

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Long Short-Term Memory

Asynchronous Generative Adversarial Network for Asymmetric Unpaired Image-to-Image Translation

MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation

GENERATIVE ADVERSARIAL NETS

A collaborative framework for 3d alignment and classification of heterogeneous subvolumes in cryo-electron tomography

HeiscurrentlyaProfessorwiththeSchoolofComputer andCommunicationEngineering,UniversityofSci-enceandTechnologyBeijing,Beijing

Supplementary generated video examples of 16 frames by Motion-VideoGAN trained on FaceForensics

Field of Study

Journal Information

Name