Towards Smooth Video Composition (2022-12-14T00:00:00.000000Z)

TL;DR

This work investigates modeling the temporal relations for composing video with arbitrary length, from a few frames to even infinite, using generative adversarial networks (GANs), and develops a novel B-Spline based motion representation to ensure temporal smoothness to achieve infinite-length video generation.

Abstract

Video generation requires synthesizing consistent and persistent frames with dynamic content over time. This work investigates modeling the temporal relations for composing video with arbitrary length, from a few frames to even infinite, using generative adversarial networks (GANs). First, towards composing adjacent frames, we show that the alias-free operation for single image generation, together with adequately pre-learned knowledge, brings a smooth frame transition without compromising the per-frame quality. Second, by incorporating the temporal shift module (TSM), originally designed for video understanding, into the discriminator, we manage to advance the generator in synthesizing more consistent dynamics. Third, we develop a novel B-Spline based motion representation to ensure temporal smoothness to achieve infinite-length video generation. It can go beyond the frame number used in training. A low-rank temporal modulation is also proposed to alleviate repeating contents for long video generation. We evaluate our approach on various datasets and show substantial improvements over video generation baselines. Code and models will be publicly available at https://genforce.github.io/StyleSV.

Authors

Bolei Zhou

20 papers

Ceyuan Yang

5 papers

Qihang Zhang

3 papers

TL;DR

Abstract

Authors

References64 items

StyleFaceV: Face Video Generation via Decomposing and Recomposing Pretrained StyleGAN3

InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images

Rewriting geometric rules of a GAN

Generating Long Videos of Dynamic Scenes

CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformer

Video Diffusion Models

Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretraining

Transframer: Arbitrary Frame Prediction with Generative Models

Latent Image Animator: Learning to Animate Images via Latent Space Navigation

Look Outside the Room: Synthesizing A Consistent Long-Term 3D Scene Video from A Single Image

Diffusion Probabilistic Modeling for Video Generation

Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks

Third Time's the Charm? Image and Video Editing with StyleGAN3

StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2

NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN

Alias-Free Generative Adversarial Networks

A Good Image Generator Is What You Need for High-Resolution Video Synthesis

DriveGAN: Towards a Controllable High-Quality Neural Simulation

GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions

VideoGPT: Video Generation using VQ-VAE and Transformers

Predicting Video with VQVAE

Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image

Rewriting a Deep Generative Model

Latent Video Transformer

Learning to Simulate Dynamic Environments With GameGAN

InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs

Human Motion Transfer from Poses in the Wild

Transformation-based Adversarial Video Prediction on Large-Scale Data

First Order Motion Model for Image Animation

Analyzing and Improving the Image Quality of StyleGAN

Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesis

Adversarial Video Generation on Complex Datasets

On the "steerability" of generative adversarial networks

GANalyze: Toward Visual Definitions of Cognitive Image Properties

Scaling Autoregressive Video Models

Video Generation From Single Semantic Label Map

VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation

Animating Arbitrary Objects via Deep Motion Transfer

A Style-Based Generator Architecture for Generative Adversarial Networks

Towards Accurate Generative Models of Video: A New Metric & Challenges

Train Sparsely, Generate Densely: Memory-Efficient Unsupervised Training of High-Resolution Temporal GAN

TSM: Temporal Shift Module for Efficient Video Understanding

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Recurrent World Models Facilitate Policy Evolution

Everybody Dance Now

Video-to-Video Synthesis

Pose Guided Human Video Generation

Stochastic Adversarial Video Prediction

Stochastic Variational Video Prediction

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Learning to Generate Time-Lapse Videos Using Multi-stage Dynamic Generative Adversarial Networks

MoCoGAN: Decomposing Motion and Content for Video Generation

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Temporal Generative Adversarial Nets with Singular Value Clipping

Video Pixel Networks

Generating Videos with Scene Dynamics

Conditional Image Generation with PixelCNN Decoders

Unsupervised Learning for Physical Interaction through Video Prediction

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

GENERATIVE ADVERSARIAL NETS

Visualizing Data using t-SNE

SUBROUTINE PACKAGE FOR CALCULATING WITH B-SPLINES.

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names