ModelScope Text-to-Video Technical Report (2023-08-12T00:00:00.000000Z)

TL;DR

The ModelScopeT2V incorporates spatio-temporal blocks to ensure consistent frame generation and smooth movement transitions and demonstrates superior performance over state-of-the-art methods across three evaluation metrics.

Abstract

This paper introduces ModelScopeT2V, a text-to-video synthesis model that evolves from a text-to-image synthesis model (i.e., Stable Diffusion). ModelScopeT2V incorporates spatio-temporal blocks to ensure consistent frame generation and smooth movement transitions. The model could adapt to varying frame numbers during training and inference, rendering it suitable for both image-text and video-text datasets. ModelScopeT2V brings together three components (i.e., VQGAN, a text encoder, and a denoising UNet), totally comprising 1.7 billion parameters, in which 0.5 billion parameters are dedicated to temporal capabilities. The model demonstrates superior performance over state-of-the-art methods across three evaluation metrics. The code and an online demo are available at \url{https://modelscope.cn/models/damo/text-to-video-synthesis/summary}.

Authors

Hangjie Yuan

5 papers

Shiwei Zhang

2 papers

Xiang Wang

3 papers

TL;DR

Abstract

Authors

References69 items

VideoComposer: Compositional Video Synthesis with Motion Controllability

Align Your Latents: High-Resolution Video Synthesis with Latent Diffusion Models

VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation

LLaMA: Open and Efficient Foundation Language Models

Composer: Creative and Controllable Image Synthesis with Composable Conditions

Structure and Content-Guided Video Synthesis with Diffusion Models

Progressive Learning without Forgetting

MagicVideo: Efficient Video Generation With Latent Diffusion Models

Imagic: Text-Based Real Image Editing with Diffusion Models

Imagen Video: High Definition Video Generation with Diffusion Models

Make-A-Video: Text-to-Video Generation without Text-Video Data

AudioLM: A Language Modeling Approach to Audio Generation

RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection

Diffusion Models: A Comprehensive Survey of Methods and Applications

Riemannian Diffusion Models

Classifier-Free Diffusion Guidance

Diffusion Models for Video Prediction and Infilling

Zero-Shot Voice Conditioning for Denoising Diffusion TTS Models

DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps

CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Flexible Diffusion Modeling of Long Videos

Fast Sampling of Diffusion Models with Exponential Integrator

Hierarchical Text-Conditional Image Generation with CLIP Latents

Video Diffusion Models

Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillation

Diffusion Probabilistic Modeling for Video Generation

Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks

Pseudo Numerical Methods for Diffusion Models on Manifolds

Truncated Diffusion Probabilistic Models

Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Quality

StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2

High-Resolution Image Synthesis with Latent Diffusion Models

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

Generative Adversarial Networks

Grounded Language-Image Pre-training

NÜWA: Visual Synthesis Pre-training for Neural visUal World creAtion

Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions

LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairs

Variational Diffusion Models

LoRA: Low-Rank Adaptation of Large Language Models

Diffusion Models Beat GANs on Image Synthesis

GODIVA: Generating Open-DomaIn Videos from nAtural Descriptions

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval

ViViT: A Video Vision Transformer

Perceiver: General Perception with Iterative Attention

Learning Transferable Visual Models From Natural Language Supervision

Taming Transformers for High-Resolution Image Synthesis

Score-Based Generative Modeling through Stochastic Differential Equations

Denoising Diffusion Implicit Models

DiffWave: A Versatile Diffusion Model for Audio Synthesis

Denoising Diffusion Probabilistic Models

Language Models are Few-Shot Learners

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

Decomposed

Towards Accurate Generative Models of Video: A New Metric & Challenges

Decoupled Weight Decay Regularization

Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Attention is All you Need

Overcoming catastrophic forgetting in neural networks

Learning without Forgetting

3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation

MSR-VTT: A Large Video Description Dataset for Bridging Video and Language

U-Net: Convolutional Networks for Biomedical Image Segmentation

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Testing the Manifold Hypothesis

AUTO-ENCODING VARIATIONAL BAYES

Progressive distillation for fast sampling of diffusion models

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type