Future Video Synthesis With Object Motion Prediction (2020-04-01T00:00:00.000000Z)

TL;DR

An approach to predict future video frames given a sequence of continuous video frames in the past by decoupling the background scene and moving objects and shows that this model outperforms the state-of-the-art in terms of visual quality and accuracy.

Abstract

We present an approach to predict future video frames given a sequence of continuous video frames in the past. Instead of synthesizing images directly, our approach is designed to understand the complex scene dynamics by decoupling the background scene and moving objects. The appearance of the scene components in the future is predicted by non-rigid deformation of the background and affine transformation of moving objects. The anticipated appearances are combined to create a reasonable video in the future. With this procedure, our method exhibits much less tearing or distortion artifact compared to other approaches. Experimental results on the Cityscapes and KITTI datasets show that our model outperforms the state-of-the-art in terms of visual quality and accuracy.

Authors

Qifeng Chen

16 papers

Jaesik Park

6 papers

Yue Wu

1 papers

TL;DR

Abstract

Authors

References59 items

High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks

SME-Net: Sparse Motion Estimation for Parametric Video Prediction Through Reinforcement Learning

Compositional Video Prediction

Cross-Domain Complementary Learning with Synthetic Data for Multi-Person Part Segmentation

3D Motion Decomposition for RGBD Future Dynamic Scene Synthesis

Deep Flow-Guided Video Inpainting

Improved Conditional VRNNs for Video Prediction

UPSNet: A Unified Panoptic Segmentation Network

SiamRPN++: Evolution of Siamese Visual Tracking With Very Deep Networks

Improving Semantic Segmentation via Video Propagation and Label Relaxation

Disentangling Propagation and Generation for Video Prediction

TrafficPredict: Trajectory Prediction for Heterogeneous Traffic-Agents

SDC-Net: Video Prediction Using Spatially-Displaced Convolution

Video-to-Video Synthesis

Uncertainty-aware Short-term Motion Prediction of Traffic Actors for Autonomous Driving

Short-term Motion Prediction of Traffic Actors for Autonomous Driving using Deep Convolutional Networks

Hierarchical Long-term Video Prediction without Supervision

SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints

Stochastic Adversarial Video Prediction

Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks

DYAN: A Dynamical Atoms-Based Network for Video Prediction

GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose

Stochastic Video Generation with a Learned Prior

Generative Image Inpainting with Contextual Attention

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Future Person Localization in First-Person Videos

Long-Term On-board Prediction of People in Traffic Scenes Under Uncertainty

Predicting Scene Parsing and Motion Dynamics in the Future

ContextVP: Fully Context-Aware Video Prediction

PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume

SMSnet: Semantic motion segmentation using deep convolutional neural networks

Dual Motion GAN for Future-Flow Embedded Video Prediction

Decomposing Motion and Content for Natural Video Sequence Prediction

Unsupervised Learning of Disentangled Representations from Video

Learning to Generate Long-term Future via Hierarchical Prediction

Predicting Deeper into the Future of Semantic Segmentation

Video Frame Synthesis Using Deep Voxel Flow

Unsupervised Learning of Long-Term Motion Dynamics for Videos

Image-to-Image Translation with Conditional Adversarial Networks

Generating Videos with Scene Dynamics

Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Social LSTM: Human Trajectory Prediction in Crowded Spaces

An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders

Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning

Unsupervised Learning for Physical Interaction through Video Prediction

The Cityscapes Dataset for Semantic Urban Scene Understanding

Deep Residual Learning for Image Recognition

Spatio-temporal video autoencoder with differentiable memory

Deep multi-scale video prediction beyond mean square error

Unsupervised Learning of Video Representations using LSTMs

Very Deep Convolutional Networks for Large-Scale Image Recognition

From Actemes to Action: A Strongly-Supervised Representation for Detailed Action Understanding

Vision meets robotics: The KITTI dataset

The Recurrent Temporal Restricted Boltzmann Machine

Image quality assessment: from error visibility to structural similarity

Multiscale structural similarity for image quality assessment

TrafﬁcPredict: Trajectory Prediction for Heterogeneous Trafﬁc-Agents

Rectifier Nonlinearities Improve Neural Network Acoustic Models

Research Showcase @ CMU

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names