Learning to Decompose and Disentangle Representations for Video Prediction (2018-06-11T00:00:00.000000Z)

TL;DR

The Decompositional Disentangled Predictive Auto-Encoder (DDPAE) is proposed, a framework that combines structured probabilistic models and deep networks to automatically decompose the high-dimensional video that the authors aim to predict into components, and disentangle each component to have low-dimensional temporal dynamics that are easier to predict.

Abstract

Our goal is to predict future video frames given a sequence of input frames. Despite large amounts of video data, this remains a challenging task because of the high-dimensionality of video frames. We address this challenge by proposing the Decompositional Disentangled Predictive Auto-Encoder (DDPAE), a framework that combines structured probabilistic models and deep networks to automatically (i) decompose the high-dimensional video that we aim to predict into components, and (ii) disentangle each component to have low-dimensional temporal dynamics that are easier to predict. Crucially, with an appropriately specified generative model of video frames, our DDPAE is able to learn both the latent decomposition and disentanglement without explicit supervision. For the Moving MNIST dataset, we show that DDPAE is able to recover the underlying components (individual digits) and disentanglement (appearance and location) as we would intuitively do. We further demonstrate that DDPAE can be applied to the Bouncing Balls dataset involving complex interactions between multiple objects to predict the video frame directly from the pixels and recover physical states without explicit supervision.

Authors

Li Fei-Fei

37 papers

Juan Carlos Niebles

9 papers

Jun-Ting Hsieh

1 papers

TL;DR

Abstract

Authors

References49 items

Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects

Visual Robot Task Planning

Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions

Folded Recurrent Neural Networks for Future Video Prediction

Hybrid VAE: Improving Deep Generative Models using Partial Observations

MoCoGAN: Decomposing Motion and Content for Video Generation

Generating the Future with Adversarial Transformers

Decomposing Motion and Content for Natural Video Sequence Prediction

Unsupervised Learning of Disentangled Representations from Video

The Pose Knows: Video Forecasting by Generating Pose Futures

Learning to Generate Long-term Future via Hierarchical Prediction

Neural Expectation Maximization

Transformation-Based Models of Video Sequences

Interaction Networks for Learning about Objects, Relations and Physics

Object-Centric Representation Learning from Unlabeled Videos

A Compositional Object-Based Approach to Learning Physical Dynamics

Video Pixel Networks

Contextual RNN-GANs for Abstract Reasoning Diagram Generation

Generating Videos with Scene Dynamics

Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Social LSTM: Human Trajectory Prediction in Crowded Spaces

An Uncertain Future: Forecasting from Static Images Using Variational Autoencoders

Tagger: Deep Unsupervised Perceptual Grouping

Dynamic Filter Networks

Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning

Unsupervised Learning for Physical Interaction through Video Prediction

Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data

View Synthesis by Appearance Flow

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models

One-Shot Generalization in Deep Generative Models

Generating Notifications for Missing Actions: Don't Forget to Turn the Lights Off!

Learning Visual Predictive Models of Physics for Playing Billiards

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Spatio-temporal video autoencoder with differentiable memory

Deep multi-scale video prediction beyond mean square error

Action-Conditional Video Prediction using Deep Networks in Atari Games

Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting

Spatial Transformer Networks

Unsupervised Learning of Video Representations using LSTMs

DRAW: A Recurrent Neural Network For Image Generation

Adam: A Method for Stochastic Optimization

Video (language) modeling: a baseline for generative models of natural videos

Sequence to Sequence Learning with Neural Networks

A Hierarchical Representation for Future Action Prediction

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Auto-Encoding Variational Bayes

Et al

Research Showcase @ CMU

Influence of cultivation temperature on the ligninolytic activity of selected fungal strains

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names