Disentangling Physical Dynamics From Unknown Factors for Unsupervised Video Prediction (2020-03-03T00:00:00.000000Z)

TL;DR

PhyDNet is introduced, a two-branch deep architecture, which explicitly disentangles PDE dynamics from unknown complementary information, and a new recurrent physical cell (PhyCell) is proposed, inspired from data assimilation techniques, for performing PDE-constrained prediction in latent space.

Abstract

Leveraging physical knowledge described by partial differential equations (PDEs) is an appealing way to improve unsupervised video forecasting models. Since physics is too restrictive for describing the full visual content of generic video sequences, we introduce PhyDNet, a two-branch deep architecture, which explicitly disentangles PDE dynamics from unknown complementary information. A second contribution is to propose a new recurrent physical cell (PhyCell), inspired from data assimilation techniques, for performing PDE-constrained prediction in latent space. Extensive experiments conducted on four various datasets show the ability of PhyDNet to outperform state-of-the-art methods. Ablation studies also highlight the important gain brought out by both disentanglement and PDE-constrained prediction. Finally, we show that PhyDNet presents interesting features for dealing with missing data and long-term forecasting.

Authors

V. Guen

2 papers

Nicolas Thomé

3 papers

TL;DR

Abstract

Authors

References83 items

Addressing Failure Prediction by Learning Model Confidence

Shape and Time Distortion Loss for Training Deep Time Series Forecasting Models

Compositional Video Prediction

Data assimilation as a learning tool to infer ordinary differential equation representations of dynamical models

Unsupervised Learning of Object Structure and Dynamics from Videos

Predicting Future Frames Using Retrospective Cycle GAN

Neural Jump Stochastic Differential Equations

Recurrent Kalman Networks: Factorized Inference in High-Dimensional Deep Feature Spaces

Improved Conditional VRNNs for Video Prediction

Differentiable Physics-informed Graph Networks

Disentangling Propagation and Generation for Video Prediction

PDE-Net 2.0: Learning PDEs from Data with A Numeric-Symbolic Hybrid Deep Network

Memory in Memory: A Predictive Neural Network for Learning Higher-Order Non-Stationarity From Spatiotemporal Dynamics

Data Driven Governing Equations Approximation Using Deep Neural Networks

SDC-Net: Video Prediction Using Spatially-Displaced Convolution

Bilinear Residual Neural Network for the Identification and Forecasting of Geophysical Dynamics

HybridNet: Classification and Reconstruction Cooperation for Semi-Supervised Learning

Flow-Grounded Spatial-Temporal Video Prediction from Still Images

Neural Ordinary Differential Equations

Learning to Decompose and Disentangle Representations for Video Prediction

Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects

Flexible Neural Representation for Physics Prediction

Structure Preserving Video Prediction

PredRNN++: Towards A Resolution of the Deep-in-Time Dilemma in Spatiotemporal Predictive Learning

Group Normalization

Convolutional neural networks combined with Runge–Kutta methods

Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions

Neural Relational Inference for Interacting Systems

Deep Hidden Physics Models: Deep Learning of Nonlinear Partial Differential Equations

PredRNN: Recurrent Neural Networks for Predictive Learning using Spatiotemporal LSTMs

Folded Recurrent Neural Networks for Future Video Prediction

Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations

Deep learning for physical processes: incorporating prior scientific knowledge

Recurrent Relational Networks

Beyond Finite Layer Neural Networks: Bridging Deep Architectures and Numerical Differential Equations

PDE-Net: Learning PDEs from Data

ContextVP: Fully Context-Aware Video Prediction

A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning

Dual Motion GAN for Future-Flow Embedded Video Prediction

MoCoGAN: Decomposing Motion and Content for Video Generation

Flexible Spatio-Temporal Networks for Video Prediction

Decomposing Motion and Content for Natural Video Sequence Prediction

Visual Interaction Networks: Learning a Physics Simulator from Video

Unsupervised Learning of Disentangled Representations from Video

The Pose Knows: Video Forecasting by Generating Pose Futures

Learning to Generate Long-term Future via Hierarchical Prediction

Image Restoration: Wavelet Frame Shrinkage, Nonlinear Evolution PDEs, and Beyond

Soft-DTW: a Differentiable Loss Function for Time-Series

A Proposal on Machine Learning via Dynamical Systems

Video Frame Synthesis Using Deep Voxel Flow

Unsupervised Learning of Long-Term Motion Dynamics for Videos

Data Assimilation: Methods, Algorithms, and Applications

Interaction Networks for Learning about Objects, Relations and Physics

Deep Spatio-Temporal Residual Networks for Citywide Crowd Flows Prediction

Data-driven discovery of partial differential equations

Generating Videos with Scene Dynamics

Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks

Dynamic Filter Networks

Unsupervised Learning for Physical Interaction through Video Prediction

Backprop KF: Learning Discriminative Deterministic State Estimators

Attend, Infer, Repeat: Fast Scene Understanding with Generative Models

Spatio-temporal video autoencoder with differentiable memory

Deep multi-scale video prediction beyond mean square error

Deep Kalman Filters

Discovering governing equations from data by sparse identification of nonlinear dynamical systems

Action-Conditional Video Prediction using Deep Networks in Atari Games

Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images

Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting

Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Unsupervised Learning of Video Representations using LSTMs

Sequence to Sequence Learning with Neural Networks

Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Image quality assessment: from error visibility to structural similarity

Long Short-Term Memory