DDLP: Unsupervised Object-Centric Video Prediction with Deep Dynamic Latent Particles (2023-06-09T00:00:00.000000Z)

TL;DR

The interpretable nature of DDLP allows us to perform ``what-if'' generation -- predict the consequence of changing properties of objects in the initial frames, and DLP's compact structure enables efficient diffusion-based unconditional video generation.

Abstract

We propose a new object-centric video prediction algorithm based on the deep latent particle (DLP) representation. In comparison to existing slot- or patch-based representations, DLPs model the scene using a set of keypoints with learned parameters for properties such as position and size, and are both efficient and interpretable. Our method, deep dynamic latent particles (DDLP), yields state-of-the-art object-centric video prediction results on several challenging datasets. The interpretable nature of DDLP allows us to perform ``what-if'' generation -- predict the consequence of changing properties of objects in the initial frames, and DLP's compact structure enables efficient diffusion-based unconditional video generation. Videos, code and pre-trained models are available: https://taldatech.github.io/ddlp-web

Authors

Aviv Tamar

4 papers

Tal Daniel

1 papers

TL;DR

Abstract

Authors

References81 items

Object-Centric Video Prediction Via Decoupling of Object Dynamics and Interactions

An Investigation into Pre-Training Object-Centric Representations for Reinforcement Learning

MAGVIT: Masked Generative Video Transformer

SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Models

Bridging the Gap to Real-World Object-Centric Learning

SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videos

Object Scene Representation Transformer

Unsupervised Image Representation Learning with Deep Latent Particles

Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videos

Transframer: Arbitrary Frame Prediction with Generative Models

Self-Supervised Keypoint Discovery in Behavioral Videos

Andrej

Conditional Object-Centric Learning from Video

Illiterate DALL-E Learns to Compose

PARTS: Unsupervised segmentation with slots, attention and independence maximization

Self-supervised Reinforcement Learning with Independently Controllable Subgoals

Accurate Grid Keypoint Learning for Efficient Video Prediction

Generative Video Transformer: Can Objects be the Words?

SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition

MarioNette: Self-Supervised Sprite Learning

GENESIS-V2: Inferring Unordered Object Representations without Iterative Refinement

VideoGPT: Video Generation using VQ-VAE and Transformers

PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning

Greedy Hierarchical Variational Autoencoders for Large-Scale Video Prediction

Predicting Video with VQVAE

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Improving Generative Imagination in Object-Centric World Models

Causal Discovery in Physical Systems from Videos

Object-Centric Learning with Slot Attention

Denoising Diffusion Probabilistic Models

Benchmarking Unsupervised Object Representations for Video Sequences

SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decomposition

High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks

Entity Abstraction in Visual Model-Based Reinforcement Learning

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer

R-SQAIR: Relational Sequential Attend, Infer, Repeat

SCALOR: Generative World Models with Scalable Object Representations

Structured Object-Aware Physics Prediction for Video Modeling and Planning

Unsupervised Keypoint Learning for Guiding Class-Conditional Video Prediction

CLEVRER: CoLlision Events for Video REpresentation and Reasoning

PHYRE: A New Benchmark for Physical Reasoning

GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations

Spatially Invariant Unsupervised Object Detection with Convolutional Neural Networks

Unsupervised Learning of Object Structure and Dynamics from Videos

Unsupervised Learning of Object Keypoints for Perception and Control

Multi-Object Representation Learning with Iterative Variational Inference

MONet: Unsupervised Scene Decomposition and Representation

Non-Adversarial Image Synthesis With Generative Latent Nearest Neighbors

Unsupervised Learning of Object Landmarks through Conditional Image Generation

Stochastic Adversarial Video Prediction

Self-Attention with Relative Position Representations

Stochastic Video Generation with a Learned Prior

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

The Perception-Distortion Tradeoff

Automatic differentiation in PyTorch

Self-Supervised Visual Planning with Temporal Skip Connections

Attention is All you Need

Learning to Generate Long-term Future via Hierarchical Prediction

CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning

Interaction Networks for Learning about Objects, Relations and Physics

beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

Unsupervised Learning for Physical Interaction through Video Prediction

Learning to Track at 100 FPS with Deep Regression Networks

Learning Structured Output Representation using Deep Conditional Generative Models

Deep spatial autoencoders for visuomotor learning

You Only Look Once: Unified, Real-Time Object Detection

Spatial Transformer Networks

Adam: A Method for Stochastic Optimization

Image quality assessment: from error visibility to structural similarity

A Learning Algorithm for Continually Running Fully Recurrent Neural Networks

Interaction-Based Disentanglement of Entities for Object-Centric World Models

Neural Block-Slot Representations

Particle Videos Revisited: Tracking Through Occlusions Using Point Trajectories

of the sampled sequences include interactions, and (2) sparse interactions, where interactions are less frequent or occur in the distant horizon

Jaesik

AUTO-ENCODING VARIATIONAL BAYES