Dynamical Variational Autoencoders: A Comprehensive Review (2020-08-28T00:00:00.000000Z)

TL;DR

A general class of models called Dynamical Variational Autoencoders (DVAEs) are introduced that encompass a large subset of these temporal VAE extensions that not only model the latent space, but also model the temporal dependencies within a sequence of data vectors and/or corresponding latent vectors.

Abstract

The Variational Autoencoder (VAE) is a powerful deep generative model that is now extensively used to represent high-dimensional complex data via a low-dimensional latent space that is learned in an unsupervised manner. In the original VAE model, input data vectors are processed independently. In the recent years, a series of papers have presented different extensions of the VAE to sequential data, that not only model the latent space, but also model the temporal dependencies within a sequence of data vectors and/or corresponding latent vectors, relying on recurrent neural networks or state space models. In this paper we perform an extensive literature review of these models. Importantly, we introduce and discuss a general class of models called Dynamical Variational Autoencoders (DVAEs) that encompass a large subset of these temporal VAE extensions. Then we present in details seven different instances of DVAE that were recently proposed in the literature, with an effort to homogenize the notations and presentation lines, as well as to relate those models with existing classical temporal models (that are also presented for the sake of completeness). We reimplemented those seven DVAE models and we present the results of an experimental benchmark that we conducted on the speech analysis-resynthesis task (the PyTorch code will be made publicly available). An extensive discussion is presented at the end of the paper, aiming to comment on important issues concerning the DVAE class of models and to describe future research guidelines.

Authors

Xavier Alameda-Pineda

4 papers

Xiaoyu Bie

2 papers

Thomas Hueber

2 papers

TL;DR

Abstract

Authors

References129 items

Generative Adversarial Networks

Neural Granular Sound Synthesis

Transformer VAE: A Hierarchical Model for Structure-Aware and Interpretable Music Representation Learning

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Feedback Recurrent Autoencoder

A Recurrent Variational Autoencoder for Speech Enhancement

Notes on the use of variational autoencoders for speech and audio spectrogram modeling

Video Compression With Rate-Distortion Autoencoders

T-CVAE: Transformer-Based Conditioned Variational Autoencoder for Story Completion

GP-VAE: Deep Probabilistic Time Series Imputation

A Transformer-Based Variational Autoencoder for Sentence Generation

Disentangled State Space Representations

An Introduction to Variational Autoencoders

Streaming Variational Monte Carlo

Low Bit-rate Speech Coding with VQ-VAE and a WaveNet Decoder

A Statistically Principled and Computationally Efficient Approach to Speech Enhancement using Variational Autoencoders

STCN: Stochastic Temporal Convolutional Networks

Unsupervised Anomaly Detection in Energy Time Series Data Using Variational Recurrent Autoencoders with Attention

Semi-supervised Multichannel Speech Enhancement with Variational Autoencoders and Non-negative Matrix Factorization

Learning Latent Dynamics for Planning from Pixels

Deep Generative Video Compression

Conv-TasNet: Surpassing Ideal Time–Frequency Magnitude Masking for Speech Separation

Acoustic Modeling Using Adversarially Trained Variational Recurrent Neural Network for Speech Synthesis

A VARIANCE MODELING FRAMEWORK BASED ON VARIATIONAL AUTOENCODERS FOR SPEECH ENHANCEMENT

Semi-blind source separation with multichannel variational autoencoder

Autoencoders for music sound synthesis: a comparison of linear, shallow, deep and variational models

Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation

A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music

Disentangled Sequential Autoencoder

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

Recurrent Neural Network-Based Semantic Variational Autoencoder for Sequence-to-Sequence Learning

Variational Recurrent Neural Machine Translation

Auxiliary Guided Autoregressive Variational Autoencoders

Z-Forcing: Training Stochastic Recurrent Networks

Neural Discrete Representation Learning

Statistical Speech Enhancement Based on Probabilistic Integration of Variational Autoencoder and Non-Negative Matrix Factorization

Stochastic Variational Video Prediction

A Disentangled Recognition and Nonlinear Dynamics Model for Unsupervised Learning

Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data

Variational Online Learning of Neural Dynamics

MoCoGAN: Decomposing Motion and Content for Video Generation

Factorized Variational Autoencoders for Modeling Audience Reactions to Movies

Decomposing Motion and Content for Natural Video Sequence Prediction

Channel-Recurrent Autoencoding for Image Modeling

Attention is All you Need

Learning Disentangled Representations with Semi-Supervised Deep Generative Models

Unsupervised Learning of Disentangled Representations from Video

Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations

Learning Latent Representations for Speech Generation and Transformation

Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders

Multiplicative Normalizing Flows for Variational Bayesian Neural Networks

Toward Controlled Generation of Text

Improved Variational Autoencoders for Text Modeling using Dilated Convolutions

A Hybrid Convolutional Variational Autoencoder for Text Generation

NIPS 2016 Tutorial: Generative Adversarial Networks

Disentangling factors of variation in deep representation using adversarial training

Variational Lossy Autoencoder

PixelVAE: A Latent Variable Model for Natural Images

beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

Recurrent switching linear dynamical systems

Structured Inference Networks for Nonlinear State Space Models

WaveNet: A Generative Model for Raw Audio

Modeling and Transforming Speech Using Variational Autoencoders

Temporal Convolutional Networks: A Unified Approach to Action Segmentation

Conditional Image Generation with PixelCNN Decoders

Improved Variational Inference with Inverse Autoregressive Flow

Sequential Neural Models with Stochastic Layers

Deep Variational Bayes Filters: Unsupervised Learning of State Space Models from Raw Data

A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues

Towards Conceptual Compression

Composing graphical models with neural networks for structured representations and fast inference

A Structured Variational Auto-encoder for Learning Deep Hierarchies of Sparse Features

Ladder Variational Autoencoders

How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks

Pixel Recurrent Neural Networks

Learning Structured Output Representation using Deep Conditional Generative Models