Action-Conditioned 3D Human Motion Synthesis with Transformer VAE (2021-04-12T00:00:00.000000Z)

TL;DR

This work designs a Transformer-based architecture, ACTOR, for encoding and decoding a sequence of parametric SMPL human body models estimated from action recognition datasets, and learns an action-aware latent representation for human motions by training a generative variational autoencoder (VAE).

Abstract

We tackle the problem of action-conditioned generation of realistic and diverse human motion sequences. In contrast to methods that complete, or extend, motion sequences, this task does not require an initial pose or sequence. Here we learn an action-aware latent representation for human motions by training a generative variational autoencoder (VAE). By sampling from this latent space and querying a certain duration through a series of positional encodings, we synthesize variable-length motion sequences conditioned on a categorical action. Specifically, we design a Transformer-based architecture, ACTOR, for encoding and decoding a sequence of parametric SMPL human body models estimated from action recognition datasets. We evaluate our approach on the NTU RGB+D, HumanAct12 and UESTC datasets and show improvements over the state of the art. Furthermore, we present two use cases: improving action recognition through adding our synthesized data to training, and motion denoising. Code and models are available on our project page [53].

Authors

Michael J. Black

39 papers

Mathis Petrovich

1 papers

Gul Varol

1 papers

TL;DR

Abstract

Authors

References83 items

SMPL: A Skinned Multi-Person Linear Model

Single-Shot Motion Completion with Transformer

AI Choreographer: Music Conditioned 3D Dance Generation with AIST++

Transformer-based Conditional Variational Autoencoder for Controllable Story Generation

We are More than Our Joints: Predicting how 3D Bodies Move

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

STAR: Sparse Trained Articulated Human Body Regressor

Learning to Generate Diverse Dance Motions with Transformer

3D Human Motion Estimation via Motion Compression and Refinement

Action2Motion: Conditioned Generation of 3D Human Motions

Perpetual Motion: Generating Unbounded Human Motion

3D Human Shape Reconstruction from a Polarization Image

Robust motion in-betweening

Learned motion matching

Bayesian Adversarial Human Motion Synthesis

GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models

Language Models are Few-Shot Learners

Transformer VAE: A Hierarchical Model for Structure-Aware and Interpretable Music Representation Learning

Variational Transformers for Diverse Response Generation

Weakly Supervised 3D Human Pose and Shape Reconstruction with Normalizing Flows

DLow: Diversifying Latent Flows for Diverse Human Motion Prediction

VIBE: Video Inference for Human Body Pose and Shape Estimation

Synthetic Humans for Action Recognition from Unseen Viewpoints

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Accelerating 3D deep learning with PyTorch3D

Human Motion Denoising Using Attention-Based Bidirectional Recurrent Neural Network

Neural state machine for character-scene interactions

Convolutional Sequence Generation for Skeleton-Based Action Synthesis

Structured Prediction Helps 3D Human Motion Modelling

Spatio-Temporal Manifold Learning for Human Motions via Long-Horizon Modeling

Language2Pose: Natural Language Grounded Pose Forecasting

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding

Expressive Body Capture: 3D Hands, Face, and Body From a Single Image

Context-Aware Human Motion Prediction

AMASS: Archive of Motion Capture As Surface Shapes

On the Continuity of Rotation Representations in Neural Networks

Learning 3D Human Dynamics From Video

Interactive character animation by learning multi-objective control

Variational Semi-Supervised Aspect-Term Sentiment Analysis via Transformer

A Large-scale RGB-D Database for Arbitrary-view Human Action Recognition

QuaterNet: A Quaternion-based Recurrent Model for Human Motion

Human Motion Modeling using DVGANs

Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

HP-GAN: Probabilistic 3D Human Motion Prediction via GAN

Deep Video Generation, Prediction and Completion of Human Action Sequences

Text2Action: Generative Adversarial Synthesis from Language to Action

Phase-functioned neural networks for character control

MoCoGAN: Decomposing Motion and Content for Video Generation

A Recurrent Variational Autoencoder for Human Motion Synthesis

Attention is All you Need

On Human Motion Prediction Using Recurrent Neural Networks

beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

A Deep Learning Framework for Character Motion Synthesis and Editing

Gaussian Error Linear Units (GELUs)

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

Learning Structured Output Representation using Deep Conditional Generative Models

Recurrent Network Models for Human Dynamics

Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

Latent structured models for human pose estimation

Modeling Human Locomotion with Topologically Constrained Latent Variable Models

Representing cyclic human motion using functional analysis

Learning Variable-Length Markov Models of Behavior

Dancing to the music

Learning Statistical Models of Human Motion

Simulating humans: computer graphics animation and control

Model-based image analysis of human motion using constraint propagation

Learn to Dance with AIST++: Music Conditioned 3D Dance Generation

AUTO-ENCODING VARIATIONAL BAYES

MoGlow: Probabilistic and controllable motion synthesis using normalising flows

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Zhang , Panna Felsen , and Jitendra Malik . Learning 3 D human dynamics from video

GENERATIVE ADVERSARIAL NETS

Embodied Hands : Modeling and Capturing Hands and Bodies Together * * Supplementary Material * *

The Visual Analysis of Human Movement: A Survey

Extraction of motion data by interactive processing

Temporal scene analysis: conceptual descriptions of object movements.