VIBE: Video Inference for Human Body Pose and Shape Estimation (2019-12-11T00:00:00.000000Z)

TL;DR

This work defines a novel temporal network architecture with a self-attention mechanism and shows that adversarial training, at the sequence level, produces kinematically plausible motion sequences without in-the-wild ground-truth 3D labels.

Abstract

Human motion is fundamental to understanding behavior. Despite progress on single-image 3D pose and shape estimation, existing video-based state-of-the-art methods fail to produce accurate and natural motion sequences due to a lack of ground-truth 3D motion data for training. To address this problem, we propose "Video Inference for Body Pose and Shape Estimation'' (VIBE), which makes use of an existing large-scale motion capture dataset (AMASS) together with unpaired, in-the-wild, 2D keypoint annotations. Our key novelty is an adversarial learning framework that leverages AMASS to discriminate between real human motions and those produced by our temporal pose and shape regression networks. We define a novel temporal network architecture with a self-attention mechanism and show that adversarial training, at the sequence level, produces kinematically plausible motion sequences without in-the-wild ground-truth 3D labels. We perform extensive experimentation to analyze the importance of motion and demonstrate the effectiveness of VIBE on challenging 3D pose estimation datasets, achieving state-of-the-art performance. Code and pretrained models are available at https://github.com/mkocabas/VIBE

Authors

Michael J. Black

39 papers

Nikos Athanasiou

2 papers

Muhammed Kocabas

3 papers

TL;DR

Abstract

Authors

References75 items

Structured Prediction Helps 3D Human Motion Modelling

Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop

Human Mesh Recovery From Monocular Images via a Skeleton-Disentangled Representation

Sim2real transfer learning for 3D pose estimation: motion to the rescue

Sim2real transfer learning for 3D human pose estimation: motion to the rescue

XNect: Real-time Multi-person 3D Human Pose Estimation with a Single RGB Camera

HoloPose: Holistic 3D Human Reconstruction In-The-Wild

NONLINEAR EQUATIONS

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

Exploiting Temporal Context for 3D Human Pose Estimation in the Wild

Convolutional Mesh Regression for Single-Image Human Shape Reconstruction

Expressive Body Capture: 3D Hands, Face, and Body From a Single Image

AMASS: Archive of Motion Capture As Surface Shapes

From Variational to Deterministic Autoencoders

Self-Supervised Learning of 3D Human Pose Using Multi-View Geometry

OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

On the Continuity of Rotation Representations in Neural Networks

Learning 3D Human Dynamics From Video

3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training

Adversarial Geometry-Aware Human Motion Prediction

Neural Body Fitting: Unifying Deep Learning and Model Based Human Pose and Shape Estimation

MultiPoseNet: Fast Multi-Person Pose Estimation using Pose Residual Network

Learning to Estimate 3D Human Pose and Shape from a Single Color Image

NTUA-SLP at SemEval-2018 Task 1: Predicting Affective Content in Tweets with Deep Attentive RNNs and Transfer Learning

BodyNet: Volumetric Inference of 3D Human Body Shapes

Detect-and-Track: Efficient Pose Estimation in Videos

End-to-End Recovery of Human Shape and Pose

Single-Shot Multi-person 3D Pose Estimation from Monocular RGB

Self-supervised Learning of Motion Capture

HP-GAN: Probabilistic 3D Human Motion Prediction via GAN

Learning 3D Human Pose from Structure and Motion

Structure-Aware and Temporally Coherent 3D Human Pose Estimation

Exploiting Temporal Information for 3D Human Pose Estimation

Wasserstein Auto-Encoders

PoseTrack: A Benchmark for Human Pose Estimation and Tracking

Towards Accurate Marker-Less Human Shape and Pose Estimation over Time

Wasserstein Generative Adversarial Networks

Attention is All you Need

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

The Kinetics Human Action Video Dataset

Adversarial Neural Machine Translation

Improving Neural Machine Translation with Conditional Sequence Generative Adversarial Nets

Unsupervised Image-to-Image Translation Networks

Unite the People: Closing the Loop Between 3D and 2D Human Representations

Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision

Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields

Image-to-Image Translation with Conditional Adversarial Networks

Semi-Supervised Classification with Graph Convolutional Networks

Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image

Improved Techniques for Training GANs

Identity Mappings in Deep Residual Networks

DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation

Adam: A Method for Stochastic Optimization

MoSh

Sequence to Sequence Learning with Neural Networks

Neural Machine Translation by Jointly Learning to Align and Translate

Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Auto-Encoding Variational Bayes

From Actemes to Action: A Strongly-Supervised Representation for Detailed Action Understanding

MODEC: Multimodal Decomposable Models for Human Pose Estimation

The Naked Truth: Estimating Body Shape Under Clothing

Combined discriminative and generative articulated pose and non-rigid shape estimation

3D People Tracking with Gaussian Process Dynamical Models

SCAPE: shape completion and animation of people

Inferring 3D structure with a statistical image-based shape model

Model-based vision: a program to see a walking person

Visual perception of biological motion and a model for its analysis

Supplementary Material to: Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera

GENERATIVE ADVERSARIAL NETS

Indirect deep structured learning for 3D human body shape and pose prediction

SMPL: A skinned multiperson linear model

Learning and Tracking Cyclic Human Motion

Zhang , Panna Felsen , and Jitendra Malik . Learning 3 d human dynamics from video

Learning phrase representations using 5260

Field of Study