3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training

Published in

Computer Vision and Pattern Recognition(2018)

External Links:

Generate Graph DownloadPDF

TL;DR

It is demonstrated that 3D poses in video can be effectively estimated with a fully convolutional model based on dilated temporal convolutions over 2D keypoints and back-projection, a simple and effective semi-supervised training method that leverages unlabeled video data is introduced.

Abstract

In this work, we demonstrate that 3D poses in video can be effectively estimated with a fully convolutional model based on dilated temporal convolutions over 2D keypoints. We also introduce back-projection, a simple and effective semi-supervised training method that leverages unlabeled video data. We start with predicted 2D keypoints for unlabeled video, then estimate 3D poses and finally back-project to the input 2D keypoints. In the supervised setting, our fully-convolutional model outperforms the previous best result from the literature by 6 mm mean per-joint position error on Human3.6M, corresponding to an error reduction of 11%, and the model also shows significant improvements on HumanEva-I. Moreover, experiments with back-projection show that it comfortably outperforms previous state-of-the-art results in semi-supervised settings where labeled data is scarce. Code and models are available at https://github.com/facebookresearch/VideoPose3D

Authors

Michael Auli

14 papers

David Grangier

6 papers

Dario Pavllo

2 papers

References64 items

Propagating LSTM: 3D Pose Estimation Based on Joint Interdependency

Can 3D Pose be Learned from 2D Projections Alone?

Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture

Ordinal Depth Supervision for 3D Human Pose Estimation

Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation

3D Human Pose Estimation in Video With Temporal Convolutions and Semi-Supervised Training

Published in

Computer Vision and Pattern Recognition(2018)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Michael Auli

14 papers

David Grangier

6 papers

Dario Pavllo

2 papers

References64 items

Propagating LSTM: 3D Pose Estimation Based on Joint Interdependency

Can 3D Pose be Learned from 2D Projections Alone?

Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture

Ordinal Depth Supervision for 3D Human Pose Estimation

Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation

Christoph Feichtenhofer

1 papers

3D Human Pose Estimation in the Wild by Adversarial Learning

Norm matters: efficient and accurate normalization schemes in deep networks

2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning

On the Convergence of Adam and Beyond

Learning Latent Representations of 3D Human Pose with Deep Neural Networks

End-to-End Recovery of Human Shape and Pose

Exploiting Temporal Information for 3D Human Pose Estimation

Cascaded Pyramid Network for Multi-person Pose Estimation

Unsupervised Machine Translation Using Monolingual Corpora Only

Learning Pose Grammar to Encode Human Body Configuration for 3D Pose Estimation

Recurrent 3D Pose Sequence Machines

Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision

A Simple Yet Effective Baseline for 3d Human Pose Estimation

Convolutional Sequence to Sequence Learning

Towards 3D Human Pose Estimation in the Wild: A Weakly-Supervised Approach

Compositional Human Pose Regression

Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image

Language Modeling with Gated Convolutional Networks

3D Human Pose Estimation = 2D Pose Estimation + Matching

Feature Pyramid Networks for Object Detection

Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision

Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose

Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation

Neural Machine Translation in Linear Time

3D Human Pose Estimation via Deep Learning from 2D Annotations

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

WaveNet: A Generative Model for Raw Audio

Wav2Letter: an End-to-End ConvNet-based Speech Recognition System

3D Human Pose Estimation Using Convolutional Neural Networks with 2D Pose Information

Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image

Structured Prediction of 3D Human Pose with Deep Neural Networks

Stacked Hourglass Networks for Human Pose Estimation

Deep Residual Learning for Image Recognition

Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video

Multi-Scale Context Aggregation by Dilated Convolutions

Direct Prediction of 3D Body Poses from Motion Compensated Sequences

Neural Machine Translation of Rare Words with Subword Units

Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

3D Human Pose Estimation from Monocular Images with Deep Convolutional Neural Network

Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

Iterated Second-Order Label Sensitive Pooling for 3D Human Pose Estimation

2D Human Pose Estimation: New Benchmark and State of the Art Analysis

Microsoft COCO: Common Objects in Context

Reconstructing 3D Human Pose from 2D Image Landmarks

Latent structured models for human pose estimation

3D Human Pose Reconstruction Using Millions of Exemplars

Rectified Linear Units Improve Restricted Boltzmann Machines

HumanEva: Synchronized Video and Motion Capture Dataset and Baseline Algorithm for Evaluation of Articulated Human Motion

3D Human Motion Analysis in Monocular Video Techniques and Challenges

Multitask Learning

Understanding back-translation at scale

Exploiting temporal information for 3D pose estimation

Vnect: Real-time 3d human pose estimation with a single rgb camera

Dropout: a simple way to prevent neural networks from overfitting

A real-time algorithm for signal analysis with the help of the wavelet transform

Ablation of receptive field and channel size

Field of Study

Computer Science

Journal Information

Name

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Page

5686-5696

Venue Information

Name

Computer Vision and Pattern Recognition

Type

conference

URL

https://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000147

Alternate Names

CVPR
Comput Vis Pattern Recognit

TL;DR

Abstract

Authors

References64 items

Propagating LSTM: 3D Pose Estimation Based on Joint Interdependency

Can 3D Pose be Learned from 2D Projections Alone?

Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture

Ordinal Depth Supervision for 3D Human Pose Estimation

Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation

TL;DR

Abstract

Authors

References64 items

Propagating LSTM: 3D Pose Estimation Based on Joint Interdependency

Can 3D Pose be Learned from 2D Projections Alone?

Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture

Ordinal Depth Supervision for 3D Human Pose Estimation

Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation

3D Human Pose Estimation in the Wild by Adversarial Learning

Norm matters: efficient and accurate normalization schemes in deep networks

2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning

On the Convergence of Adam and Beyond

Learning Latent Representations of 3D Human Pose with Deep Neural Networks

End-to-End Recovery of Human Shape and Pose

Exploiting Temporal Information for 3D Human Pose Estimation

Cascaded Pyramid Network for Multi-person Pose Estimation

Unsupervised Machine Translation Using Monolingual Corpora Only

Learning Pose Grammar to Encode Human Body Configuration for 3D Pose Estimation

Recurrent 3D Pose Sequence Machines

Adversarial Inverse Graphics Networks: Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision

A Simple Yet Effective Baseline for 3d Human Pose Estimation

Convolutional Sequence to Sequence Learning

Towards 3D Human Pose Estimation in the Wild: A Weakly-Supervised Approach

Compositional Human Pose Regression

Mask R-CNN

Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image

Language Modeling with Gated Convolutional Networks

3D Human Pose Estimation = 2D Pose Estimation + Matching

Feature Pyramid Networks for Object Detection

Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision

Coarse-to-Fine Volumetric Prediction for Single-Image 3D Human Pose

Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation

Neural Machine Translation in Linear Time

3D Human Pose Estimation via Deep Learning from 2D Annotations

On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima

WaveNet: A Generative Model for Raw Audio

Wav2Letter: an End-to-End ConvNet-based Speech Recognition System

3D Human Pose Estimation Using Convolutional Neural Networks with 2D Pose Information

Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from a Single Image

Structured Prediction of 3D Human Pose with Deep Neural Networks

Stacked Hourglass Networks for Human Pose Estimation

Deep Residual Learning for Image Recognition

Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video

Multi-Scale Context Aggregation by Dilated Convolutions

Direct Prediction of 3D Body Poses from Motion Compensated Sequences

Neural Machine Translation of Rare Words with Subword Units

Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

3D Human Pose Estimation from Monocular Images with Deep Convolutional Neural Network

Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

Iterated Second-Order Label Sensitive Pooling for 3D Human Pose Estimation

2D Human Pose Estimation: New Benchmark and State of the Art Analysis

Microsoft COCO: Common Objects in Context

Reconstructing 3D Human Pose from 2D Image Landmarks

Latent structured models for human pose estimation

3D Human Pose Reconstruction Using Millions of Exemplars

Rectified Linear Units Improve Restricted Boltzmann Machines

HumanEva: Synchronized Video and Motion Capture Dataset and Baseline Algorithm for Evaluation of Articulated Human Motion

3D Human Motion Analysis in Monocular Video Techniques and Challenges

Multitask Learning

Understanding back-translation at scale

Exploiting temporal information for 3D pose estimation

Vnect: Real-time 3d human pose estimation with a single rgb camera

Dropout: a simple way to prevent neural networks from overfitting

A real-time algorithm for signal analysis with the help of the wavelet transform

Ablation of receptive field and channel size

Field of Study

Journal Information

Name