OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

Published in

IEEE Transactions on Pattern Analysis and Machi...(2018)

External Links:

Generate Graph DownloadPDF

TL;DR

OpenPose is released, the first open-source realtime system for multi-person 2D pose detection, including body, foot, hand, and facial keypoints, and the first combined body and foot keypoint detector, based on an internal annotated foot dataset.

Abstract

Realtime multi-person 2D pose estimation is a key component in enabling machines to have an understanding of people in images and videos. In this work, we present a realtime approach to detect the 2D pose of multiple people in an image. The proposed method uses a nonparametric representation, which we refer to as Part Affinity Fields (PAFs), to learn to associate body parts with individuals in the image. This bottom-up system achieves high accuracy and realtime performance, regardless of the number of people in the image. In previous work, PAFs and body part location estimation were refined simultaneously across training stages. We demonstrate that a PAF-only refinement rather than both PAF and body part location refinement results in a substantial increase in both runtime performance and accuracy. We also present the first combined body and foot keypoint detector, based on an internal annotated foot dataset that we have publicly released. We show that the combined detector not only reduces the inference time compared to running them sequentially, but also maintains the accuracy of each component individually. This work has culminated in the release of OpenPose, the first open-source realtime system for multi-person 2D pose detection, including body, foot, hand, and facial keypoints.

Authors

T. Simon

6 papers

Yaser Sheikh

8 papers

Zhe Cao

4 papers

References81 items

SMPL: A Skinned Multi-Person Linear Model

PoseFix: Model-Agnostic General Human Pose Refinement Network

Pose Partition Networks for Multi-person Pose Estimation

Deeply Learned Compositional Models for Human Pose Estimation

Everybody Dance Now

OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

Published in

IEEE Transactions on Pattern Analysis and Machi...(2018)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

T. Simon

6 papers

Yaser Sheikh

8 papers

Zhe Cao

4 papers

References81 items

SMPL: A Skinned Multi-Person Linear Model

PoseFix: Model-Agnostic General Human Pose Refinement Network

Pose Partition Networks for Multi-person Pose Estimation

Deeply Learned Compositional Models for Human Pose Estimation

Everybody Dance Now

Gines Hidalgo

1 papers

Recycle-GAN: Unsupervised Video Retargeting

MultiPoseNet: Fast Multi-Person Pose Estimation using Pose Residual Network

CarFusion: Combining Point Tracking and Part Detection for Dynamic 3D Reconstruction of Vehicles

Learning to Refine Human Pose Estimation

Simple Baselines for Human Pose Estimation and Tracking

Multi-Scale Structure-Aware Network for Human Pose Estimation

PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model

Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies

Detect-and-Track: Efficient Pose Estimation in Videos

Using a Single RGB Frame for Real Time 3D Hand Pose Estimation in the Wild

Pose-Normalized Image Generation for Person Re-identification

Cascaded Pyramid Network for Multi-person Pose Estimation

Learning Feature Pyramids for Human Pose Estimation

Adversarial PoseNet: A Structure-Aware Convolutional Network for Human Pose Estimation

Hand Keypoint Detection in Single Images Using Multiview Bootstrapping

Multi-context Attention for Human Pose Estimation

Towards Accurate Multi-person Pose Estimation in the Wild

YOLO9000: Better, Faster, Stronger

ArtTrack: Articulated Multi-Person Tracking in the Wild

RMPE: Regional Multi-person Pose Estimation

Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors

Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields

Associative Embedding: End-to-End Learning for Joint Detection and Grouping

Joint Graph Decomposition & Node Labeling: Problem, Algorithms, Applications

Human Pose Estimation via Convolutional Part Heatmap Regression

Multi-person Pose Estimation with Local Joint-to-Person Associations

Densely Connected Convolutional Networks

Recurrent Human Pose Estimation

DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model

Stacked Hourglass Networks for Human Pose Estimation

Convolutional Pose Machines

Deep Residual Learning for Image Recognition

SSD: Single Shot MultiBox Detector

DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation

Flowing ConvNets for Human Pose Estimation in Videos

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Efficient object localization using Convolutional Networks

Pose Machines: Articulated Pose Estimation via Inference Machines

Very Deep Convolutional Networks for Large-Scale Image Recognition

Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations

Using k-Poselets for Detecting People and Localizing Their Keypoints

Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation

2D Human Pose Estimation: New Benchmark and State of the Art Analysis

Multi-source Deep Learning for Human Pose Estimation

Microsoft COCO: Common Objects in Context

DeepPose: Human Pose Estimation via Deep Neural Networks

Articulated Human Detection with Flexible Mixtures of Parts

Poselet Conditioned Pictorial Structures

Human Pose Estimation Using Body Parts Dependent Joint Regressors

Using Linking Features in Learning Non-parametric Part Models

Articulated people detection and pose estimation: Reshaping the future

Articulated part-based model for joint object detection and pose estimation

We Are Family: Joint Pose Estimation of Multiple Persons

Monocular 3D pose estimation and tracking by detection

Understanding the difficulty of training deep feedforward neural networks

Pictorial structures revisited: People detection and articulated pose estimation

Multiple Tree Models for Occlusion and Spatial Constraints in Human Pose Estimation

Measure Locally, Reason Globally: Occlusion-sensitive Articulated Pose Estimation

Beyond trees: common-factor models for 2D human pose recovery

Strike a pose: tracking people by finding stylized poses

Learning long-term dependencies with gradient descent is difficult

Introduction to graph theory

An Algorithm for Least-Squares Estimation of Nonlinear Parameters

The Hungarian method for the assignment problem

Teaching Robots to Predict Human Motion

a Research Engineer at Face-book Reality Labs

Clustered Pose and Nonlinear Appearance Models for Human Pose Estimation

Professor at the Robotics Institute at Carnegie Mellon University

Pictorial Structures for Object Recognition

Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies

The OpenCV library , ”

“ClickWorker,”

“MSCOCO keypoint leaderboard,”

Field of Study

Computer ScienceMedicine

Journal Information

Name

IEEE Transactions on Pattern Analysis and Machine Intelligence

Page

1161-1180

Volume

Venue Information

Name

IEEE Transactions on Pattern Analysis and Machine Intelligence

Type

journal

URL

http://www.computer.org/tpami/

Alternate Names

IEEE Trans Pattern Anal Mach Intell

TL;DR

Abstract

Authors

References81 items

SMPL: A Skinned Multi-Person Linear Model

PoseFix: Model-Agnostic General Human Pose Refinement Network

Pose Partition Networks for Multi-person Pose Estimation

Deeply Learned Compositional Models for Human Pose Estimation

Everybody Dance Now

TL;DR

Abstract

Authors

References81 items

SMPL: A Skinned Multi-Person Linear Model

PoseFix: Model-Agnostic General Human Pose Refinement Network

Pose Partition Networks for Multi-person Pose Estimation

Deeply Learned Compositional Models for Human Pose Estimation

Everybody Dance Now

Recycle-GAN: Unsupervised Video Retargeting

MultiPoseNet: Fast Multi-Person Pose Estimation using Pose Residual Network

CarFusion: Combining Point Tracking and Part Detection for Dynamic 3D Reconstruction of Vehicles

Learning to Refine Human Pose Estimation

Simple Baselines for Human Pose Estimation and Tracking

Multi-Scale Structure-Aware Network for Human Pose Estimation

PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model

Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies

Detect-and-Track: Efficient Pose Estimation in Videos

Using a Single RGB Frame for Real Time 3D Hand Pose Estimation in the Wild

Pose-Normalized Image Generation for Person Re-identification

Cascaded Pyramid Network for Multi-person Pose Estimation

Learning Feature Pyramids for Human Pose Estimation

VNect

Adversarial PoseNet: A Structure-Aware Convolutional Network for Human Pose Estimation

Hand Keypoint Detection in Single Images Using Multiview Bootstrapping

Mask R-CNN

Multi-context Attention for Human Pose Estimation

Towards Accurate Multi-person Pose Estimation in the Wild

YOLO9000: Better, Faster, Stronger

ArtTrack: Articulated Multi-Person Tracking in the Wild

RMPE: Regional Multi-person Pose Estimation

Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors

Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields

Associative Embedding: End-to-End Learning for Joint Detection and Grouping

Joint Graph Decomposition & Node Labeling: Problem, Algorithms, Applications

Human Pose Estimation via Convolutional Part Heatmap Regression

Multi-person Pose Estimation with Local Joint-to-Person Associations

Densely Connected Convolutional Networks

Recurrent Human Pose Estimation

DeeperCut: A Deeper, Stronger, and Faster Multi-person Pose Estimation Model

Stacked Hourglass Networks for Human Pose Estimation

Convolutional Pose Machines

Deep Residual Learning for Image Recognition

SSD: Single Shot MultiBox Detector

DeepCut: Joint Subset Partition and Labeling for Multi Person Pose Estimation

Flowing ConvNets for Human Pose Estimation in Videos

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Efficient object localization using Convolutional Networks

Pose Machines: Articulated Pose Estimation via Inference Machines

Very Deep Convolutional Networks for Large-Scale Image Recognition

Articulated Pose Estimation by a Graphical Model with Image Dependent Pairwise Relations

Using k-Poselets for Detecting People and Localizing Their Keypoints

Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation

2D Human Pose Estimation: New Benchmark and State of the Art Analysis

Multi-source Deep Learning for Human Pose Estimation

Microsoft COCO: Common Objects in Context

DeepPose: Human Pose Estimation via Deep Neural Networks

Articulated Human Detection with Flexible Mixtures of Parts

Poselet Conditioned Pictorial Structures

Human Pose Estimation Using Body Parts Dependent Joint Regressors

Using Linking Features in Learning Non-parametric Part Models

Articulated people detection and pose estimation: Reshaping the future

Articulated part-based model for joint object detection and pose estimation

We Are Family: Joint Pose Estimation of Multiple Persons

Monocular 3D pose estimation and tracking by detection

Understanding the difficulty of training deep feedforward neural networks

Pictorial structures revisited: People detection and articulated pose estimation

Multiple Tree Models for Occlusion and Spatial Constraints in Human Pose Estimation

Measure Locally, Reason Globally: Occlusion-sensitive Articulated Pose Estimation

Beyond trees: common-factor models for 2D human pose recovery

Strike a pose: tracking people by finding stylized poses