Temporally guided articulated hand pose tracking in surgical videos (2021-01-12T00:00:00.000000Z)

TL;DR

A novel hand pose estimation model, CondPose, is proposed, which improves detection and tracking accuracy by incorporating a pose prior into its prediction, and shows improvements over state-of-the-art methods which provide frame-wise independent predictions.

Abstract

Purpose Articulated hand pose tracking is an under-explored problem that carries the potential for use in an extensive number of applications, especially in the medical domain. With a robust and accurate tracking system on surgical videos, the motion dynamics and movement patterns of the hands can be captured and analyzed for many rich tasks. Methods In this work, we propose a novel hand pose estimation model, CondPose , which improves detection and tracking accuracy by incorporating a pose prior into its prediction. We show improvements over state-of-the-art methods which provide frame-wise independent predictions, by following a temporally guided approach that effectively leverages past predictions. Results We collect Surgical Hands , the first dataset that provides multi-instance articulated hand pose annotations for videos. Our dataset provides over 8.1k annotated hand poses from publicly available surgical videos and bounding boxes, pose annotations, and tracking IDs to enable multi-instance tracking. When evaluated on Surgical Hands , we show our method outperforms the state-of-the-art approach using mean Average Precision, to measure pose estimation accuracy, and Multiple Object Tracking Accuracy, to assess pose tracking performance. Conclusion In comparison to a frame-wise independent strategy, we show greater performance in detecting and tracking hand poses and more substantial impact on localization accuracy. This has positive implications in generating more accurate representations of hands in the scene to be used for targeted downstream tasks.

Authors

References41 items

Using Computer Vision to Automate Hand Detection and Tracking of Surgeon Movements in Videos of Open Surgery

Understanding Human Hands in Contact at Internet Scale

Combining Detection and Tracking for Human Pose Estimation in Videos

Evaluation of Deep Learning Models for Identifying Surgical Actions and Measuring Performance

Attention! A Lightweight 2D Hand Pose Estimation Approach

FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images

Learning Temporal Pose Estimation from Sparsely-Labeled Videos

RASNet: Segmentation for Tracking Surgical Instruments in Surgical Videos Using Refined Attention Segmentation Network

Deep Learning Based Robotic Tool Detection and Articulation Estimation With Spatio-Temporal Layers

LightTrack: A Generic Framework for Online Top-Down Human Pose Tracking

Multi-Person Articulated Tracking With Spatial and Temporal Embeddings

Using Machine Learning to Assess Physician Competence: A Systematic Review

Deep High-Resolution Representation Learning for Human Pose Estimation

Weakly supervised convolutional LSTM approach for tool tracking in laparoscopic videos

Efficient Online Multi-Person 2D Pose Tracking With Recurrent Spatio-Temporal Affinity Fields

Surgical motion analysis using discriminative interpretable patterns

Simple Baselines for Human Pose Estimation and Tracking

Tool Detection and Operative Skill Assessment in Surgical Videos Using Region-Based Convolutional Neural Networks

Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

Articulated Multi-Instrument 2-D Pose Estimation Using Fully Convolutional Networks

PoseTrack: A Benchmark for Human Pose Estimation and Tracking

Using Deep Learning for Classification of Lung Nodules on Computed Tomography Images

Large-scale Multiview 3D Hand Pose Dataset

A hand pose tracking benchmark from stereo matching

Hand Keypoint Detection in Single Images Using Multiview Bootstrapping

Concurrent Segmentation and Localization for Tracking of Surgical Instruments

Detection and Localization of Robotic Tools in Robot-Assisted Surgery Videos Using Deep Neural Networks for Region Proposal and Detection

Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields

3D Hand Pose Tracking and Estimation Using Stereo Matching

Semi-Supervised Classification with Graph Convolutional Networks

Surgical Performance Assessment

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Application of Random Forests Methods to Diabetic Retinopathy Classification Analyses

Surgical gesture classification from video and kinematic data

Unified Detection and Tracking of Instruments during Retinal Microsurgery

Sparse Hidden Markov Models for Surgical Gesture Classification and Skill Evaluation

Visual Tracking of Surgical Tools for Proximity Detection in Retinal Surgery

Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics

Dimensionality Reduction by Learning an Invariant Mapping

JHU-ISI Gesture and Skill Assessment Working Set ( JIGSAWS ) : A Surgical Activity Dataset for Human Motion Modeling

2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)