Towards Predicting Fine Finger Motions from Ultrasound Images via Kinematic Representation (2022-02-10T00:00:00.000000Z)

TL;DR

This work presents an end-to-end robotic system, which can successfully infer fine finger motions, achieved by modeling the hand as a robotic manipulator and using it as an intermediate representation to encode muscles' dynamics from a sequence of US images.

Abstract

A central challenge in building robotic prostheses is the creation of a sensor-based system able to read physiological signals from the lower limb and instruct a robotic hand to perform various tasks. Existing systems typically perform discrete gestures such as pointing or grasping, by employing electromyography (EMG) or ultrasound (US) technologies to analyze muscle states. While estimating finger gestures has been done in the past by detecting prominent gestures, we are interested in detection, or inference, done in the context of fine motions that evolve over time. Examples include motions occurring when performing fine and dexterous tasks such as keyboard typing or piano playing. We consider this task as an important step towards higher adoption rates of robotic prostheses among arm amputees, as it has the potential to dramatically increase functionality in performing daily tasks. To this end, we present an end-to-end robotic system, which can successfully infer fine finger motions. This is achieved by modeling the hand as a robotic manipulator and using it as an intermediate representation to encode muscles' dynamics from a sequence of US images. We evaluated our method by collecting data from a group of subjects and demonstrating how it can be used to replay music played or text typed. To the best of our knowledge, this is the first study demonstrating these downstream tasks within an end-to-end system.

Authors

A. Bronstein

5 papers

Dean Zadok

1 papers

Oren Salzman

1 papers

TL;DR

Abstract

Authors

References44 items

Prediction of Metacarpophalangeal Joint Angles and Classification of Hand Configurations Based on Ultrasound Imaging of the Forearm

A Wearable Ultrasound System for Sensing Muscular Morphological Deformations

Modeling Affect-based Intrinsic Rewards for Exploration and Learning

Advanced Materials and Technologies for Touch Sensing in Prosthetic Limbs

A CNN-LSTM-Based Model to Forecast Stock Prices

Parrot: Data-Driven Behavioral Priors for Reinforcement Learning

A Two-Stage Unsupervised Approach for Low Light Image Enhancement

Analysis of subject specific grasping patterns

Towards Sonomyography-Based Real-Time Control of Powered Prosthesis Grasp Synergies

GPS-based citywide traffic congestion forecasting using CNN-RNN and C3D hybrid model

Simultaneous Prediction of Wrist/Hand Motion via Wearable Ultrasound Sensing

Intuitive neuromyoelectric control of a dexterous bionic arm using a modified Kalman filter

A Lightweight Ultrasound Probe for Wearable Human–Machine Interfaces

Deep Learning in Ultrasound Imaging

Towards Zero Re-Training for Long-Term Hand Gesture Recognition via Ultrasound Sensing

SottoVoce: An Ultrasound Imaging-Based Silent Speech Interaction Using Deep Neural Networks

Autoencoder-based transfer learning in brain–computer interface for rehabilitation robot

A Two-Stage Clustering Based 3D Visual Saliency Model for Dynamic Scenarios

A Customizable Wearable Robust 3D Printed Bionic Arm: Muscle Controlled

Robust Lane Detection From Continuous Driving Scenes Using Deep Neural Networks

Learning beamforming in ultrasound imaging

Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network

Seeing Speech: Ultrasound-based Multimedia Resources for Pronunciation Learning in Indigenous Languages

Deep Learning for Electromyographic Hand Gesture Signal Classification by Leveraging Transfer Learning

Towards CT-quality Ultrasound Imaging using Deep Learning

EchoFlex: Hand Gesture Recognition using Ultrasound Imaging

The relation between tongue shape and pitch in clarinet playing using ultrasound measurements.

Performances of surface EMG and Ultrasound signals in recognizing finger motion

Recent advances in convolutional neural networks

Deep Residual Learning for Image Recognition

U-Net: Convolutional Networks for Biomedical Image Segmentation

Adam: A Method for Stochastic Optimization

Learning Spatiotemporal Features with 3D Convolutional Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

Advanced upper limb prosthetic devices: implications for upper limb prosthetic rehabilitation.

Autoencoders, Unsupervised Learning, and Deep Architectures

ImageNet: A large-scale hierarchical image database

Prosthetic hands from Touch Bionics

Tongue 'n' Groove: An Ultrasound based Music Controller

Learning internal representations by error propagation

Deep Learning in Medical Ultrasound Image Analysis: A Review

“Propriocep-tive sonomyographic control: A novel method of intuitive proportional control of multiple degrees of freedom for upper-extremity amputees,”

Myoelectric signals in control of prostheses. Studies on arm amputees and normal individuals.

“Towards predicting ﬁne ﬁnger motions from ultrasound images via kinematic representation,”

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names