Motion Fused Frames: Data Level Fusion Strategy for Hand Gesture Recognition (2018-04-19T00:00:00.000000Z)

TL;DR

This paper proposes a data level fusion strategy, Motion Fused Frames (MFFs), designed to fuse motion information into static images as better representatives of spatio-temporal states of an action.

Abstract

Acquiring spatio-temporal states of an action is the most crucial step for action classification. In this paper, we propose a data level fusion strategy, Motion Fused Frames (MFFs), designed to fuse motion information into static images as better representatives of spatio-temporal states of an action. MFFs can be used as input to any deep learning architecture with very little modification on the network. We evaluate MFFs on hand gesture recognition tasks using three video datasets - Jester, ChaLearn LAP IsoGD and NVIDIA Dynamic Hand Gesture Datasets - which require capturing long-term temporal relations of hand movements. Our approach obtains very competitive performance on Jester and ChaLearn benchmarks with the classification accuracies of 96.28% and 57.4%, respectively, while achieving state-of-the-art performance with 84.7% accuracy on NVIDIA benchmark.

Authors

G. Rigoll

11 papers

Okan Köpüklü

5 papers

Neslihan Köse

1 papers

TL;DR

Abstract

Authors

References37 items

A Unified Framework for Multi-Modal Isolated Gesture Recognition

Temporal Relational Reasoning in Videos

Automatic differentiation in PyTorch

Large-Scale Multimodal Gesture Recognition Using Heterogeneous Networks

Multimodal Gesture Recognition Based on the ResC3D Network

ConvNet Architecture Search for Spatiotemporal Feature Learning

Learning Spatiotemporal Features Using 3DCNN and Convolutional LSTM for Gesture Recognition

A Survey on Deep Learning Based Approaches for Action and Gesture Recognition in Image Sequences

Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM

Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks

Large-scale gesture recognition with a fusion of RGB-D data based on the C3D model

Large-scale Isolated Gesture Recognition using Convolutional Neural Networks

Large-scale Isolated Gesture Recognition using pyramidal 3D convolutional networks

Spatiotemporal Residual Networks for Video Action Recognition

Temporal Segment Networks: Towards Good Practices for Deep Action Recognition

Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks

ChaLearn Looking at People RGB-D Isolated and Continuous Datasets for Gesture Recognition

Convolutional Two-Stream Network Fusion for Video Action Recognition

Long-Term Temporal Convolutions for Action Recognition

Deep Residual Learning for Image Recognition

Towards Good Practices for Very Deep Two-Stream ConvNets

A Robust and Efficient Video Representation for Action Recognition

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Learning Deep Features for Scene Recognition using Places Database

Learning Spatiotemporal Features with 3D Convolutional Networks

Long-term recurrent convolutional networks for visual recognition and description

Hand Gesture Recognition in Real Time for Automotive Interfaces: A Multimodal Vision-Based Approach and Evaluations

Efficient Feature Extraction, Encoding, and Classification for Action Recognition

Large-Scale Video Classification with Convolutional Neural Networks

Two-Stream Convolutional Networks for Action Recognition in Videos

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

ImageNet classification with deep convolutional neural networks

High Accuracy Optical Flow Estimation Based on a Theory for Warping

Best practices for convolutional neural networks applied to visual document analysis

Two-Frame Motion Estimation Based on Polynomial Expansion

Recognition

Efﬁcient feature extraction, encoding and classiﬁcation for action recognition

Field of Study

Journal Information

Name

Page