Improving the Performance of Unimodal Dynamic Hand-Gesture Recognition With Multimodal Training (2018-12-14T00:00:00.000000Z)

TL;DR

This work presents an efficient approach for leveraging the knowledge from multiple modalities in training unimodal 3D convolutional neural networks (3D-CNNs) for the task of dynamic hand gesture recognition, and introduces a "spatiotemporal semantic alignment" loss (SSA) to align the content of the features from different networks.

Abstract

We present an efficient approach for leveraging the knowledge from multiple modalities in training unimodal 3D convolutional neural networks (3D-CNNs) for the task of dynamic hand gesture recognition. Instead of explicitly combining multimodal information, which is commonplace in many state-of-the-art methods, we propose a different framework in which we embed the knowledge of multiple modalities in individual networks so that each unimodal network can achieve an improved performance. In particular, we dedicate separate networks per available modality and enforce them to collaborate and learn to develop networks with common semantics and better representations. We introduce a "spatiotemporal semantic alignment" loss (SSA) to align the content of the features from different networks. In addition, we regularize this loss with our proposed "focal regularization parameter" to avoid negative knowledge transfer. Experimental results show that our framework improves the test time recognition accuracy of unimodal networks, and provides the state-of-the-art performance on various dynamic hand gesture recognition datasets.

Authors

Vishal M. Patel

23 papers

Hamid Reza Vaezi Joze

2 papers

Mahdi Abavisani

2 papers

TL;DR

Abstract

Authors

References55 items

Deep CNN-based Multi-task Learning for Open-Set Recognition

Deep Transfer Learning for Multiple Class Novelty Detection

Deep Multimodal Subspace Clustering Networks

EgoGesture: A New Dataset and Benchmark for Egocentric Hand Gesture Recognition

A Closer Look at Spatiotemporal Convolutions for Action Recognition

In2I: Unsupervised Multi-Image-to-Image Translation Using Generative Adversarial Networks

Deep Multimodal Learning: A Survey on Recent Advances and Trends

Egocentric Gesture Recognition Using Recurrent 3D Convolutional Neural Networks with Spatiotemporal Transformer Modules

SubUNets: End-to-End Hand Shape and Continuous Sign Language Recognition

Large-Scale Multimodal Gesture Recognition Using Heterogeneous Networks

LPSNet: A Novel Log Path Signature Feature Based Hand Gesture Recognition Framework

Multimodal Gesture Recognition Based on the ResC3D Network

Learning Spatiotemporal Features Using 3DCNN and Convolutional LSTM for Gesture Recognition

Recurrent Convolutional Neural Networks for Continuous Sign Language Recognition by Staged Optimization

Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset

The Kinetics Human Action Video Dataset

Multimodal Gesture Recognition Using 3-D Convolution and Convolutional LSTM

Correlation Alignment for Unsupervised Domain Adaptation

Using Convolutional 3D Neural Networks for User-independent continuous gesture recognition

Deep CORAL: Correlation Alignment for Deep Domain Adaptation

Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks

Image Style Transfer Using Convolutional Neural Networks

Hand gesture recognition with 3D convolutional neural networks

Texture Synthesis Using Convolutional Neural Networks

Multi-sensor system for driver's hand-gesture recognition

Touch-less interactive augmented reality game on vision-based wearable device

A Robust and Efficient Video Representation for Action Recognition

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Transfer of Learning

Learning Spatiotemporal Features with 3D Convolutional Networks

Long-term recurrent convolutional networks for visual recognition and description

How transferable are features in deep neural networks?

Multi-scale Deep Learning for Gesture Detection and Localization

Multi-Modal Distance Metric Learning: ABayesian Non-parametric Approach

Very Deep Convolutional Networks for Large-Scale Image Recognition

Hand Gesture Recognition in Real Time for Automotive Interfaces: A Multimodal Vision-Based Approach and Evaluations

Large-Scale Video Classification with Convolutional Neural Networks

Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks

Two-Stream Convolutional Networks for Action Recognition in Videos

Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers

Vision based hand gesture recognition for human computer interaction: a survey

Robust 3D Action Recognition with Random Occupancy Patterns

Evaluation of low-level features and their combinations for complex event detection in open source videos

Dynamic hand gesture recognition: An exemplar-based approach from motion divergence fields

Deep Learning of Representations for Unsupervised and Transfer Learning

Multimodal Deep Learning

A Survey on Transfer Learning

ImageNet: A large-scale hierarchical image database

Two-Frame Motion Estimation Based on Polynomial Expansion

Multitask Learning

Multimodal sparse and low-rank subspace clustering

Convolutional Neural Networks

Signal Processing (ICASSP)

This Paper Is Included in the Proceedings of the 12th Usenix Symposium on Operating Systems Design and Implementation (osdi '16). Tensorflow: a System for Large-scale Machine Learning Tensorflow: a System for Large-scale Machine Learning

A robust and efﬁcient video representation for action recognition

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names