Combining Residual Networks with LSTMs for Lipreading (2017-03-12T00:00:00.000000Z)

TL;DR

An end-to-end deep learning architecture for word-level visual speech recognition that is a combination of spatiotemporal convolutional, residual and bidirectional Long Short-Term Memory networks is proposed.

Abstract

We propose an end-to-end deep learning architecture for word-level visual speech recognition. The system is a combination of spatiotemporal convolutional, residual and bidirectional Long Short-Term Memory networks. We train and evaluate it on the Lipreading In-The-Wild benchmark, a challenging database of 500-size target-words consisting of 1.28sec video excerpts from BBC TV broadcasts. The proposed network attains word accuracy equal to 83.0, yielding 6.8 absolute improvement over the current state-of-the-art, without using information about word boundaries during training or testing.

Authors

Georgios Tzimiropoulos

9 papers

Themos Stafylakis

6 papers

TL;DR

Abstract

Authors

References36 items

Building Competitive Direct Acoustics-to-Word Models for English Conversational Speech Recognition

Lip Reading in the Wild

Lip Reading Sentences in the Wild

LipNet: Sentence-level Lipreading

LipNet: End-to-End Sentence-level Lipreading

Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition

Two-Stage Convolutional Part Heatmap Regression for the 1st 3D Face Alignment in the Wild (3DFAW) Challenge

Deep complementary bottleneck features for visual speech recognition

Decoding visemes: Improving machine lip-reading

Improved speaker independent lip reading using speaker adaptive training and deep neural networks

Identity Mappings in Deep Residual Networks

Lipreading with long short-term memory

Deep Residual Learning for Image Recognition

Improving lip-reading performance for robust audiovisual speech recognition using DNNs

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Audio-visual speech recognition using deep learning

A review of recent advances in visual speech decoding

Audio-visual deep learning for noise robust speech recognition

Lip reading using optical flow and support vector machines

The Natural Statistics of Audiovisual Speech

Adaptive Multimodal Fusion by Uncertainty Compensation With Application to Audiovisual Speech Recognition

An audio-visual corpus for speech perception and automatic speech recognition.

Bidirectional LSTM Networks for Improved Phoneme Classification and Recognition

Recent advances in the automatic recognition of audiovisual speech

CUAVE: A new audio-visual database for multimodal human-computer interface research

Lipreading from color video

Confusions among visually perceived consonants.

Bulat , Adrian and Tzimiropoulos , Georgios ( 2016 ) Convolutional aggregation of local evidence for large pose face alignment

Top Downloads in IEEE Xplore [Reader's Choice]

Word embeddings for speech recognition

Torch7: A Matlab-like Environment for Machine Learning

Audio-Visual Automatic Speech Recognition: An Overview

Continuous automatic speech recognition by lipreading

Connectionist Temporal Classiﬁcation: Labelling Unsegmented Sequence Data with Recurrent Neural Networks

References

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names