Deep Audio-Visual Speech Recognition

Published in

IEEE Transactions on Pattern Analysis and Machi...(2018)

External Links:

Generate Graph DownloadPDF

TL;DR

This work compares two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss, built on top of the transformer self-attention architecture.

Abstract

The goal of this work is to recognise phrases and sentences being spoken by a talking face, with or without the audio. Unlike previous works that have focussed on recognising a limited number of words or phrases, we tackle lip reading as an open-world problem – unconstrained natural language sentences, and in the wild videos. Our key contributions are: (1) we compare two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss. Both models are built on top of the transformer self-attention architecture; (2) we investigate to what extent lip reading is complementary to audio speech recognition, especially when the audio signal is noisy; (3) we introduce and publicly release a new dataset for audio-visual speech recognition, LRS2-BBC, consisting of thousands of natural sentences from British television. The models that we train surpass the performance of all previous work on a lip reading benchmark dataset by a significant margin.

Authors

O. Vinyals

41 papers

Andrew Zisserman

62 papers

A. Senior

7 papers

References58 items

LRS3-TED: a large-scale dataset for visual speech recognition

Large-Scale Visual Speech Recognition

Deep Lip Reading: a comparison of models and an online application

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

End-to-End Audiovisual Speech Recognition

Deep Audio-Visual Speech Recognition

Published in

IEEE Transactions on Pattern Analysis and Machi...(2018)

External Links:

Generate Graph DownloadPDF

TL;DR

This work compares two models for lip reading, one using a CTC loss, and the other using a sequence-to-sequence loss, built on top of the transformer self-attention architecture.

Abstract

Authors

O. Vinyals

41 papers

Andrew Zisserman

62 papers

A. Senior

7 papers

References58 items

LRS3-TED: a large-scale dataset for visual speech recognition

Large-Scale Visual Speech Recognition

Deep Lip Reading: a comparison of models and an online application

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

End-to-End Audiovisual Speech Recognition

Triantafyllos Afouras

12 papers

Joon Son Chung

11 papers

Letter-Based Speech Recognition with Gated ConvNets

An Analysis of Incorporating an External Language Model into a Sequence-to-Sequence Model

State-of-the-Art Speech Recognition with Sequence-to-Sequence Models

Learning Filterbanks from Raw Speech for Phone Recognition

Attention is All you Need

Combining Residual Networks with LSTMs for Lipreading

Residual Convolutional CTC Networks for Automatic Speech Recognition

An audio-visual corpus for multimodal automatic speech recognition

Lip Reading in the Wild

Out of Time: Automated Lip Sync in the Wild

Lip Reading Sentences in the Wild

LipNet: Sentence-level Lipreading

LipNet: End-to-End Sentence-level Lipreading

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Wav2Letter: an End-to-End ConvNet-based Speech Recognition System

Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks

Convolutional Two-Stream Network Fusion for Video Action Recognition

Deep complementary bottleneck features for visual speech recognition

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Lipreading with long short-term memory

Deep Residual Learning for Image Recognition

SSD: Single Shot MultiBox Detector

Deep Learning of Mouth Shapes for Sign Language

Audio-visual speech recognition using deep bottleneck features and high-performance lipreading

Fast and accurate recurrent neural network acoustic models for speech recognition

Attention-Based Models for Speech Recognition

Deep multimodal learning for Audio-Visual Speech Recognition

Adam: A Method for Stochastic Optimization

Audio-visual speech recognition using deep learning

End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results

Going deeper with convolutions

Sequence to Sequence Learning with Neural Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

Neural Machine Translation by Jointly Learning to Align and Translate

ImageNet Large Scale Visual Recognition Challenge

A review of recent advances in visual speech decoding

Towards End-To-End Speech Recognition with Recurrent Neural Networks

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

ImageNet classification with deep convolutional neural networks

Sequence Transduction with Recurrent Neural Networks

Deep Neural Networks for Acoustic Modeling in Speech Recognition

Audio-visual speech recognition incorporating facial depth information captured by the Kinect

Dlib-ml: A Machine Learning Toolkit

Speaker identification on the SCOTUS corpus

An audio-visual corpus for speech perception and automatic speech recognition.

Reliable Transition Detection in Videos: A Survey and Practitioner's Guide

An Iterative Image Registration Technique with an Application to Stereo Vision

Lip Reading in Profile

Listen, attend and spell

Lexicon-Free Conversational Speech Recognition with Neural Networks

Lipreading using convolutional neural network

Automatic Speech Recognition

Connectionist Temporal Classiﬁcation: Labelling Unsegmented Sequence Data with Recurrent Neural Networks

Field of Study

Computer ScienceMedicine

Journal Information

Name

IEEE Transactions on Pattern Analysis and Machine Intelligence

Page

1161-1180

Volume

Venue Information

Name

IEEE Transactions on Pattern Analysis and Machine Intelligence

Type

journal

URL

http://www.computer.org/tpami/

Alternate Names

IEEE Trans Pattern Anal Mach Intell