LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild (2018-10-16T00:00:00.000000Z)

TL;DR

This paper presents a naturally-distributed large-scale benchmark for lip-reading in the wild, named LRW-1000, which contains 1,000 classes with 718,018 samples from more than 2,000 individual speakers, and is currently the largest word-level lipreading dataset and also the only public large- scale Mandarin lip-read dataset.

Abstract

Large-scale datasets have successively proven their fundamental importance in several research fields, especially for early progress in some emerging topics. In this paper, we focus on the problem of visual speech recognition, also known as lip-reading, which has received increasing interest in recent years. We present a naturally-distributed large-scale benchmark for lip-reading in the wild, named LRW-1000, which contains 1,000 classes with 718,018 samples from more than 2,000 individual speakers. Each class corresponds to the syllables of a Mandarin word composed of one or several Chinese characters. To the best of our knowledge, it is currently the largest word-level lipreading dataset and also the only public large-scale Mandarin lip-reading dataset. This dataset aims at covering a "natural" variability over different speech modes and imaging conditions to incorporate challenges encountered in practical applications. It has shown a large variation in this benchmark in several aspects, including the number of samples in each class, video resolution, lighting conditions, and speakers’ attributes such as pose, age, gender, and make-up. Besides providing a detailed description of the dataset and its collection pipeline, we evaluate several typical popular lip-reading methods and perform a thorough analysis of the results from several aspects. The results demonstrate the consistency and challenges of our dataset, which may open up some new promising directions for future work.

LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild

TL;DR

Abstract

Authors

References26 items

Learning to lip read words by watching videos

End-to-End Multi-View Lipreading

Combining Residual Networks with LSTMs for Lipreading

Lip Reading in the Wild

Out of Time: Automated Lip Sync in the Wild

Lip Reading Sentences in the Wild

LipNet: Sentence-level Lipreading

Lip reading using a dynamic feature of lip images and convolutional neural networks

Deep Residual Learning for Image Recognition

OuluVS2: A multi-view audiovisual database for non-rigid mouth motion analysis

Learning Spatiotemporal Features with 3D Convolutional Networks

Return of the Devil in the Details: Delving Deep into Convolutional Nets

MKPLS: Manifold Kernel Partial Least Squares for Lipreading and Speaker Identification

Lip reading using optical flow and support vector machines

Classification and Feature Extraction by Simplexization

Audio-visual speech translation with automatic lip syncqronization and face tracking based on 3-D head model

Extraction of Visual Features for Lipreading

Audio-Visual Speech Modeling for Continuous Speech Recognition

An image transform approach for HMM based automatic lipreading

Toward movement-invariant automatic lip-reading and speech recognition

Lip Reading in Profile

Lipreading using convolutional neural network

AVICAR: audio-visual speech corpus in a car environment

AVICAR: audio-visual speech corpus in a car environment

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. IEEE TRANSACTIONS ON MULTIMEDIA 1 Lipreading with Local Spatiotemporal Descriptors

SeetaFace . Seetafaceengine 2

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names