Home Research Papers Datasets State of the Art Pricing

Discover, visualize, and connect AI research papers. Explore the latest trends and insights in artificial intelligence research.

Product

Home
Research Papers
About

Support

Contact
Terms of Service
Privacy Policy

© 2026 Papersgraph. All rights reserved.

computer-vision-7

Speaker-Specific Lip to Speech Synthesis

3260 papers • 126 benchmarks • 313 datasets

How accurately can we infer an individual’s speech style and content from his/her lip movements? [1] In this task, the model is trained on a specific speaker, or a very limited set of speakers. [1] Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis, CVPR 2020.

(Image credit: Papersgraph)

Benchmarks

These leaderboards are used to track progress in speaker-specific-lip-to-speech-synthesis-7

Trend

Dataset

Best Model

Actions

GRID corpus (mixed-speech)

GRID corpus (mixed-speech)

Lip2Wav (EH)

Lip2Wav (EH)

Lip2Wav (HS)

Libraries

i

Use these libraries to find speaker-specific-lip-to-speech-synthesis-7 models and implementations

Datasets

TIMIT

GRID Dataset

Subtasks

No subtasks available.

Most implemented papers

Densely Connected Convolutional Networks

Kilian Q. Weinberger, Zhuang Liu, Gao Huang•Wed Aug 24 2016

The Dense Convolutional Network (DenseNet), which connects each layer to every other layer in a feed-forward fashion, and has several compelling advantages: they alleviate the vanishing-gradient problem, strengthen feature propagation, encourage feature reuse, and substantially reduce the number of parameters.

41544

Content

Introduction Benchmarks Datasets Subtasks Libraries Papers

Lip2Wav (HS)

Lip2Wav (DL)

Lip2Wav (DL)

Lip2Wav (Chess)

Lip2Wav (Chess)

Lip2Wav (Chem)

Lip2Wav (Chem)

TCD-TIMIT corpus (mixed-speech)

TCD-TIMIT corpus (mixed-speech)

0

Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis

C. V. Jawahar, Prajwal K R, Vinay P. Namboodiri, Rudrabha Mukhopadhyay•Sat May 16 2020

This work proposes a novel approach with key design choices to achieve accurate, natural lip to speech synthesis in such unconstrained scenarios for the first time and shows that its method is four times more intelligible than previous works in this space.

132 0

Speech Reconstruction With Reminiscent Sound Via Visual Voice Memory

Minsu Kim, Joanna Hong, Se Jin Park, Yong Man Ro•Thu Dec 31 2020

The goal of this work is to reconstruct speech from silent video, in both speaker dependent and independent ways. Unlike previous works that have been mostly restricted to a speaker dependent setting, we propose Visual Voice memory to restore essential auditory information to generate proper speech from different speakers and even unseen speakers. The proposed memory takes additional auditory information that corresponds to the input face movements and stores the auditory contexts that can be recalled by the given input visual features. Specifically, the Visual Voice memory contains value and key memory slots, where value memory slots are for saving the audio features, and key memory slots are for storing the visual features in the same location of the saved audio features. Guiding each memory to properly save each feature, the model can adequately produce the speech. Hence, our method employs both video and audio information during training time but does not require any additional auditory input during inference. Our key contributions are: (1) proposing the Visual Voice memory that brings rich information of audio that complements the visual features, thus producing high-quality speech from silent video, and (2) enabling multi-speaker and unseen speaker training by memorizing auditory features and the corresponding visual features. We validate the proposed framework on GRID and Lip2Wav datasets and show that our method surpasses the performance of previous works on both multi-speaker and speaker independent settings. We also demonstrate that the Visual Voice memory contains meaningful information to reconstruct speech.

28 0

Adding a benchmark result helps the community track progress.

Speaker-Specific Lip to Speech Synthesis | State-of-the-Art