An Improved Model for Voicing Silent Speech (2021-06-03T00:00:00.000000Z)

TL;DR

An improved model for voicing silent speech, where audio is synthesized from facial electromyography (EMG) signals, which uses convolutional layers to extract features from the signals and Transformer layers to propagate information across longer distances.

Abstract

In this paper, we present an improved model for voicing silent speech, where audio is synthesized from facial electromyography (EMG) signals. To give our model greater flexibility to learn its own input features, we directly use EMG signals as input in the place of hand-designed features used by prior work. Our model uses convolutional layers to extract features from the signals and Transformer layers to propagate information across longer distances. To provide better signal for learning, we also introduce an auxiliary task of predicting phoneme labels in addition to predicting speech audio features. On an open vocabulary intelligibility evaluation, our model improves the state of the art for this task by an absolute 25.8%.

Authors

David Gaddy

2 papers

Dana Klein

2 papers

An Improved Model for Voicing Silent Speech

TL;DR

Abstract

Authors

References21 items

Digital Voicing of Silent Speech

Speech synthesis from neural decoding of spoken sentences

Multi-Task Learning of Speech Recognition and Speech Synthesis Parameters for Ultrasound-based Silent Speech Interfaces

Development of sEMG sensors and algorithms for silent speech recognition

Self-Attention with Relative Position Representations

EMG-to-Speech: Direct Generation of Speech From Facial Electromyographic Signals

Decoupled Weight Decay Regularization

Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi

Attention is All you Need

WaveNet: A Generative Model for Raw Audio

Wav2Letter: an End-to-End ConvNet-based Speech Recognition System

Deep Residual Learning for Image Recognition

Direct conversion from facial myoelectric signals to speech using Deep Neural Networks

Deep Speech: Scaling up end-to-end speech recognition

Velum Movement Detection based on Surface Electromyography for Speech Interface

Synthesizing speech from electromyography using voice transformation techniques

Towards continuous speech recognition using surface electromyography

Fundamentals of speech recognition

match the numbers in the ﬁle names. Many of the clips may be difﬁcult to hear. If this is the case, write whatever words you are able to make out, even if it does not form a complete expression

each of which will contain a sentence in English. Write your transcriptions into a spreadsheet such as Excel or Google sheets so that the row numbers

evaluators for the transcription test described in Section 3.1: Please listen to each of the attached sound ﬁles and write down what you hear

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names