HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks

Published in

Interspeech(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

This paper introduces HiFi-GAN, a deep learning method to transform recorded speech to sound as though it had been recorded in a studio, trained with multi-scale adversarial discriminators in both the time domain and the time-frequency domain.

Abstract

Real-world audio recordings are often degraded by factors such as noise, reverberation, and equalization distortion. This paper introduces HiFi-GAN, a deep learning method to transform recorded speech to sound as though it had been recorded in a studio. We use an end-to-end feed-forward WaveNet architecture, trained with multi-scale adversarial discriminators in both the time domain and the time-frequency domain. It relies on the deep feature matching losses of the discriminators to improve the perceptual quality of enhanced speech. The proposed model generalizes well to new speakers, new speech content, and new environments. It significantly outperforms state-of-the-art baseline methods in both objective and subjective experiments.

Authors

Adam Finkelstein

4 papers

Jiaqi Su

1 papers

Zeyu Jin

1 papers

References42 items

Acoustic Matching By Embedding Impulse Responses

Improving GANs for Speech Enhancement

MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

Attention Wave-U-Net for Speech Enhancement

High Fidelity Speech Synthesis with Adversarial Networks

HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networks

Published in

Interspeech(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Adam Finkelstein

4 papers

Jiaqi Su

1 papers

Zeyu Jin

1 papers

References42 items

Acoustic Matching By Embedding Impulse Responses

Improving GANs for Speech Enhancement

MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

Attention Wave-U-Net for Speech Enhancement

High Fidelity Speech Synthesis with Adversarial Networks

Impulse Response Data Augmentation and Deep Neural Networks for Blind Room Acoustic Parameter Estimation

Data Augmentation and Deep Convolutional Neural Networks for Blind Room Acoustic Parameter Estimation

MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement

Perceptually-motivated Environment-specific Speech Enhancement

Learning With Learned Loss Function: Speech Enhancement With Quality-Net to Improve Perceptual Evaluation of Speech Quality

Learning Bandwidth Expansion Using Perceptually-motivated Loss

Towards Generalized Speech Enhancement with Generative Adversarial Networks

Improved Speech Enhancement with the Wave-U-Net

Single-Channel Dereverberation Using Direct MMSE Optimization and Bidirectional LSTM Networks

Speech Denoising with Deep Feature Losses

StarGAN-VC: non-parallel many-to-many Voice Conversion Using Star Generative Adversarial Networks

Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation

Joint Separation and Dereverberation of Reverberant Mixtures with Determined Multichannel Non-Negative Matrix Factorization

Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions

Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition

A Wavenet for Speech Denoising

SEGAN: Speech Enhancement Generative Adversarial Network

Speech dereverberation and denoising using complex ratio masks

NIPS 2016 Tutorial: Generative Adversarial Networks

Statistics of natural reverberation enable perceptual separation of sound and space

Estimation of Room Acoustic Parameters: The ACE Challenge

Investigating RNN-based speech enhancement methods for noise-robust Text-to-Speech

WaveNet: A Generative Model for Raw Audio

Equalization matching of speech recordings in real-world environments

A summary of the REVERB challenge: state-of-the-art and remaining challenges in reverberant speech processing research

A Neural Algorithm of Artistic Style

Can we Automatically Transform Speech Recorded on Common Consumer Devices in Real-World Environments into Professional Production Quality Speech?—A Dataset, Insights, and Challenges

Learning Spectral Mapping for Speech Dereverberation and Denoising

The reverb challenge: A common evaluation framework for dereverberation and recognition of reverberant speech

Speech Enhancement by Online Non-negative Spectrogram Decomposition in Non-stationary Noise Environments

Speech Dereverberation Based on Variance-Normalized Delayed Linear Prediction

Speech Dereverberation

Speech enhancement based on a priori signal to noise estimation

Speech Enhancement Using Bayesian Wavenet

A Regression Approach to Speech Enhancement Based on Deep Neural Networks

Speech Enhancement Using a-Minimum Mean-Square Error Short-Time Spectral Amplitude Estimator

Field of Study

Computer ScienceEngineering

Journal Information

Name

ArXiv

Volume

abs/2005.00687

Venue Information

Name

Interspeech

Type

conference

URL

https://www.isca-speech.org/iscaweb/index.php/conferences/interspeech

Alternate Names

Conf Int Speech Commun Assoc
INTERSPEECH
Conference of the International Speech Communication Association

TL;DR

Abstract

Authors

References42 items

Acoustic Matching By Embedding Impulse Responses

Improving GANs for Speech Enhancement

MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

Attention Wave-U-Net for Speech Enhancement

High Fidelity Speech Synthesis with Adversarial Networks

TL;DR

Abstract

Authors

References42 items

Acoustic Matching By Embedding Impulse Responses

Improving GANs for Speech Enhancement

MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

Attention Wave-U-Net for Speech Enhancement

High Fidelity Speech Synthesis with Adversarial Networks

Impulse Response Data Augmentation and Deep Neural Networks for Blind Room Acoustic Parameter Estimation

Data Augmentation and Deep Convolutional Neural Networks for Blind Room Acoustic Parameter Estimation

MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement

Perceptually-motivated Environment-specific Speech Enhancement

Learning With Learned Loss Function: Speech Enhancement With Quality-Net to Improve Perceptual Evaluation of Speech Quality

Learning Bandwidth Expansion Using Perceptually-motivated Loss

Towards Generalized Speech Enhancement with Generative Adversarial Networks

Improved Speech Enhancement with the Wave-U-Net

Single-Channel Dereverberation Using Direct MMSE Optimization and Bidirectional LSTM Networks

Speech Denoising with Deep Feature Losses

StarGAN-VC: non-parallel many-to-many Voice Conversion Using Star Generative Adversarial Networks

Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation

Joint Separation and Dereverberation of Reverberant Mixtures with Determined Multichannel Non-Negative Matrix Factorization

Natural TTS Synthesis by Conditioning Wavenet on MEL Spectrogram Predictions

Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition

A Wavenet for Speech Denoising

Geometric GAN

SEGAN: Speech Enhancement Generative Adversarial Network

Speech dereverberation and denoising using complex ratio masks

NIPS 2016 Tutorial: Generative Adversarial Networks

Statistics of natural reverberation enable perceptual separation of sound and space

Estimation of Room Acoustic Parameters: The ACE Challenge

Investigating RNN-based speech enhancement methods for noise-robust Text-to-Speech

WaveNet: A Generative Model for Raw Audio

Equalization matching of speech recordings in real-world environments

A summary of the REVERB challenge: state-of-the-art and remaining challenges in reverberant speech processing research

A Neural Algorithm of Artistic Style

Can we Automatically Transform Speech Recorded on Common Consumer Devices in Real-World Environments into Professional Production Quality Speech?—A Dataset, Insights, and Challenges

Learning Spectral Mapping for Speech Dereverberation and Denoising

The reverb challenge: A common evaluation framework for dereverberation and recognition of reverberant speech

Speech Enhancement by Online Non-negative Spectrogram Decomposition in Non-stationary Noise Environments

Speech Dereverberation Based on Variance-Normalized Delayed Linear Prediction

Speech Dereverberation

Speech enhancement based on a priori signal to noise estimation

Speech Enhancement Using Bayesian Wavenet

A Regression Approach to Speech Enhancement Based on Deep Neural Networks

Speech Enhancement Using a-Minimum Mean-Square Error Short-Time Spectral Amplitude Estimator

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names