HIFI++: A Unified Framework for Bandwidth Extension and Speech Enhancement (2022-03-24T00:00:00.000000Z)

TL;DR

It is shown that with the improved generator architecture, HiFi++ performs better or comparably with the state-of-the-art in these tasks while spending significantly less computational resources.

Abstract

Generative adversarial networks have recently demonstrated outstanding performance in neural vocoding outperforming best autoregressive and flow-based models. In this paper, we show that this success can be extended to other tasks of conditional audio generation. In particular, building upon HiFi vocoders, we propose a novel HiFi++ general frame-work for bandwidth extension and speech enhancement. We show that with the improved generator architecture, HiFi++ performs better or comparably with the state-of-the-art in these tasks while spending significantly less computational resources. The effectiveness of our approach is validated through a series of extensive experiments.

Authors

D. Vetrov

7 papers

Pavel Andreev

1 papers

Aibek Alanov

1 papers

TL;DR

Abstract

Authors

References60 items

Board

VoiceFixer: A Unified Framework for High-Fidelity Speech Restoration

Dual-Branch Attention-In-Attention Transformer for Single-Channel Speech Enhancement

Dnsmos P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors

VoiceFixer: Toward General Speech Restoration With Neural Vocoder

Decoupling Magnitude and Phase Estimation with Deep ResUNet for Music Source Separation

A Two-Stage Approach to Speech Bandwidth Extension

SE-Conformer: Time-Domain Speech Enhancement Using Conformer

MetricGAN+: An Improved Version of MetricGAN for Speech Enhancement

GAN Vocoder: Multi-Resolution Discriminator Is All You Need

Towards Robust Speech Super-Resolution

TFGAN: Time and Frequency Domain Based Generative Adversarial Network for High-fidelity Speech Synthesis.

On Filter Generalization for Music Bandwidth Extension Using Deep Neural Networks

Dnsmos: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors

Real-Time Speech Frequency Bandwidth Extension

HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis

SEANet: A Multi-modal Speech Enhancement Network

Real Time Speech Enhancement in the Waveform Domain

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Conformer: Convolution-augmented Transformer for Speech Recognition

Time-Frequency Loss for CNN Based Speech Super-Resolution

Domain

CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit (version 0.92)

MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis

Music Source Separation in the Waveform Domain

Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulations

MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancement

MOSNet: Deep Learning based Objective Assessment for Voice Conversion

Bandwidth Extension on Raw Audio via Generative Adversarial Networks

Differentiable Consistency Constraints for Improved Deep Speech Enhancement

SDR – Half-baked or Well Done?

Waveglow: A Flow-based Generative Network for Speech Synthesis

A Convolutional Recurrent Neural Network for Real-Time Speech Enhancement

Wave-U-Net: A Multi-Scale Neural Network for End-to-End Audio Source Separation

The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods

An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Singing Voice Separation with Deep U-Net Convolutional Networks

Noisy speech database for training speech enhancement algorithms and TTS models

Audio Super Resolution using Neural Networks

Least Squares Generative Adversarial Networks

Generative Multi-Adversarial Networks

SUPERSEDED - CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit

Autoencoding beyond pixels using a learned similarity metric

U-Net: Convolutional Networks for Biomedical Image Segmentation

An Algorithm for Intelligibility Prediction of Time–Frequency Weighted Noisy Speech

Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs

We run the SEANet model for the same number of iterations as the HiFi++, besides that we use the same hyperparameters as described in the original paper

Speech Enhancement with a Generative Adversarial Network

GENERATIVE ADVERSARIAL NETS

The lj speech dataset

c) Did you include any new assets either in the supplemental material or as a URL?

b) Did you describe any potential participant risks, with links to Institutional Review

a) Did you include the full text of instructions given to participants and screenshots, if applicable? [Yes] See Appendix C

Upload the pages to the crowd-sourcing platform, set the number of assessors for each page to at least 30. Assessors are asked to work in headphones in a quiet environment

Insert additional 4 trapping samples into random locations on each page: 2 samples from groundtruth, and 2 samples of a noise without any speech

Filter out the results where the groundtruth samples are assessed with anything except 4 (good) and 5 (excellent), or the samples without voice are assessed with anything except 1 (bad)

Randomly shuffle this set split it into the pages of size 20

Select a set of models to be evaluated; inference their predictions on the selected subset

Select randomly 10 positions on each page where Model1’s prediction will be first

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names