On Filter Generalization for Music Bandwidth Extension Using Deep Neural Networks

Published in

IEEE Journal on Selected Topics in Signal Proce...(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

A data augmentation strategy is proposed which utilizes multiple low-pass filters during training and leads to improved generalization to unseen filtering conditions at test time, which results in a lower SNR than the band-limited input.

Abstract

In this paper, we address a subtopic of the broad domain of audio enhancement, namely musical audio bandwidth extension. We formulate the bandwidth extension problem using deep neural networks, where a band-limited signal is provided as input to the network, with the goal of reconstructing a full-bandwidth output. Our main contribution centers on the impact of the choice of low-pass filter when training and subsequently testing the network. For two different state-of-the-art deep architectures, ResNet and U-Net, we demonstrate that when the training and testing filters are matched, improvements in signal-to-noise ratio (SNR) of up to 7 dB can be obtained. However, when these filters differ, the improvement falls considerably and under some training conditions results in a lower SNR than the band-limited input. To circumvent this apparent overfitting to filter shape, we propose a data augmentation strategy which utilizes multiple low-pass filters during training and leads to improved generalization to unseen filtering conditions at test time.

Authors

Serkan Sulun

1 papers

M. Davies

1 papers

References67 items

Generative Adversarial Networks

Learning to Denoise Historical Music

Time-Frequency Loss for CNN Based Speech Super-Resolution

A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences

PyTorch: An Imperative Style, High-Performance Deep Learning Library

On Filter Generalization for Music Bandwidth Extension Using Deep Neural Networks

Published in

IEEE Journal on Selected Topics in Signal Proce...(2020)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Serkan Sulun

1 papers

M. Davies

1 papers

References67 items

Generative Adversarial Networks

Learning to Denoise Historical Music

Time-Frequency Loss for CNN Based Speech Super-Resolution

A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Speech Loss Compensation by Generative Adversarial Networks

Investigating Under and Overfitting in Wasserstein Generative Adversarial Networks

Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms

MR image reconstruction using deep learning: evaluation of network structure and loss functions.

Artificial Bandwidth Extension Using a Conditional Generative Adversarial Network with Discriminative Training

Robust Audio-visual Speech Recognition Using Bimodal Dfsmn with Multi-condition Training and Dropout Regularization

Bandwidth Extension on Raw Audio via Generative Adversarial Networks

Multi-domain adversarial training of neural network acoustic models for distant speech recognition

Improved Speech Enhancement with the Wave-U-Net

Deep Learned Frame Prediction for Video Compression

Prostate Cancer Classification on VERDICT DW-MRI Using Convolutional Neural Networks

An empirical study on evaluation metrics of generative adversarial networks

Time-Frequency Networks for Audio Super-Resolution

Residual Dense Network for Image Super-Resolution

Deep High Dynamic Range Imaging with Large Foreground Motions

Singing Voice Separation with Deep U-Net Convolutional Networks

Cell Lineage Tracing in Lens-Free Microscopy Videos

Improving Deep Learning using Generic Data Augmentation

Enhanced Deep Residual Networks for Single Image Super-Resolution

Balanced Two-Stage Residual Networks for Image Super-Resolution

The 2016 Signal Separation Evaluation Campaign

Audio Super Resolution using Neural Networks

CNN architectures for large-scale audio classification

MedleyDB 2.0: New Data and a System for Sustainable Data Collection

Inpainting of Long Audio Segments With Similarity Graphs

Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network

Wide Residual Networks

Deep Residual Learning for Image Recognition

U-Net: Convolutional Networks for Biomedical Image Segmentation

A deep neural network approach to speech bandwidth expansion

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Image Super-Resolution Using Deep Convolutional Networks

Adam: A Method for Stochastic Optimization

Non-negative matrix completion for bandwidth extension: A convex optimization approach

ViSQOL: The Virtual Speech Quality Objective Listener

A study of HMM-based bandwidth extension of speech signals

Multi-Condition Training for Unknown Environment Adaptation in Robust ASR Under Real Conditions

An HMM-based artificial bandwidth extension evaluated by cross-language training and test

Neural Network-Based Artificial Bandwidth Expansion of Speech

Bandwidth expansion of narrowband speech using non-negative matrix factorization

Neural networks versus codebooks in an application for bandwidth extension of speech signals

On artificial bandwidth extension of telephone speech

Modeling the mixtures of known noise and unknown unexpected noise for robust speech recognition

Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs

Speech enhancement via frequency bandwidth extension using line spectral frequencies

Narrowband to wideband conversion of speech using GMM based transformation

A new technique for wideband enhancement of coded narrowband speech

Generation of broadband speech from narrowband speech using piecewise linear mapping

An algorithm to reconstruct wideband speech from narrowband speech based on codebook mapping

Statistical recovery of wideband speech from narrowband speech

kuleshov/audio-super-res

HIGH FREQUENCY MAGNITUDE SPECTROGRAM RECONSTRUCTION FOR MUSIC MIXTURES USING CONVOLUTIONAL AUTOENCODERS

Convolutional Neural Networks

DNN-based speech bandwidth expansion and its application to adding high-frequency missing features for automatic speech recognition of narrowband speech

Speech bandwidth expansion based on deep neural networks

A Software Framework for Musical Data Augmentation

Dropout: a simple way to prevent neural networks from overfitting

Mel-frequency cepstral coefficient-based bandwidth extension of narrowband speech

TL;DR

Abstract

Authors

References67 items

Generative Adversarial Networks

Learning to Denoise Historical Music

Time-Frequency Loss for CNN Based Speech Super-Resolution

A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences

PyTorch: An Imperative Style, High-Performance Deep Learning Library

TL;DR

Abstract

Authors

References67 items

Generative Adversarial Networks

Learning to Denoise Historical Music

Time-Frequency Loss for CNN Based Speech Super-Resolution

A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences

PyTorch: An Imperative Style, High-Performance Deep Learning Library

Speech Loss Compensation by Generative Adversarial Networks

Investigating Under and Overfitting in Wasserstein Generative Adversarial Networks

Fréchet Audio Distance: A Reference-Free Metric for Evaluating Music Enhancement Algorithms

MR image reconstruction using deep learning: evaluation of network structure and loss functions.

Artificial Bandwidth Extension Using a Conditional Generative Adversarial Network with Discriminative Training

Robust Audio-visual Speech Recognition Using Bimodal Dfsmn with Multi-condition Training and Dropout Regularization

Bandwidth Extension on Raw Audio via Generative Adversarial Networks

Multi-domain adversarial training of neural network acoustic models for distant speech recognition

Improved Speech Enhancement with the Wave-U-Net

Deep Learned Frame Prediction for Video Compression

Prostate Cancer Classification on VERDICT DW-MRI Using Convolutional Neural Networks

An empirical study on evaluation metrics of generative adversarial networks

Time-Frequency Networks for Audio Super-Resolution

Residual Dense Network for Image Super-Resolution

Deep High Dynamic Range Imaging with Large Foreground Motions

Singing Voice Separation with Deep U-Net Convolutional Networks

Cell Lineage Tracing in Lens-Free Microscopy Videos

Improving Deep Learning using Generic Data Augmentation

Enhanced Deep Residual Networks for Single Image Super-Resolution

Balanced Two-Stage Residual Networks for Image Super-Resolution

The 2016 Signal Separation Evaluation Campaign

Audio Super Resolution using Neural Networks

CNN architectures for large-scale audio classification

MedleyDB 2.0: New Data and a System for Sustainable Data Collection

Inpainting of Long Audio Segments With Similarity Graphs

Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network

Wide Residual Networks

Deep Residual Learning for Image Recognition

U-Net: Convolutional Networks for Biomedical Image Segmentation

A deep neural network approach to speech bandwidth expansion

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Image Super-Resolution Using Deep Convolutional Networks

Adam: A Method for Stochastic Optimization

Non-negative matrix completion for bandwidth extension: A convex optimization approach

ViSQOL: The Virtual Speech Quality Objective Listener

Audio Inpainting

A study of HMM-based bandwidth extension of speech signals

Multi-Condition Training for Unknown Environment Adaptation in Robust ASR Under Real Conditions

An HMM-based artificial bandwidth extension evaluated by cross-language training and test

Neural Network-Based Artificial Bandwidth Expansion of Speech

Bandwidth expansion of narrowband speech using non-negative matrix factorization

Neural networks versus codebooks in an application for bandwidth extension of speech signals

On artificial bandwidth extension of telephone speech

Modeling the mixtures of known noise and unknown unexpected noise for robust speech recognition

Perceptual evaluation of speech quality (PESQ)-a new method for speech quality assessment of telephone networks and codecs

Speech enhancement via frequency bandwidth extension using line spectral frequencies

Narrowband to wideband conversion of speech using GMM based transformation

A new technique for wideband enhancement of coded narrowband speech

Generation of broadband speech from narrowband speech using piecewise linear mapping

An algorithm to reconstruct wideband speech from narrowband speech based on codebook mapping

Statistical recovery of wideband speech from narrowband speech

kuleshov/audio-super-res

HIGH FREQUENCY MAGNITUDE SPECTROGRAM RECONSTRUCTION FOR MUSIC MIXTURES USING CONVOLUTIONAL AUTOENCODERS

Convolutional Neural Networks

DNN-based speech bandwidth expansion and its application to adding high-frequency missing features for automatic speech recognition of narrowband speech

Speech bandwidth expansion based on deep neural networks

A Software Framework for Musical Data Augmentation

Dropout: a simple way to prevent neural networks from overfitting

Mel-frequency cepstral coefficient-based bandwidth extension of narrowband speech

Digital audio restoration

Accessed

“VGGish,”