Real-Time Target Sound Extraction
TL
TL;DR
Waveformer, an encoder-decoder architecture with a stack of dilated causal convolution layers as the encoder, and a transformer decoder layer as the decoder, is presented, the first neural network model to achieve real-time and streaming target sound extraction.
Abstract
We present the first neural network model to achieve real-time and streaming target sound extraction. To accomplish this, we propose Waveformer, an encoder-decoder architecture with a stack of dilated causal convolution layers as the encoder, and a transformer decoder layer as the decoder. This hybrid architecture uses dilated causal convolutions for processing large receptive fields in a computationally efficient manner, while also leveraging the generalization performance of transformer-based architectures. Our evaluations show as much as 2.2–3.3 dB improvement in SI-SNRi compared to the prior models for this task while having a 1.2–4x smaller model size and a 1.5–2x lower runtime. We provide code, dataset, and audio samples: https://waveformer.cs.washington.edu/.
References42 items
1
Tiny-Sepformer: A Tiny Time-Domain Transformer Network for Speech Separation
2
ClearBuds: wireless binaural earbuds for learning-based speech enhancement
3
Resource-Efficient Separation Transformer
4
Semi-supervised Time Domain Target Speaker Extraction with Attention
5