Synchformer: Efficient Synchronization From Sparse Cues
TL
TL;DR
This work proposes a novel audio-visual synchronization model, and training that decouples feature extraction from synchronization modelling through multi-modal segment-level contrastive pre-training, which achieves state-of-the-art performance in both dense and sparse settings.
Abstract
Our objective is audio-visual synchronization with a focus on ‘in-the-wild’ videos, such as those on YouTube, where synchronization cues can be sparse. Our contributions include a novel audio-visual synchronization model, and training that decouples feature extraction from synchronization modelling through multi-modal segment-level contrastive pre-training. This approach achieves state-of-the-art performance in both dense and sparse settings. We also extend synchronization model training to AudioSet a million-scale ‘in-the-wild’ dataset, investigate evidence attribution techniques for interpretability, and explore a new capability for synchronization models: audio-visual synchronizability. robots.ox.ac.uk/~vgg/research/synchformer
References32 items
1
MODEFORMER: Modality-Preserving Embedding For Audio-Video Synchronization Using Transformers
2
Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors
3
VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices
4
Audio-Visual Synchronisation in the wild
5