UAVM: Towards Unifying Audio and Visual Models
TL
TL;DR
The UAVM achieves a new state-of-the-art audio-visual event classification accuracy of 65.8% on VGGSound and finds a few intriguing properties of UavM that the modality-independent counterparts do not have.
Abstract
Conventional audio-visual models have independent audio and video branches. In this work, we <italic>unify</italic> the audio and visual branches by designing a <underline>U</underline>nified <underline>A</underline>udio-<underline>V</underline>isual <underline>M</underline>odel (UAVM). The UAVM achieves a new state-of-the-art audio-visual event classification accuracy of 65.8% on VGGSound. More interestingly, we also find a few intriguing properties of UAVM that the modality-independent counterparts do not have. Code at <uri>github.com/yuangongnd/uavm</uri>.
References31 items
1
On Robustness to Missing Video for Audiovisual Speech Recognition
2
One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Code
3
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language
4
A ConvNet for the 2020s
5