Visual Keyword Spotting with Attention
Published in
British Machine Vision Conference(2021)External Links:
TL
TL;DR
A novel architecture is proposed, the Transpotter, that uses full cross-modal attention between the visual and phonetic streams in Transformer-based models that ingest two streams, a visual encoding of the video and a phonetic encoding of a keyword, and output the temporal location of the keyword if present.
Abstract
In this paper, we consider the task of spotting spoken keywords in silent video sequences -- also known as visual keyword spotting. To this end, we investigate Transformer-based models that ingest two streams, a visual encoding of the video and a phonetic encoding of the keyword, and output the temporal location of the keyword if present. Our contributions are as follows: (1) We propose a novel architecture, the Transpotter, that uses full cross-modal attention between the visual and phonetic streams; (2) We show through extensive evaluations that our model outperforms the prior state-of-the-art visual keyword spotting and lip reading methods on the challenging LRW, LRS2, LRS3 datasets by a large margin; (3) We demonstrate the ability of our model to spot words under the extreme conditions of isolated mouthings in sign language videos.
References80 items
1
Sub-word Level Lip Reading With Visual Attention
2
Aligning Subtitles in Sign Language Videos
3
Keyword Transformer: A Self-Attention Model for Keyword Spotting
4
Read and Attend: Temporal Localisation in Sign Language Videos
5