Synopses of Movie Narratives: a Video-Language Dataset for Story Understanding
Published in
IEEE International Conference on Multimedia and...(2022)External Links:
TL
TL;DR
These results underscore the value of SYMON for advancing research in vision-language story understanding and generation, and establish benchmarks on story video-text alignment and story video narration generation.
Abstract
Computational story understanding is a crucial but under-explored area of AI, hampered by a lack of suitable datasets. To address this, we collect, preprocess and publicly release SYMON (Synopses of Movie Narratives), a new video-language dataset containing 5,193 human-narrated, short movie summary videos sourced from YouTube. SYMON features naturalistic storytelling videos for human audiences made by human creators. Compared to existing movie story datasets, the videos in SYMON are shorter yet provide higher coverage of key story events, making it ideal for computational story understanding. We establish benchmarks on story video-text alignment and story video narration generation, demonstrating significant performance improvements when models are trained on SYMON. These results underscore the value of SYMON for advancing research in vision-language story understanding and generation.
References141 items
1
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites
2
Enhancing Video-Language Representations With Structural Spatio-Temporal Alignment
3
Computational Analysis of Storylines: Making Sense of Events
4
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation
5