Cross-Modal Commentator: Automatic Machine Commenting Based on Cross-Modal Information
TL
TL;DR
This work proposes a new task: cross-model automatic commenting (CMAC), which aims to make comments by integrating multiple modal contents, and presents an effective co-attention model to capture the dependency between textual and visual information.
Abstract
Automatic commenting of online articles can provide additional opinions and facts to the reader, which improves user experience and engagement on social media platforms. Previous work focuses on automatic commenting based solely on textual content. However, in real-scenarios, online articles usually contain multiple modal contents. For instance, graphic news contains plenty of images in addition to text. Contents other than text are also vital because they are not only more attractive to the reader but also may provide critical information. To remedy this, we propose a new task: cross-model automatic commenting (CMAC), which aims to make comments by integrating multiple modal contents. We construct a large-scale dataset for this task and explore several representative methods. Going a step further, an effective co-attention model is presented to capture the dependency between textual and visual information. Evaluation results show that our proposed model can achieve better performance than competitive baselines.
References35 items
1
A Dual Reinforcement Learning Framework for Unsupervised Text Style Transfer
2
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context
3
Coarse-grain Fine-grain Coattention Network for Multi-evidence Question Answering
4
Learning Comment Generation by Leveraging User-generated Data
5