From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions
TL
TL;DR
This work proposes to use the visual denotations of linguistic expressions to define novel denotational similarity metrics, which are shown to be at least as beneficial as distributional similarities for two tasks that require semantic inference.
Abstract
We propose to use the visual denotations of linguistic expressions (i.e. the set of images they describe) to define novel denotational similarity metrics, which we show to be at least as beneficial as distributional similarities for two tasks that require semantic inference. To compute these denotational similarities, we construct a denotation graph, i.e. a subsumption hierarchy over constituents and their denotations, based on a large corpus of 30K images and 150K descriptive captions.
References39 items
1
DKPro Similarity: An Open Source Framework for Text Similarity
2
Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics
3
Choosing Linguistics over Vision to Describe Images
4
Collective Generation of Natural Image Descriptions
5