Semantic Compositional Networks for Visual Captioning (2016-11-23T00:00:00.000000Z)

TL;DR

Experimental results show that the proposed method significantly outperforms prior state-of-the-art approaches, across multiple evaluation metrics.

Abstract

A Semantic Compositional Network (SCN) is developed for image captioning, in which semantic concepts (i.e., tags) are detected from the image, and the probability of each tag is used to compose the parameters in a long short-term memory (LSTM) network. The SCN extends each weight matrix of the LSTM to an ensemble of tag-dependent weight matrices. The degree to which each member of the ensemble is used to generate an image caption is tied to the image-dependent probability of the corresponding tag. In addition to captioning images, we also extend the SCN to generate captions for video clips. We qualitatively analyze semantic composition in SCNs, and quantitatively evaluate the algorithm on three benchmark datasets: COCO, Flickr30k, and Youtube2Text. Experimental results show that the proposed method significantly outperforms prior state-of-the-art approaches, across multiple evaluation metrics.

Authors

L. Deng

8 papers

Yunchen Pu

2 papers

Zhe Gan

19 papers

TL;DR

Abstract

Authors

References60 items

StyleNet: Generating Attractive Visual Captions with Styles

Adaptive Feature Abstraction for Translating Video to Text

Adaptive Feature Abstraction for Translating Video to Language

Early Embedding and Late Reranking for Video Captioning

Variational Autoencoder for Deep Learning of Images, Labels and Captions

On Multiplicative Integration with Recurrent Neural Networks

MSR-VTT: A Large Video Description Dataset for Bridging Video and Language

Attention Correctness in Neural Image Captioning

Review Networks for Caption Generation

Factored Temporal Sigmoid Belief Networks for Sequence Learning

Theano: A Python framework for fast computation of mathematical expressions

Learning Attributes Equals Multi-Source Domain Generalization

Rich Image Captioning in the Wild

Image Captioning with Semantic Attention

Deep Residual Learning for Image Recognition

Guiding the Long-Short Term Memory Model for Image Caption Generation

Delving Deeper into Convolutional Networks for Learning Video Representations

Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data

Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks

Guiding Long-Short Term Memory for Image Caption Generation

Aligning where to see and what to tell: image caption with region-based attention and scene factorization

Mind's eye: A recurrent visual representation for image caption generation

What Value Do Explicit High Level Concepts Have in Vision to Language Problems?

Jointly Modeling Embedding and Translation to Bridge Video and Language

Language Models for Image Captioning: The Quirks and What Works

Sequence to Sequence -- Video to Text

Microsoft COCO Captions: Data Collection and Evaluation Server

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Adam: A Method for Stochastic Optimization

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Translating Videos to Natural Language Using Deep Recurrent Neural Networks

Deep visual-semantic alignments for generating image descriptions

Learning Spatiotemporal Features with 3D Convolutional Networks

CIDEr: Consensus-based image description evaluation

From captions to visual concepts and back

Long-term recurrent convolutional networks for visual recognition and description

Show and tell: A neural image caption generator

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Sequence to Sequence Learning with Neural Networks

Recurrent Neural Network Regularization

Neural Machine Translation by Jointly Learning to Align and Translate

ImageNet Large Scale Visual Recognition Challenge

Large-Scale Video Classification with Convolutional Neural Networks

Multimodal Neural Language Models

A Multiplicative Model for Learning Distributed Text-Based Attribute Representations

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation

Microsoft COCO: Common Objects in Context

Grounded Compositional Semantics for Finding and Describing Images with Sentences

From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions

Distributed Representations of Words and Phrases and their Compositionality

Generating Text with Recurrent Neural Networks

Collecting Highly Parallel Data for Paraphrase Evaluation

Factored conditional restricted Boltzmann Machines for modeling motion style

Unsupervised Learning of Image Transformations

METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

ROUGE: A Package for Automatic Evaluation of Summaries

Bleu: a Method for Automatic Evaluation of Machine Translation

Long Short-Term Memory

Finding Structure in Time

Long-Term Recurrent Convolutional Networks for Visual Recognition and Description

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names