Visual Semantic Role Labeling

Published in

arXiv.org(2015)

External Links:

Generate Graph

TL;DR

The problem of Visual Semantic Role Labeling is introduced: given an image the authors want to detect people doing actions and localize the objects of interaction and associate objects in the scene with different semantic roles for each action.

Abstract

In this paper we introduce the problem of Visual Semantic Role Labeling: given an image we want to detect people doing actions and localize the objects of interaction. Classical approaches to action recognition either study the task of action classification at the image or video clip level or at best produce a bounding box around the person doing the action. We believe such an output is inadequate and a complete understanding can only come when we are able to associate objects in the scene to the different semantic roles of the action. To enable progress towards this goal, we annotate a dataset of 16K people instances in 10K images with actions they are doing and associate objects in the scene with different semantic roles for each action. Finally, we provide a set of baseline algorithms for this task and analyze error modes providing directions for future work.

Authors

Jitendra Malik

19 papers

Saurabh Gupta

5 papers

References44 items

Fast R-CNN

Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

A dataset for Movie Description

Learning Deep Features for Scene Recognition using Places Database

Visual Semantic Role Labeling

Published in

arXiv.org(2015)

External Links:

Generate Graph

TL;DR

Abstract

Authors

Jitendra Malik

19 papers

Saurabh Gupta

5 papers

References44 items

Fast R-CNN

Multiscale Combinatorial Grouping for Image Segmentation and Object Proposal Generation

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

A dataset for Movie Description

Learning Deep Features for Scene Recognition using Places Database

Deep visual-semantic alignments for generating image descriptions

Finding action tubes

Learning a Recurrent Visual Representation for Image Caption Generation

From captions to visual concepts and back

Long-term recurrent convolutional networks for visual recognition and description

Show and tell: A neural image caption generator

Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models

Explain Images with Multimodal Recurrent Neural Networks

Very Deep Convolutional Networks for Large-Scale Image Recognition

Large-Scale Video Classification with Convolutional Neural Networks

Multimodal Neural Language Models

R-CNNs for Pose Estimation and Action Detection

2D Human Pose Estimation: New Benchmark and State of the Art Analysis

Microsoft COCO: Common Objects in Context

Towards Understanding Action Recognition

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

Diagnosing Error in Object Detectors

A database for fine grained activity detection of cooking activities

Weakly Supervised Learning of Interactions between Humans and Objects

Human action recognition by learning bases of action attributes and parts

Observing Human-Object Interactions: Using Spatial and Functional Compatibility for Recognition

Discriminative subvolume search for efficient action detection

Understanding videos, constructing plots learning a visually grounded storyline model from annotated videos

Actions in context

Action MACH a spatio-temporal Maximum Average Correlation Height filter for action recognition

Learning realistic human actions from movies

Actions as space-time shapes

Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling

Recognizing human actions: a local SVM approach

The Berkeley FrameNet Project

Classifying Actions and Measuring Action Similarity by Modeling the Mutual Context of Objects and Human Poses

Verbnet: a broad-coverage, comprehensive verb lexicon

Amazon mechanical turk. https://www.mturk.com/ mturk/welcome

Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Recognizing Human-object Interactions in Still Images by Modeling the Mutual Context of Objects and Human Poses

International Journal of Computer Vision manuscript No. (will be inserted by the editor) The PASCAL Visual Object Classes (VOC) Challenge

Action Classiﬁcation Results: VOC2012

Action Classification Results: VOC2012. http: //host.robots.ox.ac.uk:8080/leaderboard/ displaylb_noeq.php?challengeid= 11&compid=10

mis pairing : when the object is of the correct semantic class but not in the semantic role associated with this agent

when the object in the semantic role is not properly localized (IU between 0.1 and 0.5)

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/1505.04474

Venue Information

Name

arXiv.org

Type

URL

https://arxiv.org

Alternate Names

ArXiv