Scene Graph Generation by Iterative Message Passing (2017-01-10T00:00:00.000000Z)

TL;DR

This work explicitly model the objects and their relationships using scene graphs, a visually-grounded graphical structure of an image, and proposes a novel end-to-end model that generates such structured scene representation from an input image.

Abstract

Understanding a visual scene goes beyond recognizing individual objects in isolation. Relationships between objects also constitute rich semantic information about the scene. In this work, we explicitly model the objects and their relationships using scene graphs, a visually-grounded graphical structure of an image. We propose a novel end-to-end model that generates such structured scene representation from an input image. Our key insight is that the graph generation problem can be formulated as message passing between the primal node graph and its dual edge graph. Our joint inference model can take advantage of contextual cues to make better predictions on objects and their relationships. The experiments show that our model significantly outperforms previous methods on the Visual Genome dataset as well as support relation inference in NYU Depth V2 dataset.

Authors

Li Fei-Fei

37 papers

Yuke Zhu

15 papers

C. Choy

7 papers

TL;DR

Abstract

Authors

References44 items

Graph-Structured Representations for Visual Question Answering

On Support Relations and Semantic Scene Graphs

Visual Relationship Detection with Language Priors

SPICE: Semantic Propositional Image Caption Evaluation

Situation Recognition: Visual Semantic Role Labeling for Image Understanding

Semantic Object Parsing with Graph LSTM

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

Inside-Outside Net: Detecting Objects in Context with Skip Pooling and Recurrent Neural Networks

Deep Residual Learning for Image Recognition

HICO: A Benchmark for Recognizing Human-Object Interactions in Images

Structural-RNN: Deep Learning on Spatio-Temporal Graphs

Image retrieval using scene graphs

Learning semantic relationships for better action retrieval in images

Describing Common Human Visual Actions in Images

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Fast R-CNN

Scene Understanding by Reasoning Stability and Safety

Conditional Random Fields as Recurrent Neural Networks

Learning Spatial Knowledge for Text to 3D Scene Generation

Very Deep Convolutional Networks for Large-Scale Image Recognition

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

The Role of Context for Object Detection and Semantic Segmentation in the Wild

Learning the Visual Interpretation of Sentences

Holistic Scene Understanding for 3D Object Detection with RGBD Cameras

Scene Parsing by Integrating Function, Geometry and Appearance Models

3D-Based Reasoning with Blocks, Support, and Stability

ImageNet classification with deep convolutional neural networks

Indoor Segmentation and Support Inference from RGBD Images

Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials

Characterizing structural relationships in scenes using graph kernels

Recognition using visual phrases

Learning to share visual appearance for multiclass object detection

Graph Cut Based Inference with Co-occurrence Statistics

Discriminative models for static human-object interactions

Modeling mutual context of object and human pose in human-object interaction activities

Discriminative Models for Multi-Class Object Layout

Beyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers

Object categorization using co-occurrence, location and appearance

Objects in Context

The role of context in object recognition

Contextual Priming for Object Detection

Region-Based Convolutional Networks for Accurate Object Detection and Segmentation

Statistics of 3D Object Locations in Images

The scene graph classiﬁcation (SGC LS ) task is to predict the predicate as well as the object categories of the subject and the object in every pairwise relationship given a set of localized objects

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names