Image Generation from Scene Graphs

Published in

(2018)

External Links:

Generate Graph DownloadPDF

TL;DR

This work proposes a method for generating images from scene graphs, enabling explicitly reasoning about objects and their relationships, and validates this approach on Visual Genome and COCO-Stuff.

Abstract

To truly understand the visual world our models should be able not only to recognize images but also generate them. To this end, there has been exciting recent progress on generating images from natural language descriptions. These methods give stunning results on limited domains such as descriptions of birds or flowers, but struggle to faithfully reproduce complex sentences with many objects and relationships. To overcome this limitation we propose a method for generating images from scene graphs, enabling explicitly reasoning about objects and their relationships. Our model uses graph convolution to process input graphs, computes a scene layout by predicting bounding boxes and segmentation masks for objects, and converts the layout to an image with a cascaded refinement network. The network is trained adversarially against a pair of discriminators to ensure realistic outputs. We validate our approach on Visual Genome and COCO-Stuff, where qualitative results, ablations, and user studies demonstrate our method's ability to generate complex images with multiple objects.

Authors

Justin Johnson

12 papers

Li Fei-Fei

37 papers

Agrim Gupta

4 papers

References64 items

Premise Selection for Theorem Proving by Deep Graph Embedding

Photographic Image Synthesis with Cascaded Refinement Networks

Neural Scene De-rendering

Pixels to Graphs by Associative Embedding

Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars

Image Generation from Scene Graphs

Published in

(2018)

External Links:

Generate Graph DownloadPDF

TL;DR

This work proposes a method for generating images from scene graphs, enabling explicitly reasoning about objects and their relationships, and validates this approach on Visual Genome and COCO-Stuff.

Abstract

Authors

Justin Johnson

12 papers

Li Fei-Fei

37 papers

Agrim Gupta

4 papers

References64 items

Premise Selection for Theorem Proving by Deep Graph Embedding

Photographic Image Synthesis with Cascaded Refinement Networks

Neural Scene De-rendering

Pixels to Graphs by Associative Embedding

Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars

Parallel Multiscale Autoregressive Density Estimation

Scene Graph Generation by Iterative Message Passing

COCO-Stuff: Thing and Stuff Classes in Context

StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks

Improved Image Captioning via Policy Gradient optimization of SPIDEr

Image-to-Image Translation with Conditional Adversarial Networks

Conditional Image Synthesis with Auxiliary Classifier GANs

Learning What and Where to Draw

On Support Relations and Semantic Scene Graphs

Semi-Supervised Classification with Graph Convolutional Networks

Visual Relationship Detection with Language Priors

SPICE: Semantic Propositional Image Caption Evaluation

node2vec: Scalable Feature Learning for Networks

Image Style Transfer Using Convolutional Neural Networks

Conditional Image Generation with PixelCNN Decoders

Improved Techniques for Training GANs

Generative Adversarial Text to Image Synthesis

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

A Theory of Generative ConvNet

Pixel Recurrent Neural Networks

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Structural-RNN: Deep Learning on Spatio-Temporal Graphs

Gated Graph Sequence Neural Networks

A note on the evaluation of generative models

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval

Deep Convolutional Networks on Graph-Structured Data

Picture: A probabilistic programming language for scene perception

Image retrieval using scene graphs

Spatial Transformer Networks

Text to 3D Scene Generation with Rich Lexical Grounding

LINE: Large-scale Information Network Embedding

Deep Convolutional Inverse Graphics Network

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

Conditional Generative Adversarial Nets

Learning Spatial Knowledge for Text to 3D Scene Generation

Going deeper with convolutions

ImageNet Large Scale Visual Recognition Challenge

Microsoft COCO: Common Objects in Context

DeepWalk: online learning of social representations

Spectral Networks and Locally Connected Networks on Graphs

Auto-Encoding Variational Bayes

Distributed Representations of Words and Phrases and their Compositionality

Parsing Natural Scenes and Natural Language with Recursive Neural Networks

A new model for learning in graph domains

A general framework for adaptive processing of data structures

Supervised neural networks for the classification of structures

GENERATIVE ADVERSARIAL NETS

Conditional generative adversarial nets for convolutional face generation

Rectifier Nonlinearities Improve Neural Network Acoustic Models

The Graph Neural Network Model

Learning task-dependent distributed representations by backpropagation through structure

Left to Right: ground truth, my model, authors' model, scene graph

What is in the hand of boy on the bench with the hat on?" (all missed: sandwich) 2. Many questions are very easy •

Memorization and contextual clues mean answerer can "cheat" • "What color is the building?" (all correct: brown) -not even in scene graph! 7

Configurable , photo - realistic image rendering and ground truth synthesis by sampling stochastic grammars representing indoor scenes

Field of Study

Computer Science

Journal Information

Name

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

Page

586-595

TL;DR

Abstract

Authors

References64 items

Premise Selection for Theorem Proving by Deep Graph Embedding

Photographic Image Synthesis with Cascaded Refinement Networks

Neural Scene De-rendering

Pixels to Graphs by Associative Embedding

Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars

TL;DR

Abstract

Authors

References64 items

Premise Selection for Theorem Proving by Deep Graph Embedding

Photographic Image Synthesis with Cascaded Refinement Networks

Neural Scene De-rendering

Pixels to Graphs by Associative Embedding

Configurable 3D Scene Synthesis and 2D Image Rendering with Per-pixel Ground Truth Using Stochastic Grammars

Parallel Multiscale Autoregressive Density Estimation

Scene Graph Generation by Iterative Message Passing

COCO-Stuff: Thing and Stuff Classes in Context

StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks

Improved Image Captioning via Policy Gradient optimization of SPIDEr

Image-to-Image Translation with Conditional Adversarial Networks

Conditional Image Synthesis with Auxiliary Classifier GANs

Learning What and Where to Draw

On Support Relations and Semantic Scene Graphs

Semi-Supervised Classification with Graph Convolutional Networks

Visual Relationship Detection with Language Priors

SPICE: Semantic Propositional Image Caption Evaluation

node2vec: Scalable Feature Learning for Networks

Image Style Transfer Using Convolutional Neural Networks

Conditional Image Generation with PixelCNN Decoders

Improved Techniques for Training GANs

Generative Adversarial Text to Image Synthesis

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

A Theory of Generative ConvNet

Pixel Recurrent Neural Networks

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

Structural-RNN: Deep Learning on Spatio-Temporal Graphs

Gated Graph Sequence Neural Networks

A note on the evaluation of generative models

Convolutional Networks on Graphs for Learning Molecular Fingerprints

Generating Semantically Precise Scene Graphs from Textual Descriptions for Improved Image Retrieval

Deep Convolutional Networks on Graph-Structured Data

Picture: A probabilistic programming language for scene perception

Image retrieval using scene graphs

Spatial Transformer Networks

Text to 3D Scene Generation with Rich Lexical Grounding

LINE: Large-scale Information Network Embedding

Deep Convolutional Inverse Graphics Network

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Adam: A Method for Stochastic Optimization

Conditional Generative Adversarial Nets

Learning Spatial Knowledge for Text to 3D Scene Generation

Going deeper with convolutions

ImageNet Large Scale Visual Recognition Challenge

Microsoft COCO: Common Objects in Context

DeepWalk: online learning of social representations

Spectral Networks and Locally Connected Networks on Graphs

Auto-Encoding Variational Bayes

Distributed Representations of Words and Phrases and their Compositionality

Parsing Natural Scenes and Natural Language with Recursive Neural Networks

Et al

A new model for learning in graph domains

A general framework for adaptive processing of data structures

Supervised neural networks for the classification of structures

GENERATIVE ADVERSARIAL NETS

Conditional generative adversarial nets for convolutional face generation

Rectifier Nonlinearities Improve Neural Network Acoustic Models

The Graph Neural Network Model

Learning task-dependent distributed representations by backpropagation through structure

Left to Right: ground truth, my model, authors' model, scene graph

What is in the hand of boy on the bench with the hat on?" (all missed: sandwich) 2. Many questions are very easy •

Memorization and contextual clues mean answerer can "cheat" • "What color is the building?" (all correct: brown) -not even in scene graph! 7

Configurable , photo - realistic image rendering and ground truth synthesis by sampling stochastic grammars representing indoor scenes

Field of Study

Journal Information

Name