SceneGraphFusion: Incremental 3D Scene Graph Prediction from RGB-D Sequences (2021-03-27T00:00:00.000000Z)

TL;DR

This work proposes a method to incrementally build up semantic scene graphs from a 3D environment given a sequence of RGB-D frames by means of a graph neural network and proposes a novel attention mechanism well suited for partial and missing graph data present in such an incremental reconstruction scenario.

Abstract

Scene graphs are a compact and explicit representation successfully used in a variety of 2D scene understanding tasks. This work proposes a method to incrementally build up semantic scene graphs from a 3D environment given a sequence of RGB-D frames. To this end, we aggregate PointNet features from primitive scene components by means of a graph neural network. We also propose a novel attention mechanism well suited for partial and missing graph data present in such an incremental reconstruction scenario. Although our proposed method is designed to run on submaps of the scene, we show it also transfers to entire 3D scenes. Experiments show that our approach outperforms 3D scene graph prediction methods by a large margin and its accuracy is on par with other 3D semantic and panoptic segmentation methods while running at 35Hz.

Authors

N. Navab

26 papers

Federico Tombari

15 papers

Johanna Wald

4 papers

TL;DR

Abstract

Authors

References65 items

Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions

Semantic Image Manipulation Using Scene Graphs

3D-MPA: Multi-Proposal Aggregation for 3D Semantic Instance Segmentation

Latent-Graph Learning for Disease Prediction

OccuSeg: Occupancy-Aware 3D Instance Segmentation

Fusion-Aware Point Convolution for Online Semantic 3D Scene Segmentation

3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans

SuperGlue: Learning Feature Matching With Graph Neural Networks

Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping

3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera

RIO: 3D Object Instance Re-Localization in Changing Indoor Environments

3D Instance Segmentation via Multi-Task Metric Learning

Visual Relationships as Functions:Enabling Few-Shot Scene Graph Prediction

Scene Graph Prediction with Limited Labels

Deep Hough Voting for 3D Object Detection in Point Clouds

Scene Graph Generation With External Knowledge and Image Reconstruction

Fast Graph Representation Learning with PyTorch Geometric

PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things

Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery

3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans

TextureNet: Consistent Local Parametrizations for Learning From High-Resolution Signals on Meshes

Attentive Relational Networks for Mapping Images to Scene Graphs

Fusion++: Volumetric Object-Level SLAM

Fully-Convolutional Point Networks for Large-Scale Point Clouds

Incremental Object Database: Building 3D Models from Multiple Partial Observations

Graph R-CNN for Scene Graph Generation

Visual Graphs from Motion (VGfM): Scene understanding with object geometry reasoning

Real-Time Fully Incremental Scene Understanding on Mobile Platforms

Image Generation from Scene Graphs

Real-Time Progressive 3D Semantic Segmentation for Indoor Scenes

Fast and Accurate Semantic Mapping through Geometric-based Incremental Segmentation

On the Convergence of Adam and Beyond

Panoptic Segmentation

Decoupled Weight Decay Regularization

Graph Attention Networks

Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds

Scene Graph Generation from Objects, Phrases and Region Captions

Pixels to Graphs by Associative Embedding

Attention is All you Need

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Dynamic Graph Convolutional Networks

CNN-SLAM: Real-Time Dense Monocular SLAM with Learned Depth Prediction

Detecting Visual Relationships with Deep Relational Networks

Neural Message Passing for Quantum Chemistry

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

Scene Graph Generation by Iterative Message Passing

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Incremental scene understanding on dense SLAM

Graph-Structured Representations for Visual Question Answering

SemanticFusion: Dense 3D semantic mapping with convolutional neural networks

When 2.5D is not enough: Simultaneous reconstruction, segmentation and recognition on dense SLAM

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

ElasticFusion: Dense SLAM Without A Pose Graph

Real-time and scalable incremental segmentation on dense SLAM

SemanticPaint

Show, Attend and Tell: Neural Image Caption Generation with Visual Attention

Deep visual-semantic alignments for generating image descriptions

Real-time 3D reconstruction at scale using voxel hashing

Real-Time 3D Reconstruction in Dynamic Scenes Using Point-Based Fusion

SLAM++: Simultaneous Localisation and Mapping at the Level of Objects

Indoor Segmentation and Support Inference from RGBD Images

KinectFusion: Real-time dense surface mapping and tracking

A volumetric method for building complex models from range images

Table 12. The full 3D panoptic segmentation results on the ScanNet v2 open test set

Ours (skip missing) PQ 36

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names