Floors are Flat: Leveraging Semantics for Real-Time Surface Normal Prediction (2019-06-16T00:00:00.000000Z)

TL;DR

4 insights are proposed that help to significantly improve the performance of deep learning models that predict surface normals and semantic labels from a single RGB image and are demonstrated consistently improved state of the art results on several datasets.

Abstract

We propose 4 insights that help to significantly improve the performance of deep learning models that predict surface normals and semantic labels from a single RGB image. These insights are: (1) denoise the "ground truth" surface normals in the training set to ensure consistency with the semantic labels; (2) concurrently train on a mix of real and synthetic data, instead of pretraining on synthetic and finetuning on real; (3) jointly predict normals and semantics using a shared model, but only backpropagate errors on pixels that have valid training labels; (4) slim down the model and use grayscale instead of color inputs. Despite the simplicity of these steps, we demonstrate consistently improved state of the art results on several datasets, using a model that runs at 12 fps on a standard mobile phone.

Authors

K. Murphy

12 papers

Karthik Raveendran

3 papers

Irfan Essa

5 papers

Floors are Flat: Leveraging Semantics for Real-Time Surface Normal Prediction

TL;DR

Abstract

Authors

References30 items

PlaneRCNN: 3D Plane Detection and Reconstruction From a Single Image

Real-Time Joint Semantic Segmentation and Depth Estimation Using Asymmetric Annotations

GeoNet: Geometric Neural Network for Joint Depth and Surface Normal Estimation

PAD-Net: Multi-tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing

MobileNetV2: Inverted Residuals and Linear Bottlenecks

Cross-Domain Self-Supervised Multi-task Feature Learning Using Synthetic Imagery

Unsupervised Learning of Geometry with Edge-aware Depth-Normal Consistency

SceneNet RGB-D: Can 5M Synthetic Images Beat Generic ImageNet Pre-training on Indoor Segmentation?

Joint prediction of depths, normals and surface curvature from RGB images using CNNs

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks

SURGE: Surface Regularized Geometry Estimation from a Single Image

UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory

Marr Revisited: 2D-3D Alignment via Surface Normal Prediction

Deep Residual Learning for Image Recognition

U-Net: Convolutional Networks for Biomedical Image Segmentation

Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification

Designing deep networks for surface normal estimation

Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture

Discriminatively Trained Dense Surface Normal Estimation

Very Deep Convolutional Networks for Large-Scale Image Recognition

Data-Driven 3D Primitives for Single Image Understanding

Box in the Box: Joint 3D Layout and Object Reasoning from Single Images

Adaptive neighborhood selection for real-time surface normal estimation from organized point cloud data using integral images

Indoor Segmentation and Support Inference from RGBD Images

3D is here: Point Cloud Library (PCL)

Recovering Surface Layout from an Image

Lecture 6.5-rmsprop: Di-vide the gradient by a running average of its recent magnitude

google,我,萨娜

Marr revisited : 2 d - 3 d alignment via surface normal prediction Scannet : Richly - annotated 3 d reconstructions of indoor scenes

Field of Study

Journal Information

Name

Page