Frustum PointNets for 3D Object Detection from RGB-D Data (2017-11-22T00:00:00.000000Z)

TL;DR

This work directly operates on raw point clouds by popping up RGBD scans and leverages both mature 2D object detectors and advanced 3D deep learning for object localization, achieving efficiency as well as high recall for even small objects.

Abstract

In this work, we study 3D object detection from RGBD data in both indoor and outdoor scenes. While previous methods focus on images or 3D voxels, often obscuring natural 3D patterns and invariances of 3D data, we directly operate on raw point clouds by popping up RGB-D scans. However, a key challenge of this approach is how to efficiently localize objects in point clouds of large-scale scenes (region proposal). Instead of solely relying on 3D proposals, our method leverages both mature 2D object detectors and advanced 3D deep learning for object localization, achieving efficiency as well as high recall for even small objects. Benefited from learning directly in raw point clouds, our method is also able to precisely estimate 3D bounding boxes even under strong occlusion or with very sparse points. Evaluated on KITTI and SUN RGB-D 3D detection benchmarks, our method outperforms the state of the art by remarkable margins while having real-time capability.

Authors

C. Qi

15 papers

L. Guibas

28 papers

Hao Su

5 papers

TL;DR

Abstract

Authors

References44 items

2D-Driven 3D Object Detection in RGB-D Images

Vehicle detection and localization on bird's eye view elevation images using convolutional neural network

Focal Loss for Dense Object Detection

Amodal Detection of 3D Objects: Inferring 3D Bounding Boxes from 2D Ones in RGB-Depth Images

Revisiting Unreasonable Effectiveness of Data in Deep Learning Era

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Accurate Single Stage Detector Using Recurrent Rolling Convolution

Mask R-CNN

DSSD : Deconvolutional Single Shot Detector

Feature Pyramid Networks for Object Detection

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

3D Bounding Box Estimation Using Deep Learning and Geometry

3D fully convolutional network for vehicle detection in point cloud

Multi-view 3D Object Detection Network for Autonomous Driving

OctNet: Learning Deep 3D Representations at High Resolutions

Vote3Deep: Fast object detection in 3D point clouds using efficient convolutional neural networks

Three-Dimensional Object Detection and Layout Prediction Using Clouds of Oriented Gradients

Monocular 3D Object Detection for Autonomous Driving

Vehicle Detection from 3D Lidar Using Fully Convolutional Network

FPNN: Field Probing Neural Networks for 3D Data

Volumetric and Multi-view CNNs for Object Classification on 3D Data

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

SSD: Single Shot MultiBox Detector

3D Object Proposals for Accurate Object Class Detection

Deep Sliding Shapes for Amodal 3D Object Detection in RGB-D Images

VoxNet: A 3D Convolutional Neural Network for real-time object recognition

Voting for Voting in Online Point Cloud Object Detection

SUN RGB-D: A RGB-D scene understanding benchmark suite

Spatial Transformer Networks

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Data-driven 3D Voxel Patterns for object category recognition

Multi-view Convolutional Neural Networks for 3D Shape Recognition

Fast R-CNN

Sliding Shapes for 3D Object Detection in Depth Images

Very Deep Convolutional Networks for Large-Scale Image Recognition

3D ShapeNets: A deep representation for volumetric shapes

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

Vision meets robotics: The KITTI dataset

Are we ready for autonomous driving? The KITTI vision benchmark suite

Et al

B. Details on Frustum PointNets (Sec 4.2, 4.3) B.1

we provide more details on our Frustum Point-Net architectures and training parameters while explains more about our 2D detector

This supplementary document provides additional technical details, extra analysis experiments, more quantitative results and qualitative test results to the main paper

3d object detection benchmark leader board

Field of Study

Journal Information

Name

Page