Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018-04-12T00:00:00.000000Z)

TL;DR

A novel model is designed that simultaneously performs 3D reconstruction and pose estimation; this multi-task learning approach achieves state-of-the-art performance on both tasks.

Abstract

We study 3D shape modeling from a single image and make contributions to it in three aspects. First, we present Pix3D, a large-scale benchmark of diverse image-shape pairs with pixel-level 2D-3D alignment. Pix3D has wide applications in shape-related tasks including reconstruction, retrieval, viewpoint estimation, etc. Building such a large-scale dataset, however, is highly challenging; existing datasets either contain only synthetic data, or lack precise alignment between 2D images and 3D shapes, or only have a small number of images. Second, we calibrate the evaluation criteria for 3D shape reconstruction through behavioral studies, and use them to objectively and systematically benchmark cutting-edge reconstruction algorithms on Pix3D. Third, we design a novel model that simultaneously performs 3D reconstruction and pose estimation; our multi-task learning approach achieves state-of-the-art performance on both tasks.

Authors

Jiajun Wu

18 papers

J. Tenenbaum

25 papers

W. Freeman

16 papers

TL;DR

Abstract

Authors

References70 items

A Papier-Mache Approach to Learning 3D Surface Generation

MarrNet: 3D Shape Reconstruction via 2.5D Sketches

Large-Scale 3D Shape Reconstruction and Segmentation from ShapeNet Core55

SceneNet RGB-D: Can 5M Synthetic Images Beat Generic ImageNet Pre-training on Indoor Segmentation?

Synthesizing 3D Shapes via Modeling Multi-view Depth Maps and Silhouettes with Deep Generative Networks

Weakly Supervised 3D Reconstruction with Adversarial Constraint

Learning 3D Object Categories by Looking Around Them

Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency

Hierarchical Surface Prediction for 3D Object Reconstruction

Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes

T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-Less Objects

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision

Semantic Scene Completion from a Single Depth Image

OctNet: Learning Deep 3D Representations at High Resolutions

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

ObjectNet3D: A Large Scale Database for 3D Object Recognition

Unsupervised Learning of 3D Structure from Images

Single Image 3D Interpreter Network

RGBD Datasets: Past, Present and Future

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Learning a Predictable and Generative Vector Representation for Objects

A Large Dataset of Object Scans

Efficient Globally Optimal 2D-to-3D Deformable Shape Matching

Deep Residual Learning for Image Recognition

ShapeNet: An Information-Rich 3D Model Repository

Multi-view 3D Models from Single Images with a Convolutional Network

Benchmarking in Manipulation Research: Using the Yale-CMU-Berkeley Object and Model Set

SUN RGB-D: A RGB-D scene understanding benchmark suite

Aligning 3D models to RGB-D images of cluttered scenes

Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views

Adam: A Method for Stochastic Optimization

Learning to Generate Chairs, Tables and Cars with Convolutional Networks

Viewpoints and keypoints

Category-specific object reconstruction from a single image

BigBIRD: A large-scale 3D database of object instances

Seeing 3D Chairs: Exemplar Part-Based 2D-3D Alignment Using a Large Dataset of CAD Models

FAUST: Dataset and Evaluation for 3D Mesh Registration

Beyond PASCAL: A benchmark for 3D object detection in the wild

Parsing IKEA Objects: Fine Pose Estimation

3D Object Detection and Viewpoint Estimation with a Deformable 3D Cuboid Model

Indoor Segmentation and Support Inference from RGBD Images

Are we ready for autonomous driving? The KITTI vision benchmark suite

A category-level 3-D object dataset: Putting the Kinect to work

KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera

A large-scale hierarchical multi-view RGB-D object dataset

Context-based search for 3D models

SUN database: Large-scale scene recognition from abbey to zoo

Pose estimation for category specific multiview object localization

Building a database of 3D scenes from user annotations

Et al

3D generic object categorization, localization and pose estimation

Model Composition from Interchangeable Components

Robust 3D Shape Correspondence in the Spectral Domain

The Princeton Shape Benchmark

Analyzing appearance and contour based methods for object categorization

Efficient Implementation of Marching Cubes' Cases with Topological Guarantees

A distributed asynchronous relaxation algorithm for the assignment problem

Parametric Correspondence and Chamfer Matching: Two New Techniques for Image Matching

An Algorithm for Least-Squares Estimation of Nonlinear Parameters

A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES

Tanks and temples: Benchmarking large-scale scene reconstruction

SHREC’16 Track Large-Scale 3D Shape Retrieval from ShapeNet Core55

I and J

Product Quantization for Nearest Neighbor Search

Numerical Geometry of Non-Rigid Shapes

c ○ 2000 Kluwer Academic Publishers. Manufactured in The Netherlands. The Earth Mover’s Distance as a Metric for Image Retrieval

Noname manuscript No. (will be inserted by the editor) EPnP: An Accurate O(n) Solution to the PnP Problem

and as an in

Field of Study

Journal Information

Name

Page