3D-RETR: End-to-End Single and Multi-View 3D Reconstruction with Transformers (2021-10-17T00:00:00.000000Z)

TL;DR

3D-RETR is proposed, which is able to perform end-to-end 3D REconstruction with TRansformers, and is capable of 3D reconstruction from a single view or multiple views.

Abstract

3D reconstruction aims to reconstruct 3D objects from 2D views. Previous works for 3D reconstruction mainly focus on feature matching between views or using CNNs as backbones. Recently, Transformers have been shown effective in multiple applications of computer vision. However, whether or not Transformers can be used for 3D reconstruction is still unclear. In this paper, we fill this gap by proposing 3D-RETR, which is able to perform end-to-end 3D REconstruction with TRansformers. 3D-RETR first uses a pretrained Transformer to extract visual features from 2D input images. 3D-RETR then uses another Transformer Decoder to obtain the voxel features. A CNN Decoder then takes as input the voxel features to obtain the reconstructed objects. 3D-RETR is capable of 3D reconstruction from a single view or multiple views. Experimental results on two datasets show that 3DRETR reaches state-of-the-art performance on 3D reconstruction. Additional ablation study also demonstrates that 3D-DETR benefits from using Transformers.

Authors

Z. Shi

1 papers

Zhao Meng

1 papers

Yiran Xing

1 papers

TL;DR

Abstract

Authors

References56 items

Multi-view 3D Reconstruction with Transformers

Zero-Shot Text-to-Image Generation

Training data-efficient image transformers & distillation through attention

Taming Transformers for High-Resolution Image Synthesis

End-to-End Video Instance Segmentation with Transformers

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Rethinking Attention with Performers

Pix2Vox++: Multi-scale Context-aware 3D Object Reconstruction from Single and Multiple Images

Learning Texture Transformer Network for Image Super-Resolution

End-to-End Object Detection with Transformers

Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance

Reformer: The Efficient Transformer

Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision

Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images

Occupancy Networks: Learning 3D Reconstruction in Function Space

Learning Implicit Fields for Generative Shape Modeling

Scan2CAD: Learning CAD Model Alignment in RGB-D Scans

GAL: Geometric Adversarial Loss for Single-View 3D-Object Reconstruction

Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction

3D-LMNet: Latent Embedding Matching for Accurate and Diverse 3D Point Cloud Reconstruction from a Single Image

Matryoshka Networks: Predicting 3D Geometry via Nested Shape Layers

Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling

Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images

A Papier-Mache Approach to Learning 3D Surface Generation

Multi-view Consistency as Supervisory Signal for Learning Shape and Pose Prediction

3D Reconstruction of Incomplete Archaeological Objects Using a Generative Adversarial Network

Decoupled Weight Decay Regularization

Neural Discrete Representation Learning

Focal Loss for Dense Object Detection

Learning a Multi-View Stereo Machine

Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations

Synthesizing 3D Shapes via Modeling Multi-view Depth Maps and Silhouettes with Deep Generative Networks

Attention is All you Need

The Lovasz-Softmax Loss: A Tractable Surrogate for the Optimization of the Intersection-Over-Union Measure in Neural Networks

Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency

Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs

Deep Multitask Architecture for Integrated 2D and 3D Human Sensing

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Deep Residual Learning for Image Recognition

ShapeNet: An Information-Rich 3D Model Repository

Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views

Reconstruction of textureless regions using structure from motion and image-based interpolation

Fast and Accurate Image Matching with Cascade Hashing for 3D Reconstruction

KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera

StereoScan: Dense 3d reconstruction in real-time

Building Rome in a day

Distinctive Image Features from Scale-Invariant Keypoints

Ordinal Measures for Image Correspondence

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Distinctive Image Features from Scale-Invariant Keypoints

A Solution to the Simultaneous Localisation and Map Building (SLAM) Problem

Mescheder , Michael Oechsle , Michael Niemeyer , S . Nowozin , and Andreas Geiger . Occupancy networks : Learning 3 d reconstruction in function space

We conduct additional ablation studies to understand how each part of 3D-RETR contributes to the ﬁnal performance

Representing Scenes as Neural Radiance Fields for View Synthesis

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names