MVTN: Multi-View Transformation Network for 3D Shape Recognition (2020-11-26T00:00:00.000000Z)

TL;DR

The Multi-View Transformation Network (MVTN) is introduced that regresses optimal view-points for 3D shape recognition, building upon advances in differentiable rendering and can provide network robustness against rotation and occlusion in the 3D domain.

Abstract

Multi-view projection methods have demonstrated their ability to reach state-of-the-art performance on 3D shape recognition. Those methods learn different ways to aggregate information from multiple views. However, the camera view-points for those views tend to be heuristically set and fixed for all shapes. To circumvent the lack of dynamism of current multi-view methods, we propose to learn those view-points. In particular, we introduce the Multi-View Transformation Network (MVTN) that regresses optimal view-points for 3D shape recognition, building upon advances in differentiable rendering. As a result, MVTN can be trained end-to-end along with any multi-view network for 3D shape classification. We integrate MVTN in a novel adaptive multi-view pipeline that can render either 3D meshes or point clouds. MVTN exhibits clear performance gains in the tasks of 3D shape classification and 3D shape retrieval with-out the need for extra training supervision. In these tasks, MVTN achieves state-of-the-art performance on ModelNet40, ShapeNet Core55, and the most recent and realistic ScanObjectNN dataset (up to 6% improvement). Interestingly, we also show that MVTN can provide network robustness against rotation and occlusion in the 3D domain. The code is available at https://github.com/ajhamdi/MVTN.

Authors

Bernard Ghanem

28 papers

Silvio Giancola

10 papers

Abdullah Hamdi

4 papers

TL;DR

Abstract

Authors

References90 items

Surface Simplification Using Quadric Error Metrics

Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline

Point Transformer

Virtual Multi-view Fusion for 3D Semantic Segmentation

View-GCN: View-Based Graph Convolutional Network for 3D Shape Analysis

SynSin: End-to-End View Synthesis From a Single Image

AdvPC: Transferable Adversarial Perturbations on 3D Point Clouds

SGAS: Sequential Greedy Architecture Search

Accelerating 3D deep learning with PyTorch3D

Learning Relationships for Multi-View 3D Object Recognition

Multi-View PointNet for 3D Scene Understanding

DensePoint: Learning Densely Contextual Representation for Efficient Point Cloud Processing

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data

Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

MLVCNN: Multi-Loop-View Convolutional Neural Network for 3D Shape Retrieval

Point-Voxel CNN for Efficient 3D Deep Learning

Deep Multi-View Learning Using Neuron-Wise Correlation-Maximizing Regularizers

4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

KPConv: Flexible and Deformable Convolution for Point Clouds

Relation-Shape Convolutional Neural Network for Point Cloud Analysis

Towards Analyzing Semantic Robustness of Deep Neural Networks

Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning

Point Cloud Oversegmentation With Graph-Structured Deep Metric Learning

Equivariant Multi-View Networks

SADA: Semantic Adversarial Diagnostic Attacks for Autonomous Applications

Differentiable Monte Carlo ray tracing through edge sampling

InverseRenderNet: Learning Single Image Inverse Rendering

MeshNet: Mesh Neural Network for 3D Shape Representation

PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition

VERAM: View-Enhanced Recurrent Attention Model for 3D Shape Classification

Siamese CNN-BiLSTM Architecture for 3D Shape Representation Learning

3DmFV: Three-Dimensional Point Cloud Classification in Real-Time Using Convolutional Neural Networks

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Tangent Convolutions for Dense Prediction in 3D

GVCNN: Group-View Convolutional Neural Networks for 3D Shape Recognition

Multi-view Harmonized Bilinear Network for 3D Object Recognition

SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters

3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation

Triplet-Center Loss for Multi-view 3D Object Retrieval

Dynamic Graph CNN for Learning on Point Clouds

PointCNN: Convolution On X-Transformed Points

3D Semantic Segmentation with Submanifold Sparse Convolutional Networks

Large-Scale Point Cloud Semantic Segmentation with Superpoint Graphs

SGPN: Similarity Group Proposal Network for 3D Point Cloud Instance Segmentation

Neural 3D Mesh Renderer

Decoupled Weight Decay Regularization

Automatic differentiation in PyTorch

Neural Scene De-rendering

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models

Progressive Shape-Distribution-Encoder for Learning 3D Shape Representation

3D Shape Segmentation with Projective Convolutional Networks

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Generative and Discriminative Voxel Modeling with Convolutional Neural Networks

FusionNet: 3D Object Classification Using Multiple Data Representations

Unsupervised Learning of 3D Structure from Images

Volumetric and Multi-view CNNs for Object Classification on 3D Data

GIFT: A Real-Time and Scalable 3D Shape Search Engine

RotationNet: Joint Object Categorization and Pose Estimation Using Multiviews from Unsupervised Viewpoints

Group Equivariant Convolutional Networks

Deep Residual Learning for Image Recognition

ShapeNet: An Information-Rich 3D Model Repository

VoxNet: A 3D Convolutional Neural Network for real-time object recognition

Spatial Transformer Networks

Multi-view Convolutional Neural Networks for 3D Shape Recognition

Deep Convolutional Inverse Graphics Network

Adam: A Method for Stochastic Optimization

OpenDR: An Approximate Differentiable Renderer

ImageNet Large Scale Visual Recognition Challenge

3D ShapeNets: A deep representation for volumetric shapes

3D model retrieval using hybrid features and class information

Camera Constraint-Free View-Based 3-D Object Retrieval

Data-driven suggestions for creativity support in 3D modeling

3D Model Retrieval Using Probability Density-Based Shape Descriptors

Advances in Neural Information Processing Systems (NIPS)

Dimensionality Reduction of Multimodal Labeled Data by Local Fisher Discriminant Analysis