Neural 3D Mesh Renderer (2017-11-20T00:00:00.000000Z)

TL;DR

This work proposes an approximate gradient for rasterization that enables the integration of rendering into neural networks and performs gradient-based 3D mesh editing operations, such as 2D-to-3D style transfer and 3D DeepDream, with 2D supervision for the first time.

Abstract

For modeling the 3D world behind 2D images, which 3D representation is most appropriate? A polygon mesh is a promising candidate for its compactness and geometric properties. However, it is not straightforward to model a polygon mesh from 2D images using neural networks because the conversion from a mesh to an image, or rendering, involves a discrete operation called rasterization, which prevents back-propagation. Therefore, in this work, we propose an approximate gradient for rasterization that enables the integration of rendering into neural networks. Using this renderer, we perform single-image 3D mesh reconstruction with silhouette image supervision and our system outperforms the existing voxel-based approach. Additionally, we perform gradient-based 3D mesh editing operations, such as 2D-to-3D style transfer and 3D DeepDream, with 2D supervision for the first time. These applications demonstrate the potential of the integration of a mesh renderer into neural networks and the effectiveness of our proposed renderer.

Authors

Y. Ushiku

7 papers

T. Harada

4 papers

Hiroharu Kato

1 papers

TL;DR

Abstract

Authors

References38 items

Fundamentals of computer graphics

3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks

PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Deformation-driven shape correspondence via shape recognition

Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency

Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models

Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs

3D Shape Segmentation with Projective Convolutional Networks

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

SyncSpecCNN: Synchronized Spectral CNN for 3D Shape Segmentation

Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision

OctNet: Learning Deep 3D Representations at High Resolutions

A Learned Representation For Artistic Style

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Unsupervised Learning of 3D Structure from Images

Image Style Transfer Using Convolutional Neural Networks

Volumetric and Multi-view CNNs for Object Classification on 3D Data

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Perceptual Losses for Real-Time Style Transfer and Super-Resolution

ShapeNet: An Information-Rich 3D Model Repository

Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis

Multi-view 3D Models from Single Images with a Convolutional Network

Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks

VoxNet: A 3D Convolutional Neural Network for real-time object recognition

Multi-view Convolutional Neural Networks for 3D Shape Recognition

Adam: A Method for Stochastic Optimization

Going deeper with convolutions

OpenDR: An Approximate Differentiable Renderer

Very Deep Convolutional Networks for Large-Scale Image Recognition

3D ShapeNets: A deep representation for volumetric shapes

Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation

ImageNet classification with deep convolutional neural networks

GENERATIVE ADVERSARIAL NETS

Inceptionism: Going Deeper into Neural Networks

3-D Depth Reconstruction from a Single Still Image

Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

Field of Study

Journal Information

Name

Page