Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision (2019-12-16T00:00:00.000000Z)

TL;DR

This work proposes a differentiable rendering formulation for implicit shape and texture representations, showing that depth gradients can be derived analytically using the concept of implicit differentiation, and finds that this method can be used for multi-view 3D reconstruction, directly resulting in watertight meshes.

Abstract

Learning-based 3D reconstruction methods have shown impressive results. However, most methods require 3D supervision which is often hard to obtain for real-world datasets. Recently, several works have proposed differentiable rendering techniques to train reconstruction models from RGB images. Unfortunately, these approaches are currently restricted to voxel- and mesh-based representations, suffering from discretization or low resolution. In this work, we propose a differentiable rendering formulation for implicit shape and texture representations. Implicit representations have recently gained popularity as they represent shape and texture continuously. Our key insight is that depth gradients can be derived analytically using the concept of implicit differentiation. This allows us to learn implicit shape and texture representations directly from RGB images. We experimentally show that our single-view reconstructions rival those learned with full 3D supervision. Moreover, we find that our method can be used for multi-view 3D reconstruction, directly resulting in watertight meshes.

Authors

Andreas Geiger

18 papers

L. Mescheder

4 papers

Michael Oechsle

2 papers

TL;DR

Abstract

Authors

References92 items

BlockGAN: Learning 3D Object-aware Scene Representations from Unlabelled Images

Towards Unsupervised Learning of Generative Models for 3D Controllable Image Synthesis

Learning to Infer Implicit Surfaces without 3D Supervision

Implicit Surface Representations As Layers in Neural Networks

Occupancy Flow: 4D Reconstruction by Learning Particle Dynamics

Deep Mesh Reconstruction From Single RGB Images via Topology Modification Networks

Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

PointFlow: 3D Point Cloud Generation With Continuous Normalizing Flows

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Learning Non-Volumetric Depth Fusion Using Successive Reprojections

Controlling Neural Level Sets

DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction

Texture Fields: Learning Texture Representations in Function Space

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

Superquadrics Revisited: Learning 3D Shape Parsing Beyond Cuboids

KPConv: Flexible and Deformable Convolution for Point Clouds

Learning Shape Templates With Structured Implicit Functions

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer

Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference

Pix2Vox: Context-Aware 3D Reconstruction From Single and Multi-View Images

Learning Single-Image 3D Reconstruction by Generative Modelling of Shape, Pose and Shading

Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

Occupancy Networks: Learning 3D Reconstruction in Function Space

Learning Implicit Fields for Generative Shape Modeling

DeepVoxels: Learning Persistent 3D Feature Embeddings

Efficient Dense Point Cloud Object Reconstruction Using Deformation Vector Fields

GAL: Geometric Adversarial Loss for Single-View 3D-Object Reconstruction

Deep Volumetric Video From Very Sparse Multi-view Performance Capture

Shape Reconstruction Using Volume Sweeping and Learned Photoconsistency

Single-image SVBRDF capture with a rendering-aware deep network

Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

Unsupervised Training for 3D Morphable Model Regression

RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials

Learning 3D Shape Completion from Laser Scan Data with Weak Supervision

3D-RCNN: Instance-Level 3D Object Reconstruction via Render-and-Compare

Deep Marching Cubes: Learning Explicit Surface Representations

MVSNet: Depth Inference for Unstructured Multi-view Stereo

Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images

DeepMVS: Learning Multi-view Stereopsis

Learning Category-Specific Mesh Reconstruction from Image Collections

Self-Supervised Multi-level Face Model Learning for Monocular Reconstruction at Over 250 Hz

Neural 3D Mesh Renderer

Material Editing Using a Physically Based Rendering Network

Learning Representations and Generative Models for 3D Point Clouds

Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency

OctNetFusion: Learning Depth Fusion from Data

MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction

Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs

3D Shape Induction from 2D Views of Multiple Objects

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision

Learning Detailed Face Reconstruction from a Single Image

OctNet: Learning Deep 3D Representations at High Resolutions

Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling

Pixelwise View Selection for Unstructured Multi-View Stereo

Real-time height map fusion using differentiable rendering

Generative and Discriminative Voxel Modeling with Convolutional Neural Networks

Unsupervised Learning of 3D Structure from Images

Structure-from-Motion Revisited

Large-Scale Data for Multiple-View Stereopsis

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

Deep Residual Learning for Image Recognition

ShapeNet: An Information-Rich 3D Model Repository

Sparse 3D convolutional neural networks

Adam: A Method for Stochastic Optimization

OpenDR: An Approximate Differentiable Renderer

Screened poisson surface reconstruction

Using Multiple Hypotheses to Improve Depth-Maps for Multi-View Stereo

Accurate, Dense, and Robust Multi-View Stereopsis

A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms

A probabilistic framework for space carving

Photorealistic Scene Reconstruction by Voxel Coloring

A volumetric method for building complex models from range images