SDFDiff: Differentiable Rendering of Signed Distance Fields for 3D Shape Optimization (2019-12-15T00:00:00.000000Z)

TL;DR

It is demonstrated that the proposed SDFDiff, a novel approach for image-based shape optimization using differentiable rendering of 3D shapes represented by signed distance functions, can be integrated with deep learning models, which opens up options for learning approaches on 3D objects without 3D supervision.

Abstract

We propose SDFDiff, a novel approach for image-based shape optimization using differentiable rendering of 3D shapes represented by signed distance functions (SDFs). Compared to other representations, SDFs have the advantage that they can represent shapes with arbitrary topology, and that they guarantee watertight surfaces. We apply our approach to the problem of multi-view 3D reconstruction, where we achieve high reconstruction quality and can capture complex topology of 3D objects. In addition, we employ a multi-resolution strategy to obtain a robust optimization algorithm. We further demonstrate that our SDF-based differentiable renderer can be integrated with deep learning models, which opens up options for learning approaches on 3D objects without 3D supervision. In particular, we apply our method to single-view 3D reconstruction and achieve state-of-the-art results.

Authors

Matthias Zwicker

4 papers

Zhizhong Han

3 papers

Yue Jiang

1 papers

TL;DR

Abstract

Authors

References57 items

Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision

DIST: Rendering Deep Implicit Signed Distance Function With Differentiable Sphere Tracing

Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

Differentiable surface splatting for point-based geometry processing

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning

Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer

Deep Level Sets: Implicit Surface Representations for 3D Shape Inference

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

Occupancy Networks: Learning 3D Reconstruction in Function Space

Learning Implicit Fields for Generative Shape Modeling

Differentiable Monte Carlo ray tracing through edge sampling

Visual Object Networks: Image Generation with Disentangled 3D Representations

CAPNet: Continuous Approximation Projection For 3D Point Cloud Reconstruction Using 2D Supervision

Escaping Plato’s Cave: 3D Shape From Adversarial Rendering

Escaping Plato's Cave using Adversarial Training: 3D Shape From Unstructured 2D Image Collections

View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictions

Y^2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences

Unsupervised Learning of Shape and Pose with Differentiable Point Clouds

Inverse Transport Networks

GAL: Geometric Adversarial Loss for Single-View 3D-Object Reconstruction

End-to-End 6-DoF Object Pose Estimation Through Differentiable Rasterization

Adversarial Geometry and Lighting using a Differentiable Renderer

Learning to Generate and Reconstruct 3D Meshes with only 2D Supervision

RenderNet: A deep convolutional network for differentiable rendering from 3D shapes

First Experiments with Neural Translation of Informal to Formal Mathematics

LIME: Live Intrinsic Material Estimation

Neural 3D Mesh Renderer

MarrNet: 3D Shape Reconstruction via 2.5D Sketches

DeformNet: Free-Form Deformation Network for 3D Shape Reconstruction from a Single Image

Synthesizing Robust Adversarial Examples

Representation Learning and Adversarial Generation of 3D Point Clouds

Weakly Supervised 3D Reconstruction with Adversarial Constraint

Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency

Learning Category-Specific Deformable 3D Models for Object Reconstruction

3D Shape Induction from 2D Views of Multiple Objects

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision

Learning Detailed Face Reconstruction from a Single Image

Shading-Aware Multi-view Stereo

Unsupervised Learning of 3D Structure from Images

3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction

ShapeNet: An Information-Rich 3D Model Repository

Adam: A Method for Stochastic Optimization

OpenDR: An Approximate Differentiable Renderer

A New Variational Framework for Multiview Surface Reconstruction

Real-time 3D reconstruction at scale using voxel hashing

Gradient Flows for Optimizing Triangular Mesh-based Surfaces: Applications to 3D Reconstruction Problems Dealing with Visibility

KinectFusion: Real-time dense surface mapping and tracking

A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms

Level set methods and dynamic implicit surfaces

A signal-processing framework for inverse rendering

Surface splatting

Regular Article: A PDE-Based Fast Local Level Set Method

A volumetric method for building complex models from range images

Pa-parazzi: Surface editing by way of multi-view image processing

Sphere tracing: a geometric method for the antialiased ray tracing of implicit surfaces

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names