Approximate Differentiable Rendering with Algebraic Surfaces (2022-07-21T00:00:00.000000Z)

TL;DR

This work develops an approximate differentiable renderer for a compact, interpretable representation, which it is shown that the method is the only one comparable to classic techniques for pose estimation, and performs well in shape from silhouette.

Abstract

Differentiable renderers provide a direct mathematical link between an object's 3D representation and images of that object. In this work, we develop an approximate differentiable renderer for a compact, interpretable representation, which we call Fuzzy Metaballs. Our approximate renderer focuses on rendering shapes via depth maps and silhouettes. It sacrifices fidelity for utility, producing fast runtimes and high-quality gradient information that can be used to solve vision tasks. Compared to mesh-based differentiable renderers, our method has forward passes that are 5x faster and backwards passes that are 30x faster. The depth maps and silhouette images generated by our method are smooth and defined everywhere. In our evaluation of differentiable renderers for pose estimation, we show that our method is the only one comparable to classic techniques. In shape from silhouette, our method performs well using only gradient descent and a per-pixel loss, without any surrogate losses or regularization. These reconstructions work well even on natural video sequences with segmentation artifacts. Project page: https://leonidk.github.io/fuzzy-metaballs

Authors

M. Hebert

6 papers

L. Keselman

3 papers

TL;DR

Abstract

Authors

References90 items

VoGE: A Differentiable Volume Renderer using Gaussian Ellipsoids for Analysis-by-Synthesis

CryoDRGN2: Ab initio neural reconstruction of 3D protein structures from real cryo-EM images

Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction

DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras

Differentiable Surface Rendering via Non-Differentiable Sampling

Pulsar: Efficient Sphere-based Neural Rendering

Tangent Space Backpropagation for 3D Transformation Groups

Unbiased warped-area sampling for differentiable rendering

Modular primitives for high-performance differentiable rendering

Path-space differentiable rendering

GMMLoc: Structure Consistent Visual Localization With Gaussian Mixture Models

MRFMap: Online Probabilistic 3D Mapping using Forward Ray Sensor Models

Single-View View Synthesis With Multiplane Images

PointGMM: A Neural GMM Network for Point Clouds

RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

End-to-End Learning Local Multi-View Descriptors for 3D Point Clouds

Local Deep Implicit Functions for 3D Shape

Accelerating 3D deep learning with PyTorch3D

Mine Tunnel Exploration Using Multiple Quadrupedal Robots

Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer

Differentiable surface splatting for point-based geometry processing

Direct Fitting of Gaussian Mixture Models

Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning

Variable Resolution Occupancy Mapping Using Gaussian Mixture Models

Unsupervised Learning of Shape and Pose with Differentiable Point Clouds

On-Manifold GMM Registration

HGMR: Hierarchical Gaussian Mixtures for Adaptive 3D Registration

CubeSLAM: Monocular 3-D Object SLAM

Open3D: A Modern Library for 3D Data Processing

Neural 3D Mesh Renderer

Learning Efficient Point Cloud Generation for Dense 3D Object Reconstruction

Intel RealSense Stereoscopic Depth Cameras

MoFA: Model-Based Deep Convolutional Face Autoencoder for Unsupervised Monocular Reconstruction

Mask R-CNN

Pixelwise View Selection for Unstructured Multi-View Stereo

Accelerated Generative Models for 3D Point Cloud Data

Thingi10K: A Dataset of 10, 000 3D-Printing Models

The CMA Evolution Strategy: A Tutorial

CMA-ES for Hyperparameter Optimization of Deep Neural Networks

MLMD: Maximum Likelihood Mixture Decoupling for Fast and Accurate Point Cloud Registration

Adam: A Method for Stochastic Optimization

OpenDR: An Approximate Differentiable Renderer

Go-ICP: Solving 3D Registration Efficiently and Globally Optimally

Toward a Fully Autonomous UAV: Research Platform for Indoor and Outdoor Urban Search and Rescue

Fitted BVH for Fast Raytracing of Metaballs

Web-scale k-means clustering

Stochastic Transparency

Lie group integrators for animation and control of vehicles

How to Solve a Cubic Equation, Part 5: Back to Numerics

Particle splatting: Interactive rendering of particle-based simulation data

Poisson surface reconstruction

Shape-From-Silhouette Across Time Part I: Theory and Algorithms

Surface splatting

Efficient variants of the ICP algorithm

Surfels: surface elements as rendering primitives

Optimization methods for asteroid lightcurve inversion

Bundle Adjustment - A Modern Synthesis

Surface simplification using quadric error metrics

Optical Models for Direct Volume Rendering

Volumetric shape description of range data using “Blobby Model”

Ray-tracing soft objects

Marching cubes: A high resolution 3D surface construction algorithm

Data structure forsoft objects

A Generalization of Algebraic Surface Drawing

Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper

A Characterization of Ten Hidden-Surface Algorithms

Reduction of Speech Spectra by Analysis‐by‐Synthesis Techniques

for Unbiased Gradient Estimation for Differentiable Surface Splatting via Poisson Sampling

Supplementary: Approximate Differentiable Rendering with Algebraic Surfaces

Volumetric albedo framework for 3d imaging sonar

Image-Space Metaballs Using Deep Learning

Neural volumes: Learning dynamic renderable volumes from images

CMA-ES/pycma on Github

Detectron2

Beyond volumetric albedo

Mitsuba 2: A retargetable forward and inverse renderer