PlenOctrees for Real-time Rendering of Neural Radiance Fields (2021-03-25T00:00:00.000000Z)

TL;DR

It is shown that it is possible to train NeRFs to predict a spherical harmonic representation of radiance, removing the viewing direction as an input to the neural network, and PlenOctrees can be directly optimized to further minimize the reconstruction loss, which leads to equal or better quality compared to competing methods.

Abstract

We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation which supports view-dependent effects. Our method can render 800×800 images at more than 150 FPS, which is over 3000 times faster than conventional NeRFs. We do so without sacrificing quality while preserving the ability of NeRFs to perform free-viewpoint rendering of scenes with arbitrary geometry and view-dependent effects. Real-time performance is achieved by pre-tabulating the NeRF into a PlenOctree. In order to preserve view-dependent effects such as specularities, we factorize the appearance via closed-form spherical basis functions. Specifically, we show that it is possible to train NeRFs to predict a spherical harmonic representation of radiance, removing the viewing direction as an input to the neural network. Furthermore, we show that PlenOctrees can be directly optimized to further minimize the reconstruction loss, which leads to equal or better quality compared to competing methods. Moreover, this octree optimization step can be used to reduce the training time, as we no longer need to wait for the NeRF training to converge fully. Our real-time neural rendering approach may potentially enable new applications such as 6-DOF industrial and product visualizations, as well as next generation AR/VR systems. PlenOctrees are amenable to in-browser rendering as well; please visit the project page for the interactive online demo, as well as video and code: https://alexyu.net/plenoctrees.

Authors

Matthew Tancik

4 papers

Ruilong Li

3 papers

Angjoo Kanazawa

16 papers

TL;DR

Abstract

Authors

References65 items

Baking Neural Radiance Fields for Real-Time View Synthesis

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

FastNeRF: High-Fidelity Neural Rendering at 200FPS

NeX: Real-time View Synthesis with Neural Basis Expansion

Mixture of volumetric primitives for efficient neural rendering

IBRNet: Learning Multi-View Image-Based Rendering

Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes

NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis

AutoInt: Automatic Integration for Fast Neural Volume Rendering

Learned Initializations for Optimizing Coordinate-Based Neural Representations

pixelNeRF: Neural Radiance Fields from One or Few Images

DeRF: Decomposed Radiance Fields

GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

Monocular Real-Time Volumetric Performance Capture

Neural Sparse Voxel Fields

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance

Convolutional Occupancy Networks

Implicit Geometric Regularization for Learning Shapes

Volume Rendering

Differentiable Volumetric Rendering: Learning Implicit 3D Representations Without 3D Supervision

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

Inverse Rendering for Complex Indoor Scenes: Shape, Spatially-Varying Lighting and SVBRDF From a Single Image

Local light field fusion

Pushing the Boundaries of View Extrapolation With Multiplane Images

DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation

Occupancy Networks: Learning 3D Reconstruction in Function Space

Learning Implicit Fields for Generative Shape Modeling

DeepVoxels: Learning Persistent 3D Feature Embeddings

Soft 3D reconstruction for view synthesis

Learning a Multi-View Stereo Machine

Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency

Hierarchical Surface Prediction for 3D Object Reconstruction

Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs

OctNet: Learning Deep 3D Representations at High Resolutions

Adam: A Method for Stochastic Optimization

Let There Be Color! Large-Scale Texturing of 3D Reconstructions

All-frequency precomputed radiance transfer using spherical radial basis functions and clustered tensor approximation

Precomputed Radiance Transfer for Real-Time Rendering in Dynamic, Low-Frequency Lighting Environments

On the relationship between radiance and irradiance: determining the illumination from images of a convex Lambertian object.

Unstructured lumigraph rendering

Lambertian reflectance and linear subspaces

Surface light fields for 3D photography

Layered depth images

Color image quantization for frame buffer display

Stereo Matching with Transparency and Matting

Photorealistic Scene Reconstruction by Voxel Coloring

Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach

Light field rendering

Color image quantization for frame buffer display

Dispersion on a sphere

an efﬁcient JAX

JaxNeRF: an efficient JAX implementation of NeRF, 2020

Neural volumes: Learning dynamic renderable volumes from images

Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018. 2 NeRF-SH/SG Converted PlenOctree

Tanks and temples: Benchmarking large-scale scene reconstruction

O-cnn: Octree-based convolutional neural networks for 3d shape analysis

A Survey of Octree Volume Rendering Methods

The Plenoptic Function and the Elements of Early Vision

Representing Scenes as Neural Radiance Fields for View Synthesis

c ○ 2000 Kluwer Academic Publishers. Manufactured in The Netherlands. A Theory of Shape by Space Carving

Deformable neural radiance ﬁelds

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names