NeuralLift-360: Lifting an in-the-Wild 2D Photo to A 3D Object with 360° Views (2022-11-29T00:00:00.000000Z)

TL;DR

This work proposes a novel framework, dubbed NeuralLift-360, that utilizes a depth-aware neural radiance representation (NeRF) and learns to craft the scene guided by denoising diffusion models and can be guided with rough depth estimation in the wild.

Abstract

Virtual reality and augmented reality (XR) bring increasing demand for 3D content generation. However, creating high-quality 3D content requires tedious work from a human expert. In this work, we study the challenging task of lifting a single image to a 3D object and, for the first time, demonstrate the ability to generate a plausible 3D object with 360° views that corresponds well with the given reference image. By conditioning on the reference image, our model can fulfill the everlasting curiosity for synthesizing novel views of objects from images. Our technique sheds light on a promising direction of easing the workflows for 3D artists and XR designers. We propose a novel framework, dubbed NeuralLift-360, that utilizes a depth-aware neural radiance representation (NeRF) and learns to craft the scene guided by denoising diffusion models. By introducing a ranking loss, our NeuralLift-360 can be guided with rough depth estimation in the wild. We also adopt a CLIP-guided sampling strategy for the diffusion prior to provide coherent guidance. Extensive experiments demonstrate that our NeuralLift-360 significantly outperforms existing state-of-the-art baselines. Project page: https://vita-group.github.io/NeuralLift-360/

Authors

Zhiwen Fan

4 papers

Dejia Xu

3 papers

Peihao Wang

2 papers

TL;DR

Abstract

Authors

References82 items

ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Experts

Imagic: Text-Based Real Image Editing with Diffusion Models

Large-scale Text-to-Image Generation Models for Visual Artists’ Creative Works

DreamFusion: Text-to-3D using 2D Diffusion

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

Understanding Diffusion Models: A Unified Perspective

An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

Vision Transformer for NeRF-Based View Synthesis from a Single Input Image

Improved Direct Voxel Grid Optimization for Radiance Fields Reconstruction

Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentations

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Hierarchical Text-Conditional Image Generation with CLIP Latents

Unified Implicit Neural Stylization

SinNeRF: Training Neural Radiance Fields on Complex Scenes from a Single Image

Pseudo Numerical Methods for Diffusion Models on Manifolds

Instant neural graphics primitives with a multiresolution hash encoding

High-Resolution Image Synthesis with Latent Diffusion Models

Light Field Neural Rendering

More Control for Free! Image Synthesis with Semantic Diffusion Guidance

Plenoxels: Radiance Fields without Neural Networks

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields

Zero-Shot Text-Guided Object Generation with Dream Fields

3D Photo Stylization: Learning to Generate Stylized Novel Views from a Single Image

NeRFReN: Neural Radiance Fields with Reflections

Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields

Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction

Image

SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting

Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction

Neural Rays for Occlusion-aware Image-based Rendering

Depth-supervised NeRF: Fewer Views and Faster Training for Free

Variational Diffusion Models

Pivotal Tuning for Latent-based Editing of Real Images

Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging

Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis

MVSNeRF: Fast Generalizable Radiance Field Reconstruction from Multi-View Stereo

KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs

Vision Transformers for Dense Prediction

Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields

Learning Transferable Visual Models From Natural Language Supervision

IBRNet: Learning Multi-View Image-Based Rendering

Zero-Shot Text-to-Image Generation

Learning to Recover 3D Scene Shape from a Single Image

NeRD: Neural Reflectance Decomposition from Image Collections

NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis

pixelNeRF: Neural Radiance Fields from One or Few Images

Score-Based Generative Modeling through Stochastic Differential Equations

Denoising Diffusion Implicit Models

NormalGAN: Learning Detailed 3D Human from a Single RGB-D Image

Denoising Diffusion Probabilistic Models

3D Photography Using Context-Aware Layered Depth Inpainting

DiverseDepth: Affine-invariant Depth Prediction Using Diverse Data

Generative Modeling by Estimating Gradients of the Data Distribution

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer

PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

Web Stereo Video Supervision for Depth Prediction from Dynamic Scenes

Inpainting

Free-Form Image Inpainting With Gated Convolution

Monocular Relative Depth Perception with Web Stereo Data Supervision

Ordinal Depth Supervision for 3D Human Pose Estimation

MVSNet: Depth Inference for Unstructured Multi-view Stereo

Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images

MegaDepth: Learning Single-View Depth Prediction from Internet Photos

Learning Depth from Monocular Videos Using Direct Methods

MarrNet: 3D Shape Reconstruction via 2.5D Sketches

Scaling CNNs for High Resolution Volumetric Reconstruction from a Single Image

Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning

A Point Set Generation Network for 3D Object Reconstruction from a Single Image

Single-Image Depth Perception in the Wild

ShapeNet: An Information-Rich 3D Model Repository

Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Nonlinear disparity mapping for stereoscopic 3D

A geometric approach to shape from defocus

A signal-processing framework for inverse rendering

UniTune: Text-Driven Image Editing by Fine Tuning an Image Generation Model on a Single Image