DreamEditor: Text-Driven 3D Scene Editing with Neural Fields (2023-06-23T00:00:00.000000Z)

TL;DR

This paper proposes DreamEditor, a novel framework that enables users to perform controlled editing of neural fields using text prompts by representing scenes as mesh-based neural fields, and generates highly realistic textures and geometry.

Abstract

Neural fields have achieved impressive advancements in view synthesis and scene reconstruction. However, editing these neural fields remains challenging due to the implicit encoding of geometry and texture information. In this paper, we propose DreamEditor, a novel framework that enables users to perform controlled editing of neural fields using text prompts. By representing scenes as mesh-based neural fields, DreamEditor allows localized editing within specific regions. DreamEditor utilizes the text encoder of a pretrained text-to-Image diffusion model to automatically identify the regions to be edited based on the semantics of the text prompts. Subsequently, DreamEditor optimizes the editing region and aligns its geometry and texture with the text prompts through score distillation sampling [Poole et al. 2022]. Extensive experiments have demonstrated that DreamEditor can accurately edit neural fields of real-world scenes according to the given text prompts while ensuring consistency in irrelevant areas. DreamEditor generates highly realistic textures and geometry, significantly surpassing previous works in both quantitative and qualitative evaluations.

Authors

Guanbin Li

9 papers

Lingjie Liu

7 papers

Chen Wang

2 papers

TL;DR

Abstract

Authors

References58 items

TextDeformer: Geometry Manipulation using Text Guidance

Fantasia3D: Disentangling Geometry and Appearance for High-quality Text-to-3D Content Creation

SINE: Semantic-driven Image-based NeRF Editing with Prior-guided Editing Field

DreamBooth3D: Subject-Driven Text-to-3D Generation

Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions

Vox-E: Text-guided Voxel Editing of 3D Objects

BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis

TEXTure: Text-Guided Texturing of 3D Shapes

NeRF-Art: Text-Driven Neural Radiance Fields Stylization

Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation

Magic3D: High-Resolution Text-to-3D Content Creation

InstructPix2Pix: Learning to Follow Image Editing Instructions

Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures

DiffEdit: Diffusion-based semantic image editing with mask guidance

TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decomposition

Imagic: Text-Based Real Image Editing with Diffusion Models

DreamFusion: Text-to-3D using 2D Diffusion

DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation

Prompt-to-Prompt Image Editing with Cross Attention Control

An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion

NeuMesh: Learning Disentangled Neural Mesh-based Implicit Field for Geometry and Texture Editing

Deforming Radiance Fields with Cages

NeRF-In: Free-Form NeRF Inpainting with RGB-D Priors

Decomposing NeRF for Editing via Feature Field Distillation

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

NeRF-Editing: Geometry Editing of Neural Radiance Fields

Hierarchical Text-Conditional Image Generation with CLIP Latents

CLIP-Mesh: Generating textured meshes from text using pretrained image-text models

Instant neural graphics primitives with a multiresolution hash encoding

High-Resolution Image Synthesis with Latent Diffusion Models

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

Text2Mesh: Text-Driven Neural Stylization for Meshes

NeRF-SR: High Quality Neural Radiance Fields using Supersampling

Zero-Shot Text-Guided Object Generation with Dream Fields

Neural Point Light Fields

Blended Diffusion for Text-driven Editing of Natural Images

Common Objects in 3D: Large-Scale Learning and Evaluation of Real-life 3D Category Reconstruction

StyleGAN-NADA

Animatable Neural Radiance Fields from Monocular RGB Video

NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction

Editing Conditional Radiance Fields

Neural Articulated Radiance Field

NeuTex: Neural Texture Mapping for Volumetric Neural Rendering

Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans

Denoising Diffusion Implicit Models

Neural Sparse Voxel Fields

Denoising Diffusion Probabilistic Models

Implicit Geometric Regularization for Learning Shapes

BlendedMVS: A Large-Scale Dataset for Generalized Multi-View Stereo Networks

Matchable Image Retrieval by Learning from Surface Reconstruction

Attention is All you Need

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Large Scale Multi-view Stereopsis Evaluation

Embedded deformation for shape manipulation

Marching cubes: A high resolution 3D surface construction algorithm

ClimateNeRF: Physically-based Neural Rendering for Extreme Climate Synthesis

Representing Scenes as Neural Radiance Fields for View Synthesis

2022. StyleGAN-NADA: CLIP-guided domain adaptation of image generators

Field of Study

Journal Information

Name

Venue Information

Name

Type

URL

Alternate Names