InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Images (2022-07-22T00:00:00.000000Z)

TL;DR

A novel self-supervised view generation training paradigm is proposed, where the model is sample and rendering virtual camera trajectories, including cyclic ones, allowing the model to learn stable view generation from a collection of single views.

Abstract

We present a method for learning to generate unbounded flythrough videos of natural scenes starting from a single view, where this capability is learned from a collection of single photographs, without requiring camera poses or even multiple views of each scene. To achieve this, we propose a novel self-supervised view generation training paradigm, where we sample and rendering virtual camera trajectories, including cyclic ones, allowing our model to learn stable view generation from a collection of single views. At test time, despite never seeing a video during training, our approach can take a single image and generate long camera trajectories comprised of hundreds of new views with realistic and diverse content. We compare our approach with recent state-of-the-art supervised view generation methods that require posed multi-view videos and demonstrate superior performance and synthesis quality.

Authors

Noah Snavely

16 papers

Angjoo Kanazawa

16 papers

Zhengqi Li

4 papers

TL;DR

Abstract

Authors

References94 items

Generating Videos with Dynamics-aware Implicit Generative Adversarial Networks

Generative Adversarial Networks

Multimodal Conditional Image Synthesis with Product-of-Experts GANs

Palette: Image-to-Image Diffusion Models

StyleGAN of All Trades: Image Manipulation with Only Pretrained StyleGAN

StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis

CodeNeRF: Disentangled Neural Radiance Fields for Object Categories

SLIDE: Single Image 3D Photography with Soft Layering and Depth-aware Inpainting

PixelSynth: Generating a 3D-Consistent Experience from a Single Image

SLAMP: Stochastic Latent Appearance and Motion Prediction

StyleVideoGAN: A Temporal Generative Model using a Pretrained StyleGAN

OCONet: Image Extrapolation by Object Completion

Pathdreamer: A World Model for Indoor Navigation

PD-GAN: Probabilistic Diverse GAN for Image Inpainting

A Good Image Generator Is What You Need for High-Resolution Video Synthesis

Geometry-Free View Synthesis: Transformers and no 3D Priors

GANcraft: Unsupervised 3D Neural Rendering of Minecraft Worlds

Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction

Aligning Latent and Image Spaces to Connect the Unconnectable

InfinityGAN: Towards Infinite-Pixel Image Synthesis

Content-Aware GAN Compression

InOut: Diverse Image Outpainting via GAN Inversion

Unconstrained Scene Generation with Locally Conditioned Radiance Fields

CAMPARI: Camera-Aware Decomposed Generative Neural Radiance Fields

Large Scale Image Completion via Co-Modulated Generative Adversarial Networks

IBRNet: Learning Multi-View Image-Based Rendering

Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image

Taming Transformers for High-Resolution Image Synthesis

Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

Free View Synthesis

Neural Sparse Voxel Fields

World-Consistent Video-to-Video Synthesis

One shot 3D photography

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

Swapping Autoencoder for Deep Image Manipulation

Denoising Diffusion Probabilistic Models

Single-View View Synthesis With Multiplane Images

3D Photography Using Context-Aware Layered Depth Inpainting

Temporal Shift GAN for Large Scale Video Generation

Softmax Splatting for Video Frame Interpolation

SynSin: End-to-End View Synthesis From a Single Image

Analyzing and Improving the Image Quality of StyleGAN

High Fidelity Video Prediction with Large Stochastic Recurrent Neural Networks

Very Long Natural Scenery Image Prediction by Outpainting

Adversarial Video Generation on Complex Datasets

3D Ken Burns effect from a single image

Compositional Video Prediction

Boundless: Generative Adversarial Networks for Image Extension

Efficient Video Generation on Complex Datasets

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer

Generating Diverse High-Fidelity Images with VQ-VAE-2

Wide-Context Semantic Image Extrapolation

DeepView: View Synthesis With Learned Gradient Descent

SinGAN: Learning a Generative Model From a Single Natural Image

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

MSG-GAN: Multi-Scale Gradients for Generative Adversarial Networks

Extreme View Synthesis

InGAN: Capturing and Remapping the "DNA" of a Natural Image

Layer-structured 3D Scene Inference via View Synthesis

Learning to Decompose and Disentangle Representations for Video Prediction

Free-Form Image Inpainting With Gated Convolution

Stereo magnification

Non-stationary texture synthesis by adversarial expansion

Stochastic Video Generation with a Learned Prior

Going Deeper in Spiking Neural Networks: VGG and Residual Architectures

Generative Image Inpainting with Contextual Attention

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Demystifying MMD GANs

Zero-Shot Super-Resolution Using Deep Internal Learning

PredRNN: Recurrent Neural Networks for Predictive Learning using Spatiotemporal LSTMs

Progressive Growing of GANs for Improved Quality, Stability, and Variation

MoCoGAN: Decomposing Motion and Content for Video Generation

Generating the Future with Adversarial Transformers