SynSin: End-to-End View Synthesis From a Single Image (2019-12-18T00:00:00.000000Z)

TL;DR

This work proposes a novel differentiable point cloud renderer that is used to transform a latent 3D point cloud of features into the target view and outperforms baselines and prior work on the Matterport, Replica, and RealEstate10K datasets.

Abstract

View synthesis allows for the generation of new views of a scene given one or more images. This is challenging; it requires comprehensively understanding the 3D scene from images. As a result, current methods typically use multiple images, train on ground-truth depth, or are limited to synthetic data. We propose a novel end-to-end model for this task using a single image at test time; it is trained on real images without any ground-truth 3D information. To this end, we introduce a novel differentiable point cloud renderer that is used to transform a latent 3D point cloud of features into the target view. The projected features are decoded by our refinement network to inpaint missing regions and generate a realistic output image. The 3D component inside of our generative model allows for interpretable manipulation of the latent feature space at test time, e.g. we can animate trajectories from a single image. Additionally, we can generate high resolution images and generalise to other input resolutions. We outperform baselines and prior work on the Matterport, Replica, and RealEstate10K datasets.

Authors

Justin Johnson

12 papers

Georgia Gkioxari

3 papers

R. Szeliski

2 papers

TL;DR

Abstract

Authors

References78 items

PyTorch: An Imperative Style, High-Performance Deep Learning Library

3D Ken Burns effect from a single image

Boundless: Generative Adversarial Networks for Image Extension

Deep view synthesis from sparse photometric images

Neural Point-Based Graphics

The Replica Dataset: A Digital Replica of Indoor Spaces

Differentiable surface splatting for point-based geometry processing

Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations

Wide-Context Semantic Image Extrapolation

DeepView: View Synthesis With Learned Gradient Descent

Pushing the Boundaries of View Extrapolation With Multiplane Images

Transformable Bottleneck Networks

Neural Rerendering in the Wild

Habitat: A Platform for Embodied AI Research

HoloGAN: Unsupervised Learning of 3D Representations From Natural Images

Semantic Image Synthesis With Spatially-Adaptive Normalization

Multi-layer Depth and Epipolar Feature Transformers for 3D Scene Reconstruction

3D Scene Reconstruction With Multi-Layer Depth and Epipolar Transformers

Linearized Multi-Sampling for Differentiable Image Transformation

Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction

A Style-Based Generator Architecture for Generative Adversarial Networks

Extreme View Synthesis

Deep blending for free-viewpoint image-based rendering

DeepVoxels: Learning Persistent 3D Feature Embeddings

Unsupervised Learning of Shape and Pose with Differentiable Point Clouds

Large Scale GAN Training for High Fidelity Natural Image Synthesis

Multi-view to Novel View: Synthesizing Novel Views With Self-learned Confidence

Instant 3D photography

Layer-structured 3D Scene Inference via View Synthesis

Neural scene representation and rendering

Self-Attention Generative Adversarial Networks

MegaDepth: Learning Single-View Depth Prediction from Internet Photos

Learning Category-Specific Mesh Reconstruction from Image Collections

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Factoring Shape, Pose, and Layout from the 2D Image of a 3D Scene

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Soft 3D reconstruction for view synthesis

Interpretable Transformations with Encoder-Decoder Networks

Matterport3D: Learning from RGB-D Data in Indoor Environments

Learning to Synthesize a 4D RGBD Light Field from a Single Image

Unsupervised Learning of Depth and Ego-Motion from Video

Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency

Transformation-Grounded Image Generation Network for Novel 3D View Synthesis

Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

View Synthesis by Appearance Flow

Context Encoders: Feature Learning by Inpainting

Single-Image Depth Perception in the Wild

Deep Residual Learning for Image Recognition

Multi-view 3D Models from Single Images with a Convolutional Network

U-Net: Convolutional Networks for Biomedical Image Segmentation

Deep Convolutional Inverse Graphics Network

Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

Image-based rendering in the gradient domain

Vision meets robotics: The KITTI dataset

Depth synthesis and local warps for plausible image-based navigation

Indoor Segmentation and Support Inference from RGBD Images

High-performance software rasterization on GPUs

A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms

A survey of point-based techniques in computer graphics

Point-based rendering techniques

Region filling and object removal by exemplar-based image inpainting

High-quality video view interpolation using a layered representation

Image-Based Rendering Using Image-Based Priors

Surface splatting

Image inpainting

Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping

Modeling and Rendering Architecture from Photographs: A hybrid geometry- and image-based approach

Powered by AI: Turning any 2D photo into 3D using convolutional neural nets

Supplementary: Monocular Neural Image Based Rendering with Continuous View Control

Mesh R-CNN

PerspectiveNet: A Scene-consistent Image Generator for New View Synthesis in Real Indoor Environments

Neural volumes: Learning dynamic renderable volumes from images

Stereo magniﬁcation: Learning view synthesis using multiplane images

GENERATIVE ADVERSARIAL NETS

Lookingood: enhancing performance capture with real-time neural re-rendering