Aligning Latent and Image Spaces to Connect the Unconnectable (2021-04-14T00:00:00.000000Z)

TL;DR

This work develops a method to generate infinite high-resolution images with diverse and complex content based on a perfectly equivariant patch-wise generator with synchronous interpolations in the image and latent spaces and modify the AdaIN mechanism to work in such a setup.

Abstract

In this work, we develop a method to generate infinite high-resolution images with diverse and complex content. It is based on a perfectly equivariant patch-wise generator with synchronous interpolations in the image and latent spaces. Latent codes, when sampled, are positioned on the coordinate grid, and each pixel is computed from an interpolation of the neighboring codes. We modify the AdaIN mechanism to work in such a setup and train a GAN model to generate images positioned between any two latent vectors. At test time, this allows for generating infinitely large images of diverse scenes that transition naturally from one into another. Apart from that, we introduce LHQ: a new dataset of 90k high-resolution nature landscapes. We test the approach on LHQ, LSUN Tower and LSUN Bridge and outperform the baselines by at least 4 times in terms of quality and diversity of the produced infinite images. The project website is located at https://universome.github.io/alis.

Authors

Mohamed Elhoseiny

3 papers

Ivan Skorokhodov

2 papers

Grigorii Sotnikov

1 papers

TL;DR

Abstract

Authors

References80 items

Generative Adversarial Networks

Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis

Taming Transformers for High-Resolution Image Synthesis

Infinite Nature: Perpetual View Generation of Natural Scenes from a Single Image

Learning Continuous Image Representation with Local Implicit Image Function

Positional Encoding as Spatial Inductive Bias in GANs

Naturally Occurring Equivariance in Neural Networks

pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis

Image Generators with Conditionally-Independent Pixel Synthesis

Adversarial Generation of Continuous Images

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields

not-so-BigGAN: Generating High-Fidelity Images on a Small Compute Budget

NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

Interpreting Spatially Infinite Generative Models

GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis

Swapping Autoencoder for Deep Image Manipulation

GramGAN: Deep 3D Texture Synthesis From 2D Exemplars

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Differentiable Augmentation for Data-Efficient GAN Training

Implicit Neural Representations with Periodic Activation Functions

Training Generative Adversarial Networks with Limited Data

Adversarial Latent Autoencoders

PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models

LocoGAN - Locally Convolutional GAN

Image Outpainting and Harmonization using Generative Adversarial Networks

Analyzing and Improving the Image Quality of StyleGAN

SEAN: Image Synthesis With Semantic Region-Adaptive Normalization

Consistency Regularization for Generative Adversarial Networks

Very Long Natural Scenery Image Prediction by Outpainting

Boundless: Generative Adversarial Networks for Image Extension

Deep Meta Functionals for Shape Representation

Wide-Context Semantic Image Extrapolation

Few-Shot Unsupervised Image-to-Image Translation

SinGAN: Learning a Generative Model From a Single Natural Image

Making Convolutional Networks Shift-Invariant Again

Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?

HYPE: A Benchmark for Human eYe Perceptual Evaluation of Generative Models

COCO-GAN: Generation by Parts via Conditional Coordinating

Semantic Image Synthesis With Spatially-Adaptive Normalization

Gauge Equivariant Convolutional Networks and the Icosahedral CNN

A Style-Based Generator Architecture for Generative Adversarial Networks

Occupancy Networks: Learning 3D Reconstruction in Function Space

Learning Implicit Fields for Generative Shape Modeling

Large Scale GAN Training for High Fidelity Natural Image Synthesis

An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

Assessing Generative Models via Precision and Recall

Group Normalization

Spectral Normalization for Generative Adversarial Networks

Spherical CNNs

Which Training Methods for GANs do actually Converge?

Non-Autoregressive Neural Machine Translation

Wasserstein Generative Adversarial Networks

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Attention is All you Need

The Numerics of GANs

Learning Texture Manifolds with the Periodic Spatial GAN

Convolutional Sequence to Sequence Learning

Geometric GAN

Improved Training of Wasserstein GANs

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

Improved Texture Networks: Maximizing Quality and Diversity in Feed-Forward Stylization and Texture Synthesis

Texture Synthesis with Spatial Generative Adversarial Networks

A Learned Representation For Artistic Style

HyperNetworks

Instance Normalization: The Missing Ingredient for Fast Stylization

Image Style Transfer Using Convolutional Neural Networks

Improved Techniques for Training GANs

Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift

Auto-Encoding Variational Bayes

Infinite Images: Creating and Exploring a Large Photorealistic Virtual Space

Procedural modeling of buildings

Procedural modeling of cities

Swagan: A style-based wavelet-driven generative model

Spiral Generative Network for Image Extrapolation

Tilegan: synthesis of large-scale non-homogeneous textures

Image alignment and stitching: A tutorial. Foundations and Trends® in Computer Graphics and Vision