Local Relighting of Real Scenes (2022-07-06T00:00:00.000000Z)

TL;DR

This work proposes an approach for local relighting that trains a model without supervision of any novel image dataset by using synthetically generated image pairs from another model, including a stylespace-manipulated GAN.

Abstract

We introduce the task of local relighting, which changes a photograph of a scene by switching on and off the light sources that are visible within the image. This new task differs from the traditional image relighting problem, as it introduces the challenge of detecting light sources and inferring the pattern of light that emanates from them. We propose an approach for local relighting that trains a model without supervision of any novel image dataset by using synthetically generated image pairs from another model. Concretely, we collect paired training images from a stylespace-manipulated GAN; then we use these images to train a conditional image-to-image model. To benchmark local relighting, we introduce Lonoff, a collection of 306 precisely aligned images taken in indoor spaces with different combinations of lights switched on. We show that our method significantly outperforms baseline methods based on GAN inversion. Finally, we demonstrate extensions of our method that control different light sources separately. We invite the community to tackle this new task of local relighting.

Authors

Àgata Lapedriza

10 papers

A. Torralba

34 papers

Ali Jahanian

2 papers

TL;DR

Abstract

Authors

References33 items

Physically Inspired Dense Fusion Networks for Relighting

Multi-modal Bifurcated Network for Depth Guided Image Relighting

NTIRE 2021 Depth Guided Image Relighting Challenge

Multi-scale Self-calibrated Network for Image Light Source Transfer

DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

Designing an encoder for StyleGAN image manipulation

StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generation

2D Image Relighting with Image-to-Image Translation

Training Generative Adversarial Networks with Limited Data

VIDIT: Virtual Image Dataset for Illumination Transfer

GANSpace: Discovering Interpretable GAN Controls

In-Domain GAN Inversion for Real Image Editing

StyleGAN2 Distillation for Feed-forward Image Manipulation

Image Processing Using Multi-Code GAN Prior

Analyzing and Improving the Image Quality of StyleGAN

Interpreting the Latent Space of GANs for Semantic Face Editing

On the "steerability" of generative adversarial networks

A Style-Based Generator Architecture for Generative Adversarial Networks

GAN Dissection: Visualizing and Understanding Generative Adversarial Networks

Deep image-based relighting from optimal sparse samples

Unified Perceptual Parsing for Scene Understanding

The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

Scene Parsing through ADE20K Dataset

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Image-to-Image Translation with Conditional Adversarial Networks

Learning-based view synthesis for light field cameras

Deep Residual Learning for Image Recognition

Ground truth dataset and baseline evaluations for intrinsic image algorithms

A theory of locally low dimensional light transport

Clustered principal components for precomputed radiance transfer

Polynomial texture maps

Demonstrations of our user selective method. Examples on the left demonstrate different combinations of lights being turned off, and examples on the right demonstrate different lights being turned on

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names