Not Only Generative Art: Stable Diffusion for Content-Style Disentanglement in Art Analysis (2023-04-20T00:00:00.000000Z)

TL;DR

GOYA is presented, a method that distills the artistic knowledge captured in a recent generative model to disentangle content and style and shows that synthetically generated images sufficiently serve as a proxy of the real distribution of artworks, allowing GOYA to separately represent the two elements of art while keeping more information than existing methods.

Abstract

The duality of content and style is inherent to the nature of art. For humans, these two elements are clearly different: content refers to the objects and concepts in the piece of art, and style to the way it is expressed. This duality poses an important challenge for computer vision. The visual appearance of objects and concepts is modulated by the style that may reflect the author’s emotions, social trends, artistic movement, etc., and their deep comprehension undoubtfully requires to handle both. A promising step towards a general paradigm for art analysis is to disentangle content and style, whereas relying on human annotations to cull a single aspect of artworks has limitations in learning semantic concepts and the visual appearance of paintings. We thus present GOYA, a method that distills the artistic knowledge captured in a recent generative model to disentangle content and style. Experiments show that synthetically generated images sufficiently serve as a proxy of the real distribution of artworks, allowing GOYA to separately represent the two elements of art while keeping more information than existing methods.

Authors

Noa García

6 papers

Yankun Wu

1 papers

Yuta Nakashima

1 papers

TL;DR

Abstract

Authors

References90 items

Stylistic Multi-Task Analysis of Ukiyo-e Woodblock Prints

Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models

Fake it Till You Make it: Learning Transferable Representations from Synthetic ImageNet Clones

LAION-5B: An open large-scale dataset for training next generation image-text models

Diffusion-based Image Translation using Disentangled Style and Content Representation

Artistic Style Discovery with Independent Components

Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Hierarchical Text-Conditional Image Generation with CLIP Latents

StyleT2I: Toward Compositional and High-Fidelity Text-to-Image Synthesis

Pseudo Numerical Methods for Diffusion Models on Manifolds

The Met Dataset: Instance-level Recognition for Artworks

Splicing ViT Features for Semantic Appearance Transfer

High-Resolution Image Synthesis with Latent Diffusion Models

SemanticStyleGAN: Learning Compositional Generative Priors for Controllable Image Synthesis and Editing

PointCLIP: Point Cloud Understanding by CLIP

CLIPstyler: Image Style Transfer with a Single Text Condition

Towards Language-Free Training for Text-to-Image Generation

Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model

DiffusionCLIP: Text-Guided Diffusion Models for Robust Image Manipulation

Diverse Image Style Transfer via Invertible Cross-Space Mapping

Explain Me the Painting: Multi-Topic Knowledgeable Art Description Generation

Towards Generating and Evaluating Iconographic Image Captions of Artworks

DualAST: Dual Style-Learning Networks for Artistic Style Transfer

CogView: Mastering Text-to-Image Generation via Transformers

GCNBoost: Artwork Classification by Label Propagation through a Knowledge Graph

Emerging Properties in Self-Supervised Vision Transformers

Data-Efficient Language-Supervised Zero-Shot Learning with Self-Distillation

Text to Image Generation with Semantic-Spatial Aware GAN

ALADIN: All Layer Adaptive Instance Normalization for Fine-grained Style Similarity

Learning Transferable Visual Models From Natural Language Supervision

Disentangled Representations from Non-Disentangled Models

Exploring Simple Siamese Representation Learning

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

KT-GAN: Knowledge-Transfer Generative Adversarial Network for Text-to-Image Synthesis

Measuring the Biases and Effectiveness of Content-Style Disentanglement

DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesis

Improving Style-Content Disentanglement in Image-to-Image Translation

Denoising Diffusion Probabilistic Models

Arbitrary Style Transfer via Multi-Adaptation Network

ControlVAE: Controllable Variational Autoencoder

A Simple Framework for Contrastive Learning of Visual Representations

ContextNet: representation and exploration for painting classification and retrieval in context

Recognizing the Style of Visual Arts via Adaptive Cross-layer Correlation

Content and Style Disentanglement for Artistic Style Transfer

Multi-mapping Image-to-Image Translation via Learning Disentanglement

Context-Aware Embeddings for Automatic Art Analysis

Two-Stage Deep Learning Approach to the Classification of Fine-Art Paintings

Discovering Visual Patterns in Art Collections With Spatially-Consistent Feature Learning

Fine-tuning Convolutional Neural Networks for fine art classification

Style and Content Disentanglement in Generative Adversarial Networks

OmniArt

Reflecting on How Artworks Are Processed and Analyzed by Computer Vision

Deep Transfer Learning for Art Classification Problems

Weakly Supervised Object Detection in Artworks

How to Read Paintings: Semantic Art Understanding with Multi-Modal Retrieval

Image-to-image translation for cross-domain disentanglement

Multimodal Unsupervised Image-to-Image Translation

Image Style Classification Based on Learnt Deep Correlation Features

The Shape of Art History in the Eyes of the Machine

Sketching with Style: Visual Search with Sketches and Aesthetic Context

Recognizing Art Style Automatically in Painting with Deep Learning

From Part to Whole: Who is Behind the Painting?

DeepArt: Learning Joint Representations of Visual Arts

Improved ArtGAN for Conditional Synthesis of Natural Image and Artwork

Large Batch Training of Convolutional Networks

Disentangled Representation Learning GAN for Pose-Invariant Face Recognition

Unsupervised Learning of Disentangled Representations from Video

BAM! The Behance Artistic Media Dataset for Recognition Beyond Photography

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

Improved Deep Metric Learning with Multi-class N-pair Loss Objective

beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework

Ceci n'est pas une pipe: A deep convolutional network for fine-art paintings classification

Image Style Transfer Using Convolutional Neural Networks

InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets

Pandora: Description of a painting database for art movement recognition with baselines and perspectives

Deep Residual Learning for Image Recognition