ArtGPT-4: Towards Artistic-understanding Large Vision-Language Models with Enhanced Adapter (2023-05-12T00:00:00.000000Z)

TL;DR

This paper proposes ArtGPT-4, a pioneering large vision-language model tailored to address the limitations of existing models in artistic comprehension, and shows that it can render images with an artistic-understanding and convey the emotions they inspire, mirroring human interpretation.

Abstract

The success of large language models (LLMs) has inspired an emerging research field of multimodal learning. However, a grand challenge of exploiting LLMs for multimodal learning is the size of pre-trained LLMs which are always with billions of parameters. To tackle this challenge, models such as MiniGPT-4 and LLaVA have been developed to fine-tune the pre-trained models using fewer parameters. Despite their promising performance, these models remain limited in their understanding of artistic imagery. To facilitate better artistic-understanding, in this paper, we propose ArtGPT-4, a pioneering large vision-language model tailored to address the limitations of existing models in artistic comprehension. The key innovation of ArtGPT-4 lies in its craft for the sophisticated challenge of artistic image comprehension, setting it apart from other models that overlook fine details for broader themes. Specifically, it works by integrating some specialized adapter layers into the LLM, enabling the model to more efficiently and effectively parse and interpret complex visual tokens, instead of fine-tuning the whole LLM as in the existing method. ArtGPT-4 has demonstrated its outstanding performance on the efficiency: utilizing a Tesla A100 device, its training can be completed in mere 2 hours with an image-text pair dataset comprising approximately 0.52M entries. Additionally, ArtGPT-4 has also achieved state-of-the-art performance on the ArtEmis and ArtEmis-v2.0 datasets as well as the benchmarks established in this work, lagging behind professional artists' descriptions by a negligible 0.15 points on a 6-point scale. The outstanding performance of ArtGPT-4 shows that it can render images with an artistic-understanding and convey the emotions they inspire, mirroring human interpretation. The code and the pre-trained model are accessible in \url{https://github.com/DLYuanGod/ArtGPT-4}.

References44 items

MultiModal-GPT: A Vision and Language Model for Dialogue with Humans

mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality

AIM: Adapting Image Models for Efficient Video Action Recognition

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

LAION-5B: An open large-scale dataset for training next generation image-text models

MAR: Masked Autoencoders for Efficient Action Recognition

GIT: A Generative Image-to-text Transformer for Vision and Language

AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition

Flamingo: a Visual Language Model for Few-Shot Learning

It is Okay to Not Be Okay: Overcoming Emotional Bias in Affective Image Captioning by Contrastive Data Collection

Hierarchical Text-Conditional Image Generation with CLIP Latents

Visual Prompt Tuning

Training language models to follow instructions with human feedback

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

Towards a Unified View of Parameter-Efficient Transfer Learning

Multimodal Few-Shot Learning with Frozen Language Models

BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models

M6: A Chinese Multimodal Pretrainer

Learning Transferable Visual Models From Natural Language Supervision

VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

ArtEmis: Affective Language for Visual Art

Language Models are Few-Shot Learners

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Parameter-Efficient Transfer Learning for NLP

Improved ArtGAN for Conditional Synthesis of Natural Image and Artwork

VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text

Visual Prompting: Modifying Pixel Space to Adapt Pre-trained Models

image-instruction data called ArtMM . These images, characterized by their complex elements, were a mix of those found online and others generated using DALL-E 2

Introducing chatgpt

Prefix-Tuning: Optimizing Continuous Prompts for Generation

A Stage 2 training instructions B Zero-shot Testing Datasets ArtEmis

model to encode text high-dimensional embeddings. We

textblob documentation

respective compound sentiment scores. A higher score indicates better performance

emotional impact, and language-based explanations

OpenAI. 2023.

most positive

emotions and associated painting explanations. Furthermore, to further enrich our evaluation, we filtered images from the mPLUG-Owl (Ye et al., 2023) database and others, in

dataset by employing a novel contrastive data collection approach. By balancing emotional biases and incorporating 260,533 new instances with contrasting emotions

for each response

: Modularization empowers

For VADER-based similarity

2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models