FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Tasks (2023-03-04T00:00:00.000000Z)

TL;DR

This work proposes a novel FAshion-focused Multi-task Efficient learning method for Vision-and-Language tasks (FAME-ViL), which applies a single model for multiple heterogeneous fashion tasks, therefore being much more parameter-efficient.

Abstract

In the fashion domain, there exists a variety of vision-and-language (V+L) tasks, including cross-modal retrieval, text-guided image retrieval, multi-modal classification, and image captioning. They differ drastically in each individual input/output format and dataset size. It has been common to design a task-specific model and fine-tune it independently from a pre-trained V+l model (e.g., CLIP). This results in parameter inefficiency and inability to exploit inter-task relatedness. To address such issues, we propose a novel FAshion-focused Multi-task Efficient learning method for Vision-and-Language tasks (FAME-ViL) in this work. Compared with existing approaches, FAME-ViL applies a single model for multiple heterogeneous fashion tasks, therefore being much more parameter-efficient. It is enabled by two novel components: (1) a task-versatile architecture with cross-attention adapters and task-specific adapters integrated into a unified V+L model, and (2) a stable and effective multi-task training strategy that supports learning from heterogeneous data and prevents negative transfer. Extensive experiments on four fashion tasks show that our FAME-ViL can save 61.5% of parameters over alternatives, while significantly outperforming the conventional independently trained single-task models. Code is available at https://github.com/BrandonHanx/FAME-ViL.

Authors

Li Zhang

6 papers

Yi-Zhe Song

8 papers

Xiatian Zhu

3 papers

TL;DR

Abstract

Authors

References95 items

Masked Vision-language Transformer in Fashion

FaD-VLP: Fashion Vision-and-Language Pre-training towards Unified Retrieval and Captioning

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

GPPF: A General Perception Pre-training Framework via Sparsely Activated Multi-Task Learning

FashionViL: Fashion-Focused Vision-and-Language Representation Learning

Bridge-Tower: Building Bridges Between Encoders in Vision-Language Representation Learning

Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks

Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone

Neural Prompt Search

EI-CLIP: Entity-aware Interventional Contrastive Learning for E-commerce Cross-modal Retrieval

FashionVLP: Vision Language Transformer for Fashion Retrieval with Feedback

Conditioned and composed image retrieval combining and partially fine-tuning CLIP-based features

Effective conditioned and composed image retrieval combining CLIP-based features

AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition

Vision Transformer Adapter for Dense Predictions

CoCa: Contrastive Captioners are Image-Text Foundation Models

Progressive Learning for Image Retrieval with Hybrid-Modality Queries

Training and challenging models for text-guided fashion image retrieval

UIGR: Unified Interactive Garment Retrieval

Visual Prompt Tuning

ARTEMIS: Attention-based Retrieval with Text-Explicit Matching and Implicit Similarity

Conditional Prompt Learning for Vision-Language Models

CommerceMM: Large-Scale Commerce MultiModal Representation Learning with Omni Retrieval

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Multi-Task Learning as a Bargaining Game

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

FLAVA: A Foundational Language And Vision Alignment Model

UFO: A UniFied TransfOrmer for Vision-Language Representation Learning

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

Towards artificial general intelligence via a multimodal foundation model

UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning

Towards a Unified View of Parameter-Efficient Transfer Learning

Learning to Prompt for Vision-Language Models

Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models

Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Align before Fuse: Vision and Language Representation Learning with Momentum Distillation

LoRA: Low-Rank Adaptation of Large Language Models

Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback

VinVL: Revisiting Visual Representations in Vision-Language Models

CoSMo: Content-Style Modulation for Image Retrieval with Text Feedback

Dual Compositional Learning in Interactive Image Retrieval

Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learning

RTIC: Residual Learning for Text and Image Composition using Graph Convolutional Network

Kaleido-BERT: Vision-Language Pre-training on Fashion Domain

Learning Transferable Visual Models From Natural Language Supervision

UniT: Multimodal Multitask Learning with a Unified Transformer

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Multi-Task Learning with Deep Neural Networks: A Survey

Fashion Captioning: Towards Generating Accurate Descriptions with Semantic Rewards

Image Search With Text Feedback by Visiolinguistic Attention Learning

FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrieval

Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks

12-in-1: Multi-Task Vision and Language Representation Learning

PyTorch: An Imperative Style, High-Performance Deep Learning Library

AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning

UNITER: UNiversal Image-TExt Representation Learning

Unified Vision-Language Pre-Training for Image Captioning and VQA

Adaptive Scheduling for Multi-Task Learning

VL-BERT: Pre-training of Generic Visual-Linguistic Representations

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

VisualBERT: A Simple and Performant Baseline for Vision and Language

ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks

BAM! Born-Again Multi-Task Networks for Natural Language Understanding

What Makes Training Multi-Modal Classification Networks Hard?

Parameter-Efficient Transfer Learning for NLP

Composing Text and Image for Image Retrieval - an Empirical Odyssey

Multi-Task Learning of Hierarchical Vision-Language Representation

A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks

Interpretable Multimodal Retrieval for Fashion Products

Multi-Task Learning as Multi-Objective Optimization

Fashion-Gen: The Generative Fashion Dataset and Challenge

GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

Attention is All you Need