Enhancing Remote Sensing Vision-Language Models for Zero-Shot Scene Classification (2024-09-01T00:00:00.000000Z)

TL;DR

This work utilizes initial predictions based on text prompting and patch affinity relationships from the image encoder to enhance zero-shot capabilities through transductive inference, all without the need for supervision and at a minor computational cost.

Abstract

Vision-Language Models for remote sensing have shown promising uses thanks to their extensive pretraining. However, their conventional usage in zero-shot scene classification methods still involves dividing large images into patches and making independent predictions, i.e., inductive inference, thereby limiting their effectiveness by ignoring valuable contextual information. Our approach tackles this issue by utilizing initial predictions based on text prompting and patch affinity relationships from the image encoder to enhance zero-shot capabilities through transductive inference, all without the need for supervision and at a minor computational cost. Experiments on 10 remote sensing datasets with state-of-the-art Vision-Language Models demonstrate significant accuracy improvements over inductive zero-shot classification. Our source code is publicly available on Github: https://github.com/elkhouryk/RS-TransCLIP

Authors

C. Vleeschouwer

11 papers

Benoit G'erin

2 papers

Ismail Ben Ayed

11 papers

TL;DR

Abstract

Authors

References45 items

Boosting Vision-Language Models for Histopathology Classification: Predict all at once

SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding

Boosting Vision-Language Models with Transduction

On the Test-Time Zero-Shot Generalization of Vision-Language Models: Do we Really need Prompt Learning?

RS-LLaVA: A Large Vision-Language Model for Joint Captioning and Question Answering in Remote Sensing Imagery

Transductive Zero-Shot and Few-Shot CLIP

Streamlined Hybrid Annotation Framework Using Scalable Codestream for Bandwidth-Restricted UAV Object Detection

LHRS-Bot: Empowering Remote Sensing with VGI-Enhanced Large Multimodal Language Model

EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain

SkyScript: A Large and Semantically Diverse Vision-Language Dataset for Remote Sensing

Remote Sensing Vision-Language Foundation Models without Annotations via Ground Remote Alignment

A Deep Learning Application for Building Damage Assessment Using Ultra-High-Resolution Remote Sensing Imagery in Turkey Earthquake

RSGPT: A Remote Sensing Vision Language Model and Benchmark

RS5M and GeoRSCLIP: A Large-Scale Vision- Language Dataset and a Large Vision-Language Model for Remote Sensing

RemoteCLIP: A Vision Language Foundation Model for Remote Sensing

Vision-Language Models in Remote Sensing: Current progress and future trends

BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs

Does CLIP Benefit Visual Question Answering in the Medical Domain as Much as it Does in the General Domain?

Learning Transferable Visual Models From Natural Language Supervision

Deep Unsupervised Embedding for Remote Sensing Image Retrieval Using Textual Cues

Learning Visual Representations with Caption Annotations

MLRSNet: A Multi-label High Spatial Resolution Remote Sensing Dataset for Semantic Scene Understanding

Laplacian Regularized Few-Shot Learning

Deep learning in environmental remote sensing: Achievements and challenges

RSI-CB: A Large-Scale Remote Sensing Image Classification Benchmark Using Crowdsourced Data

TextRS: Deep Bidirectional Triplet Network for Matching Text to Remote Sensing Images

Prototype Rectification for Few-Shot Learning

A Baseline for Few-Shot Image Classification

Perspectives for Remote Sensing with Unmanned Aerial Vehicles in Precision Agriculture.

Scene Classification With Recurrent Attention of VHR Remote Sensing Images

Introducing Eurosat: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification

PatternNet: A Benchmark Dataset for Performance Evaluation of Remote Sensing Image Retrieval

Remote Sensing Image Scene Classification: Benchmark and State of the Art

AID: A Benchmark Data Set for Performance Evaluation of Aerial Scene Classification

Feature significance-based multibag-of-visual-words model for remote sensing image scene classification

Learning Visual Features from Large Weakly Supervised Data

STRUCTURAL HIGH-RESOLUTION SATELLITE IMAGE INDEXING

An overview of statistical learning theory

Transductive Inference for Text Classification using Support Vector Machines

From Satellite to UAV-Based Remote Sensing: A Review on Precision Agriculture

Land-cover change detection using paired OpenStreetMap data and optical high-resolution imagery via object-guided Transformer

Information Maximization for Few-Shot Learning

multimodal biomedical applicable license agreement with IEEE. Restrictions apply

biomedical foundation model pretrained from fifteen million scientific Authorized licensed use limited to the terms of the

“Towards helpful and honest remote sensing large vision language model,”

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names