Text-guided Foundation Model Adaptation for Pathological Image Classification (2023-07-27T00:00:00.000000Z)

TL;DR

CITE injects text insights gained from language models pre-trained with a broad range of biomedical texts, leading to adapt foundation models towards pathological image understanding, and achieves leading performance compared with various baselines especially when training data is scarce.

Abstract

The recent surge of foundation models in computer vision and natural language processing opens up perspectives in utilizing multi-modal clinical data to train large models with strong generalizability. Yet pathological image datasets often lack biomedical text annotation and enrichment. Guiding data-efficient image diagnosis from the use of biomedical text knowledge becomes a substantial interest. In this paper, we propose to Connect Image and Text Embeddings (CITE) to enhance pathological image classification. CITE injects text insights gained from language models pre-trained with a broad range of biomedical texts, leading to adapt foundation models towards pathological image understanding. Through extensive experiments on the PatchGastric stomach tumor pathological image dataset, we demonstrate that CITE achieves leading performance compared with various baselines especially when training data is scarce. CITE offers insights into leveraging in-domain text knowledge to reinforce data-efficient pathological image classification. Code is available at https://github.com/Yunkun-Zhang/CITE.

Authors

Xiaosong Wang

5 papers

Y. Qiao

16 papers

Shaoting Zhang

7 papers

TL;DR

Abstract

Authors

References31 items

A Large-scale Synthetic Pathological Dataset for Deep Learning-enabled Segmentation of Breast Cancer

BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

Learning Customized Visual Models with Retrieval-Augmented Knowledge

Spatially aware graph neural networks and cross-level molecular profile prediction in colon cancer histopathology: a retrospective multi-cohort study.

Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning

LinkBERT: Pretraining Language Models with Document Links

Visual Prompt Tuning

Inference of captions from histopathological patches

INTERN: A New Learning Paradigm Towards General Vision

LiT: Zero-Shot Transfer with Locked-image text Tuning

Learning to Prompt for Vision-Language Models

On the Opportunities and Risks of Foundation Models

Learning Transferable Visual Models From Natural Language Supervision

VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

An annotation-free whole-slide training approach to pathological classification of lung cancer types using deep learning

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Classification and mutation prediction based on histopathology H&E images in liver cancer using deep learning

Language Models are Few-Shot Learners

Data-efficient and weakly supervised computational pathology on whole-slide images

Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning

Deep learning-based breast cancer classification through medical imaging modalities: state of the art and research challenges

Publicly Available Clinical BERT Embeddings

BioBERT: a pre-trained biomedical language representation model for biomedical text mining

Gastric Pathology Image Classification Using Stepwise Fine-Tuning for Deep Neural Networks

Deep Residual Learning for Image Recognition

Multi-scale Convolutional Neural Networks for Lung Nodule Classification

ImageNet Large Scale Visual Recognition Challenge

Training Like a Medical Resident: Universal Medical Image Segmentation via Context Prior Learning

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Medical image classification with convolutional neural network

Text-guided Foundation Model Adaptation for Medical Image

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names