PairAug: What Can Augmented Image-Text Pairs Do for Radiology? (2024-04-07T00:00:00.000000Z)

TL;DR

A Pairwise Augmentation (PairAug) approach that contains an Inter-patient Augmentation (InterAug) branch and an Intra-patient Augmentation (IntraAug) branch that generates radiology images using synthesised yet plausible reports derived from a Large Language Model (LLM).

Abstract

Current vision-language pre-training (VLP) methodologies predominantly depend on paired image-text datasets, a resource that is challenging to acquire in radiology due to privacy considerations and labelling complexities. Data augmentation provides a practical solution to overcome the issue of data scarcity, however, most augmentation methods exhibit a limited focus, prioritising either image or text augmentation exclusively. Acknowledging this limitation, our objective is to devise a framework capable of concurrently augmenting medical image and text data. We design a Pairwise Augmentation (PairAug) approach that contains an Inter-patient Augmentation (InterAug) branch and an Intra-patient Augmentation (IntraAug) branch. Specifically, the InterAug branch of our approach generates radiology images using synthesised yet plausible reports derived from a Large Language Model (LLM). The generated pairs can be considered a collection of new patient cases since they are artificially created and may not exist in the original dataset. In contrast, the IntraAug branch uses newly generated reports to manipulate images. This process allows us to create new paired data for each individual with diverse medical conditions. Our extensive experiments on various downstream tasks covering medical image classification zero-shot and fine-tuning analysis demonstrate that our PairAug, concurrently expanding both image and text data, substantially outperforms image-/text-only expansion baselines and advanced medical VLP baselines. Our code is released at https://github.com/YtongXie/PairAug.

Authors

References48 items

CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training

DiffuseExpand: Expanding dataset for 2D medical image segmentation using diffusion models

Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review

Label-Free Liver Tumor Segmentation

GPT-4 Technical Report

Video-P2P: Video Editing with Cross-Attention Control

Does Synthetic Data Generation of LLMs Help Clinical Text Mining?

LLaMA: Open and Efficient Foundation Language Models

Advancing Radiograph Representation Learning with Masked Record Modeling

Using Large Text To Image Models with Structured Prompts for Skin Disease Identification: A Case Study

RoentGen: Vision-Language Foundation Model for Chest X-ray Generation

MedCLIP: Contrastive Learning from Unpaired Medical Images and Text

Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning

Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning

Multi-modal Masked Autoencoders for Medical Vision-and-Language Pre-training

Prompt-to-Prompt Image Editing with Cross Attention Control

Clinical-BERT: Vision-Language Pre-training for Radiograph Diagnosis and Reports Generation

Privacy for Free: How does Dataset Condensation Help Privacy?

PaLM: Scaling Language Modeling with Pathways

Pseudo Numerical Methods for Diffusion Models on Manifolds

High-Resolution Image Synthesis with Latent Diffusion Models

Masked Autoencoders Are Scalable Vision Learners

Generalized radiograph representation learning via cross-supervision between images and free-text radiology reports

GLoRIA: A Multimodal Global-Local Representation Learning Framework for Label-efficient Medical Image Recognition

ResViT: Residual Vision Transformers for Multimodal Medical Image Synthesis

Disease-Image-Specific Learning for Diagnosis-Oriented Neuroimage Synthesis With Incomplete Multi-Modality Data

CoTr: Efficiently Bridging CNN and Transformer for 3D Medical Image Segmentation

Learning Transferable Visual Models From Natural Language Supervision

Zero-Shot Text-to-Image Generation

nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

XRayGAN: Consistency-preserving Generation of X-ray Images from Radiology Reports

Language Models are Few-Shot Learners

MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports

Data Augmentation Using Learned Transformations for One-Shot Medical Image Segmentation

PadChest: A large chest x-ray image dataset with multi-label annotated reports

CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparison

Augmenting the National Institutes of Health Chest Radiograph Dataset with Expert Annotations of Possible Pneumonia.

Generative Adversarial Network in Medical Imaging: A Review

Fixing Weight Decay Regularization in Adam

Synthetic Medical Images from Dual Generative Adversarial Networks

Attention is All you Need

Medical Image Synthesis with Context-Aware Generative Adversarial Networks

LLM for Patient-Trial Matching: Privacy-Aware Data Augmentation Towards Better Performance and Generalizability

ChatAug: Leveraging ChatGPT for Text Data Augmentation

Visualizing Data using t-SNE

Stylegan-nada: Clip-guided domain adaptation of image generators

Deep Neural