Synthetic Boost: Leveraging Synthetic Data for Enhanced Vision-Language Segmentation in Echocardiography

Published in

(2023)

External Links:

Generate Graph DownloadPDF

TL;DR

This study evaluates results for two popular VLSMs using seven different kinds of language prompts derived from several attributes, automatically extracted from echocardiography images, segmentation masks, and their metadata, and shows improved metrics and faster convergence when pretraining VLSM on SDM-generated synthetic images before finetuning on real images.

Abstract

Accurate segmentation is essential for echocardiography-based assessment of cardiovascular diseases (CVDs). However, the variability among sonographers and the inherent challenges of ultrasound images hinder precise segmentation. By leveraging the joint representation of image and text modalities, Vision-Language Segmentation Models (VLSMs) can incorporate rich contextual information, potentially aiding in accurate and explainable segmentation. However, the lack of readily available data in echocardiography hampers the training of VLSMs. In this study, we explore using synthetic datasets from Semantic Diffusion Models (SDMs) to enhance VLSMs for echocardiography segmentation. We evaluate results for two popular VLSMs (CLIPSeg and CRIS) using seven different kinds of language prompts derived from several attributes, automatically extracted from echocardiography images, segmentation masks, and their metadata. Our results show improved metrics and faster convergence when pretraining VLSMs on SDM-generated synthetic images before finetuning on real images. The code, configs, and prompts are available at https://github.com/naamiinepal/synthetic-boost.

Authors

K. Poudel

2 papers

Manish Dhakal

3 papers

Prasiddha Bhandari

2 papers

References30 items

Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models

Echo from noise: synthetic ultrasound image generation using diffusion models for real image segmentation

Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study

Semantic Image Synthesis via Diffusion Models

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Synthetic Boost: Leveraging Synthetic Data for Enhanced Vision-Language Segmentation in Echocardiography

Published in

(2023)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

K. Poudel

2 papers

Manish Dhakal

3 papers

Prasiddha Bhandari

2 papers

References30 items

Exploring Transfer Learning in Medical Image Segmentation using Vision-Language Models

Echo from noise: synthetic ultrasound image generation using diffusion models for real image segmentation

Medical Image Understanding with Pretrained Vision Language Models: A Comprehensive Study

Semantic Image Synthesis via Diffusion Models

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

Rabin Adhikari

3 papers

Safal Thapaliya

3 papers

Bishesh Khanal

3 papers

Image Segmentation Using Text and Image Prompts

FLAVA: A Foundational Language And Vision Alignment Model

DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

CRIS: CLIP-Driven Referring Image Segmentation

LiT: Zero-Shot Transfer with Locked-image text Tuning

CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP

Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm

UNETR: Transformers for 3D Medical Image Segmentation

Learning Transferable Visual Models From Natural Language Supervision

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Denoising Diffusion Probabilistic Models

PhraseCut: Language-Based Image Segmentation in the Wild

Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers

Deep Learning for Segmentation Using an Open Large-Scale Dataset in 2D Echocardiography

Decoupled Weight Decay Regularization

U-Net: Convolutional Networks for Biomedical Image Segmentation

ReferItGame: Referring to Objects in Photographs of Natural Scenes

Individual Comparisons by Ranking Methods

TransBridge: A Lightweight Transformer for Left Ventricle Segmentation in Echocardiography

Ultrasound medical imaging techniques: a survey

GENERATIVE ADVERSARIAL NETS

Reaching intra-observer variability in 2-d echocardiographic image segmentation with a simple u-net architecture

We show that the VLSMs, pretrained on natural images, generalize to the real dataset (CAMUS) when finetuned on SDM-generated echocardiography images

Field of Study

Computer Science

Journal Information

Name

ArXiv

Volume

abs/2005.00687