Segment Anything (2023-04-05T00:00:00.000000Z)

TL;DR

The Segment Anything Model (SAM) is introduced: a new task, model, and dataset for image segmentation, and its zero-shot performance is impressive – often competitive with or even superior to prior fully supervised results.

Abstract

We introduce the Segment Anything (SA) project: a new task, model, and dataset for image segmentation. Using our efficient model in a data collection loop, we built the largest segmentation dataset to date (by far), with over 1 billion masks on 11M licensed and privacy respecting images. The model is designed and trained to be promptable, so it can transfer zero-shot to new image distributions and tasks. We evaluate its capabilities on numerous tasks and find that its zero-shot performance is impressive – often competitive with or even superior to prior fully supervised results. We are releasing the Segment Anything Model (SAM) and corresponding dataset (SA-1B) of 1B masks and 11M images at segment-anything.com to foster research into foundation models for computer vision. We recommend reading the full paper at: arxiv.org/abs/2304.02643.

Authors

Nikhila Ravi

3 papers

Wan-Yen Lo

2 papers

Ross B. Girshick

8 papers

TL;DR

Abstract

Authors

References148 items

Multiview Compressive Coding for 3D Reconstruction

OneFormer: One Transformer to Rule Universal Image Segmentation

SimpleClick: Interactive Image Segmentation with Simple Vision Transformers

EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations

SOCRATES: Introducing Depth in Visual Wildlife Monitoring Using Stereo Vision

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications

CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotation

Open-World Instance Segmentation: Exploiting Pseudo Ground Truth From Learned Pairwise Affinity

FocalClick: Towards Practical Interactive Image Segmentation

PaLM: Scaling Language Modeling with Pathways

Exploring Plain Vision Transformer Backbones for Object Detection

Training Compute-Optimal Large Language Models

EDTER: Edge Detection with Transformer

Semantic Segmentation in Art Paintings

Instance Segmentation for Autonomous Log Grasping in Forestry Operations

3D Instance Segmentation of MVS Buildings

Masked-attention Mask Transformer for Universal Image Segmentation

Masked Autoencoders Are Scalable Vision Learners

HyperExtended LightFace: A Facial Attribute Analysis Framework

Ego4D: Around the World in 3,000 Hours of Egocentric Video

iShape: A First Step Towards Irregular Shape Instance Segmentation

Towards Real-World Prohibited Item Detection: A Large-Scale X-ray Benchmark

On the Opportunities and Risks of Foundation Models

Learning Open-World Object Proposals without Learning to Classify

Per-Pixel Classification is Not All You Need for Semantic Segmentation

K-Net: Towards Unified Image Segmentation

BEiT: BERT Pre-Training of Image Transformers

ZeroWaste Dataset: Towards Deformable Object Segmentation in Cluttered Scenes

A Step Toward More Inclusive People Annotations for Fairness

Carbon Emissions and Large Neural Network Training

Learning Transferable Visual Models From Natural Language Supervision

Zero-Shot Text-to-Image Generation

Reviving Iterative Training with Mask Guidance for Interactive Segmentation

Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision

Occluded Video Instance Segmentation: A Benchmark

Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

LightFace: A Hybrid Deep Face Recognition Framework

TrashCan: A Semantically-Segmented Dataset towards Visual Detection of Marine Debris

Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains

Language Models are Few-Shot Learners

NDD20: A large-scale few-shot dolphin dataset for coarse and fine-grained categorisation

End-to-End Object Detection with Transformers

Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100

Getting to 99% Accuracy in Interactive Segmentation

Scaling Laws for Neural Language Models

Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy

Nucleus segmentation across imaging experiments: the 2018 Data Science Bowl

Quantifying the Carbon Emissions of Machine Learning

Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition

ilastik: interactive machine learning for (bio)image analysis

Does Object Recognition Work for Everyone?

LVIS: A Dataset for Large Vocabulary Instance Segmentation

WoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving

Model Cards for Model Reporting

Iterative Interaction Training for Segmentation Editing Networks

Interactive Image Segmentation with Latent Diversity

Places: A 10 Million Image Database for Scene Recognition

Iteratively Trained Interactive Segmentation

Datasheets for datasets

Panoptic Segmentation

Cascade R-CNN: Delving Into High Quality Object Detection

Deep Extreme Cut: From Extreme Points to Object Segmentation

Decoupled Weight Decay Regularization

Action recognition in RGB-D egocentric videos

Extreme Clicking for Efficient Object Annotation

Focal Loss for Dense Object Detection

Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints

Attention is All you Need

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Semi-supervised sequence tagging with bidirectional language models

Mask R-CNN

Finely-grained annotated datasets for image-based plant phenotyping

Semantic Understanding of Scenes Through the ADE20K Dataset

Layer Normalization