Universal Instance Perception as Object Discovery and Retrieval (2023-03-12T00:00:00.000000Z)

TL;DR

This work presents a universal instance perception model of the next generation, termed UNINEXT, which reformulates diverse instance perception tasks into a unified object discovery and retrieval paradigm and can flexibly perceive different types of objects by simply changing the input prompts.

Abstract

All instance perception tasks aim at finding certain objects specified by some queries such as category names, language expressions, and target annotations, but this complete field has been split into multiple independent sub-tasks. In this work, we present a universal instance perception model of the next generation, termed UNINEXT. UNINEXT reformulates diverse instance perception tasks into a unified object discovery and retrieval paradigm and can flexibly perceive different types of objects by simply changing the input prompts. This unified formulation brings the following benefits: (1) enormous data from different tasks and label vocabularies can be exploited for jointly training general instance-level representations, which is especially beneficial for tasks lacking in training data. (2) the unified model is parameter-efficient and can save redundant computation when handling multiple tasks simultaneously. UNINEXT shows superior performance on 20 challenging benchmarks from 10 instance-level tasks including classical image-level tasks (object detection and instance segmentation), vision-and-language tasks (referring expression comprehension and segmentation), and six video-level object tracking tasks. Code is available at https://github.com/MasterBin-IIAU/UNINEXT.

Authors

Huchuan Lu

21 papers

Jiannan Wu

2 papers

D. Wang

4 papers

TL;DR

Abstract

Authors

References140 items

Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Tracking

Video Mask Transfiner for High-Quality Video Instance Segmentation

In Defense of Online Models for Video Instance Segmentation

Towards Grand Unification of Object Tracking

XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model

You Only Infer Once: Cross-Modal Meta-Transfer for Referring Video Object Segmentation

Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks

A Unified Sequence Interface for Vision Tasks

VITA: Video Instance Segmentation via Object Token Association

A Generalist Agent

SeqTR: A Simple yet Universal Network for Visual Grounding

Exploring Plain Vision Transformer Backbones for Object Detection

Pathways: Asynchronous Distributed Dataflow for ML

Joint Feature Learning and Relation Modeling for Tracking: A One-Stream Framework

MixFormer: End-to-End Tracking with Iterative Mixed Attention

Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking

DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection

DN-DETR: Accelerate DETR Training by Introducing Query DeNoising

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework

A ConvNet for the 2020s

Detecting Twenty-thousand Classes using Image-level Supervision

Language as Queries for Referring Video Object Segmentation

SeqFormer: Sequential Transformer for Video Instance Segmentation

Grounded Language-Image Pre-training

LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

Masked-attention Mask Transformer for Universal Image Segmentation

Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks

End-to-End Referring Video Object Segmentation with Multimodal Transformers

INTERN: A New Learning Paradigm Towards General Vision

ByteTrack: Multi-Object Tracking by Associating Every Detection Box

TRAR: Routing the Attention Spans in Transformer for Visual Question Answering

Multi-Task Self-Training for Learning General Representations

Vision-Language Transformer and Query Generation for Referring Segmentation

YOLOX: Exceeding YOLO Series in 2021

Prototypical Cross-Attention Networks for Multiple Object Tracking and Segmentation

Segment as Points for Efficient and Effective Online Multi-Object Tracking and Segmentation

Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation

Video Instance Segmentation using Inter-Frame Communication Transformers

Referring Transformer: A One-step Approach to Multi-task Visual Grounding

Rethinking Cross-modal Interaction from a Top-down Perspective for Referring Video Object Segmentation

Dynamic Head: Unifying Object Detection Heads with Attentions

Cross-Modal Progressive Comprehension for Referring Segmentation

Instances as Queries

Encoder Fusion Network with Co-Attention Embedding for Referring Image Segmentation

MDETR - Modulated Detection for End-to-End Multi-Modal Understanding

TransVG: End-to-End Visual Grounding with Transformers

RefineMask: Towards High-Quality Instance Segmentation with Fine-Grained Features

Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark

Learning Spatio-Temporal Transformer for Visual Tracking

Learning Target Candidate Association to Keep Track of What Not to Track

Locate then Segment: A Strong Pipeline for Referring Image Segmentation

Transformer Tracking

Video Instance Segmentation with a Propose-Reduce Paradigm

Occluded Video Instance Segmentation: A Benchmark

TrackFormer: Multi-Object Tracking with Transformers

BoxInst: High-Performance Instance Segmentation with Box Annotations

End-to-End Video Instance Segmentation with Transformers

Sparse R-CNN: End-to-End Object Detection with Learnable Proposals

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Cascade Grouped Attention Network for Referring Expression Segmentation

Deformable DETR: Deformable Transformers for End-to-End Object Detection

LaSOT: A High-quality Large-scale Single Object Tracking Benchmark

Improving One-stage Visual Grounding by Recursive Sub-query Construction

Alpha-Refine: Boosting Tracking Performance by Precise Bounding Box Estimation

Quasi-Dense Similarity Learning for Multiple Object Tracking

Large-Scale Adversarial Training for Vision-and-Language Representation Learning

Bi-Directional Relationship Inferring Network for Referring Image Segmentation

End-to-End Object Detection with Transformers

TAO: A Large-Scale Benchmark for Tracking Any Object

A Transductive Approach for Video Object Segmentation

FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking

Tracking Objects as Points

High-Performance Long-Term Tracking With Meta-Updater

Probabilistic Regression for Visual Tracking

SOLOv2: Dynamic and Fast Instance Segmentation

Multi-Task Collaborative Network for Joint Referring Expression Comprehension and Segmentation