ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation (2020-12-09T00:00:00.000000Z)

Abstract

In this paper, we present ViP-DeepLab, a unified model attempting to tackle the long-standing and challenging inverse projection problem in vision, which we model as restoring the point clouds from perspective image sequences while providing each point with instance-level semantic interpretations. Solving this problem requires the vision models to predict the spatial location, semantic class, and temporally consistent instance label for each 3D point. ViP-DeepLab approaches it by jointly performing monocular depth estimation and video panoptic segmentation. We name this joint task as Depth-aware Video Panoptic Segmentation, and propose a new evaluation metric along with two derived datasets for it, which will be made available to the public. On the individual sub-tasks, ViP-DeepLab also achieves state-of-the-art results, outperforming previous methods by 5.1% VPQ on Cityscapes-VPS, ranking 1st on the KITTI monocular depth estimation benchmark, and 1st on KITTI MOTS pedestrian. The datasets and the evaluation codes are made publicly available1.

Authors

Liang-Chieh Chen

14 papers

A. Yuille

29 papers

Siyuan Qiao

4 papers

Abstract

Authors

References107 items

STEP: Segmenting and Tracking Every Pixel

HOTA: A Higher Order Metric for Evaluating Multi-object Tracking

Bidirectional Attention Network for Monocular Depth Estimation

Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking

ReMOTS: Self-Supervised Refining Multi-Object Tracking and Segmentation

Segment as Points for Efficient Online Multi-Object Tracking and Segmentation

DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution

Video Panoptic Segmentation

Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation

Bidirectional Graph Reasoning Network for Panoptic Segmentation

FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking

Pixel Consensus Voting for Panoptic Segmentation

Tracking Objects as Points

BANet: Bidirectional Aggregation Network With Occlusion Handling for Panoptic Segmentation

STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation

Towards Bounding-Box Free Panoptic Segmentation

MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning

Unifying Training and Inference for Panoptic Segmentation

Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching

Learning Multi-Object Tracking and Segmentation From Automatic Annotations

Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation

Track to Reconstruct and Reconstruct to Track

Towards Real-Time Multi-Object Tracking

AdaptIS: Adaptive Instance Selection Network

Adaptive Unimodal Cost Volume Filtering for Deep Stereo Matching

SSAP: Single-Shot Instance Segmentation With Affinity Pyramid

Enforcing Geometric Constraints of Virtual Normal for Depth Prediction

From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation

Learning Instance Occlusion for Panoptic Segmentation

Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth

Soft Labels for Ordinal Regression

Seamless Scene Segmentation

Spatial-Temporal Relation Networks for Multi-Object Tracking

Objects as Points

GA-Net: Guided Aggregation Net for End-To-End Stereo Matching

SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences

Tracking Without Bells and Whistles

DeeperLab: Single-Shot Image Parser

MOTS: Multi-Object Tracking and Segmentation

UPSNet: A Unified Panoptic Segmentation Network

Attention-Guided Unified Network for Panoptic Segmentation

Learning to Fuse Things and Stuff

Integrated Object Detection and Tracking with Tracklet-Conditioned Detection

Affinity Derivation and Graph Merge for Instance Segmentation

Online Multi-Object Tracking with Dual Matching Attention Networks

Monocular Depth Estimation with Affinity, Vertical Pooling, and Label Enhancement

Box2Pix: Single-Shot Instance Segmentation by Assigning Pixels to Object Boxes

Digging Into Self-Supervised Monocular Depth Estimation

Deep Ordinal Regression Network for Monocular Depth Estimation

DenseASPP for Semantic Segmentation in Street Scenes

AutoAugment: Learning Augmentation Policies from Data

PAD-Net: Multi-tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing

GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose

Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking

Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Panoptic Segmentation

Learning Depth from Monocular Videos Using Direct Methods

Recurrent Autoregressive Networks for Online Multi-object Tracking

Detect to Track and Track to Detect

The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes

Sparsity Invariant CNNs

Multiple People Tracking by Lifted Multicut and Person Re-identification

Deep Network Flow for Multi-object Tracking

Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

Multi-scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation

Simple online and realtime tracking with a deep association metric

Mask R-CNN

Semi-Supervised Deep Learning for Monocular Depth Map Prediction

Feature Pyramid Networks for Object Detection

Wider or Deeper: Revisiting the ResNet Model for Visual Recognition

Deep Watershed Transform for Instance Segmentation

Unsupervised Monocular Depth Estimation with Left-Right Consistency

Densely Connected Convolutional Networks

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

Wide Residual Networks