ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation

Published in

Computer Vision and Pattern Recognition(2020)

External Links:

Generate Graph DownloadPDF

Abstract

In this paper, we present ViP-DeepLab, a unified model attempting to tackle the long-standing and challenging inverse projection problem in vision, which we model as restoring the point clouds from perspective image sequences while providing each point with instance-level semantic interpretations. Solving this problem requires the vision models to predict the spatial location, semantic class, and temporally consistent instance label for each 3D point. ViP-DeepLab approaches it by jointly performing monocular depth estimation and video panoptic segmentation. We name this joint task as Depth-aware Video Panoptic Segmentation, and propose a new evaluation metric along with two derived datasets for it, which will be made available to the public. On the individual sub-tasks, ViP-DeepLab also achieves state-of-the-art results, outperforming previous methods by 5.1% VPQ on Cityscapes-VPS, ranking 1st on the KITTI monocular depth estimation benchmark, and 1st on KITTI MOTS pedestrian. The datasets and the evaluation codes are made publicly available1.

Authors

Liang-Chieh Chen

14 papers

A. Yuille

29 papers

Siyuan Qiao

4 papers

References107 items

STEP: Segmenting and Tracking Every Pixel

HOTA: A Higher Order Metric for Evaluating Multi-object Tracking

Bidirectional Attention Network for Monocular Depth Estimation

Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking

ReMOTS: Self-Supervised Refining Multi-Object Tracking and Segmentation

ViP-DeepLab: Learning Visual Perception with Depth-aware Video Panoptic Segmentation

Published in

Computer Vision and Pattern Recognition(2020)

External Links:

Generate Graph DownloadPDF

Abstract

Authors

Liang-Chieh Chen

14 papers

A. Yuille

29 papers

Siyuan Qiao

4 papers

References107 items

STEP: Segmenting and Tracking Every Pixel

HOTA: A Higher Order Metric for Evaluating Multi-object Tracking

Bidirectional Attention Network for Monocular Depth Estimation

Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking

ReMOTS: Self-Supervised Refining Multi-Object Tracking and Segmentation

Hartwig Adam

15 papers

Segment as Points for Efficient Online Multi-Object Tracking and Segmentation

DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution

Video Panoptic Segmentation

Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation

Bidirectional Graph Reasoning Network for Panoptic Segmentation

FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking

Pixel Consensus Voting for Panoptic Segmentation

Tracking Objects as Points

BANet: Bidirectional Aggregation Network With Occlusion Handling for Panoptic Segmentation

STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation

Towards Bounding-Box Free Panoptic Segmentation

MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning

Unifying Training and Inference for Panoptic Segmentation

Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching

Learning Multi-Object Tracking and Segmentation From Automatic Annotations

Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation

Track to Reconstruct and Reconstruct to Track

Towards Real-Time Multi-Object Tracking

AdaptIS: Adaptive Instance Selection Network

Adaptive Unimodal Cost Volume Filtering for Deep Stereo Matching

SSAP: Single-Shot Instance Segmentation With Affinity Pyramid

Enforcing Geometric Constraints of Virtual Normal for Depth Prediction

From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation

Learning Instance Occlusion for Panoptic Segmentation

Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth

Soft Labels for Ordinal Regression

Seamless Scene Segmentation

Spatial-Temporal Relation Networks for Multi-Object Tracking

Objects as Points

GA-Net: Guided Aggregation Net for End-To-End Stereo Matching

SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences

Tracking Without Bells and Whistles

DeeperLab: Single-Shot Image Parser

MOTS: Multi-Object Tracking and Segmentation

UPSNet: A Unified Panoptic Segmentation Network

Attention-Guided Unified Network for Panoptic Segmentation

Learning to Fuse Things and Stuff

Integrated Object Detection and Tracking with Tracklet-Conditioned Detection

Affinity Derivation and Graph Merge for Instance Segmentation

Online Multi-Object Tracking with Dual Matching Attention Networks

Monocular Depth Estimation with Affinity, Vertical Pooling, and Label Enhancement

Box2Pix: Single-Shot Instance Segmentation by Assigning Pixels to Object Boxes

Digging Into Self-Supervised Monocular Depth Estimation

Deep Ordinal Regression Network for Monocular Depth Estimation

DenseASPP for Semantic Segmentation in Street Scenes

AutoAugment: Learning Augmentation Policies from Data

PAD-Net: Multi-tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing

GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose

Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking

Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Panoptic Segmentation

Learning Depth from Monocular Videos Using Direct Methods

Recurrent Autoregressive Networks for Online Multi-object Tracking

Detect to Track and Track to Detect

The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes

Sparsity Invariant CNNs

Multiple People Tracking by Lifted Multicut and Person Re-identification

Deep Network Flow for Multi-object Tracking

Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

Multi-scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation

Simple online and realtime tracking with a deep association metric

Semi-Supervised Deep Learning for Monocular Depth Map Prediction

Feature Pyramid Networks for Object Detection

Wider or Deeper: Revisiting the ResNet Model for Visual Recognition

Deep Watershed Transform for Instance Segmentation

Unsupervised Monocular Depth Estimation with Left-Right Consistency

Densely Connected Convolutional Networks

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

Wide Residual Networks

Estimating Depth From Monocular Images as Classification Using Deep Fully Convolutional Residual Networks

Learning by Tracking: Siamese CNN for Robust Target Association

Single-Image Depth Perception in the Wild

Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks

The Cityscapes Dataset for Semantic Urban Scene Understanding

Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue

Simple online and realtime tracking

Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs

Towards unified depth and semantic prediction from a single image

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Efficient Decomposition of Image and Mesh Graphs by Lifted Multicuts

Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture

ImageNet Large Scale Visual Recognition Challenge

Simultaneous Detection and Segmentation

Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

Are we ready for autonomous driving? The KITTI vision benchmark suite

Perceiving in DepthVolume 1 Basic Mechanisms

Object Detection with Discriminatively Trained Part Based Models

Learning Depth from Single Monocular Images

Multiscale conditional random fields for image labeling

Perception viewed as an inverse problem

100

Watersheds in Digital Spaces: An Efficient Algorithm Based on Immersion Simulations

101

Basic mechanisms

102

Scene segmentation

103

Mapillary Planet-Scale Depth Dataset

104

Vision Science Photons To Phenomenology

105

Combined Object Categorization and Segmentation With an Implicit Shape Model

106

Generalizing the Hough transform to detect arbitrary shapes

107

Ieee Transactions on Pattern Analysis and Machine Intelligence 1 Stereo Processing by Semi-global Matching and Mutual Information

Field of Study

Computer Science

Journal Information

Name

2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Page

3996-4007

Venue Information

Name

Computer Vision and Pattern Recognition

Type

conference

URL

https://ieeexplore.ieee.org/xpl/conhome.jsp?punumber=1000147

Alternate Names

CVPR
Comput Vis Pattern Recognit

Abstract

Authors

References107 items

STEP: Segmenting and Tracking Every Pixel

HOTA: A Higher Order Metric for Evaluating Multi-object Tracking

Bidirectional Attention Network for Monocular Depth Estimation

Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking

ReMOTS: Self-Supervised Refining Multi-Object Tracking and Segmentation

Abstract

Authors

References107 items

STEP: Segmenting and Tracking Every Pixel

HOTA: A Higher Order Metric for Evaluating Multi-object Tracking

Bidirectional Attention Network for Monocular Depth Estimation

Chained-Tracker: Chaining Paired Attentive Regression Results for End-to-End Joint Multiple-Object Detection and Tracking

ReMOTS: Self-Supervised Refining Multi-Object Tracking and Segmentation

Segment as Points for Efficient Online Multi-Object Tracking and Segmentation

DetectoRS: Detecting Objects with Recursive Feature Pyramid and Switchable Atrous Convolution

Video Panoptic Segmentation

Leveraging Semi-Supervised Learning in Video Sequences for Urban Scene Segmentation

Bidirectional Graph Reasoning Network for Panoptic Segmentation

FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking

Pixel Consensus Voting for Panoptic Segmentation

Tracking Objects as Points

BANet: Bidirectional Aggregation Network With Occlusion Handling for Panoptic Segmentation

STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation

Towards Bounding-Box Free Panoptic Segmentation

MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning

Unifying Training and Inference for Panoptic Segmentation

Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching

Learning Multi-Object Tracking and Segmentation From Automatic Annotations

Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation

Track to Reconstruct and Reconstruct to Track

Towards Real-Time Multi-Object Tracking

AdaptIS: Adaptive Instance Selection Network

Adaptive Unimodal Cost Volume Filtering for Deep Stereo Matching

SSAP: Single-Shot Instance Segmentation With Affinity Pyramid

Enforcing Geometric Constraints of Virtual Normal for Depth Prediction

From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation

Learning Instance Occlusion for Panoptic Segmentation

Instance Segmentation by Jointly Optimizing Spatial Embeddings and Clustering Bandwidth

Soft Labels for Ordinal Regression

Seamless Scene Segmentation

Spatial-Temporal Relation Networks for Multi-Object Tracking

Objects as Points

GA-Net: Guided Aggregation Net for End-To-End Stereo Matching

SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences

Tracking Without Bells and Whistles

DeeperLab: Single-Shot Image Parser

MOTS: Multi-Object Tracking and Segmentation

UPSNet: A Unified Panoptic Segmentation Network

Attention-Guided Unified Network for Panoptic Segmentation

Learning to Fuse Things and Stuff

Integrated Object Detection and Tracking with Tracklet-Conditioned Detection

Affinity Derivation and Graph Merge for Instance Segmentation

Online Multi-Object Tracking with Dual Matching Attention Networks

Monocular Depth Estimation with Affinity, Vertical Pooling, and Label Enhancement

Box2Pix: Single-Shot Instance Segmentation by Assigning Pixels to Object Boxes

Digging Into Self-Supervised Monocular Depth Estimation

Deep Ordinal Regression Network for Monocular Depth Estimation

DenseASPP for Semantic Segmentation in Street Scenes

AutoAugment: Learning Augmentation Policies from Data

PAD-Net: Multi-tasks Guided Prediction-and-Distillation Network for Simultaneous Depth Estimation and Scene Parsing

GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose

Beyond Pixels: Leveraging Geometry and Shape Cues for Online Multi-Object Tracking

Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Panoptic Segmentation

Learning Depth from Monocular Videos Using Direct Methods

Recurrent Autoregressive Networks for Online Multi-object Tracking

Detect to Track and Track to Detect

The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes

Sparsity Invariant CNNs

Multiple People Tracking by Lifted Multicut and Person Re-identification

Deep Network Flow for Multi-object Tracking

Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics

Multi-scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation

Simple online and realtime tracking with a deep association metric

Mask R-CNN