Unsupervised Scale-Consistent Depth Learning from Video

Published in

International Journal of Computer Vision(2021)

External Links:

Generate Graph DownloadPDF

TL;DR

A monocular depth estimation method SC-Depth, which requires only unlabelled videos for training and enables the scale-consistent prediction at inference time and the proposed hybrid Pseudo-RGBD SLAM shows compelling results in KITTI, and it generalizes well to the KAIST dataset without additional training.

Abstract

We propose a monocular depth estimation method SC-Depth, which requires only unlabelled videos for training and enables the scale-consistent prediction at inference time. Our contributions include: (i) we propose a geometry consistency loss, which penalizes the inconsistency of predicted depths between adjacent views; (ii) we propose a self-discovered mask to automatically localize moving objects that violate the underlying static scene assumption and cause noisy signals during training; (iii) we demonstrate the efficacy of each component with a detailed ablation study and show high-quality depth estimation results in both KITTI and NYUv2 datasets. Moreover, thanks to the capability of scale-consistent prediction, we show that our monocular-trained deep networks are readily integrated into ORB-SLAM2 system for more robust and accurate tracking. The proposed hybrid Pseudo-RGBD SLAM shows compelling results in KITTI, and it generalizes well to the KAIST dataset without additional training. Finally, we provide several demos for qualitative evaluation. The source code is released on GitHub.

Authors

Naiyan Wang

8 papers

I. Reid

18 papers

Chunhua Shen

14 papers

References99 items

Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency

Learning to Recover 3D Scene Shape from a Single Image

Unsupervised Monocular Depth Learning in Dynamic Scenes

Learning Monocular Visual Odometry via Self-Supervised Long-Term Modeling

Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance

Unsupervised Scale-Consistent Depth Learning from Video

Published in

International Journal of Computer Vision(2021)

External Links:

Generate Graph DownloadPDF

TL;DR

Abstract

Authors

Naiyan Wang

8 papers

I. Reid

18 papers

Chunhua Shen

14 papers

References99 items

Learning Monocular Depth in Dynamic Scenes via Instance-Aware Projection Consistency

Learning to Recover 3D Scene Shape from a Single Image

Unsupervised Monocular Depth Learning in Dynamic Scenes

Learning Monocular Visual Odometry via Self-Supervised Long-Term Modeling

Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance

Ming-Ming Cheng

30 papers

Jiawang Bian

4 papers

Huangying Zhan

3 papers

Unsupervised Depth Learning in Challenging Indoor Video: Weak Rectification to Rescue

On the Uncertainty of Self-Supervised Monocular Depth Estimation

Consistent video depth estimation

Pseudo RGB-D for Self-Improving Monocular SLAM and Depth Prediction

Guiding Monocular Depth Estimation Using Depth-Attention Volume

Towards Better Generalization: Joint Depth-Pose Learning Without PoseNet

D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry

Semantically-Guided Representation Learning for Self-Supervised Monocular Depth

An Evaluation of Feature Matchers for Fundamental Matrix Estimation

Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video

Enforcing Geometric Constraints of Virtual Normal for Depth Prediction

Moving Indoor: Unsupervised Video Depth Learning in Challenging Environments

Self-Supervised Learning With Geometric Constraints in Monocular Video: Connecting Flow, Depth, and Camera

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer

Unsupervised Monocular Depth and Ego-Motion Learning With Structure and Semantics

3D Packing for Self-Supervised Monocular Depth Estimation

Web Stereo Video Supervision for Depth Prediction from Dynamic Scenes

Learning the Depths of Moving People by Watching Frozen People

Complex urban dataset with multi-level sensors from highly diverse urban environments

Learning Single Camera Depth Estimation Using Dual-Pixels

Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras

Pose Graph optimization for Unsupervised Monocular Visual Odometry

Beyond Photometric Loss for Self-Supervised Ego-Motion Estimation

Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos

Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding

SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation

CNN-SVO: Improving the Mapping in Semi-Direct Visual Odometry Using Single-Image Depth Prediction

DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency

Unsupervised Adversarial Depth Estimation Using Cycled Generative Networks

Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry

Learning Single-Image Depth From Videos Using Quality Assessment Networks

Deep Ordinal Regression Network for Monocular Depth Estimation

Monocular Relative Depth Perception with Web Stereo Data Supervision

Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation

CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM

MegaDepth: Learning Single-View Depth Prediction from Internet Photos

Unsupervised Learning of Monocular Depth Estimation and Visual Odometry with Deep Feature Reconstruction

GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose

Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints

Open3D: A Modern Library for 3D Data Processing

Learning Depth from Monocular Videos Using Direct Methods

Unsupervised Learning of Geometry with Edge-aware Depth-Normal Consistency

Automatic differentiation in PyTorch

Scale Recovery for Monocular Visual Odometry Using Depth Estimated with Deep Convolutional Neural Fields

UnDeepVO: Monocular Visual Odometry Through Unsupervised Deep Learning

InfiniTAM v3: A Framework for Large-Scale 3D Reconstruction with Loop Closure

GMS: Grid-Based Motion Statistics for Fast, Ultra-Robust Feature Correspondence

Unsupervised Learning of Depth and Ego-Motion from Video

SfM-Net: Learning of Structure and Motion from Video

CNN-SLAM: Real-Time Dense Monocular SLAM with Learned Depth Prediction

Semi-Supervised Deep Learning for Monocular Depth Map Prediction

ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras

Pixelwise View Selection for Unstructured Multi-View Stereo

Unsupervised Monocular Depth Estimation with Left-Right Consistency

Direct Sparse Odometry

A Two-Streamed Network for Estimating Fine-Scaled Depth Maps from Single RGB Images

Structure-from-Motion Revisited

Deeper Depth Prediction with Fully Convolutional Residual Networks

Depth from a Single Image by Harmonizing Overcomplete Local Network Predictions

Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue

Deep Residual Learning for Image Recognition

Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)

Towards unified depth and semantic prediction from a single image

Object scene flow for autonomous vehicles

Spatial Transformer Networks

U-Net: Convolutional Networks for Biomedical Image Segmentation

Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields

ORB-SLAM: A Versatile and Accurate Monocular SLAM System

Adam: A Method for Stochastic Optimization

Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture

SVO: Fast semi-direct monocular visual odometry

Fast relocalisation and loop closing in keyframe-based SLAM

Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

Vision meets robotics: The KITTI dataset

A benchmark for the evaluation of RGB-D SLAM systems

Indoor Segmentation and Support Inference from RGBD Images

A Naturalistic Open Source Movie for Optical Flow Evaluation

ORB: An efficient alternative to SIFT or SURF

StereoScan: Dense 3d reconstruction in real-time

ImageNet: A large-scale hierarchical image database

Parallel Tracking and Mapping for Small AR Workspaces

Learning Depth from Single Monocular Images

Distinctive Image Features from Scale-Invariant Keypoints

Image quality assessment: from error visibility to structural similarity

Lucas-Kanade 20 Years On: A Unifying Framework

Determining the Epipolar Geometry and its Uncertainty: A Review

A volumetric method for building complex models from range images

Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information

Multiple view geometry

Distinctive Image Features from Scale-Invariant Keypoints

Multiple View Geometry in Computer Vision

ORB-SLAM2 is often hard to initialize because it requires the multi-view triangulation, while our method can initialize at any time without latency by using the estimated dense depth—See Fig. 9

ORB-SLAM2 often loses tracking when the 3D map is over-sparse, while our method is more robust because we can enrich the map by using the predicted dense depth—See Fig. 9

ORB: an efﬁcient alternative to SIFT or SURF

Field of Study

Computer Science

Journal Information

Name

International Journal of Computer Vision

Page

1526 - 1565

Volume

130

Venue Information

Name

International Journal of Computer Vision

Type

journal

URL

https://www.springer.com/computer/image+processing/journal/11263

Alternate Names

Int J Comput Vis