MUAD: Multiple Uncertainties for Autonomous Driving, a benchmark for multiple uncertainty types and tasks (2022-03-02T00:00:00.000000Z)

TL;DR

The MUAD dataset (Multiple Uncertainties for Autonomous Driving), consisting of 10,413 realistic synthetic images with diverse adverse weather conditions, allows to better assess the impact of different sources of uncertainty on model performance and is released to allow researchers to benchmark their algorithm methodically in adverse conditions.

Abstract

Predictive uncertainty estimation is essential for safe deployment of Deep Neural Networks in real-world autonomous systems. However, disentangling the different types and sources of uncertainty is non trivial for most datasets, especially since there is no ground truth for uncertainty. In addition, while adverse weather conditions of varying intensities can disrupt neural network predictions, they are usually under-represented in both training and test sets in public datasets.We attempt to mitigate these setbacks and introduce the MUAD dataset (Multiple Uncertainties for Autonomous Driving), consisting of 10,413 realistic synthetic images with diverse adverse weather conditions (night, fog, rain, snow), out-of-distribution objects, and annotations for semantic segmentation, depth estimation, object, and instance detection. MUAD allows to better assess the impact of different sources of uncertainty on model performance. We conduct a thorough experimental study of this impact on several baseline Deep Neural Networks across multiple tasks, and release our dataset to allow researchers to benchmark their algorithm methodically in adverse conditions. More visualizations and the download link for MUAD are available at https://muad-dataset.github.io/.

Authors

G. Franchi

4 papers

Xuanlong Yu

2 papers

Andrei Bursuc

3 papers

TL;DR

Abstract

Authors

References81 items

SLURP: Side Learning Uncertainty for Regression Problems

Self-Supervised Monocular Depth Estimation with Internal Feature Fusion

ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding

Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation

Robust Semantic Segmentation with Superpixel-Mix

A survey of uncertainty in deep neural networks

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers

SegmentMeIfYouCan: A Benchmark for Anomaly Segmentation

Exploiting epistemic uncertainty of the deep learning models to generate adversarial samples

Encoding the Latent Posterior of Bayesian Neural Networks for Uncertainty Quantification

Rain Rendering for Evaluating and Improving Robustness to Bad Weather

One Thousand and One Hours: Self-driving Motion Prediction Dataset

Map-Guided Curriculum Domain Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation

Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors

On the Uncertainty of Self-Supervised Monocular Depth Estimation

Calibrating Deep Neural Networks using Focal Loss

Bayesian Deep Learning and a Probabilistic Perspective of Generalization

BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning

Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning

TRADI: Tracking deep neural network weight distributions

Scalability in Perception for Autonomous Driving: Waymo Open Dataset

A Benchmark for Anomaly Segmentation

Simulating Photo-realistic Snow and Fog on Existing Images for Enhanced CNN Training and Evaluation

Fishyscapes: A Benchmark for Safe Semantic Segmentation in Autonomous Driving

From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation

Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift

Argoverse: 3D Tracking and Forecasting With Rich Maps

How Do Neural Networks See Depth in Single Images?

Gaussian YOLOv3: An Accurate and Fast Object Detector Using Localization Uncertainty for Autonomous Driving

nuScenes: A Multimodal Dataset for Autonomous Driving

A Simple Baseline for Bayesian Uncertainty in Deep Learning

Curriculum Model Adaptation with Synthetic and Real Data for Semantic Foggy Scene Understanding

Probabilistic Object Detection: Definition and Evaluation

IDD: A Dataset for Exploring Problems of Autonomous Navigation in Unconstrained Environments

Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime

WildDash - Creating Hazard-Aware Benchmarks

Toward Driving Scene Understanding: A Dataset for Learning Driver Behavior and Causal Reasoning

BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning

Uncertainty Estimates and Multi-hypotheses Networks for Optical Flow

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Photo-Realistic Simulation of Road Scene for Data-Driven Methods in Bad Weather

The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes

Semantic Foggy Scene Understanding with Synthetic Data

The Raincouver Scene Parsing Benchmark for Self-Driving in Adverse Weather and at Night

On Calibration of Modern Neural Networks

No More Discrimination: Cross City Adaptation of Road Scene Segmenters

What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?

Feature Pyramid Networks for Object Detection

Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles

A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks

Lost and Found: detecting small road hazards for self-driving vehicles

Unsupervised Monocular Depth Estimation with Left-Right Consistency

Playing for Data: Ground Truth from Computer Games

The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes

DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs

VirtualWorlds as Proxy for Multi-object Tracking Analysis

The Cityscapes Dataset for Semantic Urban Scene Understanding

Deep Residual Learning for Image Recognition

Weight Uncertainty in Neural Network

Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

Digital Representations of the Real World - How to Capture, Model, and Render Visual Reality

Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks

Depth Map Prediction from a Single Image using a Multi-Scale Deep Network

Microsoft COCO: Common Objects in Context

ImageNet classification with deep convolutional neural networks

Are we ready for autonomous driving? The KITTI vision benchmark suite

Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management

Color image enhancement through 3-D histogram equalization

Measurement of snowflake size and falling velocity by image processing

THE DISTRIBUTION OF THE FLORA IN THE ALPINE ZONE.1

NeW CRFs: Neural window fully-connected CRFs for monocular depth estimation

Deterministic Neural Networks with Appropriate Inductive Biases Capture Epistemic and Aleatoric Uncertainty

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

baranidesign

Virtual arXiv:2001