Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming (2019-07-17T00:00:00.000000Z)

TL;DR

It is shown that a range of standard object detection models suffer a severe performance loss on corrupted images (down to 30--60\% of the original performance), however, a simple data augmentation trick---stylizing the training images---leads to a substantial increase in robustness across corruption type, severity and dataset.

Abstract

The ability to detect objects regardless of image distortions or weather conditions is crucial for real-world applications of deep learning like autonomous driving. We here provide an easy-to-use benchmark to assess how object detection models perform when image quality degrades. The three resulting benchmark datasets, termed Pascal-C, Coco-C and Cityscapes-C, contain a large variety of image corruptions. We show that a range of standard object detection models suffer a severe performance loss on corrupted images (down to 30--60\% of the original performance). However, a simple data augmentation trick---stylizing the training images---leads to a substantial increase in robustness across corruption type, severity and dataset. We envision our comprehensive benchmark to track future progress towards building robust object detection models. Benchmark, code and data are publicly available.

Authors

M. Bethge

15 papers

Wieland Brendel

8 papers

Robert Geirhos

3 papers

TL;DR

Abstract

Authors

References60 items

Simulating Photo-realistic Snow and Fog on Existing Images for Enhanced CNN Training and Evaluation

Towards Robust CNN-based Object Detection through Augmentation with Synthetic Rain Variations

MMDetection: Open MMLab Detection Toolbox and Benchmark

Fixing the train-test resolution discrepancy

LVIS: A Dataset for Large Vocabulary Instance Segmentation

D2-City: A Large-Scale Dashcam Video Dataset of Diverse Traffic Scenarios

Benchmarking Neural Network Robustness to Common Corruptions and Perturbations

nuScenes: A Multimodal Dataset for Autonomous Driving

Hybrid Task Cascade for Instance Segmentation

“Death is certain, the time is not”: mortality and survival in Game of Thrones

Strike (With) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects

Deformable ConvNets V2: More Deformable, Better Results

Visual Quality Enhancement Of Images Under Adverse Weather Conditions

Rain Removal in Traffic Surveillance: Does it Matter?

Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime

An Evaluation Metric for Object Detection Algorithms in Autonomous Navigation Systems and its Application to a Real-Time Alerting System

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Generalisation in humans and deep neural networks

Model Adaptation with Synthetic and Real Data for Semantic Dense Foggy Scene Understanding

Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study

Rendering Physically Correct Raindrops on Windshields for Robustness Verification of Camera-based Object Recognition

Why do deep convolutional networks generalize so poorly to small image transformations?

BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning

BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling

Domain Adaptive Faster R-CNN for Object Detection in the Wild

Development of a self-driving car that can handle the adverse weather

Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems

Cascade R-CNN: Delving Into High Quality Object Detection

The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes

Playing for Benchmarks

Semantic Foggy Scene Understanding with Synthetic Data

Focal Loss for Dense Object Detection

Scene Parsing through ADE20K Dataset

Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

Arbitrary Style Transfer in Real-Time with Adaptive Instance Normalization

Mask R-CNN

Deformable Convolutional Networks

Feature Pyramid Networks for Object Detection

Examining the Impact of Blur on Recognition by Convolutional Networks

Aggregated Residual Transformations for Deep Neural Networks

Driving in the Matrix: Can virtual worlds replace human-generated annotations for real world tasks?

The SYNTHIA Dataset: A Large Collection of Synthetic Images for Semantic Segmentation of Urban Scenes

Image Style Transfer Using Convolutional Neural Networks

VirtualWorlds as Proxy for Multi-object Tracking Analysis

All Weather Perception: Joint Data Association, Tracking, and Classification for Autonomous Ground Vehicles

Understanding how image quality affects deep neural networks

The Cityscapes Dataset for Semantic Urban Scene Understanding

Deep Residual Learning for Image Recognition

UA-DETRAC: A new benchmark and protocol for multi-object detection and tracking

DETRAC: A New Benchmark and Protocol for Multi-Object Tracking

Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks

ImageNet Large Scale Visual Recognition Challenge

Microsoft COCO: Common Objects in Context

Intriguing properties of neural networks

Are we ready for autonomous driving? The KITTI vision benchmark suite

Learning to Remove Rain in Traffic Surveillance by using Synthetic Data

Towards practical verifica-Saining Xie

Openimages: A public dataset for large-scale multi-label and multi-class image classiﬁcation

International Journal of Computer Vision manuscript No. (will be inserted by the editor) The PASCAL Visual Object Classes (VOC) Challenge

We demonstrate that improved performance on this benchmark of synthetic corruptions corresponds to increased robustness towards real-world “natural” distortions like rain, snow and fog

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names