Is RobustBench/AutoAttack a suitable Benchmark for Adversarial Robustness? (2021-12-02T00:00:00.000000Z)

TL;DR

It is argued that the alternation of data by AutoAttack with l-inf, eps = 8/255 is unrealistically strong, resulting in close to perfect detection rates of adversarial samples even by simple detection algorithms and human observers, and that other attack methods are much harder to detect while achieving similar success rates.

Abstract

Recently, RobustBench (Croce et al. 2020) has become a widely recognized benchmark for the adversarial robustness of image classification networks. In its most commonly reported sub-task, RobustBench evaluates and ranks the adversarial robustness of trained neural networks on CIFAR10 under AutoAttack (Croce and Hein 2020b) with l-inf perturbations limited to eps = 8/255. With leading scores of the currently best performing models of around 60% of the baseline, it is fair to characterize this benchmark to be quite challenging. Despite its general acceptance in recent literature, we aim to foster discussion about the suitability of RobustBench as a key indicator for robustness which could be generalized to practical applications. Our line of argumentation against this is two-fold and supported by excessive experiments presented in this paper: We argue that I) the alternation of data by AutoAttack with l-inf, eps = 8/255 is unrealistically strong, resulting in close to perfect detection rates of adversarial samples even by simple detection algorithms and human observers. We also show that other attack methods are much harder to detect while achieving similar success rates. II) That results on low-resolution data sets like CIFAR10 do not generalize well to higher resolution images as gradient-based attacks appear to become even more detectable with increasing resolutions.

Authors

Janis Keuper

4 papers

P. Lorenz

2 papers

Dominik Strassel

1 papers

Is RobustBench/AutoAttack a suitable Benchmark for Adversarial Robustness?

TL;DR

Abstract

Authors

References17 items

Detecting AutoAttack Perturbations in the Frequency Domain

SpectralDefense: Detecting Adversarial Attacks on CNNs in the Fourier Domain

RobustBench: a standardized adversarial robustness benchmark

Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks

Square Attack: a query-efficient black-box adversarial attack via random search

Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attack

advertorch v0.1: An Adversarial Robustness Toolbox based on PyTorch

A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets

Towards Deep Learning Models Resistant to Adversarial Attacks

Towards Evaluating the Robustness of Neural Networks

Adversarial examples in the physical world

Wide Residual Networks

DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks

Explaining and Harnessing Adversarial Examples

Deep Learning Face Attributes in the Wild

An algorithm for the machine calculation of complex Fourier series

Foolbox: A Python toolbox to benchmark the robustness of machine learning models

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type

URL

Alternate Names