Visual Prompting for Adversarial Robustness (2022-10-12T00:00:00.000000Z)

TL;DR

This work proposes a new VP method, termed Class-wise Adversarial Visual Prompting (C-AVP), to generate class-wise visual prompts so as to not only leverage the strengths of ensemble prompts but also optimize their interrelations to improve model robustness.

Abstract

In this work, we leverage visual prompting (VP) to improve adversarial robustness of a fixed, pre-trained model at test time. Compared to conventional adversarial defenses, VP allows us to design universal (i.e., data-agnostic) input prompting templates, which have plug-and-play capabilities at test time to achieve desired model performance without introducing much computation overhead. Although VP has been successfully applied to improving model generalization, it remains elusive whether and how it can be used to defend against adversarial attacks. We investigate this problem and show that the vanilla VP approach is not effective in adversarial defense since a universal input prompt lacks the capacity for robust learning against sample-specific adversarial perturbations. To circumvent it, we propose a new VP method, termed Class-wise Adversarial Visual Prompting (C-AVP), to generate class-wise visual prompts so as to not only leverage the strengths of ensemble prompts but also optimize their interrelations to improve model robustness. Our experiments show that C-AVP outperforms the conventional VP method, with 2.1× standard accuracy gain and 2× robust accuracy gain. Compared to classical test-time defenses, C-AVP also yields a 42× inference time speedup. Code is available at https://github.com/Phoveran/vp-for-adversarial-robustness.

Authors

Yuguang Yao

3 papers

Pin-Yu Chen

17 papers

P. Lorenz

2 papers

TL;DR

Abstract

Authors

References50 items

Text-Visual Prompting for Efficient 2D Temporal Video Grounding

ClawSAT: Towards Both Robust and Accurate Code Models

Fairness Reprogramming

How to Robustify Black-Box ML Models? A Zeroth-Order Optimization Perspective

Reverse Engineering of Imperceptible Adversarial Image Perturbations

Evaluating the Adversarial Robustness of Adaptive Test-time Defenses

Model Reprogramming: Resource-Efficient Cross-Domain Machine Learning

When Does Contrastive Learning Preserve Adversarial Robustness from Pretraining to Finetuning?

Stable Neural ODE with Lyapunov-Stable Equilibrium Points for Defending Against Adversarial Attacks

Why Adversarial Reprogramming Works, When It Fails, and How to Tell the Difference

Voice2Series: Reprogramming Acoustic Models for Time Series Classification

Adversarial purification with Score-based generative models

Combating Adversaries with Anti-Adversaries

Adversarial Attacks are Reversible with Natural Supervision

Learning Transferable Visual Models From Natural Language Supervision

On Fast Adversarial Robustness Adaptation in Model-Agnostic Meta-Learning

Cross-modal Adversarial Reprogramming

Towards Robust Neural Networks via Close-loop Control

Unadversarial Examples: Designing Objects for Robust Vision

Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources

Proper Network Interpretability Helps Adversarial Robustness in Classification

Adversarial Examples Detection and Analysis with Layer-wise Autoencoders

Language Models are Few-Shot Learners

Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning

Denoised Smoothing: A Provable Defense for Pretrained Classifiers

Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks

Rethinking Randomized Smoothing for Adversarial Robustness

ML-LOO: Detecting Adversarial Examples with Feature Attribution

Unlabeled Data Improves Adversarial Robustness

You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle

Adversarial Training for Free!

Adversarial Robustness vs. Model Compression, or Both?

Theoretically Principled Trade-off between Robustness and Accuracy

Feature Denoising for Improving Adversarial Robustness

Adversarial Reprogramming of Neural Networks

Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

Certified Defenses against Adversarial Examples

Provable defenses against adversarial examples via the convex outer adversarial polytope

BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain

Towards Deep Learning Models Resistant to Adversarial Attacks

MagNet: A Two-Pronged Defense against Adversarial Examples

On the (Statistical) Detection of Adversarial Examples

On Detecting Adversarial Perturbations

Towards Evaluating the Robustness of Neural Networks

Explaining and Harnessing Adversarial Examples

Visual Prompting: Modifying Pixel Space to Adapt Pre-trained Models

Prefix-Tuning: Optimizing Continuous Prompts for Generation

Online Adversarial Purification based on Self-Supervision

Voice2Series: Reprogramming Acoustic Models for Time Series Classification

Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names