Ignore Previous Prompt: Attack Techniques For Language Models
TL
TL;DR
This work investigates two types of attacks -- goal hijacking and prompt leaking -- and demonstrates that even low-aptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3's stochastic nature, creating long-tail risks.
Abstract
Transformer-based large language models (LLMs) provide a powerful foundation for natural language tasks in large-scale customer-facing applications. However, studies that explore their vulnerabilities emerging from malicious user interaction are scarce. By proposing PromptInject, a prosaic alignment framework for mask-based iterative adversarial prompt composition, we examine how GPT-3, the most widely deployed language model in production, can be easily misaligned by simple handcrafted inputs. In particular, we investigate two types of attacks -- goal hijacking and prompt leaking -- and demonstrate that even low-aptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3's stochastic nature, creating long-tail risks. The code for PromptInject is available at https://github.com/agencyenterprise/PromptInject.
References42 items
1
A Case Report On The "A.I. Locked-In Problem": social concerns with modern NLP
2
Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examples
3
A Holistic Approach to Undesired Content Detection in the Real World
4
X-Risk Analysis for AI Research
5