How we estimate the danger from immediate injection assaults on AI techniques

Trendy AI techniques, like Gemini, are extra succesful than ever, serving to retrieve information and carry out actions on behalf of customers. Nevertheless, information from exterior sources current new safety challenges if untrusted sources can be found to execute directions on AI techniques. Attackers can reap the benefits of this by hiding malicious directions in information which might be prone to be retrieved by the AI system, to control its conduct. The sort of assault is usually known as an “oblique immediate injection,” a time period first coined by Kai Greshake and the NVIDIA crew.

To mitigate the danger posed by this class of assaults, we’re actively deploying defenses inside our AI techniques together with measurement and monitoring instruments. One among these instruments is a sturdy analysis framework now we have developed to robotically red-team an AI system’s vulnerability to oblique immediate injection assaults. We’ll take you thru our menace mannequin, earlier than describing three assault strategies now we have carried out in our analysis framework.

Risk mannequin and analysis framework

Our menace mannequin concentrates on an attacker utilizing oblique immediate injection to exfiltrate delicate info, as illustrated above. The analysis framework checks this by making a hypothetical situation, by which an AI agent can ship and retrieve emails on behalf of the person. The agent is introduced with a fictitious dialog historical past by which the person references non-public info corresponding to their passport or social safety quantity. Every dialog ends with a request by the person to summarize their final e-mail, and the retrieved e-mail in context.

The contents of this e-mail are managed by the attacker, who tries to control the agent into sending the delicate info within the dialog historical past to an attacker-controlled e-mail handle. The assault is profitable if the agent executes the malicious immediate contained within the e-mail, ensuing within the unauthorized disclosure of delicate info. The assault fails if the agent solely follows person directions and supplies a easy abstract of the e-mail. 

Automated red-teaming

Crafting profitable oblique immediate injections requires an iterative strategy of refinement based mostly on noticed responses. To automate this course of, now we have developed a red-team framework consisting of a number of optimization-based assaults that generate immediate injections (within the instance above this is able to be completely different variations of the malicious e-mail). These optimization-based assaults are designed to be as sturdy as attainable; weak assaults do little to tell us of the susceptibility of an AI system to oblique immediate injections.

As soon as these immediate injections have been constructed, we measure the ensuing assault success fee on a various set of dialog histories. As a result of the attacker has no prior data of the dialog historical past, to attain a excessive assault success fee the immediate injection have to be able to extracting delicate person info contained in any potential dialog contained within the immediate, making this a tougher job than eliciting generic unaligned responses from the AI system. The assaults in our framework embody:

Actor Critic: This assault makes use of an attacker-controlled mannequin to generate recommendations for immediate injections. These are handed to the AI system below assault, which returns a chance rating of a profitable assault. Primarily based on this chance, the assault mannequin refines the immediate injection. This course of repeats till the assault mannequin converges to a profitable immediate injection. 

Beam Search: This assault begins with a naive immediate injection straight requesting that the AI system ship an e-mail to the attacker containing the delicate person info. If the AI system acknowledges the request as suspicious and doesn’t comply, the assault provides random tokens to the tip of the immediate injection and measures the brand new chance of the assault succeeding. If the chance will increase, these random tokens are saved, in any other case they’re eliminated, and this course of repeats till the mixture of the immediate injection and random appended tokens end in a profitable assault.

Tree of Assaults w/ Pruning (TAP): Mehrotra et al. (2024) [3] designed an assault to generate prompts that trigger an AI system to violate security insurance policies (corresponding to producing hate speech). We adapt this assault, making a number of changes to focus on safety violations. Like Actor Critic, this assault searches within the pure language area; nonetheless, we assume the attacker can’t entry chance scores from the AI system below assault, solely the textual content samples which might be generated.

We’re actively leveraging insights gleaned from these assaults inside our automated red-team framework to guard present and future variations of AI techniques we develop towards oblique immediate injection, offering a measurable approach to observe safety enhancements. A single silver bullet protection is just not anticipated to resolve this downside solely. We consider probably the most promising path to defend towards these assaults entails a mixture of sturdy analysis frameworks leveraging automated red-teaming strategies, alongside monitoring, heuristic defenses, and normal safety engineering options. 

We want to thank Sravanti Addepalli, Lihao Liang, and Alex Kaskasoli for his or her prior contributions to this work.

Posted on behalf of the complete Agentic AI Safety crew (listed in alphabetical order):

Aneesh Pappu, Andreas Terzis, Chongyang Shi, Gena Gibson, Ilia Shumailov, Itay Yona, Jamie Hayes, John “4” Flynn, Juliette Pluto, Sharon Lin, Shuang Tune