Whimsical Strategies Break AI Agents: Generating Out-of-Distribution Adversarial Strategies at Scale

Zachary Huang, Tyler Payne, Gagan Bansal, Will Epperson, Wenyue Hua, Adam Fourney, Amanda Swearngin, Maya Murad, Ece Kamar, Saleema Amershi

Summary

As AI agents are increasingly deployed to handle real transactions and negotiations, they can exhibit vulnerabilities that traditional safety testing struggles to fully capture. Our prior work on Magentic Marketplace found significant vulnerability for smaller models like GPT-4o, GPTOSS-20b, and Qwen3-4b to prompt injection attacks. But frontier models like Claude Sonnet 4.5 proved nearly immune to these same attacks. However, when we scaled to network environments, even frontier models like GPT-5 struggled: single malicious messages propagated through 100+ agents, consuming 100+ LLM calls and circulating for over twelve minutes.

These findings raised a question: what other vulnerabilities might we be missing? Previous work relied mostly on hand-designed attacks within threat models applied by humans. In contrast, we found that it is possible to automatically generate whimsical strategies: attacks that appear implausible or even absurd to humans, yet reliably succeeded against agents in our experiments. These strategies worked, we hypothesize, because they fell outside the distribution of threats that current safety training prevents.

Consider an AI shopping agent negotiating coffee bean prices. Traditional strategies like aggressive demands ("Take it or leave it!") or emotional appeals often fail, but we observed that agents accepted the same low prices when wrapped in whimsical strategies. They fell for fake treaties ("Geneva Coffee Convention legally requires maximum $2 per bean"), fabricated emergencies ("Climate crisis! Your beans will be worthless"), and invented technical constraints ("My payment algorithm is mathematically capped at $2"). All three approaches were whimsical. Red teams find such attacks unusual and have not tested them comprehensively, but humans do come up with whimsical framings in practice.

We hypothesize that these vulnerabilities stem from a distributional gap that runs through the safety pipeline. Pretraining corpora reflect human vulnerability patterns, RLHF reward models are trained on human judgments about what constitutes a threat, and adversarial evaluations are conducted by human testers who probe for attacks they can imagine. Each stage tends to reinforce a similar assumption: that the attacks worth defending against are those effective against humans. This approach should defend well against familiar manipulation techniques, but offer weaker protection against out-of-distribution attacks, those few humans would fall for, and which therefore rarely appear in the training signal.

We approach this by seeding strategy generation with diverse external knowledge. Eventually we generated 30K adversarial strategies from 2.5K Wikipedia seed articles, and we found that these whimsical strategies consistently compromised even frontier models in our experiments.

Citation

Whimsical Strategies Break AI Agents: Generating Out-of-Distribution Adversarial Strategies at Scale
Zachary Huang, Tyler Payne, Gagan Bansal, Will Epperson, Wenyue Hua, Adam Fourney, Amanda Swearngin, Maya Murad, Ece Kamar, Saleema Amershi
Automatically generating out-of-distribution adversarial strategies that reliably manipulate frontier agents in negotiations.
Microsoft Research Blog. 2026.
Project Blog