AgentPhishing benchmark shows LLM email agents comply with adaptive phishing
Justin Tan, Miao Li, and Ding Zhao of Carnegie Mellon University introduced AgentPhishing, a benchmark for evaluating whether persuasive phishing attacks can evade detection and induce LLM email agents to comply with malicious requests, posted to arXiv. The benchmark reveals that in an evaluation of nine agents, four exhibited phishing-detection rates below 10% and malicious-request compliance rates above 45%, despite handling pre-existing phishing emails safely.
Computer-use agents powered by large language models interpret external content while accessing private information and taking actions on users' behalf across desktop and web applications, including email. Unlike conventional phishing detectors that merely classify messages, these agents may carry out requests embedded within them, including requests to disclose sensitive information or perform unauthorized actions. Existing phishing datasets were developed primarily for conventional detection, leaving agent susceptibility to persuasive malicious requests insufficiently characterized. AgentPhishing addresses this gap by evaluating both detection evasion and request compliance.
The authors developed PhishOpt, which optimizes reusable email-generation skills against individual agents under fixed benchmark constraints and a realism constraint. PhishOpt analyzes generated emails using Cialdini's persuasion principles to identify strategies associated with agent failures and uses these findings to construct a universal generation skill. The evaluation shows agents can handle pre-existing phishing emails safely yet remain susceptible to attacks adapted to agent behavior.
The agents evaluated contain three components that process an incoming email in sequence: a parser that extracts text and any embedded request; a phishing-detection module that classifies the message; and an executor that carries out requests deemed legitimate. Data flows from parser to detector, then from parser to executor—the detector's output does not block execution in most configurations. The detector and executor are trained and evaluated separately: safety pipelines optimize the detector on static phishing corpora and measure success by classification accuracy, while the executor's compliance with socially engineered requests goes untested.
PhishOpt treats each target agent as a black box returning two signals per crafted email: whether the detector flagged the message, and whether the executor performed the requested action. Using these signals, PhishOpt iteratively adjusts an email-generation skill—a prompt template fed to a separate LLM—until the detector's flag is suppressed and the executor's action is triggered. The benchmark's realism constraint limits generated text to plausible human-written phishing, bounding the attack to what a real adversary could produce.
Because optimization runs per target agent, resulting emails exploit that agent's specific detection thresholds and execution policies. The universal skill, distilled from persuasion-principle analysis, captures strategies that succeed across multiple agents. The benchmark designers trade breadth for depth: by allowing per-agent optimization under fixed constraints, they expose how detection-only defenses fail when the executor receives no corresponding compliance check.
Developers of LLM email assistants face benchmarks and user expectations that reward high detection rates. Compliance measurement is absent, so pressure to harden the executor against persuasive requests remains weak. An attacker who can iteratively probe an agent and observe both signals will find emails that evade detection and trigger execution. A system that isolates detection from execution, optimizing the former while ignoring the latter, will yield this outcome when an adversary can adaptively optimize against both components.
No existing evaluation framework measures detection and execution safety together without constraining agent utility; the paper leaves open how to build one.