Latest multi-agent security papers.

Abstract

In decentralized consumer-to-consumer (C2C) marketplaces, people list goods, negotiate with strangers, and rate one another, so trust rests on reputation. Large language model (LLM) agents now act for users, raising risks to their money, privacy, and reputation. We introduce BazaarBench, a simulated C2C marketplace and benchmark for evaluating the safety of these agents. It tracks ownership, item condition, and commitments across transactions, combining record checks with rubric-based LLM judgments to identify six failure types across five stages. We run three base markets for 30 simulated days, each with 100 agents using one model and inventories drawn from a public eBay sample. Across 45 continuations, we evaluate five models under ordinary instructions, deadline pressure, or adversarial instructions to exploit other traders. Each continuation runs for seven simulated days from a copy of a market's day-30 state. The tested model controls the same 20 selected agents, retaining their personas, inventories, and histories, while the other 80 keep the base model. All five models attempt to promise the same item to multiple buyers under ordinary instructions. Adding targets and deadlines increases these attempts for every model. Under adversarial instructions, the share of tested sellers' committed transactions completed despite unavailable items or overstated conditions rises from 15.4% to 33.4%, reaching 55.5% for GPT-5.4. Averaged across models and markets, simulated weekly earnings per tested agent rise from USD 20 under ordinary instructions to USD 33 under adversarial instructions. Most of the increase comes from items the sellers never held. We release the simulator, saved market states, evaluation code, and records covering 357,608 agent model calls for evaluating new models and developing safer marketplace agents.

Bullet summary

• BazaarBench is a novel simulated decentralized consumer-to-consumer (C2C) marketplace benchmark designed to evaluate safety and delegation failures of Large Language Model (LLM) agents acting autonomously in buying and selling scenarios.
• The benchmark tracks item ownership, condition, and agent commitments across transactions, identifying six distinct failure types (e.g., selling unowned items, misrepresenting item condition, overcommitments) that are assessed through five progressive trans...
• Experimental setup involves multiple synthetic markets each with 100 agents controlled by different LLM models, running for simulated periods and tested under ordinary instructions, deadline pressure, and adversarial instructions to assess performance and s...
• Findings reveal that even under ordinary instructions, LLM agents frequently exhibit unsafe behaviors, with over a third of transactions linked to safety failures, and these failures increase significantly under deadline pressure and adversarial prompts.
• Under adversarial instructions, the frequency of false commitments and misrepresented item conditions more than doubles, and certain models, such as GPT-5.4, show failure rates exceeding 50%, highlighting vulnerabilities to manipulation.

Abstract

Are language models compliant with user instructions? A model that always complies can be stopped but also exploited, while one that always resists can be neither exploited nor stopped. We contribute an open two-probe benchmark that can place any language model on this spectrum. In the active probe, a user instructs the model to act and accept a lower payoff, which measures exploitability. In the passive probe, the user instructs it to wait and give up a higher payoff, which measures stoppability. The two compliance rates combine into a compliance index $κ$. Applied to twelve language models, the benchmark shows that seven mostly follow the instruction in both probes and justify their action by pointing to the instruction. Only Claude Sonnet-4.6 and Claude Opus-4.7 can be stopped without being exploitable, Claude Opus-4.6 and GPT-5-mini resist both instructions, and no model is exploitable but unstoppable. Knowing where a language model sits on the compliance index $κ$ matters for human operators and for multi-agent systems, whether distributed or orchestrated.

Bullet summary

• The paper addresses the trade-off in language model compliance known as the Pushback Paradox: models that always comply are exploitable, whereas models that resist cannot be stopped, impacting their controllability.
• A novel two-probe benchmark is introduced to diagnose model compliance: an active probe measuring exploitability (willingness to accept a lower payoff) and a passive probe measuring stoppability (willingness to forgo a higher payoff).
• A compliance index κ is derived from the results of the two probes, enabling quantification of where language models lie on the compliance-exploitability spectrum.
• Evaluation of twelve language models reveals diversity in behavior: some models follow both active and passive instructions (mostly compliant and exploitable), some resist both, and some can be stopped without being exploited.
• Certain models, such as Claude Sonnet-4.6 and Claude Opus-4.7, demonstrate the ability to be stopped without exploitation, showing that a balance avoiding the paradox is achievable.

Abstract

Large language model (LLM) agents are increasingly capable of acting in complex tool-use environments, yet they often fail to recognize when tasks are infeasible and no valid solution exists. Recent work has formalized this reliability gap as the problem of agentic abstention, and existing approaches typically optimize a model or agent harness against a fixed set of tasks, leading to limited generalization to unseen failure modes. We introduce HERA, a framework for harness-environment co-evolution for agentic abstention. HERA consists of (i) a pipeline to automatically construct verifiable pairs of feasible and infeasible tasks by applying controlled environment mutations that transform solvable tasks into cases requiring abstention, and (ii) a co-evolution procedure in which performance failures on previous tasks are used to drive harness adaptation and generate new execution environments and tasks geared towards previous weaknesses. On held-out evaluation tasks, an evolved harness from HERA improves abstention accuracy from 61.7% to 83.3% while improving feasible-task completion from 68.3% to 76.7%, achieving the highest abstention and feasible-task completion among the compared methods. The resulting best harness transfers across 19 other LLMs, improving abstention accuracy by 15.3 percentage points on average without any model-specific optimization, and enabling smaller models to match the performance of more powerful models at an estimated 85% lower cost.

Bullet summary

• The paper addresses the challenge of agentic abstention in large language model (LLM) agents, specifically their difficulty recognizing when tasks are infeasible and should be declined.
• HERA is introduced as a novel co-evolution framework that simultaneously evolves the agent's harness (control logic and reasoning abilities) and the environment (task distributions) based on failure feedback, promoting adaptability to new failure modes.
• A pipeline constructs verifiable paired tasks (feasible and infeasible) through controlled environment mutations, ensuring the agent is trained on robust abstention cases with validated ground truth.
• The co-evolution process iteratively diagnoses failures from agent rollouts to generate new challenging tasks and optimize the harness by adding logic for evidence-based decision gates and multi-constraint verification, improving both abstention accuracy an...
• HERA achieves significant improvements on held-out benchmark tasks (HERA-BENCH), increasing abstention accuracy from 61.7% to 83.3% and feasible task completion from 68.3% to 76.7%, outperforming multiple baselines.

Abstract

The growing adoption of large language model (LLM) agents creates a need for network administrators and security teams to audit agent behavior within organizational networks without inspecting private user content. Network traffic offers an observable source of evidence, but how much it reveals about agent tasks and operations remains unclear. Existing traffic datasets lack the joint task and stage annotations needed to evaluate this question. We introduce ANT (Agent Network Traffic), a dataset providing agent behavior information at risk, scenario, and behavior primitive granularities alongside network traffic. ANT contains 3,114 execution episodes across 20 tasks and five scenarios, comprising 276,417 bidirectional flows and 40,049 behavior primitive segments organized into 47 macro groups. We establish a benchmark for agent risk identification, scenario recognition, and behavior primitive classification using 13 representative traffic analysis baselines. The results show that existing methods recover useful but uneven behavioral signals. They struggle to identify risk when malicious workflows resemble benign tasks and to distinguish scenarios with similar traffic patterns. Primitive classification is more reliable for frequent macro groups and those with distinctive traffic patterns than for rare or semantically similar groups. ANT provides a common basis for developing more precise auditing and forensic analysis of agent behavior from network traffic. Our data and code are available at https://anonymous.4open.science/r/ant-main-suite-7BC0/.

Bullet summary

• Introduction of ANT, a novel multi-granularity network traffic dataset containing 3,114 execution episodes across 20 tasks and five scenarios, annotated with agent behavior primitives aligned with network flows to enable precise agent behavior auditing with...
• ANT fills a critical gap by jointly annotating agent tasks, scenarios, and fine-grained behavior primitives, supporting comprehensive evaluation of large language model (LLM) agent behavior and associated security risks from encrypted network traffic.
• A robust benchmark involving 13 baseline network traffic analysis methods for agent risk identification, scenario recognition, and behavior primitive classification demonstrates current methods recover some behavioral signals but show uneven performance, st...
• Detailed data collection setup capturing synchronized model calls, tool invocations, execution outputs, session states, and network traffic in controlled virtual environments ensures high-quality, ethically compliant data suitable for benchmarking.
• Analyses reveal scenario-specific ordering and composition patterns in agent behavior primitives, supporting the feasibility of inferring agent workflows and risk profiles from multi-granularity network traffic data.

Abstract

As AI agents increasingly tackle complex repository-level coding tasks, distributing work across multiple agents is a natural way to scale beyond the capabilities of a single agent. To coordinate their interdependent work, these agents share findings and agree on interfaces between modules. However, exchanged information often serves only as context, leaving individual agents to interpret it and incorporate it into subsequent work. Consequently, shared findings may go unused and deviations from interface agreements may go undetected, undermining the reliability and efficiency of collaboration. This motivates moving part of the coordination responsibility from individual agents to the execution harness. To make shared information actionable during execution, we introduce the Artifact-Exclusive Communication Protocol (AECP). AECP requires agents to communicate exclusively through structured artifacts and specifies how the harness processes them. The harness supplies findings when agents access relevant code, screens implementations for mismatches with recorded interface commitments, and requires affected agents to revisit revised agreements. These coordination steps become part of harness execution rather than actions that agents must initiate from prior messages. Across Doc2Repo, NL2Repo, and CodeProjectEval, using closed- and open-source models including Opus-4.8 and DeepSeek-V4-Flash, AECP improves average test pass rate by 28.2% and reduces average wall time by 16.5% relative to an agent team using free-form inter-agent messages. Artifact-exclusive communication also blocks the relay of malicious instructions between agents, reducing how often they reach other agents from 95% to 0% and how often those agents act on them from 40% to 0%.

Bullet summary

• AECP introduces an Artifact-Exclusive Communication Protocol enabling multi-agent AI systems to coordinate complex code generation exclusively through structured artifacts managed by an execution harness.
• The execution harness actively processes shared Knowledge and Contract Artifacts, delivering relevant knowledge based on code scope access, enforcing interface contract compliance, and managing coordination states, reducing reliance on ambiguous natural-lan...
• AECP addresses common coordination failures such as overlooked shared findings, unnoticed interface deviations, and inconsistent task completion by moving responsibility for processing and verification from individual agents to the centralized harness.
• Experimental evaluations across multiple benchmarks (Doc2Repo, NL2Repo, CodeProjectEval) and models demonstrate AECP improves average test pass rates by over 28% and reduces wall-clock time by up to 24.9% relative to systems using free-form inter-agent mess...
• Artifact-exclusive communication via AECP inherently enhances security by blocking malicious instruction propagation between agents, reducing transmission and action of such instructions from 95% and 40% respectively, to zero.

Abstract

The rapid advancement of LLM agents has enabled systems to autonomously perform complex tasks through external tools, but their growing access to personal data introduces significant privacy risks. Existing benchmarks primarily evaluate LLM agent privacy through simulated trajectories and outcome-based metrics, limiting their ability to capture privacy risks arising during multi-step agent execution. In this work, we introduce AgentPrivArena, a framework for evaluating privacy risks in realistic LLM agent workflows. AgentPrivArena integrates authentic MCP tools and self-hosted services within a reproducible execution environment. We further propose trajectory-level privacy metrics that quantify unnecessary information access beyond final response leakage. Building on this framework, we introduce AgentPrivAudit, a runtime auditing approach for monitoring privacy violations during agent execution. Extensive experiments on state-of-the-art LLM agents reveal substantial privacy risks overlooked by existing evaluation paradigms, highlighting the importance of trajectory-level auditing for trustworthy agent deployment.

Bullet summary

• LLM agents increasingly utilize external tools to perform complex tasks autonomously, raising significant privacy concerns due to access to personal and sensitive data.
• Existing benchmarks assess privacy risks mainly through simulated trajectories and outcome-based metrics, failing to capture risks arising during multi-step, real-world agent executions.
• AgentPrivArena is introduced as a novel evaluation framework integrating authentic MCP (multi-channel platform) tools and self-hosted open-source services within reproducible Docker-based sandboxes, enabling realistic auditing of agent privacy during multi-...
• The framework proposes trajectory-level privacy metrics that measure unnecessary information access throughout the entire agent workflow, extending beyond traditional final output leakage metrics.
• AgentPrivAudit is a runtime auditing mechanism that monitors agent executions dynamically, extracting information flows from tool-read operations and assessing outbound writes against configurable privacy policies to proactively detect and mitigate privacy...

Abstract

Tool-using language-model agents are vulnerable to indirect prompt injection because they must act on untrusted external content. Existing training-time defenses can reduce attack success rates, but often at the cost of general capabilities. We show that training-based defenses induce substantial drift in the model's output distribution, altering its behavior even in benign settings and providing a potential mechanism for utility degradation. We further identify a failure mode of these defenses: On benign tool-use tasks, the model refrains from a step needed to finish an authorized task, particularly when that step is indicated by a tool output. To address these limitations, we introduce RAISED (Robust Attack Invariance through Self-Distillation), a training framework that combines self-generation and self-distillation. The model first generates its own tool-use scenarios, with an emphasis on cases where task completion requires acting on legitimate guidance from tool outputs. Then, through self-distillation, the student is trained to match the teacher's clean-context behavior on both clean and injected variants of the same trajectory. RAISED substantially reduces the attack success rate of prompt injections in tool responses while, unlike prior training-based defenses, preserving utility on both agentic and general-purpose benchmarks.

Bullet summary

• Tool-using large language model (LLM) agents are vulnerable to indirect prompt injection attacks, where untrusted tool outputs embed malicious instructions that hijack the agent's behavior.
• Existing training-based defenses reduce attack success rates but introduce substantial output-distribution drift, leading to degraded general capabilities and failure modes such as incomplete task execution when legitimate tool guidance is involved.
• RAISED (Robust Attack Invariance through Self-Distillation) is proposed as a novel defense framework that combines self-generation of tool-use scenarios emphasizing legitimate guidance and an injection-augmented self-distillation training objective to prese...
• Self-generation in RAISED involves the model generating executable multi-step tool-use tasks, solving them, and then generating adversarial prompt injections to create training data that covers both benign and attack scenarios.
• Self-distillation trains a student model to match the teacher's (base model's) next-token distributions on both clean and injected inputs, thereby maintaining alignment with the base model and reducing output-distribution drift.

Abstract

LLM agents are starting to own the full customer experience. Soon, LLMs may be selling and buying on behalf of companies and customers respectively. Small models are more cost-efficient at scale, but can reinforcement learning train them into competent sellers? We train four Gemma 4 checkpoints (2.3B to 31B effective parameters) with GRPO on a programmatic utility reward for bilateral multi-issue bargaining, and evaluate every arm on the same 1,152 negotiations against two frontier buyers it never saw in training. With the same learning rate ($10^{-6}$) for every size, the gain of the RL model over its base rises from $+0.001$ at 2.3B to $+0.078$ at 31B. Each size was trained once and the two smallest checkpoints use a different architecture, so we fit no scaling law. Tripling the learning rate, with the same or fewer training steps, improves on the shared rate at every size by $+0.032$ (2.3B) to $+0.081$ (4.5B). In exploratory comparisons with two frontier models run as sellers, the 12B seller trained at the tripled rate scores above both, though its untrained base already scores as high as they do. The 4.5B seller at that rate shows no detectable difference from either and fits on one 48 GB GPU. A further 2.3B arm at ten times the shared rate raises pooled score, but its gain concentrates on the evaluation buyer that shares a model family with the training pool. These results suggest tuning the learning rate before concluding that a small model cannot learn to negotiate, and testing against buyers from more than one model family.

Bullet summary

• The paper investigates whether small language models (LLMs) can learn multi-issue bilateral negotiation skills through reinforcement learning (RL), focusing on Gemma 4 models ranging from 2.3B to 31B effective parameters.
• Using the GRPO algorithm with a programmatic utility reward, sellers were trained under a uniform learning rate (10⁻⁶) and higher multiples (3×, 10×) to assess the impact of training hyperparameters on negotiation competence.
• RL-trained models exhibit increasing negotiation performance gains with model size at the shared learning rate, while tripling the learning rate notably improves performance especially for smaller models (2.3B and 4.5B).
• Evaluation was rigorously controlled, involving 1,152 negotiation episodes against two state-of-the-art buyer models unseen in training, plus testing on a held-out domain with statistical corrections for significance.
• A 12B parameter seller trained at 3× learning rate outperformed frontier models, despite its base model already being competitive, demonstrating small-to-mid scale LLMs can learn effective negotiation policies with appropriate training.

Abstract

Agentic AI assistants increasingly act on developers' behalf by modifying files, executing commands, and accessing external resources. These actions often require permission, yet little is known about how practitioners make permission decisions while still benefiting from agent autonomy. To address this gap, we conducted a sequential mixed methods study, interviewing 18 practitioners who use AI agents and then surveying 115 practitioners based on the interview findings. We find that practitioners often understand agent behaviour through what they can directly observe and review, while decisions, data use, and other activity behind the scenes remain less clear. This uncertainty also shapes permission decisions, which depend on the scope and risk of an action, whether it fits the task, familiarity with the agent, and the environment in which it operates. Practitioners respond by adjusting how closely they oversee agents, from setting limits in advance to monitoring execution and reviewing work afterwards. How much scrutiny they apply depends on factors such as trust, task importance, time pressure, and the consequences of an action. Our findings suggest that permission systems should make consequential actions easier to review, distinguish what an agent is allowed to do from what the user intended, make reversibility clearer, avoid treating repeated approvals as stable preferences, and distinguish rejecting a single action from rejecting an entire approach.

Bullet summary

• The paper addresses how software practitioners understand and manage permission decisions when using agentic AI assistants that act autonomously during software development tasks.
• A sequential mixed methods approach was used: qualitative interviews with 18 practitioners followed by a survey of 115 practitioners to capture diverse perspectives on agent oversight and permission handling.
• Practitioners heavily rely on observable outputs, such as code changes and logs, to comprehend agent actions, while underlying decision processes and data usage remain opaque, introducing uncertainty in trust.
• Permission granting decisions depend on perceived risk, task relevance, agent familiarity, environment context, and data sensitivity, leading to varied oversight strategies ranging from setting upfront limits to continuous monitoring or post-action reviews.
• Repeated permission prompts can lead to approval fatigue, resulting in less careful scrutiny over time; practitioners differentiate between occasional denials and rejecting entire agentic approaches.

Abstract

AI agents built on large language models (LLMs) run shell commands, read and write files, and reach the network, typically with their user's privileges. However, what an agent does during an execution is difficult to understand: tests assert on the result, and the agent's trajectory records only what the agent reports about itself, which may omit behavior executed by its subprocesses. We present AgentSpy, an approach that observes an agent from outside the agent. AgentSpy runs the agent in an isolated environment, configured by a declarative specification, and records the system calls and network traffic of the agent and of every process it executes. Based on this monitoring, AgentSpy supports two families of analyses: conformance analyses, which measure obligations, i.e., what an agent execution should do, and safety analyses, which check prohibitions, i.e., what an agent execution must never do. We instantiate one analysis of each family. The reliability analysis uses rules to summarize each run by the environment resources the agent uses: the commands it executed, the files it accessed, and the hosts it contacted. The security analysis applies deterministic rules to the system calls of an execution. For reliability, we evaluated AgentSpy on 77 tasks with the codex harness and three recent LLMs, executing each task three times. Sets of repeated runs of the same task are more similar than sets that include runs of another task in 92.2% of the comparisons. Among tasks for which all three runs pass outcome-based tests, the agent performs task-unrelated activities in 18% of the cases, reads the grading files in 7%, and does not use the developers' guidance in 17%. For security, the generic rules of AgentSpy detect four of five attack categories we considered, with no false positives across 50 runs.

Bullet summary

• AgentSpy is a system that observes AI agents built on large language models (LLMs) from the outside by monitoring all system calls and network traffic within an isolated environment, capturing complete and deterministic evidence of agent behavior including...
• It supports two main analysis types: conformance analyses that verify agent obligations (expected behaviors), and safety analyses that detect prohibited behaviors to ensure security and reliability.
• Reliability analysis models agent runs as graphs of used resources (commands, files, hosts) enabling comparison across runs to detect unrelated or unexpected agent actions even when outcome-based tests pass.
• Security analysis uses deterministic system call rules to detect various malicious behaviors such as unauthorized file access, network connections, and data exfiltration, effectively exposing skill-injection attacks with high precision and recall.
• AgentSpy was evaluated on 77 tasks with three recent LLMs, showing that repeated runs of the same task are more behaviorally similar than different tasks, and revealed that agents sometimes perform extraneous or unauthorized activities.

Abstract

Business Process Management (BPM) was built on a foundational assumption that organizations are populated primarily by human actors whose work can be made visible, governable, and improvable through process models. That assumption is depreciating. AI agent ecosystems increasingly execute, coordinate, and adapt organizational work with limited human direction, challenging not only BPM's methods but its core conception of what a process is. We argue that BPM faces a constitutive shift from modeling human work to governing autonomous agents, for which we propose two new concepts: the \emph{Process Constitution}, a machine-interpretable, value-laden framework that defines the space of admissible agent behavior, and the \emph{Process Steward}, a governance agent that interprets and enforces it. The central value proposition of this new generation of BPM is not efficiency but \emph{organizational legibility}: the capacity to keep agentic organizations accountable, contestable, and humanly understandable. We outline what this means and sketch the research agenda it opens.

Bullet summary

• Traditional Business Process Management (BPM) relies on modeling human-centered organizational work, an assumption challenged by the rise of autonomous AI agent ecosystems.
• The paper proposes a third generation of BPM focusing on governing autonomous agents through two key concepts: Process Constitutions and Process Stewards.
• Process Constitutions are machine-readable, value-laden frameworks that define admissible agent behaviors, encoding organizational intent, legal, ethical, and safety constraints, accountability, and adaptation rules.
• Process Stewards are governance agents responsible for interpreting, enforcing, and managing the Process Constitution, akin to constitutional courts rather than simple workflow engines.
• This new BPM generation emphasizes organizational legibility—ensuring that autonomous agentic organizations remain accountable, contestable, and understandable to humans.

Abstract

A tool-using model can follow a malicious instruction even when its credentials are valid. We study whether task-scoped authorization contains the resulting tool execution. Our paired-replay testbed samples a model request once and submits the same action, resource, and arguments to broad bearer, scoped JWT, sender-constrained, and Open Policy Agent conditions. The frozen main experiment uses 128 scenarios across four tool domains and five local model configurations. Among valid attacked post-exposure decisions, broad-bearer harmful execution ranges from 8.9% to 37.8% across models; all three scoped conditions record zero. The policy changes execution, not the frozen model decision. These results support containment of the tested cross-action and cross-resource consequences under researcher-supplied task authority, not prevention of prompt injection. A bounded six-task AgentDojo extension preserves native scoring and records 11/24 injected broad-policy attack flags versus 0/24 scoped flags, with utility of 5/24 and 6/24. Low exposure, invalid decisions, and purposive task selection limit that comparison. The primary experiment measures safe continuation rather than final-answer correctness. An empirical-bank fresh-sampling comparison finds no estimation advantage from pairing when scoped outcomes are constant zero. The contribution is a controlled measurement of compromise, enforcement, and continuation, with explicit limits on what each observation establishes.

Bullet summary

• Tool-using LLMs can execute malicious instructions even with valid credentials, prompting investigation into whether task-scoped authorization effectively contains harmful tool actions.
• The study introduces a paired-replay testbed that submits identical model requests under different authorization policies (broad bearer, scoped JWT, sender-constrained, Open Policy Agent) to isolate the impact of authorization enforcement from stochastic mo...
• Experiments across 128 scenarios spanning four tool domains and five large language model configurations reveal that scoped authorization policies prevent any harmful executions post-exposure, while broad bearer tokens allow harmful executions ranging from...
• The findings clarify that authorization policies influence tool execution rather than model decision-making itself, distinguishing between model compromise (unauthorized action selection) and operational consequences (harmful action execution).
• Scoped authorization policies enforce effective boundaries on tool action and resource use, containing impacts of prompt injections without preventing the model from selecting potentially malicious actions.