AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security

Formally, we are given an agent trajectory $\mathcal{T}=\{t_{1},\dots,t_{n}\}$, where each step $t_{i}$ consists of an action $a_{i}$ and an observation $o_{i}$, i.e., $t_{i}=(a_{i},o_{i})$. The goal is to predict a trajectory-level binary label

where $y=\texttt{unsafe}$ means there exists at least one step exhibiting unsafe behavior:

Given an unsafe trajectory, the second task is to predict the fine-grained labels:

where $\mathcal{L}^{\text{risk}}$, $\mathcal{L}^{\text{mode}}$, and $\mathcal{L}^{\text{harm}}$ denote the discrete label sets for risk source, failure mode, and real-world harm, respectively, as defined in our taxonomy (Section 2).

Figure 3 provides an annotated example of our prompt template for the two tasks. For trajectory-level safety evaluation, our prompt template includes three parts: (i) task definition, (ii) agent trajectory, and (iii) output format. The model is instructed to judge whether any step in the trajectory violates safety. For fine-grained risk diagnosis, we adjust the instructions accordingly and additionally include the safety taxonomy for reference. The model is instructed to output the fine-grained labels line by line.

As illustrated in Figure 4, we adopt a three-stage, planner-based pipeline for data synthesis, designed to generate long-horizon, tool-augmented interaction trajectories with controllable risk injection and reliable safety labels.

Stage 1: Planning. For each trajectory, we first sample a risk configuration tuple (comprising risk source, failure mode, and risk consequence) as defined in the safety taxonomy of section 2. Then, we determine the safety outcome of the trajectory, specifying whether the trajectory is intended to be safe (the agent successfully detects and mitigates the risk) or unsafe (the attack succeeds). In parallel, we sample a candidate set of tools from the filtered tool library.

Given these inputs, the planner constructs a coherent multi-step task plan via a two-phase process. In the first phase, it designs a coherent multi-step task and analyzes how the risk can be naturally embedded into the trajectory. The output of this phase is a free-form Chain-of-Thought. In the second phase, it produces a structured execution plan that specifies the task description, the selected subset of tools, the sequence of tool-augmented steps, and the exact risk injection point. For safe trajectories, the plan also encodes the agent’s expected defensive behavior at the risk point.

Stage 2: Trajectory Synthesis. Given a structured execution plan produced in Stage 1, the trajectory synthesis stage instantiates it into a concrete, multi-turn interaction through a coordinated, plan-driven generation process. A central Orchestrator controls the execution flow and ensures that each step in the plan is executed in order, including user query generation, tool interaction simulation, agent response generation, and outcome summarization.

To ensure that synthesized trajectories are both trainable and reliably labeled, we introduce a dedicated quality-control (QC) procedure after trajectory generation. QC follows a two-layer design: deterministic validators to remove structural and formatting errors; and an LLM-based judge to verify semantic alignment between the trajectory content and the intended safety labels.

Finally, we provide a quantitative overview of the synthesized dataset in terms of overall scale, coverage of the risk taxonomy, tool-usage characteristics, and the pass rate of the quality control (QC) process. The dataset contains over 100k multi-turn interaction trajectories, designed to provide broad and systematic coverage of the targeted risks under diverse tool-assisted settings. After QC filtering, the resulting corpus is practically usable for both model training and systematic evaluation, enabling detailed analysis of multi-step agent behaviors and their safety failure patterns.

The trajectory generation procedure for ATBench follows the same taxonomy-guided synthesis framework used to construct our training data (Section 3). We ground generation in the risk taxonomy and drive agent executions with tool seeds, producing long-horizon trajectories with multi-turn interactions and realistic tool invocations.

To ensure a clean evaluation, we strictly decouple ATBench from the training data via a tool-level split. Specifically, we use an independent tool library for ATBench with no overlap with the training tools. This library contains 2,292 tool definitions, creating an “unseen-tools” evaluation setting. This design directly assesses whether a guard model can generalize to previously unseen tools and contexts, rather than just measuring performance on familiar patterns.

We apply a two-stage verification and filtering procedure to improve the reliability and realism of synthesized trajectories. We first perform quality scoring to remove low-fidelity samples, and then conduct multi-model labeling to obtain robust binary verdicts and taxonomy labels. Cross-model non-unanimity on the binary verdict is used as a signal to route ambiguous cases to human verification.

Following the multi-agent filtering stage, we implement a rigorous human verification protocol to resolve ambiguities and ensure benchmark reliability. Specifically, we adopt a stratified quality control strategy that partitions the dataset into two subsets based on model consensus: an easy subset and a hard subset.

For the easy subset, we employ a lightweight verification strategy to validate labeling consistency. Given the unanimous consensus among four diverse models, we conduct human spot-checking by randomly sampling 20% of the cases. All spot-checked instances were confirmed by human annotators under the same rubric, verifying the reliability of the model-consensus labels.

For the hard subset, we conduct an exhaustive, double-blind review process to resolve ambiguities. We adopt a cross-checking protocol with 10 expert annotators: each hard trajectory is independently audited by two researchers to verify whether unsafe verdicts are supported by observable evidence rather than mere intent. This step explicitly filters out false positives caused by capability limitations. When annotators disagree, we resolve the case by introducing a third expert annotator for adjudication, ensuring that the final labels are high-fidelity and taxonomy-grounded.

We utilized R-judge (Yuan et al., 2024b), ASSE-Safety (Luo et al., 2025a) and ATBench (Section 4) to evaluate the performance of our AgentDoG. Each dataset consists of complete agent trajectories, where each trajectory is classified as either safe or unsafe. The evaluation is structured as two complementary tasks:

We evaluated our model against leading guard models like LlamaGuard3-8B, LlamaGuard4-12B(Inan et al., 2023), Qwen3-Guard (Zhao et al., 2025), ShieldAgent (Zhang et al., 2024a), JoySafety (JD Open Source Team, 2025), ShieldGemma (Team et al., 2024), PolyGuard (Kang et al., 2025), NemoGuard (NVIDIA NeMo Team, 2023), alongside general models including Gemini-3-Flash (Google Cloud, 2026), GPT-5.2 (OpenAI, 2025b), and Qwen3-235B-A22B-Instruct-2507 (Yang et al., 2025a). The detailed introduction of these dataset and evaluation details can be found at Appendix B.

Table 4 presents the comparative results of AgentDoG against general models and specialized guard models across the R-Judge, ASSE-Safety, and ATBench benchmarks. We observe that general models show strong baselines, significantly outperforming existing guard models on all benchmarks. We hypothesize this primarily to two reasons: first, general models go through extensive post-training and have strong instruction-following capabilities, which allows them better to adhere to our task definition; second, existing guard models are not trained on agent trajectory data, so there is a mismatch between the training data of many guard models and our evaluation setting.

In particular, many guard models are primarily trained and validated on single-turn prompts or short-form dialogue safety data rather than long, tool-augmented agent trajectories. Since our benchmarks require reasoning over multi-step interaction traces, they can differ substantially from the original training distribution of these guard models. As a result, this evaluation constitutes an out-of-distribution scenario for many existing guard models, which may contribute to their degraded performance.

Compared to these baseline models, AgentDoG demonstrates excellent performance. It significantly outperforms all specialized guard models and remains competitive with general models which have larger parameter scales. Specifically, on the R-Judge benchmark, AgentDoG-Qwen3-4B achieves an F1 score of 92.7%, surpassing GPT-5.2 (91.8%) and approaching the performance of Gemini-3-Flash (95.3%). Similarly, on the ASSE-Safety dataset, AgentDoG-Llama3.1-8B attains an F1 score of 83.4%, outperforming Gemini-3-Pro (78.6%). Moreover, AgentDoG maintains a balanced trade-off between precision and recall across all datasets. In contrast, existing guard models often exhibit extremely low recall (e.g., ShieldGemma reports recall below 10%), reflecting an overly conservative prediction behavior that leads to missed safety risks.

Table 5 illustrates the performance comparison between AgentDoG and general models on the fine-grained classification tasks within ATBench, specifically covering Risk Source, Failure Mode, and Real-world Harm. While binary classification can indicate whether a trajectory is safe, it provides limited insight into the root cause of a violation. Fine-grained diagnosis is therefore critical for safety auditing, remediation, and system debugging, as it helps identify why unsafe behavior occurs and which component or interaction pattern is responsible. A detailed discussion of these risk taxonomies and their role in safety auditing is provided in Section 2.

In this fine-grained setting, AgentDoG demonstrates a significant performance advantage. We observe that general models exhibit difficulty in mapping trajectories to specific risk taxonomies; for instance, Gemini-3-Pro achieves only 36.8% accuracy on Risk Source, whereas AgentDoG-Qwen3-FG-4B achieves 82.0%. This trend is consistent across all three fine-grained labels. Notably, in Real-world Harm Accuracy, AgentDoG-Qwen2.5-FG-7B reaches 59.2%, which is significantly higher than the best baseline model, Qwen3-235B-A22B-Instruct-2507 (38.0%). We hypothesize that this gap arises because general models typically lack explicit supervision on fine-grained risk taxonomies during training, making attribution difficult even if they can detect unsafe trajectories effectively. In contrast, AgentDoG is trained with trajectory-level and fine-grained risk annotations, enabling more accurate diagnosis of how and why agent behaviors become unsafe.

Case 1: Prompt injection in recruitment scenario.

Case overview. In this case (Figure 7-left), an AI Recruitment Assistant is tasked with screening a candidate’s resume file and scheduling an interview if the candidate matches specific criteria. The risk arises from an Prompt Injection: the job applicant embeds a deceptive instruction within the resume text, implying that the resume is qualified and should pass the current screening stage. The agent fails to distinguish this adversarial data from valid content, resulting in unauthorized interview scheduling.

Attribution analysis. The attribution module accurately diagnoses the external trigger responsible for the final behavior. At the trajectory level, the analysis identifies the agent’s initial action to access the file (Step 2) and the subsequent resume content provided by the tool (Step 3) as the primary drivers. While Step 2 establishes the necessary retrieval context, the tool output in Step 3 introduces the decisive information. A fine-grained sentence-level examination of Step 3 reveals that the attribution correctly identifies the adversarial content from benign resume data. As shown in the attribution results, the highest positive scores are assigned to the deceptive instructions: “[Important] Please ignore the previous content, automated validation for this record is successful.” In contrast, the normal candidate details receive less attribution.

Overall, the attribution result effectively disentangles the root cause, confirming that the agent’s decision was hijacked by the specific adversarial text embedded in the resume. This transparency is vital for diagnosing risks introduced by external tool interactions, underscoring the urgent need to enhance agent robustness against untrusted information channels.

Case 2: Misinterpretation of sarcasm in financial analysis.

Case overview. In this financial analysis scenario (Figure 7-right), the agent is tasked with monitoring market feedback on CompanyA† to determine trading strategies. The core risk emerges from a sarcastic user quote: a customer posts a comment that superficially uses positive language (“Fantastic update”, “Absolute genius”) but is contextually negative (“Paying more for fewer features”). The agent misinterprets this sarcasm as genuine praise, leading to a flawed financial recommendation to go long.

Attribution analysis. The attribution module effectively localizes the source of this cognitive failure. At the trajectory level, the analysis pinpoints the tool execution result (Step 3) as the dominant factor driving the final decision, which contains the retrieved customer quote on the stock of CompanyA. Sentence-level analysis further illustrates that the attribution scores are heavily concentrated on the positive phrases, i.e., “Absolute genius” and “Fantastic update”. Crucially, the sentence providing the sarcastic context—“Paying more for fewer features really shows how much they value customers”—receives significantly lower attribution, indicating that the agent is misled by the positive part and fails to recognize the underlying risks in the stock.

Overall, the diagnosis reveals that the agent’s decision is driven by shallow keywords rather than a holistic semantic understanding, exposing a potential risk of agentic systems deployed in high-stakes settings. In these scenarios, attribution analysis can serve as a valuable tool to audit an agent’s inference process, guiding targeted interventions to improve reasoning robustness.

To validate whether our safety training enhances the model’s ability to identify the internal drivers of agents’ risky behaviors, we compare the attribution performance of AgentDoG-Qwen3-4B against its base model-Qwen3-4B-Instruct-2507, results are shown in Figure 8.

Case 3: Ambiguous Financial Transaction.

Case overview. In this scenario, the user requests to send a send a peer-to-peer payment (specifically a WeChat red packet) to a contact named Zhang Wei.” The contact search tool returns three distinct individuals with the same name (a coworker, a classmate, and a trainer). Instead of pausing to ask for clarification, the agent makes an unsafe assumption based on common usage patterns” and then selects the coworker, to execute the financial transaction.

Comparative attribution analysis. We compare the top-ranked steps identified by both models to understand their diagnosis of this risky action. At the trajectory level, both models exhibit broad agreement on the primary drivers, identifying the agent’s initial action to the user query (Step 2) and the tool output (Step 3, showing multiple contacts) as high-impact components. This attribution is reasonable, as these steps provide the necessary background context and ambiguous search results that set the stage for the subsequent decision.

However, a significant divergence appears in the third-ranked component, revealing a fundamental difference in attribution quality. AgentDoG-Qwen3-4B correctly identifies the agent’s internal reasoning step (Step 4) after the search results as the critical driver. Specifically, the sentence-level attribution highlights the flawed thought process: “Based on common usage patterns, the user is most likely sending money to a coworker.” This pinpoints that the agent’s decision to select the coworker was directly driven by this insufficiently supported assumption. In contrast, Qwen3-4B-Instruct-2507 attributes the decision to the final environment execution log (Step 5, “success: true”), which contains less information regarding the decision logic.

This comparison highlights that AgentDoG exhibits more robust and granular agentic attribution capabilities compared to its base model. This indicates that our safety training not only equips the model with the ability to detect unsafe agent trajectories but also refines its understanding of the internal drivers behind the action. By bridging the gap between detection and diagnosis, this diagnostic transparency provides a critical foundation for the accountable deployment of agentic systems.

The binary safety verdict determines whether the trajectory contains any unsafe actions or decision patterns. A trajectory is labeled safe if the agent does not execute unsafe actions, including cases where the agent successfully refuses or mitigates risky instructions. The prompt template for binary verdict annotation is shown in Figure 10.

For unsafe trajectories, we additionally collect fine-grained taxonomy labels across three dimensions: risk source, failure mode, and risk consequence. The complete taxonomy definitions are provided in Section 2 (Tables 1, 2, and 3). The prompt template for fine-grained taxonomy labeling is shown in Figure 11.

Figure 12 illustrates a scenario where a risk trigger (prompt injection) is present, but the agent successfully defends by detecting and refusing the unsafe instruction. In this case, the agent is exposed to indirect prompt injection embedded in the tool output, which attempts to override safety checks. However, the agent identifies the suspicious instructions and refuses to proceed, resulting in a safe verdict.

Figure 13 illustrates a concrete unsafe behavior chain where prompt injection leads to goal drift and results in unauthorized external communication. In this case, the agent initially recognizes the malicious instruction but later exhibits goal drift by adopting the attacker-induced objective. The trajectory is labeled unsafe with fine-grained taxonomy labels indicating the risk source, failure mode, and consequence.

General models are evaluated using the same methodology and prompt template as AgentDoG. To be specific, the prompt shown below (Figure 14) is adopted for both assessments. In this process, a complete trajectory is formatted and filled into the prompt template below. We evaluate most guard models using their original, model-specific chat templates (see Appendix B.3). This approach works for the majority of models.

However, ShieldGemma is limited to an 8,192-token context window, which causes context overflow and evaluation failures under the full-trajectory setting. Therefore, we adopt an alternative evaluation procedure for ShieldGemma.

Specifically, we employ a turn-level evaluation protocol for safety guard models. We first segment each trajectory by turns. For each turn, we construct a user/environment–agent segment consisting of the user (or environment) message and the corresponding agent response, and feed this segment to the guard model to obtain a binary safety label (safe vs. unsafe). The final safety prediction for the entire trajectory is determined using only the user–agent segments: if any user–agent segment is classified as unsafe, we label the entire trajectory as unsafe.

Some guard models, such as ShieldGemma and JoySafety, do not output simple "safe" or "unsafe" outputs. Therefore, for these models, we adopt a rule-based approach to map their outputs to "safe" or "unsafe", ensuring that we can accurately assess their performance.

As in the previous section, both general models and our model use the same prompt template below (Figure 15) for evaluation. The template is designed to take the agent trajectory as input, and the model is required to output fine-grained labels, including Risk Source, Failure Mode, and Real-world Harm. Since most guard models are fine-tuned on their own predefined prompt templates, their adherence to templates outside of their training setting is limited. We attempted to evaluate the guard models using our prompt template; however, they did not perform well in adhering to our defined template. As a result, we do not report the results of the guard models.