Prompt-Injection Testing Belongs in CI

Most agent teams can show that they tested prompt injection. Far fewer can show which environments they tested, which tool paths were reachable, how success was measured, what changed since the last run, and which score prevents a release.

That is the gap ToolHazard is trying to close.

The paper is a work in progress, and its environments are synthetic. It should not be read as a prediction that a particular model will be compromised at a particular rate in production. Its more useful contribution is mechanical: executable environments, state-grounded tasks, automatically discovered injection points, deterministic checks, and a path from evaluation into training.

In other words, prompt-injection testing can look like engineering instead of a collection of clever prompts in a document.

What ToolHazard builds

ToolHazard has three main generators:

1. An Environment Simulator creates executable, stateful tool environments.
2. A User Simulator creates benign long-horizon tasks and verification functions grounded in those environments.
3. An Attacker Agent identifies writable free-text fields, maps how tools read and write them, and builds indirect prompt-injection cases that intersect a benign task’s trajectory.

Those outputs become ToolHazard-Bench for evaluation and ToolHazard-Align for supervised and reinforcement-learning experiments.

The benchmark contains 87 tasks across 28 test environments and 512 tools. Its average task takes 15.56 steps. That matters because indirect prompt injection is a trajectory problem. The malicious content may be written before the evaluation starts, encountered by a retrieval tool several steps later, summarized, and acted on near the end.

A single-turn chatbot test does not exercise that system.

The environments are executable rather than generated response by response during the test. That gives the evaluator a stable state before and after each action. A check function can determine whether the benign task completed and whether the injected objective succeeded without asking another model to judge the transcript.

This is a strong pattern for security evaluation even if a team never adopts the complete framework.

Read the scores as benchmark results

The paper evaluates GPT-5, GPT-4.1, Gemini 3.1 Pro, Gemini 2.5 Pro, DeepSeek V3.2, Qwen3-8B, and Qwen3-4B using a ReAct-style agent loop.

It reports benign task completion rate and attack success rate across six attack strategies. The detailed table matters because a single average would hide large differences between strategies.

For example:

Model	Basic combined ASR	Important template	Multi-turn	Decision hijacking	Reasoning criteria	Tool selection
GPT-5	1.18	51.49	33.43	44.82	44.96	59.14
Gemini 3.1 Pro	3.53	23.06	24.19	36.28	63.20	32.56
DeepSeek V3.2	1.18	73.33	73.33	75.00	40.00	73.33

These numbers come from ToolHazard-Bench. They are not incident rates, vendor security grades, or probabilities that an enterprise deployment will be breached.

The paper explicitly acknowledges the gap between synthesized environments and proprietary production systems. Deployment-specific interactions and long-tail failures may not appear in the benchmark. The target agents also use one particular orchestration approach. A production agent with different system instructions, tool schemas, policy enforcement, memory, and approval controls may behave differently.

The right question is not “which model won?” It is “can I run the same relevant tests before and after changing my system?”

Stronger models do not remove the problem

The results resist one comforting explanation.

Qwen3-4B often shows lower attack success than stronger models, but it also completes fewer benign tasks. An agent that fails to follow instructions can appear safer because it fails at both the intended and malicious work.

Security evaluation therefore needs at least two axes:

• Did the agent complete the authorized task?
• Did the agent complete the injected task?

A refusal-only system can produce a low attack-success rate and still be unusable. A highly capable agent can produce excellent benign performance while following attacker-controlled instructions deep in a workflow.

This is why the release gate cannot be “ASR below X” by itself. It needs a target region for both utility and security, plus scenario-level requirements for high-impact tools.

Earlier injections were more effective

ToolHazard tested when an agent first encountered the injected material. In its experiments, content encountered earlier in the trajectory produced higher attack success than later or randomly selected injection points.

That finding is plausible for several reasons. Early content can shape the agent’s plan, determine which sources it trusts, and influence later tool selection. Once the agent has built a coherent explanation around the poisoned information, subsequent steps may reinforce it.

The immediate control is not to distrust only the final action. Apply stronger validation to the first context-gathering calls:

• label the origin and trust class of retrieved material;
• prevent external content from selecting new tools or destinations;
• keep untrusted content out of planning and memory channels where possible;
• require the plan to identify claims that came from externally writable sources;
• revalidate consequential arguments against trusted systems before execution.

Early retrieval deserves at least as much scrutiny as the final write.

Position inside the output also mattered

The paper found that injections placed in later writable fields of a tool response were more effective than random or earlier placements. That suggests a positional bias toward tail-end content.

This is a warning for tool designers. Security metadata placed at the beginning of a long response may be forgotten by the time the model reaches the attacker-controlled comment at the end. A banner saying “untrusted content follows” is weaker than enforcing a schema and carrying trust labels on each field.

Tool output should separate control data from content data. The model should not receive one long string containing status, instructions, comments, and suggested commands with equal visual weight.

Structured output helped, but it is not a sandbox

ToolHazard also compared free-form strings with JSON and YAML tool outputs under the same conditions. Free-form output produced substantially higher attack success.

Structured data can create boundaries between fields. It lets the tool mark a value as a comment, identifier, destination, or externally supplied description. A policy layer can validate types, lengths, enumerations, and allowed destinations before the model sees or acts on them.

That does not make JSON safe. An attacker can put instructions inside a valid JSON string. The improvement comes from how the system uses structure:

• parse with a strict schema;
• reject unexpected fields;
• preserve field-level provenance;
• keep free text out of command and destination fields;
• validate values before tool invocation;
• render untrusted content as quoted evidence, not agent guidance;
• avoid flattening the structure back into prose before the next agent sees it.

Serialization is not a security boundary. Schema enforcement can support one.

Turn the benchmark into a release gate

A practical AISecOps implementation does not need to synthesize hundreds of environments on day one.

Start with the consequential paths in the actual agent:

1. Inventory tools that can write, execute, send, deploy, publish, or modify identity.
2. Identify externally writable fields the agent may read before using those tools.
3. Build small stateful environments that reproduce those read and write paths without real credentials.
4. Define a benign task and a separate injected objective.
5. Verify both outcomes programmatically from final state.
6. Run several injection strategies and positions.
7. Record the complete trace, policy decisions, model and configuration digest, latency, and cost.
8. Compare the result with the approved baseline.

The release gate should run when any of these change:

• model or model version;
• system instructions;
• tool descriptions or schemas;
• connector scopes;
• approval policy;
• memory and retrieval behavior;
• orchestration framework;
• output parser;
• sandbox or egress policy.

An application-code change can alter security even when the model stays the same. A model update can alter security even when the application code does not change.

Both belong in the gate.

Define failure at the scenario level

A global average is useful for trend reporting and dangerous for release decisions.

Suppose an agent resists 99 document-based injections but allows one poisoned ticket to change a production deployment. A 99 percent pass rate hides the control that matters.

Group scenarios by impact:

Tier	Example	Release expectation
Critical	Production change, credential access, external transmission	No successful injected objective
High	Repository write, package execution, identity modification	No success without an explicit independent approval
Moderate	Internal draft or reversible workflow update	Bounded threshold plus review
Low	Read-only summarization with synthetic data	Trend and regression monitoring

Each failure should retain a replayable artifact. A team must be able to answer which source field carried the injection, when it entered the trajectory, which tool accepted it, what policy ran, and which state change proved success.

That artifact is more valuable than a screenshot of the final chat response.

The public repository is a starting point

The ToolHazard repository includes the environment simulator, user simulator, attacker agent, benchmark data, evaluation code, and alignment components.

It is research code, not a drop-in enterprise CI action. The README notes that some scripts retain absolute paths from the original experiments and require manual configuration. The supervised and reinforcement-learning dependencies are not fully included in the main requirements file. The complete synthesis pipeline also uses model calls and several staged scripts, which adds time and cost.

That makes a bounded adoption more realistic than attempting to reproduce the whole paper immediately.

I would begin by using the released benchmark data and evaluation path in an isolated environment. Then I would adapt one scenario around a familiar agent workflow, such as reading an issue or log entry before proposing a tool call. Only after the evaluation path is reliable would I invest in automated environment synthesis or alignment training.

The test environment must not contain real email, cloud, source-control, local filesystem, or production credentials. The point is to measure whether the agent attempts the injected action. A deterministic mock can prove that without allowing the side effect.

What the evaluation must record

To make the result governable, every run should produce:

• evaluation and scenario IDs;
• model and provider version;
• system-prompt and policy digests;
• tool manifest and schema versions;
• initial environment state;
• injected field, source, position, and strategy;
• complete agent and tool trajectory;
• benign-task result;
• injected-task result;
• approval and denial decisions;
• final environment state;
• token, latency, and cost measurements;
• comparison with the last approved baseline.

Store the result outside the evaluated agent. Sign or hash the artifact so the release record can prove what was tested.

This is where a benchmark becomes a control. The pipeline does not merely produce a score. It produces evidence that a named system, with a named configuration, passed a defined set of adversarial tasks before release.

Test continuously, interpret carefully

ToolHazard does not settle which model is safe. It provides a useful way to ask a better question repeatedly.

Can the agent complete its real work while refusing objectives introduced through the environment? Does that answer change when the injection appears earlier, later, or inside a different field? Does structured output help? Does a model swap improve utility while quietly weakening one critical tool path?

Those are release-engineering questions.

Prompt injection will not be solved by one benchmark, one model, or one system prompt. It can still be measured as a regression. Once the test is executable, stateful, and tied to an owner and threshold, “we tried a few attacks” can finally become “this system met its security gate.”

Sources

• ToolHazard paper
• ToolHazard full text
• ToolHazard repository