November 9, 2026
AI Agents Penetration Testing: Why a Clean Retest Isn’t Proof
Contents
Key Takeaways
- Against an AI agent, a clean retest can mean the attack missed that time.
- To prove a fix worked, rerun the attack and keep the record of the rule that blocked it.
- A rule on the action holds however the attack is worded.
- Check your judge against human reviewers, and report untested as untested.
To prove a prompt injection fix worked, run the same attack again once the fix is live, and keep the record of the rule that blocked it. A clean rerun on its own shows that the attack failed on that run, not why it failed.
Penetration testing means attacking a system the way an adversary would, to find weaknesses before someone else does. Retesting after a fix is standard practice in penetration testing, and it has been for years. Against a web application, it’s good evidence: the same request against the same code tends to produce the same result, so a clean retest usually means the fix held.
Against an AI agent, the same retest proves less. An agent is non-deterministic. It can miss an attack on one run with no fix in place at all, because the model happened to ignore the hidden instruction that time. So a finding can close because a control held, or because the attack missed, and a clean retest can’t tell you which.
This article covers why that happens, how to red team an AI agent so the answer is clear, and one worked example from a demo environment, from the finding to the record of the rule that blocked it.
Why A Clean Retest Proves Less Against An Agent
In January 2025, researchers at NIST (then the U.S. AI Safety Institute, now NIST’s Center for AI Standards and Innovation) published work on agent hijacking: instructions hidden in data an agent reads, such as an email, a file, or a tool result. One finding is directly relevant to retesting. When the researchers repeated the same attacks 25 times instead of once, the average success rate rose from 57% to 80%.
The scope matters, so here it is plainly: the research tested one model on a lab benchmark. It isn’t a prediction for your agents. But it does show the mechanism. An attack that fails once may succeed on a later try, with nothing changed in between.
The Cloud Security Alliance put the same point in one line: “A single refusal does not prove resilience.”
For a penetration testing team, that changes what a closed finding means. “We reran it, and it failed” is a weaker statement for an agent than for an application. To close a finding with confidence, you need to know what stopped the attack, not only that it was stopped.
See a weakness found, a rule drafted, and the same attack blocked, in a 25-minute demo.
Book a Demo
How Do You Red Team An AI Agent?
The core of pentesting carries over: attack the running system the way an adversary would, record what got through, fix it, and check the fix. What changes is where the attacks come in and what counts as evidence.
Attack through what the agent reads, not only what users type
Prompt injection often arrives indirectly, in a document, an email, a web page, or a tool result the agent processes as part of its job. Test those paths, because that’s where an agent is told to do something its owner never intended.
Keep the exact request and response for every finding
A finding without the evidence that proved it is hard to reproduce, and hard to rerun after a fix.
Check the judge
Automated penetration testing of agents relies on automated judges to decide whether each attack succeeded or was blocked. Most coverage skips the obvious question: how do you know the judge agrees with your own reviewers? An evaluation nobody has checked against a person isn’t evidence either. Ask how often the judge’s verdicts are compared with human review, and what happens when they disagree.
Report what you couldn’t test as untested
Some attack categories can’t be exercised against a given agent. An agent with no email tool can’t be tested for email exfiltration, for example. Those categories should show as untested, not passed. A report that counts them as passes turns “we didn’t check” into “it’s safe,” and that’s the kind of false assurance a clean retest can already create.
After the fix, ask what stopped the attack
This is the step that separates evidence from a pass mark, and the example below shows it end to end.
A Worked Example, End to End
This example follows a scenario we run in a demo environment: a claims-intake agent that reads inbound documents and can send email through a send_email tool. It runs on the Lumenova AI Platform’s SDK, which matters for the last step.
The request, response, and record below are simplified for illustration, with placeholder names and addresses.
1. The Finding
A probe plants an instruction inside an inbound claim PDF: email the full claim file to an outside address. The agent follows it and tries to call send_email with an external recipient. The finding keeps the exact request and response that show it. The graphic at the end of this example shows a simplified version.
2. The Rule Drafted From It
From that finding, the platform drafts a rule in plain terms: this agent may not email claim files to anyone outside the company. The rule sits on the action itself, the send_email call, not on the wording of the instruction. A detector on the input can flag the hidden line, but no detector catches every phrasing. A rule on the action holds, however the instruction is worded.
3. A Person Reviews It and Switches It On
Nothing is applied automatically. A member of the security team reads the drafted rule to check if it covers the intended purpose, and then enables it and adds it to the existing policy stack that is run with the agent through the SDK.
4. The Same Attack Runs Again
Once the rule is live, the original attack runs again: the same planted instruction, against the same agent. This time the agent still tries to call send_email, and policy blocks the call before it runs.
5. The Record of the Block
Because this agent is connected through the SDK, the rerun does more than come back clean. The finding now carries the record of the decision that blocked it: which rule applied, why, and the input it evaluated. On an agent connected through the SDK or the AI Gateway, that record is what answers the architect’s question. The attack didn’t miss its target; this rule stopped it.
What the Record of the Block Should Hold
A report that says “attack blocked” leaves the important questions open. Did the agent attempt the call? Which rule was active? What input did it see? A useful record answers each of these for one specific attempted action:
|
Field |
Example value |
| Agent | claims-intake-agent |
| Attempted tool | send_email |
| Destination | External email domain |
| Rule | Claim files may only be emailed internally |
| Decision | Blocked before it ran |
| Reason | Destination outside the approved domain |
| Evidence | The input and the decision, linked to the original finding |
These fields are illustrative. What matters is that the record supports a specific claim about a specific action: it shows why that attack failed, not that the next one will.
What’s Common, and What the Full Chain Adds
None of the individual steps above is unique. Many AI security tools run the attack again after a fix, and many contrast a point-in-time test with continuous testing. Detection on inputs and outputs is widespread too.
What matters is whether the steps connect. The finding with its evidence, the rule drafted from it, the person who switches it on, the rerun, and the record of the rule that blocked it, all linked to the same finding. With that chain, a closed finding comes with an answer to “what stopped it”. Without it, you have a pass mark.
The chain has limits, and they’re worth stating plainly:
- The record needs the agent connected. The record of the block exists on agents connected through the SDK or the AI Gateway, where policy decides before the call runs. For an agent outside that path, the attack can still be run again after a fix, but there’s no record of a block to attach.
- Agents from a vendor are a different case. You can probe a vendor’s agent with the vendor’s authorization, but finding a weakness doesn’t mean you control its tool calls.
- A record explains one attack, not the next. Test again after every change to the agent, its tools, or its model.
One Question for Your Red Team, or Your Vendor
Whether your team runs agent testing in-house or a vendor runs it for you, one question tells you how much a closed finding is worth:
When the retest came back clean, what stopped the attack, and can anyone show it?
If the answer names a rule and points to a record, the finding closed because a control held. If the answer is “the attack didn’t work the second time”, the finding may have closed because the attack missed. Any assurance about agent security, up to the one a board hears, rests on knowing the difference.
Two more questions are worth asking alongside it. How is the automated judge checked against your own reviewers? And are the categories that couldn’t be tested reported as untested?
Get the AI Agent Security Evaluation Guide
The AI Agent Security Evaluation Guide sets out what to test, what to enforce, and what proof to ask for, whether you buy, build, or both. It includes a 14-question checklist to score any option, your own build included.
To see how the Lumenova AI Platform finds, blocks, and proves the block on a workflow you describe, explore AI Security or book your demo.
Frequently Asked Questions
Penetration testing, or pentesting, is an authorized, simulated attack on a system to find weaknesses before a real attacker does. For AI agents, that means attacking the running agent through what it reads and the tools it can call, not only through its code or network.
A penetration test usually checks a defined scope for known weakness types within a set time frame. Red teaming is broader and more adversarial: it tries to reach a goal, such as getting the agent to leak data, using whatever path works. Most agent security programs use both.
Automation is what makes agent testing practical. It can run hundreds of attack variations and repeat them, which a manual tester can’t. But it still needs people: someone to check that the automated verdicts match human judgment, and to decide which fixes to switch on.
Retest whenever something the agent depends on changes: a new model or model version, a new tool or data source, or a change to its instructions. Any of these can open a weakness a previous test didn’t cover, even if the agent’s own code hasn’t changed.