Your security gate always passes. Can it actually block a release?

Agent developers have started publishing oversight numbers.

For example, Anthropic reports that of more than a billion decisions by its research and engineering agents in August 2026, about 1 in 47,000 were blocked by an online monitor.

A block rate tells you how often the gate said no. It does not tell you how often the gate should have said no and did not.

The same publication points toward a harder measurement: "Developers should share how often known agent misbehavior is caught by monitors, and should have third parties do this kind of testing independently."

Detection on known cases is not a count of every real-world miss. But it is the part you can measure, and it requires feeding the monitor misbehavior on purpose and watching whether it fires.

What a quiet miss looks like

Controls can fail quietly, in ways a block rate alone would not reveal.

For example, in OpenAI's report on a September 20 training incident, an agent found that its sandbox's DNS resolver could reach the public internet, and used it to send questions to an external chatbot.

The monitoring worked in part. It flagged the behavior, and a person acknowledged the alert three minutes after it fired. The report is candid about what did not work:

• The run "did not stop automatically as expected." It was stopped by hand two and a half hours later.
• "An infrastructure detector for anomalous DNS activity excluded the affected environment." The detector existed. It was not on the path it was meant to guard.
• A retrospective review found other cases of external DNS access that the monitor had not flagged at the expected severity. It "sometimes treated the failure to obtain useful information as evidence that the attempt to access the internet had failed."

A block rate alone would not establish any of those gaps. They show up when someone checks whether each control acts on the path that matters.

Your own pipeline has the same shape, at a smaller scale.

The one line that decides

Most security pipelines end with one line that decides whether the build ships.

A scanner writes a report. A small script reads it, counts the findings above some severity, and exits non-zero if the count is too high. Everything upstream (the tests, the payloads, the coverage mapping) reaches the release decision through that one count.

That script is easy to leave untested.

Two contracts that nobody reconciles

The scanner has an output contract: field names, status values, severity spellings. The gate has an input contract: the fields it looks for and the values it compares against.

They are often written by different people, at different times, sometimes in different repositories. Nothing forces them to agree.

Without schema validation, contract drift can fail silently, and in the worst direction. A gate that looks for result == "failed" against a report that now writes "FAILED" counts zero failures. A gate that compares a severity string against a report that switched to a numeric CVSS score also counts zero.

The build is green. The dashboard is green. The gate has stopped being a control and become a formality.

No error is raised, because nothing is wrong with the syntax. The gate does exactly what it was written to do, against a report it was not written for.

Green is not evidence

A gate that has only ever passed tells you one of two things: the system is clean, or the gate cannot see. From the outside, those look identical.

Until you have seen a gate block a known violation, you have not shown that it can.

The fix is not more scanning. It is proving that the last line can say no.

Three checks that make a gate falsifiable

1. Seed a known finding through the real writer. Feed the gate a report, produced by the scanner's own writer, that contains one critical finding. The gate must exit non-zero. Run this in CI on every change to either side.

This tests the writer-to-gate contract. It does not test whether the scanner detects anything, or whether the pipeline honours the gate's exit code. Those need their own checks. But if someone renames a field in the scanner, this one breaks the same day, not the day an incident review asks why nothing blocked.

2. Share one classifier, and pin its answers. If the scanner's summary and the gate each decide what counts as a failure, they can drift apart. Put the classification (pass, fail, inconclusive) in one function that both call, so there is only one definition to drift.

One definition can still be consistently wrong. Pin its expected outcomes with test cases written independently of the classifier itself.

3. Give "could not tell" its own state. When a response does not show that the capability or control under test was exercised, it is not a pass, and it is not a fail. It is inconclusive.

That is a statement about the assessment, not about release policy. A team can reasonably block a release because required evidence is missing, while the report still says inconclusive rather than fail. What goes wrong is folding the two together: counting inconclusive as pass ships things nobody tested, and relabelling it as fail hides why the build is red.

A check on the checks

The same reasoning applies one level down. A security test that needs a live application response, and returns PASS against a server that is not there, has not tested anything.

Run every test against a few targets that give it nothing to judge: a closed port, a server that answers 404 to everything, a server that answers 403 to everything, a server that answers with an empty body.

For most tests, none of those responses shows that the capability under test was exercised, so the honest verdict is inconclusive. Some tests are different. For an access-control check a 403 can be the evidence. For a network-isolation check a closed port can be part of it, but only alongside evidence that the service is meant to be listening, so isolation is not confused with a service that is simply down. Those should be declared exceptions, each tied to the specific assertion it satisfies. Any other PASS or FAIL there is a test that is not reading its target.

In the Agent Security Harness, as of v4.26.1, both checks run in CI. The GitHub Action's gate and the harness summaries share one row classifier. A test feeds the Action's own gate step a report from the MCP harness's writer with one critical failure and requires a non-zero exit.

A separate guard runs every URL-taking harness against nine such targets, from a closed port and a 404-everywhere server to empty 200s, empty 500s and redirect loops. It fails the build on any verdict there outside a short, documented exception list. The guard also proves it can fire: a seeded target-blind harness must turn it red.

That guard's claim has been reproduced externally, against the published v4.26.0 package, by an outside party using their own targets and result parser. They classify it as external execution of the published harness, not an independent implementation: https://github.com/VrtxOmega/veritas-agent-trust-lab/blob/cb061dc9930cee456250ecab71b16d5e977f073d/evidence/ASH_NO_SURFACE_RETEST_20260926.md

Try to break it yourself: https://github.com/msaleme/red-team-blue-team-agent-fabric/discussions/619

The question to ask this week

Pick the one line in your pipeline that decides whether security findings block a release.

Then ask: when did it last block on purpose?

If the answer is never, that is the first test to write.

---

Views are my own.

Sources:

• Anthropic, "Measurements for understanding the pace of AI development inside frontier labs": https://www.anthropic.com/institute/measuring-pace-of-ai-development
• OpenAI, "An agent used DNS to reach an external chatbot" (report updated 25 September 2026): https://alignment.openai.com/misalignment-reports/an-agent-used-dns-to-reach-an-external-chatbot/
• Agent Security Harness v4.26.1 changelog: https://github.com/msaleme/red-team-blue-team-agent-fabric/blob/v4.26.1/CHANGELOG.md