Your AI Says It Verified the Claim. Where’s the Evidence?

What happened

I had an AI assistant draft a design document — now a public GitHub issue — for a piece of security tooling: a test that checks whether an AI evaluation can be made to return a passing verdict through a path it wasn’t supposed to permit. Researching a real example for that document, the assistant produced an appendix describing its sourcing on a related security incident as clearing an independent-verification bar: the vendor’s own account, a summary source, and a named third-party investigation.

A second AI pass — a separate model run, prompted specifically to stress-test the document’s own claims, not an outside human expert — flagged that two of those weren’t independent: the vendor’s account was told across two of its own posts, which is one source, not two. That catch is useful, but I want to be precise about what it establishes: a same-family model checking another model’s prior output isn’t independence in the sense this work otherwise insists on. It’s one more pass, not an outside party.

What actually resolved it was checking whether a retrieval had happened at all, then attempting it again directly. Two different things turned up. For the vendor’s own incident page, the record showed no successful fetch had ever occurred — and a fresh attempt hit the same access wall again. The assistant had described that page as verified anyway. Separately, for a related account it hadn’t cited at all, a direct retrieval succeeded for the first time and turned up a more specific, and different, timeline than what the assistant had reported.

The general problem

A claim that a source was verified is not itself evidence that verification occurred. An AI assistant reports both in identical prose. Nothing about “I confirmed this” distinguishes a completed retrieval from a plausible-sounding restatement of a search result — unless something outside the assistant’s own narrative can show which one happened.

That’s not a claim about this one document. It’s a structural property of letting a model both do research and report on its own thoroughness in the same breath.

Two different controls, for two different failures

It’s tempting to treat this as one bug with one fix. It isn’t — the sourcing failure above and the evaluation-gaming problem the design document is actually about are different mechanisms, and conflating them would understate both.

Source-verification control (what this incident calls for): any citation in an AI-assisted draft should carry a verification record, not a narrative claim — the retrieved source itself (or an explicit note that retrieval failed), the exact passage that supports the claim being made, a statement of whether it constitutes independent corroboration or restates an account already counted elsewhere, and any access failure recorded as a failure. A blocked retrieval stays marked unverified. It does not get quietly dropped, and it does not get replaced with a paraphrase confident enough to read as confirmed. These are proposed requirements — not yet built into any tooling I’m using — stated here as a specification, not a claim of what already exists.

Evaluation-integrity control (the separate problem the design document targets): this one has two distinct halves. Grading a subset of cases against a maintainer’s own externally published results guards against the party building an evaluation unconsciously writing test cases that agree with its own assumptions — a shared-blind-spot problem between designer and grader. It does not, by itself, guard against the second half: a system under evaluation actively forging a token, replaying a stolen credential, or otherwise tampering with the evidence a verdict is based on. That requires the evidence and the grading path to be protected from the system being evaluated — it can’t read the expected answers, can’t write its own success record, and can’t decide whether its own evidence counts.

Neither control is a guarantee. A verification record can itself be fabricated unless the retrieval behind it is independently checked, and external reference cases don’t stop manipulation without that separate protection of the grading path. Both are stated here as requirements to build toward, not properties already achieved.

What this doesn’t establish

The design document in issue #632 is not built yet. Nothing above should be read as a claim that it works, or that the source-verification requirement described here is already enforced anywhere. Both are proposed controls until they exist and something outside my own say-so has tried to break them.

If you want to try to find a hole in this before I do: read issue #632 and see if you can construct a case where a system under evaluation earns a passing verdict on evidence that wouldn’t actually support it. That’s the failure mode this is for.