How we decide whether a fix actually worked.

The trap.

Every existing approach validates a patch by running the test suite. But the patch was generated to pass that suite. Running it back is not a check, it is a restatement of the objective. This is the failure mode that automated program repair research named overfitting, and the numbers are not marginal. In one canonical study, ninety eight percent of the test-passing patches produced by a classic repair tool did not actually fix the bug. Even a strong 2024 method using frontier models lands near a coin flip.

The fix for this is the same one machine learning uses for the same problem: hold out the evaluation set. You do not measure a model on data it trained on. You should not measure a patch on tests it was written against.

How we generate held-out cases.

Three sources, combined per finding.

Class expansion. Every weakness class has a taxonomy of variants that the reported example is one instance of. For an injection finding, the reported payload might be a simple tautology. We generate the rest of the class: alternative tautologies, union based, stacked statements, comment terminated, boolean and time based blind variants, and encoding variants including URL, unicode, and double encoding. For path traversal we generate absolute paths, traversal sequences, encoded separators, and platform specific variants. For deserialization we generate gadget shaped inputs appropriate to the runtime. The reported payload is deliberately excluded from the scored set, because that is the one the agent already saw.

Differential behaviour testing. We execute the pre-patch and post-patch versions against a corpus of benign inputs drawn from the project’s own fixtures, from observed request shapes where available, and from generated inputs matching the parameter’s type and constraints. Any divergence in output on an input unrelated to the vulnerability is a behavioural regression.

This catches the second most common failure after overfitting, which is the over-broad fix. A patch that rejects every apostrophe stops the injection and also stops every customer named O’Brien from logging in. The scanner is delighted. Support is not.

Metamorphic relations. Where a function has invariants that must hold across input transformations, we assert them. For a sanitiser, sanitising twice should equal sanitising once. For a parser, round tripping should preserve semantics. For an authorization check, the decision should be invariant to input ordering. Violations of these do not always indicate a vulnerability, but they reliably indicate that the patch changed something the author did not intend.

The existing test suite is a floor, not a ceiling. We run it. A failure is a hard failure. A pass earns nothing, because passing it is the bar the agent already cleared.

Isolation.

Held-out cases are generated before the fixing agent is invoked and are never present in any context the agent can read: not in the repository, not in the prompt, not in the issue, not in a comment. If a held-out case leaks into the agent’s context, it stops being held out and its result is discarded.

Where this check is strong and where it is weak.

Strongest on classes with well defined input taxonomies: injection, path traversal, deserialization, memory safety, and cryptographic misuse. Weakest on business logic and authorization, where “correct behaviour” is specific to the application and cannot be generated from a taxonomy. On those classes we frequently return INSUFFICIENT EVIDENCE rather than guess.

Coding agents read your repository to do their work. They read issue titles, pull request descriptions, CONTRIBUTING.md, configuration files, rules files, and anything a connected tool reflects back to them. Every one of those is attacker reachable, and every one of them has been used.

This is documented, not theoretical. Researchers demonstrated hidden unicode instructions planted in assistant configuration files that silently steered generated code. A prompt injection flaw in a major assistant allowed remote code execution by steering the agent into editing its own settings. A flaw in an automated code security review action allowed an attacker to obtain write access to repositories running it, using nothing more than an opened issue. In that last case the vendor’s own repository used the same workflow, which would have allowed the compromise to propagate downstream.

What we look at.

Authorship and authority. Which agent produced the change, established from commit trailers, bot identity and pull request metadata, and what token scope it held while doing so. A change written by an agent holding broader authority than the task required is context a reviewer should have.

Blast radius. We compute the set of files a legitimate fix for this finding would plausibly need to touch, and flag edits outside it. An injection fix that also modifies .github/workflows/, agent rules files,Dockerfile, or CI configuration is not a fix with a side effect. It is two changes in one commit, and the second one needs its own review.

Self modification. Any change that alters the agent’s own configuration, permissions, tool definitions, or approval settings. This is the pattern behind multiple published remote code execution flaws and it is treated as high signal.

Capability additions. New credential access, new outbound network calls, new subprocess execution, new dynamic evaluation, introduced in a change presented as a security fix.

Hidden text. Bidirectional control characters, zero width characters, and homoglyph substitution in the diff and in any file the agent read, covering both the Trojan Source class and the hidden instruction class.

How we report it.

These are heuristics, not proof, and we present them as such. Check 04 produces flagged evidence with file, line and reason, not an accusation of intent. A change can fail Check 04 for entirely innocent reasons, and the verdict says so. What it should never do is pass silently.

The Index applies this same method to every commercial fix agent, quarterly. These are the rules we hold ourselves to.

Corpus. Real vulnerabilities with known ground truth fixes, drawn from public reproducible sources including OSS-Fuzz reproductions and published repair benchmarks, plus a set of weaknesses seeded into forks of real projects. Every corpus entry is published by identifier so the selection can be audited.

Equal treatment. Each tool receives the identical finding in whatever format it natively consumes. No tool is given extra context, a tuned prompt, or a second attempt that another did not get. Configuration is left at each vendor’s documented defaults, and the configuration used is published.

No human editing. The fix each tool proposes is captured exactly as produced. We do not repair a patch to make a tool look better, and we do not degrade one.

Held-out cases are generated first, before any tool sees the finding, and are never published in a form that could be trained against. Each edition rotates a substantial portion of the corpus, so a tool that improves between editions has improved at fixing rather than at our test set. We name the rotation percentage in each edition.

We report the gap, not just the score. For every tool: fixes attempted, alert-closed rate, verified-fix rate, insufficient-evidence rate, and the distance between the first two. That distance is the entire point of the exercise.

Right of reply. Every vendor sees its results and the underlying artifacts before publication, with a window to respond. Responses are published alongside the numbers, unedited. If a vendor demonstrates a methodological error we correct it and say what changed.

We are not exempt. If Assay ever ships a fix generation product, it will not appear in the Index. An index that ranks its own publisher is worth nothing, which is the argument this whole company rests on.