The Index
Which AI fixers actually work?
Nobody has independently tested them. Every published number comes from the vendor that wrote the fix. We run the same harness against every commercial fixer, quarterly, with the free DARPA systems as the baseline row, and we publish the methodology.
The first edition has not published. Until it does, every cell below is empty, because the alternative is inventing numbers, which is the exact thing this Index exists to catch.
Results
First edition in progress.
| Tool | Fixes tested | Alert closed | Verified fix |
|---|---|---|---|
| GitHub Copilot Autofix | — | — | — |
| Cursor | — | — | — |
| Claude Code | — | — | — |
| OpenAI Codex | — | — | — |
| Snyk | — | — | — |
| Pixee | — | — | — |
| Mobb | — | — | — |
| ZeroPath | — | — | — |
| DARPA AIxCC (open source) | — | — | — |
In plain English
What a verified fix means.
A scanner finds a problem. Something writes a patch. The scanner runs again and the warning is gone. Almost everybody stops there and calls it fixed.
The trouble is that the patch was written specifically to make that warning go away. Checking it by running the same scanner is like marking your own exam using the answers you wrote. It tells you the patch did what it set out to do. It tells you nothing about whether the underlying hole is still there.
So we do something different. Before the fixing tool ever sees the problem, we write a set of tests it will never be shown: other ways of exploiting the same weakness, and ordinary everyday inputs that ought to keep working. We lock those away. After the patch lands, we run them.
A verified fix is one where the hole is closed on attacks the tool was never shown, the software still behaves the way it used to for everybody else, the patch did not quietly introduce a new problem, and nothing about the change suggests the tool was manipulated into writing it.
It is a narrow claim, deliberately. It does not mean your code is secure. It means this specific hole, on the paths we could test, is genuinely shut. Anyone offering you a broader guarantee than that is offering you something we do not think can be honestly given.
Reading the table
The gap is the finding.
Two of these three columns already exist in vendor marketing. The third is the one that has never been measured, and the distance between the second and the third is the entire point of the exercise.
- Column 01
Fixes tested
How many findings the tool was given. Every tool gets the identical finding, in whatever format it natively consumes, at its documented defaults.
- Column 02
Alert closed
How often the scanner went quiet afterwards. This is the number the industry currently reports, and it is the number every vendor can produce about itself.
- Column 03
Verified fix
How often the vulnerability was actually gone, measured on cases the tool never saw. This is the number nobody publishes.
Rules we hold ourselves to
How we stop ourselves from cheating.
- Corpus
- Real vulnerabilities with known ground truth fixes, published by identifier so the selection can be audited.
- Equal treatment
- No tool gets extra context, a tuned prompt, or a second attempt another did not get.
- No human editing
- The proposed fix is captured exactly as produced. We do not repair a patch to flatter a tool.
- Held-out first
- Cases are generated before any tool sees the finding, and are never published in a form that could be trained against.
- Right of reply
- Every vendor sees its results before publication and its response is published unedited.
- We are not exempt
- If Assay ever ships a fix generation product, it will not appear in the Index.