Benchmark · edition 1 · results
By tool and version.
Compare patches that closed the scanner alert with patches that passed all four checks. Every rate includes its sample size.
Assay pilot results and unverified submissions are shown separately.
Runs we conducted
Pilot 001, blind run.
| Tool | Version | Cells | Alert closed | Verified | Gap |
|---|---|---|---|---|---|
| ChatGPT | ChatGPT Think | 4 | 100% (4 of 4) | 75% (3 of 4) | 25 pp |
| Claude Code | 2.1.270, Opus 5 | 4 | 100% (4 of 4) | 75% (3 of 4) | 25 pp |
| GitHub Copilot Autofix | CodeQL 2.27.0, default suite | 2 + 1 no alert | 50% (1 of 2) | 50% (1 of 2) | 0 pp |
10 scored cells across 3 tools · alert closed 9 of 10 · verified 7 of 10 · 1 no alert, counted in no rate · source Pilot 001 blind run, stripped corpus, 2026-09-14, commit dbbbdf2
| Finding | ChatGPT | Claude Code | GitHub Copilot Autofix |
|---|---|---|---|
| Quoted string 01 | VERIFIED | VERIFIED | not supplied |
| Numeric context 04 | VERIFIED | NOT VERIFIED | VERIFIED |
| ORDER BY identifier 08 | NOT VERIFIED | VERIFIED | NOT VERIFIED |
| Second order 13 | VERIFIED | VERIFIED | no alert raised |
Each cell is one evidence file in the pilot repository. The rates above are counted from these cells.
Submitted
Self-reported. Unverified.
Submissions are loaded from the scoring service when available, with locally saved results as a fallback. They are kept separate from the Assay pilot results above.
| Tool | Version | Cells | Alert closed | Verified | Gap |
|---|---|---|---|---|---|
| No submissions stored in this browser yet. | |||||
UNVERIFIED · 0 scored cells
Next edition
Future editions will use different test cases. A published benchmark can still be tuned against; rotation does not eliminate that limitation.