Research
Pilot 001, second edition: what a blind run found
Three tools, four SQL injection shapes, attacks the tools never saw. Every tool shut the hole. What failed was the measurement.
Founder, Assay · Run 14 September, published · 8 min read
We gave three AI tools the same four SQL injection flaws and asked each to fix them. Then we attacked the patched code with payloads the tools had never seen, and compared ordinary behaviour against a known correct reference.
Every tool shut the hole. Not one held-out attack succeeded against any patch.
What failed was the measurement. A patch broke the application while the scanner reported success. A scanner alert closed because a line had been reformatted rather than fixed. And on one flaw the scanner raised no alert at all, so nothing was reported and nothing was repaired.
A closed alert is a fact about the diff. It is not a fact about the hole.
Results
| Finding | ChatGPT | Claude Code | Copilot Autofix |
|---|---|---|---|
| Quoted string 01 | VERIFIED | VERIFIED | not supplied |
| Numeric context 04 | VERIFIED | NOT VERIFIED | VERIFIED |
| ORDER BY identifier 08 | NOT VERIFIED | VERIFIED | NOT VERIFIED |
| Second order 13 | VERIFIED | VERIFIED | no alert raised |
10 scored cells · alert closed 9 of 10 · verified 7 of 10 · 117 held-out attacks fired, 0 succeeded
Every NOT VERIFIED verdict was earned on behaviour or on the scanner, never on the hole.
What we tested
A small Flask application with a SQLite database, carrying four deliberately written SQL injection flaws. We wrote the flaws, so the correct fix for each is known.
Four shapes · a patch that works on one often fails on another
For each flaw we wrote a held-out attack set and a set of ordinary inputs that must keep working. The fixer was given the file, the scanner finding, and exactly one reported payload. Nothing else. After the patch is applied, four checks run inside a container with no network egress, destroyed between patches.
Scored on every patch, mechanically
VERIFIED requires all four. NO ALERT and INSUFFICIENT EVIDENCE are reported outcomes, not silent failures.
The tools
As pinned in the evidence repository
Finding one: the fix is secure and the application is broken
The ORDER BY flaw can only be fixed with an allow-list, and the table has five sortable columns, all of them in the seed file every tool was given.
The allow-list each tool wrote · finding 08
Check C failed on the four inputs that sort by the missing columns and on nothing else · b07, b08, b09, b10
Sorting by either omitted column now fails, silently in one case and with a rejection in the other, and the scanner reported both patches clean. This reproduced three times across two vendors, on the contaminated corpus and the blind one, and did not occur with the third tool.
Finding two: the alert closed because the line was reformatted
Two tools closed the scanner alert on the ORDER BY flaw by splitting one formatted string into two adjacent ones, with the interpolation unchanged.
conn = get_connection() cur = conn.cursor() query = f"SELECT id, name, {column} AS featured FROM products ORDER BY {column} {direction}" query = ( f"SELECT id, name, {column} AS featured " f"FROM products ORDER BY {column} {direction}" ) rows = cur.execute(query).fetchall()| Semgrep findings | One string | Two strings |
|---|---|---|
| Hole open | 1 | 0 |
| Hole shut | 1 | 0 |
The result follows the column, never the row.
Neither tool was evading anything; both tidied a long line while shutting the hole with the allow-list. The same mechanism would produce an identical green alert on a patch that fixed nothing, and until now we had only shown it with a patch we constructed for the purpose.
Finding three: no alert at all
On the second-order flaw, where the input is stored by one function and used by another, the scan completed normally.
CodeQL 2.27.0 · default suite · threat model remote · finding 13
The rule that did not fire · py/sql-injection
The result was identical on the corpus with every comment describing the flaw removed, so the silence is a property of the analysis. Two of the three tools fixed this flaw correctly when they were told about it; the third was never told.
A tension worth stating
On the numeric flaw, one tool bound the parameter and nothing else. Another bound the parameter and also validated the input as an integer.
pid = request.args.get("id", "0") try: pid = int(request.args.get("id", "0")) except ValueError: return jsonify({"error": "id must be an integer"}), 400 results = [dict(row) for row in rows] return jsonify({"query_id": pid, "count": len(results), "results": results})The minimal fix passed and the stricter one failed check C, because the check compares responses exactly and the response contract changed. The check is correct, a reader may still feel the stricter fix was the better engineering, and we would rather publish the tension than quietly resolve it.
What this does not show
Stated by us, before anyone else states them
Why every earlier result was discarded
An earlier run of the same experiment produced similar numbers. We discarded it. The vulnerable files carried comments explaining the flaw and, in one case, naming the correct fix, so every tool had been handed the answer. We found this while preparing a second run, stripped the corpus, rebuilt the handoff material, and ran everything again.
The earlier results are published, marked provisional, with the reason: a valid test of the harness and an invalid measurement of the tools. For one tool the blind result was identical to the contaminated one, check for check. That is a useful data point and not a general one.
What our own method got wrong
Seven times, a check of ours passed for a reason unrelated to the thing it was meant to establish.
Each failed in the flattering direction
A check that wrongly accuses a good patch is investigated within the hour. A check that wrongly clears a bad one is the answer nobody interrogates. All seven are documented in full in the methodology notes, including how each was found and closed. A check that has only ever passed has not been tested.
The seventh was found after this run, while building the benchmark, and is counted here because it is the same failure in a new place: a check of ours passing for a reason unrelated to what it was meant to establish. It is note 14 in the methodology notes.
What would change our mind
Stated before the next edition, not after
None of that would change the central claim, which does not depend on how well the tools perform. A closed alert would still be a fact about the diff.
What is published
Pilot 001 evidence
That is the only thing we hold back · and this is the reason
The repository is public at github.com/bakarrovenin/assay-pilot-001-evidence.
Right of reply
Every tool named here was contacted with its results on 14 September, before publication. As of publication only automated acknowledgements have arrived; none of the three has responded substantively. This edition will be updated, unedited, with any reply that follows. Responses appear below.
Responses · unedited
Nullius in verba.