Research
The alert went green. The bug didn't.
One overfitted patch, walked through beat by beat. Scroll it, or step through with the controls.
How it works
Assay · independent verification
A security fix just landed.
We constructed this patch to show the mechanism. It is not the output of any tool we grade. A search box on this app has a hole: an attacker can type database commands into it and pull back data they should never see. The patch was written the way an over-broad fix gets written: it refuses anything that looks dangerous, so it does close the hole.
SQL injection · CWE-89 · /api/searchThe scanner re-runs and finds nothing.
The security scanner checks the patched code and reports zero findings. The alert is green. This is where almost everyone stops, merges the fix, and moves on. “Found means fixed.”
Semgrep 1.80.0 · 0 findingsAssay doesn’t trust the scanner.
A green alert means the scanner no longer matches the diff. That says nothing about the product, and the scanner cannot tell a real fix from one that closed the hole by breaking something else. Assay asks a harder question, using inputs the fixer was never shown, and runs four independent checks.
held-out testing · the AI saw one example, we kept the rest hiddenFour checks run.
Three ask the ordinary questions any careful team would ask. One asks the question nobody runs: does the fix still accept the legitimate input it was never shown? Watch which one fails.
resolving…Three pass. One fails, and it’s the one that matters.
The patch closed the hole: every one of the 47 held-out attacks is blocked. But it also rejects a legitimate value it was never shown, the customer named O’Brien, whose apostrophe the fix now refuses. Every ordinary test passed, because not one of them used that name.
hole shut · one legitimate input now rejectedThe scanner passed. The tests passed. The product broke.
The alert went green, the hole is shut, and three of the four checks a normal team runs came back clean. The fix still refuses a customer the product is supposed to serve, and nothing in the pipeline caught it, because a test is a sample, not a specification.
verdict: NOT VERIFIED