Independent verification for AI-authored code
Your AI fixed it.
We check whether
it actually did.
Every AI security tool now writes its own patches and certifies its own work. Assay is the third party that checks. One verdict on every AI-authored change, before it merges.
Free forever on public repositories · Installs in under a minute
Verifies fixes from
- GitHub Copilot Autofix
- Cursor
- Claude Code
- OpenAI Codex
- Snyk
- Pixee
- Mobb
- ZeroPath
- DARPA AIxCC systems
We are vendor neutral by design. We do not write fixes.
The verification gap
A patch can pass every test and leave the hole wide open.
Software is checked by tests. A test is a handful of examples of what the code should do. It is a sample, not a specification.
When you tell an AI to make the tests pass, it optimises for the sample. Passing the tests tells you the sample is satisfied and very little else.
Researchers have studied this for over a decade and named it overfitting.
The industry checks AI fixes by re-running the scanner. The AI was optimising to make that scanner go quiet. It is a circular test.
Test-passing patches that were actually correct
- Classic repair tool (GenProg)2%
98% were overfitting
- Strong 2024 frontier method (D4C)~50%
roughly a coin flip
- Five frontier models on 20 real CVEs, 202650%
best solve rate
Sources: Qi et al. via Invalidator (arXiv 2301.01113) · D4C 2024 · independent CVE study, May 2026
| Approach | Patches that were actually correct | Note |
|---|---|---|
| Classic repair tool (GenProg) | 2% | 98% were overfitting |
| Strong 2024 frontier method (D4C) | ~50% | roughly a coin flip |
| Five frontier models on 20 real CVEs, 2026 | 50% | best solve rate |
of AI-generated code introduces an OWASP-class vulnerability, and the rate is flat across model generations.
Veracode, 100+ LLMs, 2025
of committed code is now AI-generated.
Sonar, 1,100+ developers, 2026
of developers merge AI code without reviewing it at all.
Qodo, 2025
of AI code changes need debugging in production.
Lightrun, 2026
“The alert turning green and the bug being fixed are two different events. The industry measures the first one.”
Anatomy of an overfitted patch
It fixed the test, not the bug.
A bug report says an attacker gets in using the text 1' OR '1'='1. The agent writes a fix that rejects exactly that string. The scanner re-runs, the finding closes, CI goes green.
The injection is untouched.
Why the correct fix looks different
The real fix never inspects the payload at all. It stops building the query by string concatenation: db.query("... id = %s", [customer_id]).
Any check that asks “is the reported finding gone” approves the wrong patch. Only a check that asks “is this class of injection still reachable” catches it.
// the agent's "fix"
def get_customer(customer_id):
if "OR '1'='1" in customer_id:
raise ValueError(...)
return db.query(f"... id = '{customer_id}'")
// still works:
1' OR 'a'='a
1'; DROP TABLE customers; --
1' UNION SELECT password FROM users --The structural problem
Every vendor grades its own homework.
- Role 01Writes the fixthe same vendor
- Role 02Reviews the fixthe same vendor
- Role 03Certifies the fixthe same vendor
- Role 04, vacantIndependently verifies the fixnobody
OpenAI’s Aardvark writes the patch with Codex, then scans its own patch. Google’s CodeMender runs its own validation framework, and Google still puts a human researcher on every patch before it ships. Snyk, Pixee, Mobb and ZeroPath all certify their own work. This is not a scandal. It is a structural gap, and it is the one thing an author can never sell you.
Second beat
The human backstop broke too. Developers using AI ship three to four times more code. Review capacity did not move.
How it works
Findings in. Verdict out.
Four checks that nobody runs today, on every AI-authored change.
- Check 01
Closed
Re-scan after the patch. Is the original finding actually gone, or did it move somewhere else in the file?
- Check 02
Behaviour held
We generate test cases the agent never saw and run the patched code against those. Re-running the agent’s own suite is the overfitting trap, not a check. This is the hard part, and the reason the category exists.
- Check 03
No regression
Re-scan the change itself. Fixes routinely trade one finding for another.
- Check 04
Provenance clean
Which agent authored the change, under what token scope, and does the diff show injection markers: self-config edits, unexpected file touches, credential access.
If you let a student write their own exam, they will pass it. You test them on questions they have not seen.
In your workflow
One status check on the pull request.
- build & unit testspassed
- scanner re-runalert closed
- assay / proof-of-fixNOT VERIFIED
- held-out case sqli_tautology_v2 still reachable
- patch narrows on reported payload only
- merge blocked by policy · appeal in thread
Vendor neutral.
Verifies fixes from any agent, including the free DARPA systems.
Blocking or advisory.
Your policy, per repository.
Signed record.
Every verdict is an attestation naming the agent, the authority it held, and who approved.
The Index
Which AI fixers actually work?
Nobody has independently tested them. Every published number comes from the vendor that wrote the fix. We run the same harness against every commercial fixer, quarterly, with the free DARPA systems as the baseline row, and we publish the methodology.
| Tool | Fixes tested | Alert closed | Verified fix |
|---|---|---|---|
| GitHub Copilot Autofix | — | — | — |
| Cursor | — | — | — |
| Claude Code | — | — | — |
| OpenAI Codex | — | — | — |
| Snyk | — | — | — |
| Pixee | — | — | — |
| Mobb | — | — | — |
| ZeroPath | — | — | — |
| DARPA AIxCC (open source) | — | — | — |
The record
Prove a human oversaw the machine.
From December 2027 the EU AI Act requires logged, documented human oversight of high-risk AI systems. Singapore’s IMDA framework already requires an audit trail of which agent acted under whose authority. Assay produces a signed attestation for every AI-authored change: which agent wrote it, what authority it held, what verified it, and who approved. Independence is the part nobody can self-certify.
- EU AI Act Art. 12 & 14
- NIST AI agent standards
- CSA agentic trust
Integrations
Wherever the change already lands.
GitHub, GitLab, Bitbucket, Azure DevOps
A status check on every pull request. Blocking or advisory.
SARIF in, verdict out
Works with whichever scanner you already run.
Slack and Linear
Verdicts where your team already triages.
MCP and REST API
Pull verdicts into Cursor, Claude Code or Codex so the agent sees its own score.
Pricing
Free where the work is public.
Open source
Free forever
Public repositories
Unlimited verdicts on public repositories. Every verdict is public.
Team
Per repository
Monthly
Private repos, blocking policy, held-out test generation, dashboards.
Pricing in progress · talk to us
Talk to usEnterprise
Custom
Priced per agent under governance
Signed attestation, audit export, SSO, org-wide policy, priced per agent under governance.
Talk to us
We have no customers yet. These are the terms we intend to hold to.
Take nobody’s word for it.
Point Assay at a public repository and see what the last AI fix actually did.