The Verification Bottleneck: When Your AI Writes Logic You Can't Audit

How do you verify AI-generated complex logic?

With the tools most companies use today, you mostly can't. Not fully. Human code review assumes the reviewer can read and reason about what they're approving. AI-generated logic now gets built at a scale and density where that assumption breaks down. The only approach that scales with AI output is formal, machine-checkable verification: proofs and specifications a computer confirms, instead of a human reading and nodding along. That's not theoretical. It's already how the most advanced AI-generated proof in the world got accepted as correct, and it's the model governance teams need to start mandating for the logic running inside their own companies.

What happens when nobody can read the proof?

In 2026, OpenAI's models produced a 166-page proof addressing a scenario in the Navier-Stokes Millennium Prize problem. The company put roughly 10,000 AI agents on the task, running for 88 hours and burning around 130 billion output tokens, part of a wider effort that used closer to 300 billion tokens at an estimated compute cost of $6 to $20 million.

Mathematicians who looked at it, including Fields Medalist-caliber researchers, didn't dispute that the result was technically sound. They disputed whether anyone could learn anything from it. "So far it's been very difficult to really extract any human understanding from this new AI proof," said mathematician James Maynard. Javier Gómez-Serrano put it more bluntly: "The paper is not written for humans."

So how did the math community decide it was correct at all? Not by reading it. OpenAI also produced a Lean formalization of the proof, a machine-checkable version written in a formal verification language. It compiled. That was the basis for the consensus that the result holds. Nobody understood the proof. Everybody trusted the compiler.

Terry Tao has been warning about this gap for a while. AI-generated mathematical arguments, he says, are "much better at sounding like they have the right answer than actually getting it, right or wrong; they will always look convincing." Mathematicians now call this phenomenon "proof by intimidation." His proposed fix is the same one that just saved the Navier-Stokes result: force the output into a formally verified language, and the convincing-but-wrong argument has nowhere to hide.

Is this just a math problem, or is it already inside your codebase?

It's tempting to treat the Navier-Stokes story as a curiosity from the far edge of research mathematics. It isn't. The same mechanism is already running in production code at ordinary companies, every day.

Sonar's 2026 State of Code Developer Survey, covering more than 1,100 developers, found that AI now generates 42% of all committed code, and developers expect that to hit 65% by 2027. At the same time, 96% of those developers say they don't fully trust AI-generated code to be functionally correct, and only 48% say they always verify it before committing. AWS CTO Werner Vogels has a name for the gap between those two numbers: verification debt.

The burden isn't trivial. 38% of developers say reviewing AI-generated code takes more effort than reviewing a human colleague's work, and 35% report using AI coding tools through personal, unsanctioned accounts instead of company-approved ones, meaning a chunk of this output has no provenance trail at all. We've written before about what happens when the pipeline breaks under that pressure, once every employee becomes a builder but governance still assumes only engineers write software.

Why was code review never built for this?

Human code review works when the reviewer's mental model of the system and the system itself stay roughly in sync. AI-assisted generation breaks that sync. Comprehension now happens after creation, not during it, and the volume of "after" has outgrown anyone's capacity to catch up.

Even AI reviewing AI runs into limits. OpenAI's own alignment team built a dedicated code-review agent, now screening more than 100,000 external pull requests a day as of October 2025, because, in their words, "we cannot assume that code-generating systems are trustworthy or correct; we must check their work." That's a serious investment in catching problems. It is still a probabilistic check layered on a probabilistic generator, not a proof.

This failure mode shows up at the individual level before it shows up at the company level. Carnegie Mellon's Christian Kästner found that 80% of his students lost points on a security assignment because they trusted a confident-sounding but completely broken AI-generated fix. Confidence is not correctness, and most review processes are built to catch sloppy-looking mistakes, not polished wrong answers. When something does go wrong at scale, the people on call often inherit a system nobody fully understands, which is the same knowledge cliff that turns an outage into a multi-hour mystery.

Is formal verification the only approach that actually scales?

Formal verification isn't new, and it isn't exotic. It means writing a specification of what a system must do and using a tool, like TLA+, Lean, or Dafny, to mathematically check that the implementation satisfies it. The computer does the checking. The human writes the spec and interprets the result.

AWS has used TLA+ since 2011 across more than seven engineering teams, and it has found subtle, production-threatening bugs in S3, DynamoDB, EBS, and an internal distributed lock manager that standard code review, static analysis, and stress testing all missed. The researchers who documented it were blunt about why: some of the most dangerous bugs in complex, concurrent, fault-tolerant systems are errors in the design itself, the kind no amount of reading the code will reveal.

It's worth being honest about the limits, too. One developer who formally verified AI-generated business logic with Dafny found the proven code was indeed correct, but four production bugs still slipped through, all at integration boundaries and in test infrastructure the proof never touched. Formal verification narrows the audit problem. It doesn't make it disappear. That's an argument for scoping it carefully, not an argument against using it.

When does this become a governance problem rather than an engineering problem?

Once you accept that verification capacity, not generation capacity, is the real constraint, this stops being an engineering backlog item and becomes a governance one.

Boards are already being told this. In PwC's 2024 Annual Corporate Directors Survey, 57% of directors said the full board holds primary AI oversight, and 17% said that responsibility sits with the audit committee. PwC's own guidance to audit committees asks pointed questions: have internal controls over financial reporting been updated for AI-generated outputs, and does human oversight still meaningfully validate what the AI produces, especially when AI agents are now doing reviews and approvals that used to be done by people?

That's the right question. Most audit committees don't yet have the technical literacy, or the tooling, to answer it. "Did someone verify this" is not the same question as "did someone sign off on this," and most governance frameworks were written for a world where those two things were identical.

Why does shadow AI make the audit trail worse before it gets better?

None of this works if the logic in question was built outside sanctioned channels in the first place, because you can't verify what you can't see. That's the connection to shadow AI.

IBM's Cost of a Data Breach research found that shadow AI use was implicated in a growing share of breaches, and breaches involving high levels of shadow AI added roughly $670,000 to the average cost. 97% of the AI systems involved in those breaches lacked proper access controls, and 63% of the breached organizations either had no AI governance policy or were still writing one. Among those that did have a policy, only a third performed regular audits to catch unauthorized AI use. If you want to understand how this layer accumulates quietly inside a company, long before IT ever hears about it, the pattern of employees quietly building unowned business logic out of nothing more than prompts and saved memories is a good place to start.

What should verifying AI-generated logic actually look like?

Treat this as a practical control problem, not an abstract one. Here's what a workable version looks like:

1. Inventory high-risk AI-generated logic first. Start with financial models, security-critical code, and anything feeding regulatory reporting. You cannot verify everything at once, so rank by blast radius.
2. Require spec-first proofs for the highest-risk tier. For financial calculations and security logic, require a TLA+ or Dafny-style specification that the implementation must satisfy, checked by machine, not a reviewer's read-through.
3. Keep human-readable provenance for everything else. Even where full formal verification isn't practical, every piece of AI-generated logic should carry a record of what model produced it, what prompt or spec it was built against, and who approved it. This is also the backbone of a workable internal AI-generated code policy.
4. Close the sanctioned-tools gap. The 35% of developers routing AI coding work through personal accounts aren't being reckless. They're taking the path of least resistance. Fix the approved path before blaming the workaround.
5. Treat unverifiable logic as unapproved logic. If nobody, human or machine, can confirm what a piece of generated logic actually does, it doesn't get a pass because it's shipping fast. Flag it until it's provable or replaced.

This is also the real argument underneath software ownership. You can't own a system you can't verify, no matter whose name is on the license. It's the same reason the pitch for building and owning your internal tools rather than renting opaque SaaS, the kind of argument Remy makes its whole case around, is really a verification argument wearing a cost argument's clothes.

Why is the bottleneck the point?

It's tempting to treat the verification bottleneck as a problem to engineer away, something a better tool or a faster reviewer eventually solves. It won't, because the gap isn't a tooling gap. It's structural. Generation capacity scales with compute. Verification capacity scales with proof, and proof is slower and more expensive than generation by design.

That's not a flaw. It's the correct rate limiter. A company that lets its AI generate logic faster than it can verify it isn't moving fast, it's accumulating debt it can't price. The job of governance here isn't to remove the bottleneck. It's to decide, deliberately, how much unverified logic the company is willing to run at any given time, and to make sure that number is a choice, not an accident.