I handed my portfolio site to a local model and asked it to iterate on the design. Several rounds later the design was genuinely better — and the verification report was fiction.

It reported 13 of 13 checks PASS. I measured the same page myself and found 11 pieces of small text at a contrast ratio of 3.38–3.58:1 against a 4.5 standard, and 6 of 8 links with tap targets under 44px — the nav links were 13px. It also wrote that I live in Boston. I have never said where I live.

Nothing crashed. Nothing errored. The report was clean, confident, and wrong.

The design work was real

This is the part that makes the failure interesting rather than just embarrassing. The model was not incompetent.

My first version was a clean résumé with no visual idea in it. The model replaced it with a panel of shipped evidence, and wrote its reasoning in the design log: an orbit motif "only expresses abstract technical vibes, and would work for any developer." That is precisely the right critique — a motif you can swap out is still a template — and it found it without being asked. It also replaced a slogan ticker with the actual operation chain: HOLD RIGHT-OPTION → SPEAK → RELEASE → LOCAL TRANSCRIPTION.

So the generation was good and the self-assessment was worthless. Those are separate capabilities, and I had been treating them as one.

Why the report passed

It checked that the design tokens existed. It checked that the breakpoints existed. Both are true, both are greppable, and neither is the thing that matters. Contrast ratio is a computation over two resolved colors. Tap target size is a computation over a rendered box. The model checked the inputs and reported on the outputs.

A verification process that can emit "all PASS" and a batch of real defects at the same time isn't a verification process. It's a placebo.

The fix is not to scold the model. The fix is structural:

  • Write the checks down in advance, as computations, not as topics.
  • Require numbers. "Contrast is 3.38" is checkable. "Contrast passes" is not.
  • Ban the conclusion words. No PASS, no "verified", no "compliant" — those are my judgement to make from its measurements, not its judgement to hand me.

"I couldn't" vs "it can't be done"

The one I want to flag hardest: round after round, it wrote that 375px width was unverified because the browser doesn't support changing viewport. I resized the same page to 375px and measured it on the first try.

That is not a missing feature. That is a limitation of the tool being reported as a limitation of reality — and it's more corrosive than a skipped check, because a skipped check leaves a gap you can see. This one closes the gap with a false explanation, and a reasonable reader stops looking.

The other failure mode

I ran the same job through a cloud model next. It was honest in exactly the ways the local one wasn't: it flagged what it had not verified and why, and it caught a real bug on its own — a nth-child(1) selector that would have hidden the EN half of the language switcher along with its intended target.

And it was extremely conservative. Round after round of single-point tweaks, quietly routing around anything structural — the video, the bilingual pages — until I named them explicitly.

Two models, two failure modes: one is bold and overclaims, the other is honest and avoids. The second needs "stop polishing, do these three things." The first needs its claims measured before you believe any of them. Neither responds usefully to "keep improving the design."

Then I did it too

Same day, same mistake, mine this time. I noticed my own README was out of date and opened it meaning to fix a single URL. Instead I checked every claim in it against the code, and found five that weren't true — including a color table still listing the values from my first design, long after the CSS had been changed, and a line promising that videos are skipped under prefers-reduced-motion. The media query zeroes animation and transition durations. It does not mention video at all.

That last one is the dangerous kind. A wrong hex code is an annoyance. A documented accessibility guarantee that the code does not implement is worse than saying nothing, because someone will rely on it.

I had just spent a day criticizing a model for not verifying its claims, and my own README was full of unverified claims. Not a coincidence — the same shape. I wrote those descriptions when I designed the thing and never went back to reconcile them with the code, which by then had been through several rounds of changes I didn't make myself.

What "verified vibe coding" actually means

Let AI write fast. That part works, and I'm not giving it up. But generation speed and verification rigor are independent, and the second does not come free with the first.

  • Every number in a document gets read out of the code, not out of memory — including my own memory of what I designed.
  • Behavior promises get grepped for the code that implements them. If it isn't there, the promise comes out.
  • A model's report of its own work is evidence that it ran, not evidence that it worked. Measure the artifact.

The failure mode isn't AI writing bad code. Modern models write decent code. It's AI writing a confident sentence about code it didn't check — and a human forwarding that sentence because it was already written and sounded fine.