The only rigorous randomized trial on AI coding productivity found that developers using AI tools were 19% slower. They believed they were 20% faster.
That’s the finding from METR’s 2025 study — 16 experienced open-source developers, 246 real tasks, properly randomized. Not a vendor survey, not self-reported estimates. An actual experiment, run by a team with no financial stake in the result.
The perception gap is 39 percentage points. Participants predicted a 24% speedup before the study. During the study, they believed they were 20% faster. The actual result: 19% slower. The 95% confidence interval runs from −40% to −2%, meaning the study can’t rule out that AI tools nearly halved expert productivity in this context.
This is the most important piece of evidence in the vibe coding debate, and almost no one is leading with it.
The Feeling of Speed
The METR finding isn’t surprising once you understand what vibe coding actually does.
Using AI doesn’t make you faster at the task you were already doing. It changes the cognitive texture of the work. The effort of syntax recall, of remembering where the semicolons go, of looking up library methods — that friction disappears. What replaces it is a different friction: reading AI output, evaluating it, catching what’s wrong, debugging what looks right but isn’t.
The METR researchers call the net result the “Verification Tax.” The time saved on generation is more than consumed by validation. But here’s the perceptual catch: syntax friction feels like work. Verification friction feels like progress. You’re reading something that looks finished. You’re clicking “accept.” The sensation is fluency, even when the underlying reality is slower.
This is what makes vibe coding a productivity placebo. It reduces the sensation of effort without reducing — and potentially while increasing — actual time spent.
The danger isn’t that people use AI tools and slow down. The danger is that they slow down and are confident they’re accelerating. Decisions made on false premises compound. If you believe AI is saving you 20% of your time, you stop looking for the hours it’s actually consuming.
The Wrong Question
The productivity framing is understandable. “Does AI make developers faster?” is a measurable question that generates publishable papers. But it’s the wrong question for most of the people actually vibe coding — because vibe coding isn’t about making existing developers faster at the things they already do.
Scott H. Young built a flashcard application — not to compete with Anki, but to embed a specific cognitive model that no commercial product would implement. The app incorporates Zipf’s law, ACT-R modeling (how human memory encodes and retrieves), and Bayesian estimation for difficulty calibration. The result is a personal learning tool tuned to his own research and how his own memory works.
The detail worth noting: “ChatGPT never suggested the innovations spontaneously.” Young provided the structure. The AI provided the implementation. Without his domain expertise, the app would have been a generic flashcard tool indistinguishable from hundreds of existing ones. With it, the app is something that couldn’t have been commissioned, because no engineer would have understood the brief well enough to build it, and the economics wouldn’t have supported the attempt anyway.
This is the question that actually matters: what becomes possible that wasn’t possible before?
The Economics of the Threshold
Simon Willison, Django co-creator, has built more than 77 single-purpose tools using AI assistance — each linked to its originating conversation. The collection is a working demonstration of a shift in economic logic.
Traditional software economics followed a brutal threshold: a tool was worth building only if its utility justified the cost of building it. In practice, for most knowledge workers, that meant most tools never got built. The problem wasn’t that they didn’t know what they needed. It was that what they needed was too specific, too personal, too small to justify any developer’s time.
Willison’s observation is that the threshold has moved. When a tool takes fifteen minutes instead of fifteen hours, the entire calculation of what’s worth building changes. Tools that were economically irrational to build are now rational. Problems that were solved with spreadsheet workarounds can be solved with something designed for the specific problem.
This isn’t a productivity story in the conventional sense. Willison isn’t doing the same work faster. He’s doing work that previously didn’t get done. The METR study can’t measure that. It wasn’t designed to. The 16 developers in the study were working on existing tasks in existing codebases. They weren’t measuring whether AI unlocks previously impossible work — they were measuring whether it accelerates currently-being-done work. For that specific question, the answer is no. But that’s not the question that explains why practitioners keep building with it.
Programming Liberated from the Requirement to Be Professional
Robin Sloan put the conceptual frame most precisely: “When you liberate programming from the requirement to be professional and scalable, it becomes a different activity altogether, just as cooking at home is nothing like cooking in a restaurant.”
Cooking professionally means consistency at scale, ingredient cost management, kitchen logistics, health codes. Cooking at home means making what you want to eat, adjusting for your own taste, feeding four people instead of four hundred. The technical skills overlap, but the activity is different. A home cook isn’t a failed restaurant chef.
Vibe coding is programming for the home cook. Not in the sense of being low quality — Young’s flashcard app is technically sophisticated. In the sense of being liberated from the constraints that professional software requires: maintainability for teams, security for production deployment, scalability for users you don’t know. When those constraints dissolve, what you can build changes.
This matters for knowledge workers in particular. The population of people who have strong domain expertise but lack coding fluency is enormous. They understand their work — what it requires, what would make it better, what a useful tool would actually do. What they lack is the ability to translate that understanding into software. Vibe coding doesn’t give them that ability in any traditional sense. It gives them a different path to the same destination: an artifact that does what they understand.
MIT Sloan’s Rama Ramakrishnan describes AI coding tools as enabling knowledge workers who can think computationally — who can decompose a problem into logical steps, define inputs and outputs, articulate what “working” means — to build things that previously required a software engineer to execute. The skill that transfers isn’t coding. It’s structured thinking.
Where the Gains Actually Accrue
The most carefully measured productivity data we have comes from Daniotti et al. (2026, Science) — 160,097 developers, 30 million commits, the largest study of AI coding assistance yet conducted. The headline number is a 3.6% improvement in productivity.
That number requires unpacking. The gains are real. And they are concentrated.
Across that population of 160,097 developers, the 3.6% average conceals a highly uneven distribution. Experienced developers capture disproportionate benefit. Beginners, the study notes, “hardly benefit at all.” The tools that are theoretically democratizing coding are, in measured reality, primarily accelerating the people who already knew how to code.
This is not a reason to dismiss vibe coding. It is a reason to ask a sharper question: what exactly is being amplified?
The METR study and the Daniotti data together suggest the same underlying dynamic. AI tools change the work. They don’t uniformly improve it. The people who benefit most are those who bring enough structural knowledge to the interaction to direct the AI effectively, catch what it gets wrong, and know when to override it. The people who benefit least are those who don’t have that knowledge and therefore can’t compensate for the AI’s limitations.
This is the amplifier pattern. Not a rising tide. A multiplier applied to whatever existing capability you bring.
What Vibe Coding Is Actually Doing
The productivity placebo frame isn’t a dismissal. The placebo effect is real and clinically significant. Something happening is better than nothing happening. But it’s a different kind of something than people believe.
Vibe coding is not making knowledge workers faster at their existing work. The evidence doesn’t support that, and the mechanics of how the tools work make it structurally unlikely for anyone without a strong existing capability base.
What vibe coding is doing is shifting the threshold for what’s worth building. Tools that were economically irrational to commission become rational to build in a weekend. Problems that were solved with workarounds get solved with something designed for the actual problem. Domain experts who think computationally can externalize their expertise into artifacts that didn’t exist before and wouldn’t have been commissioned.
That’s a genuine capability. But it accrues to people who already have something to amplify.
Which raises the question the productivity framing entirely misses: if the gains concentrate with experts, what happens to everyone else? And if vibe coding is an amplifier rather than a democratizer, what separates the people it amplifies from the people it leaves behind?
Those are the questions worth asking. The next piece tries to answer them.
The evidence base behind this article is searchable at the Evidence Explorer. Every claim links to the underlying study or source.
Tim Dickinson leads Learning Experience Enablement at Novartis. He writes about AI capability, organizational design, and the gap between how technology is described and how it actually works.