(Crossposting with LinkedIn)
Much virtual ink has been spilled recently discussing “vibe” coding. Broadly defined, vibe coding is an AI-driven software development approach where developers use natural language prompts to generate and refine code. In this approach, the developer focuses more on the app’s functionality (“the vibe”) rather than writing code line-by-line. This, in theory, allows folks with little to no programming knowledge to create applications.
I’ve been experimenting with vibe coding, recently. I’ve built iPhone apps to help me focus, web apps to track family chores, and command line tools to help keep my computer organized. In each case, I’ve done minimal (approaching zero) direct writing of code and have relied on natural language prompts to the LLMs to produce the code and make adjustments. And in each case I’ve been impressed with the ability for LLMs to take my (sometimes vague) input, parse it, and turn it into compilable code.
So naturally, I’ve been wondering: if vibe coding can produce a working iPhone app from a few prompts, can the same approach work for analytics? I’ve spent large parts of my career borrowing concepts and best practices from the world of software engineering and applying them to the realm of social science analytics, and I’ve seen huge gains from doing so. But the more I think about it, the more I believe that the jump from vibe coding to “vibe analytics” isn’t just a lateral move — it’s a category error. While analytic programming and traditional programming look similar on the surface, they are fundamentally different in ways that make purely vibe-coded analytics not just impractical but genuinely dangerous.
How do they differ? Perhaps most importantly, the output of an analysis is, ultimately, a judgement. In traditional programming, correctness is usually binary and verifiable. The app either crashes or it doesn’t. In analytics, the output is a a number, a chart, or a conclusion requiring domain judgement. A query can run perfectly and return a completely misleading result. The LLM has no way to know that the statistics are off because it double-counted arrests. And neither will you unless you already understand the data well enough to sanity-check it.
That leads naturally to the difference between code, which is structured and has meaning that can be understood generically, and data, which is structured but has meaning that is almost entirely local to your organization. What does “active user” mean in your schema? Why does the orders table have two date columns and which one should you use? Why are there nulls in that field — is that meaningful or an artifact? LLMs can guess (and boy do they) but they cannot know, and bad assumptions here produce plausible-looking wrong answers.
And if our data requires local domain knowledge, well so do our programming goals. Traditional programming usually starts with a reasonably well-defined spec. Analytics often starts with a vague business question where part of the job is figuring out what the right question even is. “What’s driving churn?” isn’t a spec — it’s a research agenda. LLMs are good at executing defined tasks but less good at the iterative negotiation between data, business context, and question refinement that characterizes real analytics work.
The local nature of both data and purpose lead to another problem: the subtlety of errors. One of the most common analytics bugs is joining tables at the wrong level or direction and silently inflating or deflating metrics. A traditional programmer writing a wrong join usually gets an obvious error. An analyst writing a wrong join gets a confident-looking number that might be off by 300%. LLMs are particularly prone to this because they’ll generate syntactically correct SQL that joins on whatever seems reasonable without understanding the context of your specific tables.
I’ve been amazed by LLMs’ ability to write code, test itself and then correct itself. But in software engineering there’s a rich culture of unit tests, integration tests, and CI pipelines that catch regressions. In analytics, most work is exploratory and one-off, so there’s rarely a test suite. When you vibe code an app feature, you can at least click around and see if it works. When you vibe code an analysis, the only real test is whether someone with deep domain knowledge reviews the logic.
None of this means LLMs have no role in analytics — far from it. My own early experiments using LLMs to assist me in analysis have been promising. LLMs are excellent at accelerating the mechanical parts of the work: drafting boilerplate SQL, suggesting visualization approaches, writing documentation, and even helping think through edge cases in a dataset. The key word, though, is assist. Every one of those tasks still benefits from — and in most cases requires — a human who understands the data, the business context, the questions, and, yes, the code well enough to evaluate what the LLM produces.
