Tag: AI

A Test for Bias

August 15, 2026 / Martin

I belong to a music league. Each week, about 20 friends and I create a communal playlist around some theme—“songs to dance to,” for example. We listen, then vote on which songs best fit the theme.

It’s a blast. It’s a good way to discover—or sometimes rediscover—music, and a good excuse to hang out with friends online.

But there’s a question that’s been nagging at me: is there systematic bias in Music League voting based on where a song appears in the playlist?

Maybe songs near the beginning get more votes simply because we hear them first—a primacy effect. Or maybe the opposite happens: songs near the end are fresher in our minds when it comes time to vote, producing a recency effect. Or perhaps both are true, with songs at either end benefiting at the expense of those stuck in the forgettable middle.

Fortunately, thanks to the hard work of another league member, I had the data needed to answer the question. And, as part of my continuing exploration of agentic programming, I decided to throw the problem at Claude Code and see what it would do.

A deliberately underspecified experiment

I’ve written before about the limits I see in “vibe analysis.” I was—and remain—skeptical that agents can replace humans in the analytic pipeline. There are too many places where analysis depends on understanding context, asking the right question, recognizing when the data don’t mean quite what you think they mean, and exercising judgment. But replacing analysts is a different question from making analysts dramatically faster. That’s where I see much more promise.

For this experiment, at least, I decided not to carefully walk Claude through the problem. I wanted to see how far it could get on its own.

So, after a couple of simple warm-up prompts—“read the data,” “produce a simple scatterplot of playlist order by votes,” and so on—I gave Claude this:

I’m interested in if there is bias in voting. Do earlier (or later) playlister order scores account for higher votes. Perhaps earlier order get more votes or perhaps later order gets more votes. Or perhaps both and there is a curvilinear relationship. Write a quarto report to explore this question in detail. Calculate appropriate statistics and illustrate with graphs. The output should be a quarto file that I can run to get the report.

And that was basically it.

I didn’t explain Music League. I didn’t explain how rounds or voting worked. I didn’t describe the structure of the data beyond pointing it toward the variables I cared about. In retrospect, there’s quite a bit I could have—and probably should have—added to the prompt.

But Claude took it and ran with it (as it always does).

So, how did it do?

It produced this report.

For those dying to know the answer: Claude concluded that there is a very slight primacy effect in our league. Songs appearing earlier in the playlist seem to have a small advantage. Having looked through the analysis, I agree with that conclusion.

More interesting to me, though, was the analysis itself.

And the analysis is…completely reasonable.

Not great. Not brilliant. But reasonable.

There’s too much filler, although I suppose I did explicitly tell it to “explore this question in detail.” It isn’t the polished report I would hand to a stakeholder, but I also didn’t ask it to produce a stakeholder-ready report. There are places where I would have made different choices or explained things differently.

Still, I found the overall analytic path sensible, the statistical choices defensible, and the conclusions appropriately supported by the evidence. It considered multiple possible relationships between playlist position and voting, visualized them, calculated relevant statistics, and ran several robustness checks rather than latching onto the first interesting result it found.

That last part is worth emphasizing. For a silly side project like this, I probably would have made a few plots, run a model or two, satisfied my curiosity, and moved on. Claude was more thorough than I would have bothered to be.

And the code?

Solid.

I read through the roughly 930-line Quarto file it generated. The code was straightforward and clear. Nothing particularly clever, but also nothing that made me nervous. It did have a little help from an analytics helper skill I’ve been developing, so this wasn’t quite an out-of-the-box test. Still, the vast majority of the analytic work was generated from that very short prompt.

That leaves me in a somewhat different place from where I started.

I still don’t think this demonstrates that we can hand an agent a dataset and dispense with the analyst. Claude didn’t know enough about the context to identify all the ways the analysis might go wrong, and it couldn’t know whether the question I asked was actually the right question. Those remain important limitations.

But that may also be setting the bar in the wrong place.

The more interesting question is what happens when you give a capable analyst tools that can take a loosely specified question and, in minutes, produce a credible first-pass analysis: clean code, visualizations, statistical tests, robustness checks, and a written report.

In this case, the answer was: something surprisingly useful.

The analysis took a fraction of the time it would have taken me to produce something comparable myself, while being more comprehensive than I probably would have bothered to be for a question of such…ummm…importance.

I remain skeptical of “vibe analysis.”

But I’m becoming considerably less skeptical of the idea that this is going to change how analysts work.

A Safe Approach to AI-Assisted Analysis

March 25, 2026 / Martin

(Crossposting with LinkedIn)

In a recent essay, I argued that “Vibe Analysis” couldn’t be a real thing—that analysis is too different from traditional programming for AI to fully replace human analysts. But as I noted in that essay, this does not mean AI has no place in analytic work. Far from it. AI has enormous potential to accelerate analytic programming; it simply must be applied cautiously, with careful attention to quality, rigor, and security.

Security—and how to enable analysts to work with AI in a secure way—is the primary focus here. It is a central aspect of my work at Urban Labs. While much of our analytic code is intended for public release, the same cannot be said of our data. In fact, it is rarely our data to begin with: the vast majority consists of administrative data entrusted to us by partners. As a result, we are obligated—ethically, legally, and reputationally—to safeguard that data and use it only in contractually specified ways. Each dataset is governed by agreements that precisely define who may access it and how it can be used.

Unsurprisingly, none of these agreements permit sending partner data to Anthropic or OpenAI. Nearly all require that data be stored and processed on servers owned and maintained by the University of Chicago. These environments adhere to strict security standards that ensure partners can trust their data is protected and used only for agreed purposes. Installing new applications in this environment is already difficult; installing an AI coding agent is simply not feasible.

Each analyst does have a local machine used to connect to the secure server, and in theory, these could connect to AI tools. However, for analytic purposes, these machines function largely as “thin clients”—machines that store no analytic code or data. Even so, they may still contain sensitive business information. Allowing AI agents to run freely on them raises legitimate security concerns.

This creates a fundamental problem: we have no safe place to run LLM-based tools, and we cannot expose our data to them even if we did. Compounding this is the reality of how analysts work. Our team—and every analyst I’ve worked with—develops code interactively and iteratively: write a small piece, run it, inspect results, revise, repeat. Writing an entire analytic pipeline in one pass without repeatedly validating against real data is not how analysis works in practice.

Taken together, these constraints make a straightforward AI coding workflow impossible:

Constraint 1: Our data is private and can never leave the secure server environment, including being sent to external LLMs.
Constraint 2: Our servers must meet strict security standards and cannot host LLM tools like Claude Code or Codex.
Constraint 3: Local machines may contain sensitive information and cannot safely run unrestricted AI agents.
Constraint 4: Analysts work iteratively, and any solution must support that workflow.

So how do we proceed? How can analysts benefit from AI without compromising security or disrupting how they actually work?

I don’t have a final answer, but I believe I do have a workable approach built on four components: LLM coding agents, containers, Git, and synthetic datasets. Together, these can form a secure and practical AI-assisted analytic environment.

At a high level, the approach works as follows: an LLM coding agent runs inside an isolated container. The LLM has access to project code via Git, but not to real data, which never leaves the secure server. Instead, the LLM interacts with synthetic datasets that mimic the structure of real data. This setup allows analysts to collaborate with AI during development, then push finalized code to the secure server for execution on real data.

Breaking this down further:

The process begins with containers. Containers allow you to create isolated computing environments—effectively “computers within computers.”¹ The container is completely separated from the host system. You define its operating system and installed software, and that’s all it can access. Running the LLM inside a container limits its scope entirely. It cannot read arbitrary files, execute unknown programs, or affect the host system—because, from its perspective, those resources do not exist.

This isolation creates a safe sandbox, but it raises a new question: how does the LLM do anything useful without access to real resources?

This is where Git comes in. Git is one of the tools installed inside the container. By cloning a repository into the container, the LLM gains access to the project’s codebase—allowing it to read, modify, and suggest improvements. Crucially, this includes only code, not data (and if you are storing data in Git, that is a separate problem). From the LLM’s perspective, the data still does not exist.

However, code alone is often insufficient context. Many analytic tasks depend on understanding the structure and content of the data itself. Asking an AI to summarize variables in a dataset is impossible if it cannot see those variables. But, as described above, we can’t expose our data to AI.

The solution, I believe, is synthetic data. Synthetic datasets replicate the structure and statistical properties of real data without containing any actual sensitive information². By providing synthetic data within the container, the LLM gains enough context to be useful while still preserving data security. Analysts can continue their normal iterative workflow—writing, testing, and refining code—using synthetic data alongside the LLM.

When development is complete, the workflow is simple: push the code via Git to the secure server, execute it against real data in the secure environment of that server, and generate results.

This approach does not eliminate all challenges, but it creates a viable path forward. LLMs will not replace analysts, but they will augment them. The responsibility is on us to ensure that augmentation happens in a way that is secure, controlled, and aligned with how analysis is actually done.

In practice, containers are more complex and offer many additional capabilities, but this captures the essential concept. ↩︎
Generating high-quality synthetic data can be complex. In this case, however, the requirements are modest—it only needs to approximate the structure and behavior of the real data for development purposes. ↩︎

Is “Vibe Analysis” A Thing?

March 2, 2026 / Martin

(Crossposting with LinkedIn)

Much virtual ink has been spilled recently discussing “vibe” coding. Broadly defined, vibe coding is an AI-driven software development approach where developers use natural language prompts to generate and refine code. In this approach, the developer focuses more on the app’s functionality (“the vibe”) rather than writing code line-by-line. This, in theory, allows folks with little to no programming knowledge to create applications.

I’ve been experimenting with vibe coding, recently. I’ve built iPhone apps to help me focus, web apps to track family chores, and command line tools to help keep my computer organized. In each case, I’ve done minimal (approaching zero) direct writing of code and have relied on natural language prompts to the LLMs to produce the code and make adjustments. And in each case I’ve been impressed with the ability for LLMs to take my (sometimes vague) input, parse it, and turn it into compilable code.

So naturally, I’ve been wondering: if vibe coding can produce a working iPhone app from a few prompts, can the same approach work for analytics? I’ve spent large parts of my career borrowing concepts and best practices from the world of software engineering and applying them to the realm of social science analytics, and I’ve seen huge gains from doing so. But the more I think about it, the more I believe that the jump from vibe coding to “vibe analytics” isn’t just a lateral move — it’s a category error. While analytic programming and traditional programming look similar on the surface, they are fundamentally different in ways that make purely vibe-coded analytics not just impractical but genuinely dangerous.

How do they differ? Perhaps most importantly, the output of an analysis is, ultimately, a judgement. In traditional programming, correctness is usually binary and verifiable. The app either crashes or it doesn’t. In analytics, the output is a a number, a chart, or a conclusion requiring domain judgement. A query can run perfectly and return a completely misleading result. The LLM has no way to know that the statistics are off because it double-counted arrests. And neither will you unless you already understand the data well enough to sanity-check it.

That leads naturally to the difference between code, which is structured and has meaning that can be understood generically, and data, which is structured but has meaning that is almost entirely local to your organization. What does “active user” mean in your schema? Why does the orders table have two date columns and which one should you use? Why are there nulls in that field — is that meaningful or an artifact? LLMs can guess (and boy do they) but they cannot know, and bad assumptions here produce plausible-looking wrong answers.

And if our data requires local domain knowledge, well so do our programming goals. Traditional programming usually starts with a reasonably well-defined spec. Analytics often starts with a vague business question where part of the job is figuring out what the right question even is. “What’s driving churn?” isn’t a spec — it’s a research agenda. LLMs are good at executing defined tasks but less good at the iterative negotiation between data, business context, and question refinement that characterizes real analytics work.

The local nature of both data and purpose lead to another problem: the subtlety of errors. One of the most common analytics bugs is joining tables at the wrong level or direction and silently inflating or deflating metrics. A traditional programmer writing a wrong join usually gets an obvious error. An analyst writing a wrong join gets a confident-looking number that might be off by 300%. LLMs are particularly prone to this because they’ll generate syntactically correct SQL that joins on whatever seems reasonable without understanding the context of your specific tables.

I’ve been amazed by LLMs’ ability to write code, test itself and then correct itself. But in software engineering there’s a rich culture of unit tests, integration tests, and CI pipelines that catch regressions. In analytics, most work is exploratory and one-off, so there’s rarely a test suite. When you vibe code an app feature, you can at least click around and see if it works. When you vibe code an analysis, the only real test is whether someone with deep domain knowledge reviews the logic.

None of this means LLMs have no role in analytics — far from it. My own early experiments using LLMs to assist me in analysis have been promising. LLMs are excellent at accelerating the mechanical parts of the work: drafting boilerplate SQL, suggesting visualization approaches, writing documentation, and even helping think through edge cases in a dataset. The key word, though, is assist. Every one of those tasks still benefits from — and in most cases requires — a human who understands the data, the business context, the questions, and, yes, the code well enough to evaluate what the LLM produces.