Overthinking it

A Test for Bias

August 15, 2026 / Martin

I belong to a music league. Each week, about 20 friends and I create a communal playlist around some theme—“songs to dance to,” for example. We listen, then vote on which songs best fit the theme.

It’s a blast. It’s a good way to discover—or sometimes rediscover—music, and a good excuse to hang out with friends online.

But there’s a question that’s been nagging at me: is there systematic bias in Music League voting based on where a song appears in the playlist?

Maybe songs near the beginning get more votes simply because we hear them first—a primacy effect. Or maybe the opposite happens: songs near the end are fresher in our minds when it comes time to vote, producing a recency effect. Or perhaps both are true, with songs at either end benefiting at the expense of those stuck in the forgettable middle.

Fortunately, thanks to the hard work of another league member, I had the data needed to answer the question. And, as part of my continuing exploration of agentic programming, I decided to throw the problem at Claude Code and see what it would do.

A deliberately underspecified experiment

I’ve written before about the limits I see in “vibe analysis.” I was—and remain—skeptical that agents can replace humans in the analytic pipeline. There are too many places where analysis depends on understanding context, asking the right question, recognizing when the data don’t mean quite what you think they mean, and exercising judgment. But replacing analysts is a different question from making analysts dramatically faster. That’s where I see much more promise.

For this experiment, at least, I decided not to carefully walk Claude through the problem. I wanted to see how far it could get on its own.

So, after a couple of simple warm-up prompts—“read the data,” “produce a simple scatterplot of playlist order by votes,” and so on—I gave Claude this:

I’m interested in if there is bias in voting. Do earlier (or later) playlister order scores account for higher votes. Perhaps earlier order get more votes or perhaps later order gets more votes. Or perhaps both and there is a curvilinear relationship. Write a quarto report to explore this question in detail. Calculate appropriate statistics and illustrate with graphs. The output should be a quarto file that I can run to get the report.

And that was basically it.

I didn’t explain Music League. I didn’t explain how rounds or voting worked. I didn’t describe the structure of the data beyond pointing it toward the variables I cared about. In retrospect, there’s quite a bit I could have—and probably should have—added to the prompt.

But Claude took it and ran with it (as it always does).

So, how did it do?

It produced this report.

For those dying to know the answer: Claude concluded that there is a very slight primacy effect in our league. Songs appearing earlier in the playlist seem to have a small advantage. Having looked through the analysis, I agree with that conclusion.

More interesting to me, though, was the analysis itself.

And the analysis is…completely reasonable.

Not great. Not brilliant. But reasonable.

There’s too much filler, although I suppose I did explicitly tell it to “explore this question in detail.” It isn’t the polished report I would hand to a stakeholder, but I also didn’t ask it to produce a stakeholder-ready report. There are places where I would have made different choices or explained things differently.

Still, I found the overall analytic path sensible, the statistical choices defensible, and the conclusions appropriately supported by the evidence. It considered multiple possible relationships between playlist position and voting, visualized them, calculated relevant statistics, and ran several robustness checks rather than latching onto the first interesting result it found.

That last part is worth emphasizing. For a silly side project like this, I probably would have made a few plots, run a model or two, satisfied my curiosity, and moved on. Claude was more thorough than I would have bothered to be.

And the code?

Solid.

I read through the roughly 930-line Quarto file it generated. The code was straightforward and clear. Nothing particularly clever, but also nothing that made me nervous. It did have a little help from an analytics helper skill I’ve been developing, so this wasn’t quite an out-of-the-box test. Still, the vast majority of the analytic work was generated from that very short prompt.

That leaves me in a somewhat different place from where I started.

I still don’t think this demonstrates that we can hand an agent a dataset and dispense with the analyst. Claude didn’t know enough about the context to identify all the ways the analysis might go wrong, and it couldn’t know whether the question I asked was actually the right question. Those remain important limitations.

But that may also be setting the bar in the wrong place.

The more interesting question is what happens when you give a capable analyst tools that can take a loosely specified question and, in minutes, produce a credible first-pass analysis: clean code, visualizations, statistical tests, robustness checks, and a written report.

In this case, the answer was: something surprisingly useful.

The analysis took a fraction of the time it would have taken me to produce something comparable myself, while being more comprehensive than I probably would have bothered to be for a question of such…ummm…importance.

I remain skeptical of “vibe analysis.”

But I’m becoming considerably less skeptical of the idea that this is going to change how analysts work.

Trans-Europe Express

August 11, 2026 / Martin

By Kraftwerk (1977, #92)

I was expecting some “Sprokets” parody of itself. This isn’t that. It wasn’t my cup-of-tea but I see why it’s on this list. For the time it was released, I can imagine this being quite the revelation.

Moondance

May 28, 2026 / Martin

By Van Morrison (1970, #94)

What a great, frick’n, album! I didn’t have high hopes since I’ve heard the title track many times and don’t really enjoy it. But this whole album is just really enjoyable. Lots to recommend here.

Back in Black

May 27, 2026 / Martin

By AC/DC (1980, #95)

This is what I needed today: just a fun, high engergy Rock ’n Roll album. I’m quite sure this was my first full listen, even if its most famous song (You Shook Me All Night Long) is unavoidable. But this is indeed a great album. Lots of really good songs and a cohesive whole.

A Safe Approach to AI-Assisted Analysis

March 25, 2026 / Martin

(Crossposting with LinkedIn)

In a recent essay, I argued that “Vibe Analysis” couldn’t be a real thing—that analysis is too different from traditional programming for AI to fully replace human analysts. But as I noted in that essay, this does not mean AI has no place in analytic work. Far from it. AI has enormous potential to accelerate analytic programming; it simply must be applied cautiously, with careful attention to quality, rigor, and security.

Security—and how to enable analysts to work with AI in a secure way—is the primary focus here. It is a central aspect of my work at Urban Labs. While much of our analytic code is intended for public release, the same cannot be said of our data. In fact, it is rarely our data to begin with: the vast majority consists of administrative data entrusted to us by partners. As a result, we are obligated—ethically, legally, and reputationally—to safeguard that data and use it only in contractually specified ways. Each dataset is governed by agreements that precisely define who may access it and how it can be used.

Unsurprisingly, none of these agreements permit sending partner data to Anthropic or OpenAI. Nearly all require that data be stored and processed on servers owned and maintained by the University of Chicago. These environments adhere to strict security standards that ensure partners can trust their data is protected and used only for agreed purposes. Installing new applications in this environment is already difficult; installing an AI coding agent is simply not feasible.

Each analyst does have a local machine used to connect to the secure server, and in theory, these could connect to AI tools. However, for analytic purposes, these machines function largely as “thin clients”—machines that store no analytic code or data. Even so, they may still contain sensitive business information. Allowing AI agents to run freely on them raises legitimate security concerns.

This creates a fundamental problem: we have no safe place to run LLM-based tools, and we cannot expose our data to them even if we did. Compounding this is the reality of how analysts work. Our team—and every analyst I’ve worked with—develops code interactively and iteratively: write a small piece, run it, inspect results, revise, repeat. Writing an entire analytic pipeline in one pass without repeatedly validating against real data is not how analysis works in practice.

Taken together, these constraints make a straightforward AI coding workflow impossible:

Constraint 1: Our data is private and can never leave the secure server environment, including being sent to external LLMs.
Constraint 2: Our servers must meet strict security standards and cannot host LLM tools like Claude Code or Codex.
Constraint 3: Local machines may contain sensitive information and cannot safely run unrestricted AI agents.
Constraint 4: Analysts work iteratively, and any solution must support that workflow.

So how do we proceed? How can analysts benefit from AI without compromising security or disrupting how they actually work?

I don’t have a final answer, but I believe I do have a workable approach built on four components: LLM coding agents, containers, Git, and synthetic datasets. Together, these can form a secure and practical AI-assisted analytic environment.

At a high level, the approach works as follows: an LLM coding agent runs inside an isolated container. The LLM has access to project code via Git, but not to real data, which never leaves the secure server. Instead, the LLM interacts with synthetic datasets that mimic the structure of real data. This setup allows analysts to collaborate with AI during development, then push finalized code to the secure server for execution on real data.

Breaking this down further:

The process begins with containers. Containers allow you to create isolated computing environments—effectively “computers within computers.”¹ The container is completely separated from the host system. You define its operating system and installed software, and that’s all it can access. Running the LLM inside a container limits its scope entirely. It cannot read arbitrary files, execute unknown programs, or affect the host system—because, from its perspective, those resources do not exist.

This isolation creates a safe sandbox, but it raises a new question: how does the LLM do anything useful without access to real resources?

This is where Git comes in. Git is one of the tools installed inside the container. By cloning a repository into the container, the LLM gains access to the project’s codebase—allowing it to read, modify, and suggest improvements. Crucially, this includes only code, not data (and if you are storing data in Git, that is a separate problem). From the LLM’s perspective, the data still does not exist.

However, code alone is often insufficient context. Many analytic tasks depend on understanding the structure and content of the data itself. Asking an AI to summarize variables in a dataset is impossible if it cannot see those variables. But, as described above, we can’t expose our data to AI.

The solution, I believe, is synthetic data. Synthetic datasets replicate the structure and statistical properties of real data without containing any actual sensitive information². By providing synthetic data within the container, the LLM gains enough context to be useful while still preserving data security. Analysts can continue their normal iterative workflow—writing, testing, and refining code—using synthetic data alongside the LLM.

When development is complete, the workflow is simple: push the code via Git to the secure server, execute it against real data in the secure environment of that server, and generate results.

This approach does not eliminate all challenges, but it creates a viable path forward. LLMs will not replace analysts, but they will augment them. The responsibility is on us to ensure that augmentation happens in a way that is secure, controlled, and aligned with how analysis is actually done.

In practice, containers are more complex and offer many additional capabilities, but this captures the essential concept. ↩︎
Generating high-quality synthetic data can be complex. In this case, however, the requirements are modest—it only needs to approximate the structure and behavior of the real data for development purposes. ↩︎