I belong to a music league. Each week, about 20 friends and I create a communal playlist around some theme—“songs to dance to,” for example. We listen, then vote on which songs best fit the theme.
It’s a blast. It’s a good way to discover—or sometimes rediscover—music, and a good excuse to hang out with friends online.
But there’s a question that’s been nagging at me: is there systematic bias in Music League voting based on where a song appears in the playlist?
Maybe songs near the beginning get more votes simply because we hear them first—a primacy effect. Or maybe the opposite happens: songs near the end are fresher in our minds when it comes time to vote, producing a recency effect. Or perhaps both are true, with songs at either end benefiting at the expense of those stuck in the forgettable middle.
Fortunately, thanks to the hard work of another league member, I had the data needed to answer the question. And, as part of my continuing exploration of agentic programming, I decided to throw the problem at Claude Code and see what it would do.
A deliberately underspecified experiment
I’ve written before about the limits I see in “vibe analysis.” I was—and remain—skeptical that agents can replace humans in the analytic pipeline. There are too many places where analysis depends on understanding context, asking the right question, recognizing when the data don’t mean quite what you think they mean, and exercising judgment. But replacing analysts is a different question from making analysts dramatically faster. That’s where I see much more promise.
For this experiment, at least, I decided not to carefully walk Claude through the problem. I wanted to see how far it could get on its own.
So, after a couple of simple warm-up prompts—“read the data,” “produce a simple scatterplot of playlist order by votes,” and so on—I gave Claude this:
I’m interested in if there is bias in voting. Do earlier (or later) playlister order scores account for higher votes. Perhaps earlier order get more votes or perhaps later order gets more votes. Or perhaps both and there is a curvilinear relationship. Write a quarto report to explore this question in detail. Calculate appropriate statistics and illustrate with graphs. The output should be a quarto file that I can run to get the report.
And that was basically it.
I didn’t explain Music League. I didn’t explain how rounds or voting worked. I didn’t describe the structure of the data beyond pointing it toward the variables I cared about. In retrospect, there’s quite a bit I could have—and probably should have—added to the prompt.
But Claude took it and ran with it (as it always does).
So, how did it do?
It produced this report.
For those dying to know the answer: Claude concluded that there is a very slight primacy effect in our league. Songs appearing earlier in the playlist seem to have a small advantage. Having looked through the analysis, I agree with that conclusion.
More interesting to me, though, was the analysis itself.
And the analysis is…completely reasonable.
Not great. Not brilliant. But reasonable.
There’s too much filler, although I suppose I did explicitly tell it to “explore this question in detail.” It isn’t the polished report I would hand to a stakeholder, but I also didn’t ask it to produce a stakeholder-ready report. There are places where I would have made different choices or explained things differently.
Still, I found the overall analytic path sensible, the statistical choices defensible, and the conclusions appropriately supported by the evidence. It considered multiple possible relationships between playlist position and voting, visualized them, calculated relevant statistics, and ran several robustness checks rather than latching onto the first interesting result it found.
That last part is worth emphasizing. For a silly side project like this, I probably would have made a few plots, run a model or two, satisfied my curiosity, and moved on. Claude was more thorough than I would have bothered to be.
And the code?
Solid.
I read through the roughly 930-line Quarto file it generated. The code was straightforward and clear. Nothing particularly clever, but also nothing that made me nervous. It did have a little help from an analytics helper skill I’ve been developing, so this wasn’t quite an out-of-the-box test. Still, the vast majority of the analytic work was generated from that very short prompt.
That leaves me in a somewhat different place from where I started.
I still don’t think this demonstrates that we can hand an agent a dataset and dispense with the analyst. Claude didn’t know enough about the context to identify all the ways the analysis might go wrong, and it couldn’t know whether the question I asked was actually the right question. Those remain important limitations.
But that may also be setting the bar in the wrong place.
The more interesting question is what happens when you give a capable analyst tools that can take a loosely specified question and, in minutes, produce a credible first-pass analysis: clean code, visualizations, statistical tests, robustness checks, and a written report.
In this case, the answer was: something surprisingly useful.
The analysis took a fraction of the time it would have taken me to produce something comparable myself, while being more comprehensive than I probably would have bothered to be for a question of such…ummm…importance.
I remain skeptical of “vibe analysis.”
But I’m becoming considerably less skeptical of the idea that this is going to change how analysts work.