Evals Are a Product Manager's Job

Years ago I ran a page-speed initiative across 500+ B2C pages. We took Lighthouse scores from 40 to 92, and it moved revenue quite a lot. But the part I remember isn't just the optimization work, it's that for the first few weeks we couldn't agree on whether anything we shipped was actually helping. We all had our opinions, but noone had an impact number.

Once we had the number, the work got easy. Not easy as in trivial, easy as in directional: we could try something, see the score move, and keep or discard. The measurement wasn't the boring prerequisite to the real work. The measurement was the unlock. And though AI evals did not exists back then (and neither did AI), the direction was the same.

I've been going deep on LLM evaluation over the last weeks, mostly through open resources from Hamel Husain's writing — his posts Your AI Product Needs Evals and Using LLM-as-a-Judge for Evaluation are, as far as I can tell, the reference texts on this. Read them. But also deeplearning.ai and other eval texts and podcast are helpful. This means, that this post isn't a substitute; it's what landed for me as a product manager rather than an engineer, and what I'm now doing about it.

The trap: shipping without measuring

Building an AI feature involves three distinct activities: measuring whether output is any good, debugging why it isn't, and changing the system to make it better. Hamel's observation, which matches everything I've seen, is that teams pour almost all their energy into the third one: prompt tweaking, model swapping, adding examples, and skip the first two entirely.

The result is a product that gets to demo quality fast and then stops. You recognize the symptoms:

• You fix one failure and two new ones appear somewhere else
• Nobody can say whether last week's prompt change helped or hurt
• The prompt has grown into a 900-word wall of edge cases nobody wants to touch
• Quality assessment is "we tried it a few times and it seemed fine"

That last one has a name in my head now: vibe-checking. And it's a PM problem before it's an engineering problem, because "is this good enough to ship?" was never an engineering question.

Why this is product work, not engineering work

Here's the reframe that made this click for me.

In deterministic software, quality is largely a correctness question. The button works or it doesn't. In an LLM product, the same input can produce a hundred acceptable outputs and a hundred unacceptable ones, and the line between them isn't technical, it's stochastic, it's a judgment about users, brand, risk, and business context. That's the PM's job description.

Three specifics where this shows up:

The pass rate is a product decision. Unlike traditional unit tests, you don't need everything green. What failure rate is acceptable depends on the cost of a failure: a wrong shop recommendation is annoying, a wrong price is a trust event.

Someone has to be the arbiter of "good." Hamel makes a point I initially underrated: you need one principal domain expert whose judgment is the standard. Not a committee, not an average of opinions, just one person, deeply familiar with the domain or the user. Everyone will have views on whether the AI's answer was acceptable; if you try to satisfy all of them you'll satisfy none. Finding that person, protecting their time, and making their judgment operational is stakeholder work of exactly the kind PMs already do. (But you can also let him train a LLM Judge, which makes things even more efficient later)

You cannot specify the criteria up front. This is the finding I keep thinking about. Shankar call it criteria drift: to grade outputs you need defined criteria, but the act of grading is what teaches you what your criteria actually are. You look at fifty real outputs and discover you care about things you never would have written into a spec. It is truly more of a learning experience to sketch down and define what matters to you, something you only see once confronted with the wrong output.

If you've ever watched a stakeholder look at a prototype and immediately articulate three requirements they couldn't produce in a workshop the week before, this is the same phenomenon. PMs already know that people discover their requirements by reacting to concrete artifacts. Evals are that loop, formalized and pointed at AI output.

The practical shape of it

What I took away as an actual method, in rough order of cost:

Cheap assertions first. Break the product into features, then into scenarios per feature, then write fast programmatic checks for each. Did the query return the expected number of results? Did an internal ID leak into the user-facing text? These run on every change, cost nothing, and catch regressions. Write them from observed failures, every bug you find should leave behind a test.

You can generate your test inputs. You don't need production traffic to start. An LLM will happily produce fifty realistic user requests across the scenarios you define, and by all accounts this works better than skepticism suggests. Note the boundary: you generate the inputs, then run them through your actual system to get real outputs. You're not simulating your product, just its users.

Then look at the data. Actually look at it. Log full traces, and remove every scrap of friction between you and reading them. If understanding one interaction means opening three tools, you will stop doing it by Thursday. Build the ugly internal viewer that puts everything on one screen. This is a half-day of work and it's the highest-leverage half-day in the whole process.

Judge binary, then explain. The single most counterintuitive recommendation, and the one I'd have gotten wrong: don't rate outputs 1–5 across eight dimensions. Use pass/fail, plus a written critique explaining the reasoning. Scored dashboards feel rigorous and are mostly noise, nobody knows what to do with a 3.4, and those scores tend not to correlate with what the domain expert actually cares about. A binary verdict forces the real question: did this achieve the user's goal or not? The critique then carries the nuance.

Automate the judgment only after you've done it manually. Once you have expert pass/fail decisions with critiques, those critiques become the few-shot examples for an LLM judge. You iterate on the judge prompt until it agrees with your expert often enough to trust. One caveat worth internalizing: raw agreement is a misleading metric when your pass/fail classes are imbalanced: measure precision and recall separately instead.

Error analysis is funnel analysis. Once a judge can label at volume, segment failure rates by feature, scenario, and user type. You'll find the failures aren't evenly spread, one persona in one scenario is generating most of your pain. Then hand-classify a batch of failures by root cause and you have a prioritized fix list. This is the most familiar part of the whole process for a PM; it's the same analysis I'd run on a checkout funnel, pointed at a different artifact.

I turned this order of operations into a one-page cheat sheet — the same steps above, laid out as something you can actually pin next to your desk. It's also sitting in Resources if you just want the file.

The part I keep coming back to

Hamel ends with a line that reframes the entire exercise: the judge isn't what creates the value. Building it is a mechanism that forces you to sit down and look carefully at your data, and that's where the insight comes from. He half-admits the judge is a trick to get teams to do the thing they'd otherwise skip.

For a PM I'd put it even more strongly. Reading a few hundred real interactions between users and your AI is the closest thing to user research that exists for these products, and most teams are shipping without doing it at all.

What I'm doing with this

I'm building NearBuy — a hyper-local marketplace connecting people with shops near them, and the next matching layer is AI. So the first eval set is going against exactly that: a shop owner wants to add a product, and the system proposes all of the details (descriptions, price proposal, title, etc.).

The plan, deliberately unambitious:

1. Define the dimensions that matter: shop, product, location
2. Generate ~30 test inputs covering the combinations, run them through the system as it exists
3. Sit down and mark each result pass/fail with a written critique. I'm the domain expert here, which is a privilege of being small and a risk I should be honest about
4. Fix the obvious breakage before building anything clever
5. Only then consider an LLM judge

---

Credit where it's due: the substance here comes from Hamel Husain's work, and he and Shreya Shankar maintain evals.info as a free reference. The framing errors are mine.