Evaluation Is the Product

The most honest hour in an agent company is the hour when someone asks, “Did this get better?” and the room reaches for anecdotes. A founder has a favorite trace. A researcher has a notebook of vibes. A PM has a tweet. None of that is lying. All of it is unshippable as a process.

Chat products can hide in qualitative fog for a while. Agent products cannot. They touch files, money, tickets, customers. If you do not own evaluation, your users will evaluate you in public, and they will not use your rubric.

A rubric you can run overnight

I want four numbers on a wall, updated by a machine, not a narrative.

• Task success: did the stop condition fire for the right reason?
• Cost: tokens, minutes, tool calls. Success that bankrupt you is a hobby.
• Damage: how often did we mutate something we had to undo?
• Ask-rate: how often did the loop need a human. Down is not always better.

Ask-rate is the one teams game. They celebrate an agent that never asks, then spend Friday cleaning up a confident mess. A good agent asks when the world is ambiguous and acts when the world is checked. That ratio is a product decision, not a model property.

Gold sets rot

Static evals are necessary and they go stale. The repository moves. The style guide changes. The “correct” patch from April now fails formatting. If your gold set cannot be refreshed, your graphs will eventually measure nostalgia.

The practical move is to treat production traces as a quarry. Sample failures weekly. Promote a few into the gold set. Retire the ones that no longer resemble the product. This is unglamorous editorial work. It is also the only way evaluation stays pointed at the thing customers have.

An eval you are afraid to fail is an eval you will stop running.

I have watched teams hide a red suite because a launch was close. I have also watched the same teams spend a quarter arguing about prompts that would have been settled by that suite. Evaluation is not a gate you add at the end. It is the product’s way of knowing itself. If that sounds dramatic, try shipping a loop without it. The drama arrives anyway, just later, and with screenshots.