Agent evaluation (agent evals)

Single-response metrics — exact match, similarity to a reference answer — assume the unit under test is one piece of text. An agent's unit of work is a trajectory: did it pick the right tool with the right arguments, keep track of what the user said four turns ago, stop when the job was done, and leave the external state the user wanted? An agent can produce a fluent, polite, individually-reasonable transcript and still fail the task, and a per-response metric will grade every message in that transcript highly.

Agents are also non-deterministic. The same input can produce different trajectories across runs, so agent evals report pass rates over repeated runs rather than a single pass or fail, and the failures worth engineering against usually live in the tail runs rather than the median one.

In practice the field splits grading along two axes: outcome evals check whether the end state matches expectations, and trajectory evals check whether the steps taken to get there were reasonable. Both are needed. Outcome-only grading passes agents that succeed expensively or dangerously; trajectory-only grading passes agents that take beautiful steps to the wrong destination.

The decision that determines whether any of this is worth running is none of the four. It is where the cases come from. A suite whose cases were written from imagination measures the team's imagination; a suite whose cases were pulled from production measures the agent. The rest of this page keeps returning to that distinction because every other design choice is downstream of it.

The method took over because it scales like code while grading, approximately, like a human. The MT-Bench work that made it credible measured strong LLM judges agreeing with human preferences over 80% of the time — roughly the rate at which humans agree with each other. That result is why LLM-as-a-judge is now the default grader for open-ended agent behavior.

The same research is also the canonical catalog of its failure modes: position bias (favoring whichever answer is shown first), verbosity bias (favoring longer answers), self-enhancement bias (favoring text the judge model itself might have produced), and weak grading of subtle reasoning errors. None of these are exotic. All of them appear in production eval suites that nobody calibrated.

The working discipline is unglamorous. Prefer narrow, binary rubrics per criterion over one 1-to-10 aggregate score. Grade the judge itself against a set of human-labeled examples before trusting it, and re-calibrate whenever the judge model or the rubric changes. A judge nobody has checked against human labels is a random number generator with good grammar.

For agents specifically, judges should read the trajectory, not just the final message — wrong-but-plausible tool arguments and quietly skipped steps are invisible in closing text. And the rubric should be derived from failures observed in production, not imagined ones: a judge tuned to catch failures no user ever hits is a cost center with a confidence interval.

The real options span a spectrum. OpenAI Evals established the registry-of-benchmarks pattern in open source. Hosted platforms such as Braintrust, Langfuse, and LangSmith ship dataset management, judges, and CI hooks as a product. Open-source libraries like DeepEval and promptfoo suit teams that want the whole suite living in the repository. All of them are competent at the running part.

The trap is the one Hamel Husain's widely-cited essay on evals names directly: tooling makes running evals easy, which lets teams skip the hard part — error analysis, the unglamorous work of reading real transcripts to decide what is worth testing. A framework filled with hand-written cases executes flawlessly and measures nothing.

The standard failure sequence: a team hand-writes an eval set at launch, encoding what they imagined users would do. Traffic arrives and drifts away from those guesses. Six months later the suite is green on every run while users hit failures the suite never modeled. The suite did not rot because the tooling was bad; it rotted because nothing connected it to production.

Moda approaches the problem from the other side. It is not an eval-running platform and does not replace one. Moda is a harness engineering platform: it treats the eval set as one component of the agent harness — alongside prompts, tools, skills, and memory — and changes it from production traces. Behavioral failures detected in real conversations become eval cases. Intent clusters with no eval coverage become new suites. Proposed harness changes are verified against replayed production conversations before they ship.

The practical division of labor: keep whichever runner your team already likes. What changes is the provenance of the cases it runs — evidence from production instead of guesses from launch week.

The harness is everything around the model that makes it your agent: prompts, tools, skills, evals, memory. Every component encodes assumptions about users, and every component drifts as users change. A prompt written for last quarter's traffic goes stale in a familiar way. An eval set written for last quarter's traffic is worse, because the eval set is the instrument that was supposed to detect staleness in everything else. When it drifts, the whole improvement loop reports green while the agent regresses.

Treated as a harness layer, evals get the same operational properties as the rest: cases live in version control next to the prompts and tool schemas they gate; every case traces to evidence — a production failure it reproduces or an intent cluster it covers; the set is refreshed on a cadence tied to traffic shift rather than the calendar; and changes to the eval set itself go through review like any other code.

This is the sense in which Moda works on evals — the same loop it runs for prompts and tools: detect a failure in production traces, attribute it to the responsible harness component, propose a diff, verify against replays. When the component is the eval set, the diff is new cases built from real failures and retired cases that no longer reflect how users behave.

What makes the switch dangerous is that the harness was tuned, mostly implicitly, to the old model's dispositions: how literally it follows instructions, how it formats tool calls, how much it hedges, what it does with ambiguity. The new model breaks those assumptions silently. Generic benchmark deltas between the two models say nothing useful here, because your agent's behavior is the model and the harness together, and the benchmark only saw the model.

The only honest gate is an eval set built from your own production conversations: replay the same real traffic through both configurations and compare per intent cluster and per failure mode, not as one aggregate score. An aggregate can hold steady while the three clusters your revenue depends on quietly regress.

The loop that makes the switch survivable is compensate-and-re-gate: run the gate, see which clusters regressed under the new model, fix the harness for it — a prompt adjusted to the new model's instruction-following, a tool schema tightened against its argument habits, a skill added for a workflow it handles differently — and re-run until the gate holds. The switch succeeds or fails on the harness and the evals that gate it, not on the announcement benchmarks of the incoming model.

This is the first job Moda's eval work is built around: hold agent performance through the switch, with the gate built from your traffic and every compensating harness change verified against replayed conversations before it ships.

Where Moda fits is the supply side of this playbook: the detection of behavioral failures across the whole conversation population, the intent clustering that defines coverage, the attribution that says which harness component a failure belongs to, and the replay verification that gates a change. The runner, the CI wiring, and the deploy pipeline stay yours.