The new hype of classifiers is LARP

TL;DR: Moda provides production monitoring and evals for AI agents. Our in-house classifiers flag user frustration and other signals that help teams find failures. We tested twelve decision systems and versions as possible replacements. Changing choice order often changed emotion scores. Jev also changed its top emotion on some identical requests, breaking the one trust factor we have: it's supposed to be deterministic and repeatable.

Every classifier we deploy at Moda is built in-house. Our classifiers help flag user frustration and other signals in production traces. A trace records one complete agent run. Teams use these signals to find runs that need investigation.

We test frontier APIs and open models to see whether they can do this work better. This study began with a practical question: could we replace one of those classifiers with a released decision model? Before comparing emotion accuracy, we checked whether its scores stayed consistent when the evidence stayed fixed.

We started by changing the order of the emotion choices. The text and definitions stayed fixed, but the scores moved. A test that checked only the winning emotion would have missed it.

We tested twelve systems and versions on 1,000 text samples, across 504,000 planned outcomes. This test is something we did internally and decided to share with everyone. Our main Jev example uses 1.14.0, released on October 14, 2026. We keep 1.13 as a version comparison.

The winning emotion hid the change

Our smoke test used an invented sentence:

I don't understand why this is still not ready; I need it today.

We supplied seven emotion choices in 14 orders, with three repeats per order. Jev 1.14's mean frustration score moved from 77.67% to 56.67%. In those two orders, anxiety moved from 15.67% to 24.00%. Frustration remained the winner in all 42 replies.

A test that checked only the selected label would have passed.

Under a rule that sends frustration scores above 70% to review, these two orders would take different paths. The sentence and winning label would be identical.

This sentence makes the behavior visible. The larger evaluation set tells us how often it occurs.

Twelve systems on the same task

Decision models return scores for a set of allowed answers. Those scores can control routing, review, or another bounded decision. We tested each system through its native probability interface.

For each sample and emotion, we averaged the three replies within each order. We took the highest order mean minus the lowest. The largest of the seven emotion ranges became that sample's score range. We set a ten-percentage-point threshold before the full run.

The chart uses the same 721 random samples for every system. Each sample has all 42 score replies from every system. In every reply, the chosen label must have the highest reported score. The methodology explains the shared sample rule and reports excluded cases.

Von and NanoJev showed very small changes. Von's mean largest range was 0.000277 points, with a maximum of 0.01. NanoJev's was 0.00000199 points, with a maximum of 0.00000894. Neither changed the winning emotion in this shared set.

At the other end, Laya, OpenAI Decisions, and SemIf crossed ten points on 94.6% to 99.9% of samples. SubQ Snap crossed it on 82.1%. Clef stayed below that threshold, but its input handling changes what the result means.

Jev 1.14 crossed ten points on 58.0% of the shared samples. Its median largest range was 12.67 points, and its 95th-percentile range was 26.67 points.

For comparison, Jev 1.13 crossed ten points on 56.6% of samples, with the same median and a 95th-percentile range of 29.67 points. The paired change from 1.13 to 1.14 was +1.39 percentage points, with a 95% interval of -0.81 to +3.52. The interval includes zero, so this threshold result does not show a clear change. Jev 1.14 ran on October 14 and 1.13 on October 10. Serving changes can also contribute. This task does not assess multilingual performance.

We borrowed the shared-task and separate-metric approach from HELM. Kev's benchmark and the Primacy Effect of ChatGPT study also test shuffled choices. Our score-range measure catches changes that a winner-only check misses.

Repeated calls do not explain the result

An identical request can produce different scores. We compared pairs of replies with the same order against pairs with different orders, averaging absolute score differences across seven emotions and the shared samples.

Jev 1.14 differed by 0.648 points across identical orders and 1.647 points across different orders. OpenAI Decisions differed by 0.051 and 7.973 points, respectively. SemIf returned identical scores for repeated orders, but differed by 9.739 points across orders. Across eight of the twelve systems, changing order added more than one point beyond repeat variation.

Jev 1.14 also changed its top emotion between identical requests. On the shared 721 samples, this happened for 108 samples, in at least one of 14 fixed orders. Jev 1.13 did so on 97 samples. These counts exclude pairs with tied top scores. This undermines the repeatability we need when a classifier controls a workflow. The repeat audit verifies the request bodies and records a separate check on invented sentences.

These pairwise differences average across all emotions. The range measure uses each sample's largest emotion range. Both matter when a score can cross a review threshold or change a ranking while the winning label stays fixed.

Stable scores need an input check

We tested Cloudflare Clef's released local 27B checkpoint and Python interface. Its encoder sorts choice keys before building the token sequence.

For each of the 1,000 prepared texts, all 14 client orders produced one logical encoded record before batch padding. No text was truncated. We ran every planned evaluation on a Modal H200 with BF16 weights.

Clef's mean largest range was 0.000102 points, with a maximum of 0.0733, on the shared random samples. We did not isolate the cause of that small variation.

This measures the released interface after sorting. It does not test different token positions inside Clef. Hosted Workers AI serving was not tested. Changing a client-side list does not always change what reaches the model.

When scores drive workflow decisions

We use classifier scores to route work, set review thresholds, and decide when to act. For those uses, we need to know how consistently a score responds to the same evidence.

This test shows how choice order can affect those decisions. A score change can move a sample across a threshold or change its highest-scoring emotion. Checking that sensitivity helps us choose a system and set thresholds for the workflow it will serve.

Native prompts, precision, calibration, and training tasks differ. The full methodology records those settings and limits. This compares released systems, including their serving behavior.

We tested 14 of 5,040 possible orders. Moving a choice also changes its neighbors, so the design cannot isolate a pure position effect. Results may differ on other texts. Identifier removal can also change the evidence for an emotion.

What we now check before using a score

Our classifier evaluations now include balanced choice orders, repeated identical requests, and model-input inspection when code is available. We check score ranges, winner changes, threshold crossings, and coverage.

An accuracy comparison needs a “no emotion” option and human-reviewed labels. That comparison also needs response time and cost measurements.

Replacing an in-house classifier needs evidence about both the score and the decision it controls. This is why we test the new releases against the workflows we need them to support.

Read the methodology and full results for the protocol, detailed tables, coverage checks, and downloads. Explore the interactive comparison or recorded invented examples.

If your agents produce more than 1,000 traces a week, book a demo.