Classifier choice-order study: methodology

Contents

This page records the protocol and full results behind The new hype of classifiers is LARP. It covers the same twelve-system study. The figures and downloads use the saved primary replies.

Study design

We used an internal evaluation set of 1,000 text samples. An automated process removed identifiers before inference. Each request contained one prepared text, with no conversation history or evaluation labels. The source texts remain private.

The set contained 800 random samples with 747 distinct texts, plus 200 added-coverage samples selected to include more existing emotion labels. Three texts appeared in both groups, leaving 944 distinct texts overall. Those labels came from a separate GLM classifier. They were model predictions, not human judgments, and never entered the requests.

The added-coverage group is not a random sample. We report its score-sensitivity rates separately from the random group in the comparison viewer.

The choices were frustration, disapproval, sadness, confusion, anxiety, distrust, and positive emotion. For each text, we tested seven rotations of a seeded random order and seven rotations of its reverse. Each emotion appeared twice in every position. We repeated each order three times and shuffled the request sequence.

The text, question, and seven definitions stayed fixed. We used each system's supported request format, so prompt wrapping and calibration are part of the released-system comparison.

That gives 1,000 samples x 14 orders x 3 repeats = 42,000 planned outcomes per system. Across twelve systems and versions, the total is 504,000. Collection ran from October 10 to October 14, 2026.

We tested Jev 1.14.0, Jev 1.13.0, Kev-4B, Laya, Von, SemIf with Qwen3.5-4B, Bespoke Nimble-9B, NanoJev, Rizzo Flow-4B, OpenAI Decisions with gpt-6-luna, local Clef-27B, and SubQ Snap Preview. The article uses 1.14 for its main Jev example and retains 1.13 as a version comparison. Jev versions and open model code and weights were pinned. OpenAI and SubQ returned aliases without immutable build identifiers.

SubQ's own benchmark also varies option order. Our test measures emotion scores, not its long-context reasoning claims.

Question and choices

Each request used this question:

Which emotion family best describes the user's reaction to an AI assistant in this sentence? Choose the best match among the seven supplied families. Judge only the user's words. Use each family's supplied definition.

Each choice used the definition below. The trust key denotes distrust.

  • frustration: Annoyance or hostility toward the assistant or its work, including anger, repeated-mistake exasperation, insults, profanity, sarcasm, or requests to escalate because of the assistant.
  • disapproval: Calm rejection of the assistant's work or proposal without annoyance or hostility. A plain edit instruction alone is not disapproval.
  • sadness: Disappointment with the assistant's result, resignation or giving up on the task, self-blame, or personal distress expressed in the exchange.
  • confusion: Not understanding the assistant's message, work, state, or behavior.
  • anxiety: Worry or time pressure directed at the assistant or its actions. A deadline mentioned without worry or pressure is not enough.
  • trust: Distrust: doubt about the assistant's honesty, accuracy, claims, or whether it did what it said.
  • positive: Gratitude to the assistant, satisfaction with its work, or delight at its output.

Score ranges and winner changes

For each sample and emotion, we average three replies within each of 14 client orders. The emotion's range is its highest order mean minus its lowest, in percentage points. The largest of the seven emotion ranges becomes that sample's score range.

The main chart reports the share of samples whose largest range is at least ten points. This threshold was fixed before the full run. The tables also report the median, 95th percentile, and maximum of those per-sample ranges. These summaries describe different parts of the distribution.

Untied winner changes count samples whose highest-scoring emotion changes across order means, with no tied highest score in any order mean. The denominator includes every shared sample, including those with a tie. The comparison viewer reports ties separately.

Emotion score sensitivity to choice order

Same 721 strict-complete random samples for every system

How far did scores move?

Lower is better for stability

  • Clef-27B (local)

    Median range (points)
    0.00
    95th percentile (points)
    0.00
    Maximum range (points)
    0.07

  • Von

    Median range (points)
    0.00
    95th percentile (points)
    0.00
    Maximum range (points)
    0.01

  • NanoJev

    Median range (points)
    2.98e-6
    95th percentile (points)
    5.96e-6
    Maximum range (points)
    8.94e-6

  • Jev 1.13.0

    Median range (points)
    12.67
    95th percentile (points)
    29.67
    Maximum range (points)
    40.33

  • Jev 1.14.0

    Median range (points)
    12.67
    95th percentile (points)
    26.67
    Maximum range (points)
    40.33

  • Kev-4B

    Median range (points)
    13.00
    95th percentile (points)
    22.67
    Maximum range (points)
    42.03

  • SubQ Snap Preview

    Median range (points)
    16.87
    95th percentile (points)
    29.86
    Maximum range (points)
    42.01

  • Bespoke Nimble-9B

    Median range (points)
    22.66
    95th percentile (points)
    56.46
    Maximum range (points)
    87.39

  • Rizzo Flow-4B

    Median range (points)
    39.41
    95th percentile (points)
    77.93
    Maximum range (points)
    92.04

  • Laya

    Median range (points)
    46.05
    95th percentile (points)
    62.58
    Maximum range (points)
    78.84

  • OpenAI Decisions

    Median range (points)
    55.00
    95th percentile (points)
    81.00
    Maximum range (points)
    92.33

  • SemIf / Qwen3.5-4B

    Median range (points)
    60.41
    95th percentile (points)
    80.88
    Maximum range (points)
    89.44

How often did the outcome change?

Lower is better for stability

  • Clef-27B (local)

    Range at least 10 points
    0.0%95% CI: 0.0–0.0%
    Untied winner changes
    0.0%

  • Von

    Range at least 10 points
    0.0%95% CI: 0.0–0.0%
    Untied winner changes
    0.0%

  • NanoJev

    Range at least 10 points
    0.0%95% CI: 0.0–0.0%
    Untied winner changes
    0.0%

  • Jev 1.13.0

    Range at least 10 points
    56.6%95% CI: 51.8–61.3%
    Untied winner changes
    11.8%

  • Jev 1.14.0

    Range at least 10 points
    58.0%95% CI: 53.1–62.6%
    Untied winner changes
    11.0%

  • Kev-4B

    Range at least 10 points
    73.5%95% CI: 70.0–76.9%
    Untied winner changes
    36.3%

  • SubQ Snap Preview

    Range at least 10 points
    82.1%95% CI: 78.8–85.2%
    Untied winner changes
    22.1%

  • Bespoke Nimble-9B

    Range at least 10 points
    74.2%95% CI: 70.4–77.9%
    Untied winner changes
    32.5%

  • Rizzo Flow-4B

    Range at least 10 points
    76.7%95% CI: 71.3–81.2%
    Untied winner changes
    46.6%

  • Laya

    Range at least 10 points
    99.9%95% CI: 99.6–100.0%
    Untied winner changes
    81.1%

  • OpenAI Decisions

    Range at least 10 points
    98.8%95% CI: 97.8–99.6%
    Untied winner changes
    56.3%

  • SemIf / Qwen3.5-4B

    Range at least 10 points
    94.6%95% CI: 92.6–96.4%
    Untied winner changes
    38.7%
Each interface received the same texts, question, and seven emotion definitions in 14 client choice orders with three repeats. Intervals use whole-text bootstrap resampling. Small nonzero ranges use scientific notation: 2.98e-6 means 0.00000298 points. Lower values indicate greater order stability; this figure does not measure classification accuracy. Untied winner changes count samples whose highest-scoring emotion changes across orders with no tied highest score in any order mean. The denominator includes all 721 shared samples, including samples with a tied highest score. Clef sorts choice keys before encoding. Its row measures score stability after that step.

Small nonzero values use scientific notation. For example, 2.98e-6 means 0.00000298 percentage points. A printed zero can reflect native rounding or the displayed precision. It is not a guarantee about future inputs.

Shared samples and coverage

A score-complete sample has all 42 score replies from a system. A strict-complete sample also passes the chosen-label check in every reply: the chosen label must have the highest reported score. The main comparison takes the intersection across all twelve systems, then selects the random group. That leaves 721 samples containing 674 distinct texts.

This keeps the inputs identical across rows. It excludes cases that any system could not score or did not validate. The shared results therefore do not describe those excluded inputs. We report their coverage here.

We checked score coverage separately from whether the chosen label had the highest reported score.

Both Jev versions returned all 42,000 score distributions. Jev 1.14 had 52 label mismatches across 43 samples. The 1.13 comparison had 32 across 28 samples. In these replies, the chosen label scored one point below the highest choice.

Jev 1.14 also had 19 temporary API errors. All recovered on the next attempt within our four-attempt transport budget. We kept the error records and did not retry valid scores or label mismatches.

OpenAI Decisions returned 178 refusals without scores across 34 samples. The other nine systems completed all 1,000 samples under the strict rule. The shared intersection leaves the 721 random samples used above.

Score coverage

Higher is better for completeness

42,000 planned evaluations per system across 1,000 samples

  • Clef-27B (local)

    Scored replies
    42,000
    Score sets
    1,000
    Label check
    1,000

  • Von

    Scored replies
    42,000
    Score sets
    1,000
    Label check
    1,000

  • NanoJev

    Scored replies
    42,000
    Score sets
    1,000
    Label check
    1,000

  • Jev 1.13

    Scored replies
    42,000
    Score sets
    1,000
    Label check
    972

  • Jev 1.14

    Scored replies
    42,000
    Score sets
    1,000
    Label check
    957

  • Kev

    Scored replies
    42,000
    Score sets
    1,000
    Label check
    1,000

  • SubQ Snap

    Scored replies
    42,000
    Score sets
    1,000
    Label check
    1,000

  • Nimble

    Scored replies
    42,000
    Score sets
    1,000
    Label check
    1,000

  • Rizzo

    Scored replies
    42,000
    Score sets
    1,000
    Label check
    1,000

  • Laya

    Scored replies
    42,000
    Score sets
    1,000
    Label check
    1,000

  • OpenAI Decisions

    Scored replies
    41,822
    Score sets
    966
    Label check
    966

  • SemIf

    Scored replies
    42,000
    Score sets
    1,000
    Label check
    1,000

Jev 1.14 returned 41,981 score replies in the initial collection. Its 19 temporary API failures recovered on the next attempt. The row includes those recovered scores. No valid score or label-mismatch reply was retested.

Can retries restore complete samples?

A separate follow-up repeated each affected request up to three more times. It stopped at the first valid result. The original results above remain in the benchmark.

  • Jev 1.13

    Requests retested
    32
    Recovered
    32
    Complete samples, before / after
    972 / 1,000

  • OpenAI Decisions

    Requests retested
    178
    Recovered
    1
    Complete samples, before / after
    966 / 966

Jev 1.13 used 32 extra calls. OpenAI used 532 extra calls and still refused 177 requests across 34 samples. All inputs and settings stayed fixed.

A score-complete sample has all 42 score replies. The label check requires the chosen label to have the highest score in every reply. Jev 1.13 returned all score sets; 32 replies failed that check. OpenAI returned 178 refusals with no scores. These diagnostics measure completeness, not emotion accuracy.

A separate check allowed up to three extra attempts for Jev 1.13's label mismatches and OpenAI's refusals, stopping at the first valid result. All 32 Jev replies passed on their first retry. OpenAI recovered one of 178 refusals. Refusals are a documented Decisions answer type, but supply no emotion scores.

These retry scores do not replace primary observations. Selecting a replacement because it passes a score or label check would change the comparison. Transport recovery instead fills requests that returned no score reply.

Repeat controls and uncertainty

The same-order control uses all three pairs of repeats within each order. The different-order control uses every unordered pair of orders, comparing repeats one and two in both cross-order directions. Each comparison uses two individual replies. We average absolute differences across seven emotions and then across samples.

We rechecked Jev's repeat control against the saved native replies. For each version, all 14,000 groups of three repeats used identical serialized request bodies. On the shared 721 samples, the same-order mean absolute difference was 0.648 points for Jev 1.14 and 0.568 for the 1.13 comparison. These match the plotted values.

We also checked whether the highest-scoring emotion changed between repeats of the same order. A pair counts only when both replies have a unique highest score and those emotions differ. This occurred in 254 of 10,094 sample-order groups for Jev 1.14 and 212 for Jev 1.13. At least one such group occurred in 108 and 97 of the 721 samples, respectively. Each sample has 14 opportunities for a change. These sample counts are not per-call failure rates or accuracy estimates.

A separate check on October 15 sent each of the three existing invented smoke-test sentences 20 times to each pinned Jev version. It kept the question, definitions, and choice order fixed, shuffled the 120 requests, and used two workers. All 120 returned scores. The two clear sentences returned identical score vectors. On the mixed-emotion sentence, the largest single-emotion range was 8 points for Jev 1.14 and 5 for Jev 1.13. The selected label stayed fixed for each sentence and version. This confirms score variation on that invented input. It does not reproduce the larger study's winner changes or replace its primary observations. The repeat verification download contains the audit counts, exact synthetic inputs, and all 120 native answers.

These checks measure hosted API behavior. We did not isolate the cause of the variation inside the model or service.

The excess difference is the different-order mean minus the same-order mean. It compares order variation against variation across identical requests. It does not isolate position from neighbor effects.

Intervals use 2,000 whole-text cluster bootstrap resamples. All copies of the same text stay together, including their orders and repeats. The primary estimates give each sample equal weight. The aggregate download also gives each distinct text equal weight as a sensitivity check. The main chart's intervals describe sample shares, not uncertainty around an individual emotion score.

The Jev version comparison is paired on the same 721 random samples. The change from 1.13 to 1.14 in the share crossing ten points is +1.39 percentage points, with a 95% interval of -0.81 to +3.52. The interval includes zero. Jev 1.14 ran on October 14 and 1.13 on October 10, 2026. Serving changes can contribute to the difference.

Model settings and input handling

We used native probability scores without renormalization or calibration fitted to this evaluation set. Native prompt formats and calibration differ. Kev uses its shipped temperature of 2.406; Laya uses 1.000; Von uses input-conditioned calibration. The comparison includes these settings. It does not isolate architecture from calibration or serving behavior.

Jev reports two decimal places and Clef four. Rounding can hide small changes. OpenAI Decisions and SubQ Snap returned model aliases without immutable build identifiers. The aggregate download contains the recorded versions, code revisions, precision, and runtime settings for every system.

Clef uses the released local 27B checkpoint on a Modal H200 with BF16 weights. Its encoder sorts choice keys. For each of the 1,000 prepared texts, all 14 client orders produced one logical encoded record before batch padding. No text was truncated. Its results measure the released interface after sorting, not different token positions inside the model. Hosted Workers AI serving was not tested.

The encoder's choice handling is equivalent to:

def choice_options(q):
    return sorted(
        (str(key), value)
        for key, value
        in q["criteria"].items()
    )

Laya processed every input without runtime truncation, but 64 samples exceeded its published 512-token training budget. NanoJev's root checkpoint documents training for games and navigation. This task does not establish either model's emotion accuracy or test multilingual, image, or long-context performance claims.

Scope and downloads

The seven choices share one probability mass. They are relative-choice scores, not independent probabilities that each emotion is present. Several emotions can coexist, but they compete in this interface. We omitted a “no emotion” option. The separate classifier flagged no emotion in 599 of the 800 random samples.

We tested 14 balanced orders out of 5,040 possible orders. Moving a choice also changes its neighbors. This measures choice-order sensitivity and cannot isolate a pure position effect. Results may differ on other texts. Automated identifier removal can also change the evidence for an emotion.

The study does not rank general decision quality or emotion accuracy. A constant scorer could have perfect stability and no emotion information. An accuracy comparison needs a “no emotion” option and human-reviewed labels.

Use the interactive comparison to switch sample groups and validation rules. The invented-sentence explorer shows recorded examples without making new API calls.

Download the comparison CSV, full aggregates and pinned settings, retry check counts, and repeat verification. Public files omit the evaluation set's source texts, identifiers, individual records, and random seed values. The repeat verification includes only invented sentences and their replies.

Back to the research post