Explore the classifier comparison

The same prepared texts, seven emotion definitions, fourteen choice orders, and three repeats. Change the sample group and shared sample rule to inspect the recorded results.

Contents

Every row uses the same 721 samples and 674 distinct prepared texts. The original random sample has 800 planned samples.

Score stability on the shared samples

Samples with a range of at least 10 points

Lower is better for stability

Share of samples, with 95% confidence intervals

0%20%40%60%80%100%
Cloudflare Clef-27B (local)0.0%
Von0.0%
NanoJev0.0%
Jev 1.13.056.6%
Jev 1.14.058.0%
Kev-4B73.5%
SubQ Snap Preview82.1%
Bespoke Nimble-9B74.2%
Rizzo Flow-4B76.7%
Laya99.9%
OpenAI Decisions98.8%
SemIf / Qwen3.5-4B94.6%
Lines show 95% intervals from 2,000 whole-text bootstrap resamples. All copies of one text stay together. A zero interval describes this sample. Clef sorts choice keys before encoding. Its row measures stability after that step.

How far did scores move?

Lower is better for stability

  • Cloudflare Clef-27B (local)

    Median range (points)
    0.00
    95th percentile (points)
    0.00
    Maximum range (points)
    0.07

  • Von

    Median range (points)
    0.00
    95th percentile (points)
    0.00
    Maximum range (points)
    0.01

  • NanoJev

    Median range (points)
    2.98e-6
    95th percentile (points)
    5.96e-6
    Maximum range (points)
    8.94e-6

  • Jev 1.13.0

    Median range (points)
    12.67
    95th percentile (points)
    29.67
    Maximum range (points)
    40.33

  • Jev 1.14.0

    Median range (points)
    12.67
    95th percentile (points)
    26.67
    Maximum range (points)
    40.33

  • Kev-4B

    Median range (points)
    13.00
    95th percentile (points)
    22.67
    Maximum range (points)
    42.03

  • SubQ Snap Preview

    Median range (points)
    16.87
    95th percentile (points)
    29.86
    Maximum range (points)
    42.01

  • Bespoke Nimble-9B

    Median range (points)
    22.66
    95th percentile (points)
    56.46
    Maximum range (points)
    87.39

  • Rizzo Flow-4B

    Median range (points)
    39.41
    95th percentile (points)
    77.93
    Maximum range (points)
    92.04

  • Laya

    Median range (points)
    46.05
    95th percentile (points)
    62.58
    Maximum range (points)
    78.84

  • OpenAI Decisions

    Median range (points)
    55.00
    95th percentile (points)
    81.00
    Maximum range (points)
    92.33

  • SemIf / Qwen3.5-4B

    Median range (points)
    60.41
    95th percentile (points)
    80.88
    Maximum range (points)
    89.44

How often did the outcome change?

Lower is better for stability

  • Cloudflare Clef-27B (local)

    Range at least 10 points
    0.0%95% CI: 0.0–0.0%
    Untied winner changes
    0.0%

  • Von

    Range at least 10 points
    0.0%95% CI: 0.0–0.0%
    Untied winner changes
    0.0%

  • NanoJev

    Range at least 10 points
    0.0%95% CI: 0.0–0.0%
    Untied winner changes
    0.0%

  • Jev 1.13.0

    Range at least 10 points
    56.6%95% CI: 51.8–61.3%
    Untied winner changes
    11.8%

  • Jev 1.14.0

    Range at least 10 points
    58.0%95% CI: 53.1–62.6%
    Untied winner changes
    11.0%

  • Kev-4B

    Range at least 10 points
    73.5%95% CI: 70.0–76.9%
    Untied winner changes
    36.3%

  • SubQ Snap Preview

    Range at least 10 points
    82.1%95% CI: 78.8–85.2%
    Untied winner changes
    22.1%

  • Bespoke Nimble-9B

    Range at least 10 points
    74.2%95% CI: 70.4–77.9%
    Untied winner changes
    32.5%

  • Rizzo Flow-4B

    Range at least 10 points
    76.7%95% CI: 71.3–81.2%
    Untied winner changes
    46.6%

  • Laya

    Range at least 10 points
    99.9%95% CI: 99.6–100.0%
    Untied winner changes
    81.1%

  • OpenAI Decisions

    Range at least 10 points
    98.8%95% CI: 97.8–99.6%
    Untied winner changes
    56.3%

  • SemIf / Qwen3.5-4B

    Range at least 10 points
    94.6%95% CI: 92.6–96.4%
    Untied winner changes
    38.7%

Range means the highest score minus the lowest score across 14 order means. Each mean uses three replies. All ranges use percentage points. Small nonzero values use scientific notation. For example, 2.98e-6 means 0.00000298 points. Untied winner changes count samples with a different highest-scoring emotion across orders and no tied highest score in any order mean. The percentage uses all 721 shared samples.

Which emotion scores move?

Each cell shows the mean score range for one emotion, in percentage points. Both tables use the same color scale for the selected sample set.

Mean score range: Frustration, Disapproval, Sadness, Confusion

Lower is better for stability

  • Cloudflare Clef-27B (local)

    Frustration
    0.00
    Disapproval
    1.02e-4
    Sadness
    8.32e-5
    Confusion
    0.00

  • Von

    Frustration
    0.00
    Disapproval
    2.77e-5
    Sadness
    2.77e-5
    Confusion
    1.39e-5

  • NanoJev

    Frustration
    7.57e-7
    Disapproval
    1.09e-6
    Sadness
    6.63e-7
    Confusion
    2.79e-7

  • Jev 1.13.0

    Frustration
    3.17
    Disapproval
    8.54
    Sadness
    1.35
    Confusion
    6.49

  • Jev 1.14.0

    Frustration
    3.24
    Disapproval
    7.83
    Sadness
    1.73
    Confusion
    6.64

  • Kev-4B

    Frustration
    4.23
    Disapproval
    8.05
    Sadness
    4.00
    Confusion
    7.61

  • SubQ Snap Preview

    Frustration
    4.23
    Disapproval
    12.73
    Sadness
    4.43
    Confusion
    6.27

  • Bespoke Nimble-9B

    Frustration
    9.61
    Disapproval
    13.36
    Sadness
    6.88
    Confusion
    11.40

  • Rizzo Flow-4B

    Frustration
    8.45
    Disapproval
    27.55
    Sadness
    13.96
    Confusion
    25.09

  • Laya

    Frustration
    40.51
    Disapproval
    32.44
    Sadness
    6.83
    Confusion
    4.64

  • OpenAI Decisions

    Frustration
    12.73
    Disapproval
    45.30
    Sadness
    18.81
    Confusion
    33.57

  • SemIf / Qwen3.5-4B

    Frustration
    20.77
    Disapproval
    46.91
    Sadness
    26.04
    Confusion
    41.36

Mean score range: Anxiety, Distrust, Positive

Lower is better for stability

  • Cloudflare Clef-27B (local)

    Anxiety
    0.00
    Distrust
    9.25e-6
    Positive
    9.25e-6

  • Von

    Anxiety
    5.55e-5
    Distrust
    1.11e-4
    Positive
    4.16e-5

  • NanoJev

    Anxiety
    6.12e-7
    Distrust
    6.55e-7
    Positive
    1.78e-6

  • Jev 1.13.0

    Anxiety
    2.04
    Distrust
    3.24
    Positive
    7.94

  • Jev 1.14.0

    Anxiety
    2.02
    Distrust
    3.31
    Positive
    7.76

  • Kev-4B

    Anxiety
    5.03
    Distrust
    4.86
    Positive
    8.14

  • SubQ Snap Preview

    Anxiety
    3.43
    Distrust
    4.38
    Positive
    13.00

  • Bespoke Nimble-9B

    Anxiety
    4.57
    Distrust
    9.85
    Positive
    15.96

  • Rizzo Flow-4B

    Anxiety
    10.40
    Distrust
    8.93
    Positive
    16.86

  • Laya

    Anxiety
    29.14
    Distrust
    2.40
    Positive
    19.54

  • OpenAI Decisions

    Anxiety
    18.95
    Distrust
    15.67
    Positive
    31.64

  • SemIf / Qwen3.5-4B

    Anxiety
    23.18
    Distrust
    28.77
    Positive
    32.99

Repeated requests and score concentration

Same order, compared with different orders

Lower is better for stability

  • Cloudflare Clef-27B (local)

    Same-order difference (points)
    1.25e-5
    Different-order difference (points)
    1.48e-5
    Extra difference (points)
    2.40e-695% CI: 0.000–7.36e-6

  • Von

    Same-order difference (points)
    0.000
    Different-order difference (points)
    1.27e-5
    Extra difference (points)
    1.27e-595% CI: 6.88e-6–1.92e-5

  • NanoJev

    Same-order difference (points)
    0.000
    Different-order difference (points)
    3.20e-7
    Extra difference (points)
    3.20e-795% CI: 2.99e-7–3.44e-7

  • Jev 1.13.0

    Same-order difference (points)
    0.568
    Different-order difference (points)
    1.641
    Extra difference (points)
    1.07395% CI: 0.983–1.159

  • Jev 1.14.0

    Same-order difference (points)
    0.648
    Different-order difference (points)
    1.647
    Extra difference (points)
    0.99995% CI: 0.926–1.070

  • Kev-4B

    Same-order difference (points)
    0.000
    Different-order difference (points)
    2.077
    Extra difference (points)
    2.07795% CI: 2.016–2.136

  • SubQ Snap Preview

    Same-order difference (points)
    0.025
    Different-order difference (points)
    2.305
    Extra difference (points)
    2.28095% CI: 2.194–2.369

  • Bespoke Nimble-9B

    Same-order difference (points)
    0.000
    Different-order difference (points)
    3.433
    Extra difference (points)
    3.43395% CI: 3.230–3.633

  • Rizzo Flow-4B

    Same-order difference (points)
    0.000
    Different-order difference (points)
    5.235
    Extra difference (points)
    5.23595% CI: 4.863–5.606

  • Laya

    Same-order difference (points)
    0.000
    Different-order difference (points)
    6.476
    Extra difference (points)
    6.47695% CI: 6.362–6.587

  • OpenAI Decisions

    Same-order difference (points)
    0.051
    Different-order difference (points)
    7.973
    Extra difference (points)
    7.92295% CI: 7.674–8.171

  • SemIf / Qwen3.5-4B

    Same-order difference (points)
    0.000
    Different-order difference (points)
    9.739
    Extra difference (points)
    9.73995% CI: 9.377–10.078

Differences compare pairs of individual replies and average across seven emotions. Extra difference is the different-order difference minus the same-order difference. Lower values indicate more stable scores. The repeat verification checks the Jev results against the recorded requests and a fresh test.

Score concentration and ties

  • Cloudflare Clef-27B (local)

    Mean highest score
    64.7%
    Samples with a tied order mean
    0.0%

  • Von

    Mean highest score
    61.5%
    Samples with a tied order mean
    0.0%

  • NanoJev

    Mean highest score
    30.2%
    Samples with a tied order mean
    0.0%

  • Jev 1.13.0

    Mean highest score
    79.6%
    Samples with a tied order mean
    1.5%

  • Jev 1.14.0

    Mean highest score
    76.3%
    Samples with a tied order mean
    1.9%

  • Kev-4B

    Mean highest score
    44.7%
    Samples with a tied order mean
    0.0%

  • SubQ Snap Preview

    Mean highest score
    61.7%
    Samples with a tied order mean
    0.0%

  • Bespoke Nimble-9B

    Mean highest score
    69.9%
    Samples with a tied order mean
    0.0%

  • Rizzo Flow-4B

    Mean highest score
    72.7%
    Samples with a tied order mean
    0.0%

  • Laya

    Mean highest score
    48.5%
    Samples with a tied order mean
    6.9%

  • OpenAI Decisions

    Mean highest score
    61.1%
    Samples with a tied order mean
    11.1%

  • SemIf / Qwen3.5-4B

    Mean highest score
    64.7%
    Samples with a tied order mean
    35.9%

The highest score describes how scores concentrate on one emotion. The tie measure is the share of samples with at least one tied highest score across the order means. These are context for the stability results. Emotion accuracy was not tested.

Coverage across all planned samples

Each system had 42,000 planned evaluations across 1,000 samples. These counts use the full run and stay the same when you change the sample filters above.

Score coverage

Higher is better for completeness

  • Cloudflare Clef-27B (local)

    Score replies / 42,000
    42,000
    Complete score samples / 1,000
    1,000
    Strict-complete samples / 1,000
    1,000

  • Von

    Score replies / 42,000
    42,000
    Complete score samples / 1,000
    1,000
    Strict-complete samples / 1,000
    1,000

  • NanoJev

    Score replies / 42,000
    42,000
    Complete score samples / 1,000
    1,000
    Strict-complete samples / 1,000
    1,000

  • Jev 1.13.0

    Score replies / 42,000
    42,000
    Complete score samples / 1,000
    1,000
    Strict-complete samples / 1,000
    972

  • Jev 1.14.0

    Score replies / 42,000
    42,000
    Complete score samples / 1,000
    1,000
    Strict-complete samples / 1,000
    957

  • Kev-4B

    Score replies / 42,000
    42,000
    Complete score samples / 1,000
    1,000
    Strict-complete samples / 1,000
    1,000

  • SubQ Snap Preview

    Score replies / 42,000
    42,000
    Complete score samples / 1,000
    1,000
    Strict-complete samples / 1,000
    1,000

  • Bespoke Nimble-9B

    Score replies / 42,000
    42,000
    Complete score samples / 1,000
    1,000
    Strict-complete samples / 1,000
    1,000

  • Rizzo Flow-4B

    Score replies / 42,000
    42,000
    Complete score samples / 1,000
    1,000
    Strict-complete samples / 1,000
    1,000

  • Laya

    Score replies / 42,000
    42,000
    Complete score samples / 1,000
    1,000
    Strict-complete samples / 1,000
    1,000

  • OpenAI Decisions

    Score replies / 42,000
    41,822
    Complete score samples / 1,000
    966
    Strict-complete samples / 1,000
    966

  • SemIf / Qwen3.5-4B

    Score replies / 42,000
    42,000
    Complete score samples / 1,000
    1,000
    Strict-complete samples / 1,000
    1,000

Jev 1.13 returned every score distribution. Its strict check excludes 32 chosen-label mismatches. Jev 1.14 recovered 19 temporary API errors on the next attempt. Its row includes those scores. Its 52 chosen-label mismatches remain, leaving 957 strict-complete samples. OpenAI answered every call, but 178 answers were refusals with no scores. An unchanged retry check recovered all 32 Jev 1.13 mismatches and one OpenAI refusal. The retry check does not replace the original Jev 1.13 or OpenAI results. See the coverage and retry methods.

Model settings and records

Each system used its native prompt format, precision, and calibration. This compares released systems on one task. Clef uses the released local Python model. Its encoder sorts named choice keys. The hosted Workers AI service was not tested.

Native precision and calibration

  • Cloudflare Clef-27B (local)

    Precision
    BF16 model; FP32 softmax; native probabilities rounded to four decimals
    Temperature / calibration
    1.000

  • Von

    Precision
    Native FP32
    Temperature / calibration
    Shipped input-conditioned calibration, clipped to 0.3–12

  • NanoJev

    Precision
    Native BF16 autocast
    Temperature / calibration
    1.000

  • Jev 1.13.0

    Precision
    Provider controlled
    Temperature / calibration
    Provider controlled

  • Jev 1.14.0

    Precision
    Native probabilities rounded to two decimal places
    Temperature / calibration
    Provider controlled

  • Kev-4B

    Precision
    BF16
    Temperature / calibration
    2.406

  • SubQ Snap Preview

    Precision
    Provider controlled
    Temperature / calibration
    Provider controlled

  • Bespoke Nimble-9B

    Precision
    BF16
    Temperature / calibration
    1.000

  • Rizzo Flow-4B

    Precision
    Q8_0 weights
    Temperature / calibration
    1.000

  • Laya

    Precision
    Native BF16 autocast
    Temperature / calibration
    1.000

  • OpenAI Decisions

    Precision
    Provider controlled
    Temperature / calibration
    Provider controlled

  • SemIf / Qwen3.5-4B

    Precision
    BF16
    Temperature / calibration
    1.000

The full methodology records input handling, checkpoints, calibration, and comparison limits.

Inspect the recorded invented examples. Download the comparison CSV or full aggregates and model revisions. These files contain no source texts or per-sample records.

Back to the research post