Every saved run, scored out of 60.

1,004 saved passes on the 60 reviews, each scored on four decisions, with what each pass cost.

All results

Solar Decide led the decision models at 55 of 60 on its first base-task pass, and Opus 5.5 at high effort reached 59. Each row is one configuration with its score for every prompt version and pass, the four decisions, and the cost of one pass.

Accuracy on each decision

Jev's six misses come to 11 differing decisions: 4 on sentiment, 2 on follow-up, 3 on serious concern and 2 on testimonial. Every model below is scored out of 60 on each decision, first pass, base prompt.

Ask again and the answers move

31 of 50 decision-model setups gave identical answers in all three passes. 11 of 202 general-model setups did.

What one pass cost

A Jev pass over the 60 reviews cost $0.0059 on OpenRouter. The same pass on Opus 5.5 at high effort costs $0.2221† at API prices, about 38 Jev passes.

What a pass cost against what it matched

Each point is one complete pass over the 60 reviews: across, what the pass cost on a log scale; up, the reviews where all four answers matched.

All 60 reviews against the decision models

26 reviews matched the reference on all seven decision models. 2 missed on all seven: DEV-029, the off-topic soup review, and DEV-030.

The same review can produce different decisions.

Each review with the four answers from the seven decision models, first pass, base prompt.

Loading 60 source-bound reviews…

Loading the saved answers…

Every answer in one run

Each of the 60 reviews with the run's four answers beside the reference, then request time, tokens and cost.

Loading run evidence…

When two models agree

21 pairs of decision models, each accepting a review only when both give the same four answers.

Scores under alternative labels

DEV-006, DEV-013 and DEV-030 rescored under each alternative label, with no new model calls.

Downloads

Earlier analysesThe panels we built while the study ran. Every number above comes from the same saved answers.

Decision models, general models, or both

The chart and the run list below follow this filter.

Loading the saved-run view…

Loading repeat and continuation passes.

Saved runs, ranked by all-four matches

One row per setup, best pass first.

Jev on the direct TypeSafe API

Jev on TypeSafe's direct API matched 54 of 60 on its first pass. That route sends no bill; through OpenRouter the same model cost $0.0059 a pass.

All four match, base taskn/a/ 60

Loading the saved Jev run…

Jev beside a second run

Agreement with the reference on each of the four decisions, Jev against the second run.

The second run, review by review ↓

Tokens and cost for one run

Three prompt versions on the same setup

From P1 to P2, 4 of 39 setups scored higher, 14 held and 21 scored lower.

P0

The task, scoring rules and required response format.

P1

The base task with a classifier role and more explicit instructions.

P2

The base task with step-by-step rules for the four decisions.

For native decision models, P1 and P2 change the questions sent to the model. For chat models, they add prompt instructions.

Loading model setups…

Why some prompt versions are missing
Browse every saved run Filter by task, verified training, or answer interface

Every saved run

A model can appear more than once when it has different prompts, settings or attempts.

Category describes the model's task, and one setup can fit more than one. The training filter shows models whose weights were changed for this task; anything without a source says not verified. Output interface is how the setup returned answers.

Model and runPromptValid of 60All four of 60Usage and cost

Loading saved outcomes…

What we learned from the answers.

One total hides which review was missed and whether the next run answers the same way.

P0 is the base task, P1 adds a classifier role and P2 adds ordered decision rules.

Jev's answers differed on six comments.

On TypeSafe's own API, Jev matched 54 of the 60 comments, the same score as its first pass through OpenRouter. Every response could be scored, and six comments had at least one answer that differed from the reference.

Missing a follow-up request and misreading praise lead to different actions, so the type of mistake matters as much as the count.

The six disagreements ↓Next finding What changed when we added instructions
54/ 60 comments
matched on all four answers
All four match At least one disagreement
Jev against another run →

In the first pass, 21 of 39 setups scored lower with decision rules.

On the first pass we compared three prompt versions: the base task (P0), classifier instructions (P1) and step-by-step decision rules (P2).

Moving from P1 to P2 improved four setups, left 14 unchanged and lowered 21. More detail did not buy more matches here.

The paired comparisons ↓Next finding What changed when we asked again

Adding a decision tree: P1 → P2

Higher score
4
Same score
14
Lower score
21
39 hosted and subscription setups.

4 higher scores, 14 unchanged, 21 lower scores.

Prompt versions by setup →

In one repeated Gemma test, 11 of 60 comments got different answers.

In two Gemma E2B thinking-on runs with the same P1 instructions, the score moved from 36 to 39 out of 60. Both runs returned 60 responses we could score.

Four comments gained a full match with the reference; one lost it. Other answers changed without changing whether all four matched. An answer change means at least one of the four decisions changed, even if the total score stayed the same.

A three-point difference on one run can hide that much movement underneath.

Repeat passes for every model ↓Explore the data Compare saved results
First P1 pass36/60
Second P1 pass39/60
Same answers At least one answer changed
One of the 11 changed comments
Second pass records → First pass records →
Gemma E2B findings →

Decision models against general LLMs

First-pass answers from 149 general-model passes and the seven decision models, decision by decision.

Which reviews split the models

First-pass answers from decision models and general LLMs, review by review.

Clef and Clef Flash: first-pass results

Before the OpenRouter runs, Clef and Clef Flash each ran one plain-prompt pass on Cloudflare's direct route. The OpenRouter pass in the results table gave Clef 54 of 60.

Similar scores can hide changed answers.

Sonnet 5.5 now has three passes of each prompt at four effort levels. Xhigh scored 58/60 every time, but changed individual answers under P1 and P2. The extra instructions did not consistently improve agreement.

Loading the latest comparison…

Effort is the requested reasoning level. Each range covers three passes, and changed counts reviews with any changed decision across them. Token costs are at API rates for nine development runs per effort.

Perplexity Decider and Clef both matched 54 of 60 on their first base-task pass.

They cost $0.0154 and $0.0318 a pass on OpenRouter. Same total, different misses.

Their answers side by side

Qwen 0.6B repeated the same mistakes.

This small local model returned all four labels in the required format in every pass. Yet no review matched all four reference answers. With the base prompt, it marked all 60 comments as potential testimonials; the reference key marks nine. Its answers were identical across three passes of each prompt.

Qwen 0.6B field by field

Prompt, repeat, error, cost and review analysesPrompt changes, repeat tests, Jev disagreements, observed costs and comments worth another look.Comparison sections

Decision rules lowered more setups than they raised

P1 against P2 for the 39 audited setups.

Did the added instructions help the same setups?

Loading paired comparisons…

Seven more hosted comparisons show gains and losses from longer instructions. Among reviews with valid answers under both prompts, classifier instructions gained six all-four matches and lost six; decision-tree instructions gained seven and lost ten. Interrupted runs changed their schedule, so they sit outside the paired comparison above. Field changes, failures and examples.

What changed when Kev received different decision instructions?

P1 and P2 kept the route, labels, criteria and parser. Only the instruction text inside each native Choice question changed.

Did the added decision rules improve the same four answers?

Loading the Kev prompt comparison…

How did Jev change across three instruction sets?

Jev through OpenRouter's native Choice route, three passes each of P0, P1 and P2. Invalid and unsent answers stay in the counts.

Did the categorical answers and probability distributions change together?

Loading the Jev prompt comparison…

Did the answers change when we asked again?

Same score, different answers: Gemma E2B scored 35 of 60 in all three base-task passes while 9 reviews changed a decision.

Which comments changed, and were all responses usable?

An unfinished pass has no score, and it does not count as zero.

Loading repeat comparisons…

Qwen 27B at medium effort scored lower with classifier instructions in all three passes. Those scores ranged from 54 to 58 matches out of 60, and four reviews changed at least one answer across passes. Both effort settings, all three prompts and the changed answers.

More findings, by model and question

Findings across four topics.

When instructions changeCompare prompt variants within the saved setups.12 notes

Sonnet 5.5 completed three passes of every prompt at four effort levels. Xhigh scored 58/60 throughout, but individual answers still changed under classifier and decision-tree instructions. More instructions did not consistently add matches. See scores, changed answers, token use and API-rate costs.

The six fresh Codex comparisons also show unstable prompt gains. Neither added-instruction prompt beat the base task in all three passes for any of these six configurations. Classifier instructions switched between a gain and a loss in four configurations. GPT-6 Luna low ranged from 51 to 55 all-four matches with those instructions; its decision-tree prompt lost two or three matches against the base task in every pass. Compare all six three-pass results.

DeepSeek Flash scored lower with both added-instruction prompts in all three passes. With reasoning off, the base prompt matched all four labels on 48–49 of 60 reviews, classifier guidance on 45–47, and the decision-tree prompt on 43–46. Testimonial agreement improved, while sentiment agreement fell. The same prompt still produced changed classifications on two to four reviews across passes. See the field-level trade-offs and repeat variation.

Across the 17 Claude configurations reviewed on 28 September, neither added-instruction prompt improved all-four agreement over the base task in every pass. Seven configurations switched between a gain and a loss for classifier instructions; seven did so for decision-tree instructions. Those groups overlap. Compare all 17 configurations and the limits of this finding.

Gemma E2B's three fresh local passes scored 35/60 with the original prompt each time, while nine reviews changed a decision. Classifier instructions scored 33–34/60; adding a decision procedure scored 31–33/60 and changed 15 reviews. The earlier local runs are kept separate. Read the field-level differences and local timing limits.

Haiku's classifier-instruction score ranged from 53 to 59 matches out of 60 across three fresh passes. Its advantage over the base prompt changed from two fewer matches to two more. Read the Haiku repeat comparison.

Opus 5 high effort gained two matches from decision-tree instructions in its first two passes, then lost two in the third. See the Opus prompt comparisons.

Sonnet 5 medium gained two and one matches from classifier instructions in its first two passes, then lost four in the third. Opus 5.5 medium scored 58/60 with those instructions in every pass, while one review changed classification. Sonnet results and Opus 5.5 results.

Across 18 Codex configurations, classifier instructions improved agreement in every pass for two configurations; decision-tree instructions did so for none. Twelve configuration/condition groups kept the same score while individual answers changed. Compare all 18 configurations.

Qwen's decision-tree prompt gained a small number of matches in all three passes. On reviews valid in both conditions, P2 gained two, one and two all-field matches over P0. P1's advantage changed direction. Two provider failures required later continuations, so this is a descriptive comparison with interrupted dispatch. See all nine results, changed reviews and retained failures.

DeepSeek low now has results for all nine scheduled passes, including failed answers. Its first P0/P1/P2 scores were 58/57/53 out of 60. The interrupted P2 includes 56 valid answers, one invalid answer and three failures; no reviews remain unsent. In the final pass, P0/P1/P2 scored 58/57/58, with 59/59/60 valid answers. These totals retain failures and do not establish nine clean repeats. See the first-pass outcomes. Compare the final passes and changed answers. Earlier reports: first P0/P1 report and interruption report.

OpenJev generated-off and generated-on have each closed all nine development phases. Across three passes, generated-off P0 scored 48, 51 and 49 all-field matches out of 60; P1 scored 48, 50 and 49; P2 scored 51, 54 and 51. Generated-on P0 scored 54, 53 and 49; P1 scored 50, 46 and 49; P2 scored 46, 47 and 42. Among reviews valid in all three generated-on passes, 8/54 P0, 2/39 P1 and 8/35 P2 reviews changed a four-field answer. Invalid outputs remain in the fixed 60-review scores. These comparisons do not establish an instruction effect. See the separate generated findings.

When the answers changeA similar total can hide different decisions.3 notes

The Gemini repeats show why both scores and individual answers matter: Gemini 3.7 medium kept the same P0 score while changing three reviews. Gemini 3.8 medium sometimes ran out of output tokens, leaving a batch of ten reviews invalid. Read the Gemini findings and limitations.

Gemini 3.8 Flash low scored 57/60 with the original prompt in all three passes, yet changed one review's classification. The classifier prompt scored 56/57/56 and changed five reviews; the decision-procedure prompt scored 56/55/56 and changed three. Neither added prompt improved the score over the original in this series. See the changed reviews, costs and limits.

OpenJev completed all three native P0 triplets. Fixed scored 53/60 and adaptive 52/60 all-field matches per pass, with no within-mode classification flips. Thinking scored 53/60, 54/60 and 54/60; its three pairwise comparisons changed at least one decision on 9, 9 and 4 reviews. All nine passes had 60 valid outputs. Cross-mode scores do not isolate a causal effect. See the class-level findings and measurement limits.

When scores stay steadyA repeated score can still disagree with the draft answers.8 notes

Stable answers can still be wrong under the rubric. Three Laya native configurations returned unchanged classifications across all three passes, but each matched all four reference decisions on 0 of 60 reviews. Testimonial suitability matched on only 10–13 reviews. See field scores and class distributions.

SemIf's testimonial agreement hides missed positive reviews. All three native modes match 54/60 testimonial labels, but identify only four of the nine reference-positive testimonials in their completed passes. See the class-level findings and repeat coverage.

AnyJev raw repeated the same mistakes. All three native passes matched all four reference fields on 0 of 60 reviews. It called every review neutral and marked every review for follow-up and serious concern. No classifications changed across passes. This result concerns the raw readout; calibrated variants are separate. Read the field-level analysis.

AnyJev L0 repeated its answers, but matched all four reference fields on only 4/60 reviews. All three passes had 60 valid outputs and no changed classifications. For follow-up, it answered "insufficient information" 44 times against one such reference label. The raw and L0 methods differ in several ways, so the score gap does not tell us which change caused it. Read the class-level findings.

Alex 0.8B marked nearly every review as a potential testimonial. Each of three fresh native passes had 60 valid outputs but only 3/60 all-four reference matches. It predicted testimonial yes 59 times, including 49 reference-negative reviews. No classifications changed across the three passes. Read the field-level findings and measurement limits.

Alex 4B repeated the same strengths and misses. All three native passes matched all four labels on 39/60 reviews, with no changed classifications. It found 17 of 25 reference-positive serious concerns and missed the same eight each time. Stable answers can still disagree with the reference. See the class-level comparison and runtime measurements.

AnyJev L1 returned identical classifications in three calibrated passes, matching all four labels on 6/60 reviews each time. All 60 four-label predictions matched across passes, with zero pairwise flips and a three-pass range of 6–6/60. In each pass it returned "insufficient information" for follow-up on 41 reviews and missed 16 of 25 reference-positive serious concerns. Calibration used five held-out folds. Repeated agreement on this fixed set does not prove deterministic behavior elsewhere. Read the class-level findings and calibration method.

AnyJev L2 repeated the same classifications in three verified native passes, with 13/60 all-four matches each. All three pairwise comparisons have zero flips across 60 valid reviews. It identified 29 of 35 reference-positive follow-up cases, but also marked nine of 24 reference-negative cases for follow-up. These fitted-head results use held-out development folds; agreement here does not establish accuracy elsewhere. Read the class-level findings and timing limits.

When an answer cannot be scoredFailures and unusable formats remain in the record.3 notes

A provider failure left one Gemma 26B comment without an answer. Its decision-procedure test stopped after six answers and one error. A separate continuation processed the 53 remaining comments. Together they produced 57 all-four matches out of 60, with two disagreements and one missing answer. This interrupted result is separate from the matched repeat study. Read the disagreements, recorded charges and timing limits.

SemIf sometimes returned the schema instead of a classification. Across three fresh passes per prompt, P0 scored 35/60 all-field matches each time, P1 scored 26/60 and P2 scored 43/60. The same eight, 22 and two reviews respectively returned the schema each time. No answers changed among reviews valid in all three passes: 0/52 for P0, 0/38 for P1 and 0/58 for P2. These repeated results describe the fixed review set. Read the generated-output findings.

AnyJev's generated JSON control completed all nine full phases. P0 and P1 returned 60 fenced, invalid responses and scored 0/60 in each of three passes; neither has valid classifications to compare. P2 returned the same 30 valid and 30 invalid reviews each time, scored 1/60, and had 0/30 observed four-field changes among reviews valid in all three passes. This control is separate from native AnyJev readouts. Read the source-bound findings.

Six reviews where Jev differed from the reference

The four decisions can fail in different ways.

Where did Jev disagree, and did other runs miss the same comments?

Loading decision counts…

High confidence did not remove the misses. Keeping only Jev's testimonial answers at confidence 0.9 or higher would retain 54 of 60, and one of those still differs from the reference. No answer was withheld in the run. Calibration is measured on these 60 reviews only. Thresholds for all four decisions.

Higher effort cost more in these two Gemini comparisons

Each higher-effort run matched one fewer review and cost 2.6 to 4 times as much.

What did these scored runs cost?

Loading cost records…

Reviews the first-pass setups missed most

A repeated miss may reveal an ambiguous comment or reference answer.

Which comments should a person review first?

Loading review counts…

What the scores mean

Missing or unusable answers score zero; the denominator stays 60.

How the test works

Four questions on each of 60 reviews, asked under three prompt versions and repeated in fresh passes.

Fictional feedback, including difficult cases

The 60 comments include praise, complaints, mixed experiences and unclear or off-topic text. They describe the candidate's experience of recruitment. They are not interviewers' assessments of candidates.

How the reviews and the reference were made →