Do models like Jev get it right when correctness is business-critical?

Nice staff. After the trial shift the manager kept asking me out even after I said no twice. I've asked the recruiter to stop him contacting me.

Is this a serious concern? Does someone need to follow up? You probably knew both answers by the second line. That gut call is what Kahneman calls System 1, and TypeSafe borrowed the name for Jev. I wanted to know how far it carries when the decision matters. Each dot is one of the 60 reviews, and the orange one is this review.

Adam Kovacs, Chief AI Strategist, AI Enablement Academy
Agentics Foundation Meetup, Budapest, 8 October 2026

I'm Adam Kovacs.

Co-founder and Chief AI Strategist, AI Enablement Academy
Co-founding Member and International Ambassador, Agentics Foundation
Executive Director, OPEN Talent Society
Before this, I spent six years in recruiting at Amazon.
I build hands-on AI training for teams, and open benchmarks on recruiting problems like this one.
github.com/adambkovacs

What a decision model is

A decision model takes text and a typed question, and returns one of your options with a probability in one pass.

System 1: the quick gut call

It works when the answer is one of a short list and each question asks about one thing. It can't invent a label.

Is a harassment report a serious concern? Jev caught all 25.

System 2: stopping to ask if it's the right question

It takes your words literally and can't say why it answered. It won't tell you when the question doesn't fit.

"Train got cancelled... Really helpful of them." Is that a testimonial?

The launch wave

Jev shipped on 15 September. By 7 October, Cloudflare, AWS, Perplexity and OpenAI had each shipped a decision model of their own.

They take your text and typed questions and return one option, and most of them work the same way. Within a day of launch, nearly 13% of paid teams on Vercel's AI Gateway were using Jev. Credit to TypeSafe: they published a page listing where Jev fails. Most vendors don't.

15 SepJev 1.13TypeSafe
1 OctClef, Strands DeciderCloudflare and AWS, same day
7 OctPerplexity Deciderlisted on OpenRouter
6 OctOpenAI Decisionspublic beta

The test

Every model got the same 60 fictional reviews and four questions about each. Matched means all four answers agree with our key.

How was the experience?sentiment, negative, positive, neutral, mixed, can't tell

Does someone need to follow up?follow-up, yes, no, can't tell

Does it report a serious concern?concern, harassment, discrimination or a privacy breach; a bad experience alone doesn't count, yes, no, can't tell

Could it work as a testimonial?testimonial, yes, no, can't tell

An AI assistant drafted the reviews and the key, and people checked all 60. Three labels are still disputed.
P0 is the plain task. P1 tells the model it's a classifier. P2 adds step-by-step instructions for deciding.
39 general models and settings ran all three prompts, most of them three times from scratch, next to the seven decision models and Jev.

The answer: four decisions per review

On 60 test reviews, Jev matched our answer key on all four decisions for 54. General models from Gemma 26B up to Opus 5.5 matched 58 or 59, at four to forty times the cost.

Jev 1.13direct API, first pass, plain prompt

5 reviews Opus matched and Jev missed

OpenRouter, three passes: 54, 53, 52

Opus 5.5high effort, ten reviews per request

three passes: 59, 58, 58

Matched means agreeing with our answer key on all four. Three of the key's labels are still disputed.

If the decision matters, create a rule that hands the hard reviews to a person.

Show of handsHands up if you've shipped an LLM classifier. Keep it up if a second model checks it.

Proof 1: the gap is five reviews

Every run shows the same gap: Jev matched 52 to 54 of 60, and Opus 5.5 matched 58 or 59.

Each grid is one model. Columns are the prompt versions P0, P1 and P2, rows are three fresh passes, and a blank cell means that run doesn't exist.

all four matched, of 60: 40 to 60blank means no run

Sonnet 5.5 matched 58 in all nine runs. Gemma 26B and Qwen 27B, both free to download, each matched 59 on their first pass.

Proof 1: one pass over 60 reviews

A Jev pass over all 60 reviews cost $0.006, and an Opus pass about $0.22. Opus matched five more reviews, and at this volume that's pocket change, so the question is which five.

Jev, OpenRouter P0 pass

$0.006

Opus 5.5 high, ten reviews per request

$0.22

Jev caught all 25 reviews the key flags as a serious concern. On the harassment review that opened the talk, it still said nobody needed to follow up, at 0.49 confidence.

Proof 1: what the general models taught us

More effort didn't buy more matches. Bigger models didn't buy more matches either. The big models gave the same answers, wrong ones included. Multiple runs proved non-deterministic outcomes.

Gemini 3.1 Pro matched 56 of 60 at low effort for $0.063, and 55 at high effort for $0.257.
44 runs from 7 models across three vendors gave exactly the same answers.
31 of 50 decision-model setups gave identical answers across three runs. 11 of 202 general-model setups did.

Why five reviews matter

Typed output fixes the format, and the decision can still be wrong.

Perplexity Decider matched 54 of 60 in all nine runs, with the same six misses. A model that gives the same wrong answer every time never shows you it's wrong.

Miss a harassment report and the candidate posts about it on X and Reddit. Hiring gets a lot harder after that.

Great soup, tiny portions, wouldn't eat there again.

Shout it outSeven decision models read this review. How many of them flagged it as off-topic?

Replay of saved answers, 2026-10

Proof 2: seven decision models, one review

Great soup, tiny portions, wouldn't eat there again.

0 of 7 decision models matched the key.

Jev 1.13 is not one of the seven. Its saved answer:

It gave the same wrong answer: no serious concern, at 0.91 confidence.

89 of the 113 general-model configurations with a valid answer said can't tell on all four.

The models answered the question on the form and didn't notice it didn't apply. Asking "is this even the right question?" is System 2, and it isn't in the model.

Replay of saved answers, 2026-10

Proof 2: six reviews a person would pause on

Four of these are judgment calls and two are plain misses.

These are exactly the six reviews Jev's first pass missed, and three carry labels a second review of our key disputes. The misses weren't random, so you can route reviews like these to a person. Under each card: the key's four answers, then seven tiles, one per decision model, with its four answers as squares. An orange square is an answer that differs from the key.

All 60 reviews by how many of the 7 decision models missed

260

151

122

DEV-030fixed, I think

The accessibility issue from the assessment has been dealt with, I think. Someone said it was fixed, but I don't know whether that means I get another assessment or just that they've closed the ticket.

DEV-029off-topic

Great soup, tiny portions, wouldn't eat there again.

DEV-006it happened again

Same as last time. That thing happened again and it still isn't sorted.

DEV-013a friend heard a rumour

A friend told me they heard this firm asks inappropriate questions. My own interview was straightforward, and I have no complaint about it.

DEV-027kind words, cancelled train

Train got cancelled so they switched me to video with the same interviewers and gave me five mins to catch my breath. Really helpful of them.

DEV-059serious concern, no complaint

Nice staff. After the trial shift the manager kept asking me out even after I said no twice. I've asked the recruiter to stop him contacting me.

[Not the six hardest: DEV-056 (4 of 7 models missed) ranks above DEV-059 (2 of 7). DEV-059 is here because it opened the talk.]

And five more reviews split the models the same way. [S13, all six cards on screen] "Someone said it was fixed, I think": uncertain resolution, and none of the seven decision models got it right. "That thing happened again": vague recurrence, one of seven got it right. "A friend told me they heard": second-hand rumour, two of seven. "Great soup": off-topic, zero of seven. "Really helpful of them" after a cancelled train: kindness during a logistics failure, three of seven. And the harassment review itself: a serious concern with no formal complaint, five of seven. Three of those are the labels a second review of our key, done by AI, disputes. So the misses weren't random, and that's good news, because you can route reviews like these to a person.

Replay of saved answers, 2026-10

Proof 3: the number next to the answer

Every answer comes with a confidence score. On these 60 reviews Jev's score tracked how often it was right, and it was still 0.96 confident on a wrong testimonial.

Jev 1.13 P0, testimonial field, 60 answers sorted by confidence

cutoff 0.90

0.9 cutoff keeps 54 of 60

0.7 keeps 58

0.5 keeps 58

DEV-027, the train review: key, then Jev's answer Sentiment? key positive, Jev negative at 0.71
Follow-up? key no, Jev no
Serious concern? key no, Jev no
Testimonial? key yes, Jev no at 0.96

DEV-029, the soupNot a testimonial, at 0.88. Key: can't tell.

keptwithheldwrong answer (red: DEV-027)

A 0.9 cutoff keeps only answers at that confidence or above. It keeps 54 of 60 here, the wrong one included, though it would have caught the harassment follow-up miss at 0.49. A low score marks a hard review. A high score doesn't mean the answer is right.

How far each model's confidence sat from how often it was right, over 720 answers on these 60 reviews, smaller is better: Jev 0.011, Clef Flash 0.358. It describes these reviews and doesn't predict the next batch.

Proof three is the number next to the answer.

On these 60 reviews, Jev's confidence tracked how often it was right: pooled over 720 answers the average gap was 0.011, while Clef Flash's was 0.358. And Jev called the cancelled-train review not a testimonial at 0.96, and the key says the opposite. On sentiment, a 0.9 cutoff threw away nine good answers to catch four bad ones. On the testimonial field the same cutoff still let the train review through. In fairness, that cutoff would have caught the harassment follow-up miss at 0.49. Jev's ten least-confident reviews include seven of the ten hardest, the soup among them, but only through the sentiment field. On follow-up, concern and testimonial, the fields where you'd act, it said no at 0.84 to 0.91. So a low score marks a hard review, and a high score doesn't mean the answer is right.

Proof 3: the fixes I'd reach for first

The obvious fixes didn't reliably help.

A longer prompt

P0 is the plain task: the rubric and the answer schema. P1 adds a classifier role and the rule to use only the feedback as evidence. P2 keeps P1 and adds a numbered procedure: is the text usable, then each field in turn. The procedure lowered 21 of 39 setups.

Clef, matched of 60, same every pass

54P0

51P1

49P2

39 saved setups

P0 to P1, adding framing

15 up, 15 same, 9 down

P1 to P2, adding the tree

4 up, 14 same, 21 down

A majority vote

Voting Jev's three passes never beat its best single pass. A vote of three gave strong general models 0.14 more matches out of 60. That's nothing, so ask a different model instead.

Fine-tuning

It starts paying off around 200 labels, and we have 9 testimonials.

Cloudflare's Clef may only read the first 2,000 tokens, so it might never have seen the whole set of instructions.

The rule: agree or defer

Agree or defer. Run two cheap models that make different mistakes. Accept a review only when both give the same four answers. Send the rest to a person.

53 accepted, 0 disagreed with the key

7 to a person

S + P, Solar Decide and Perplexity Decider, $0.037 for both runs

58 accepted, 0 disagreed with the key

2 to a person

Q + G, Qwen 27B and Gemma 26B, $0.069 for both runs. Other Gemma runs let one error through.

[Clicks: 1. The slide enters sorted by Solar + Perplexity; the click re-sorts by Qwen + Gemma.]

The rule is agree or defer. Two cheap models on every review, accept only identical answers, send the rest to a person.

I fixed the rule before computing any pair, then looked at all 21 pairs of the seven decision models. Five pairs let zero errors through. [S16 is sorted on entry] Solar Decide plus Perplexity Decider agreed on 53 and sent 7 to a person, for under four cents for both, and I picked it after seeing all 21, so it's an example. It deferred the soup, the three disputed labels and the 0.96 testimonial, and both got the harassment review right, so the rule overrules Jev's miss.

Zero errors on these 60 is not zero errors on the next 60. And agreement only helps when the models fail differently: six of the 21 pairs accepted the soup with the same wrong answer.

[S16 re-sorts] Across 895 costed pairs, Qwen 27B low plus Gemma 26B thinking-on accepted 58 with zero errors and deferred 2, for about seven cents, though other Gemma passes let one error through.

The rule: the policy and the queue

The rule says what a model may auto-accept. The escalation lines say who reads the rest.

Run two decision models on every review.
Auto-accept only when all four answers are identical.
If either model flags a serious concern, a person reads it.
If either model says can't tell, a person reads it.
Keep the confidence score for the record, and don't use it to decide.

Solar + Perplexity, all 60 reviews, 35 reach a person

7the two models disagreed, to a person

24agreed, a serious concern was flagged, to a person

4agreed, one said can't tell, to a person

25auto-accepted

We put a lot of serious concerns in this set on purpose, so a normal pile would make a shorter queue.

Show of handsOne model at 0.96 confidence, or two cheap models that agree?

The rule: every pair

All 21 pairs and the human queue.

This table shows every pair of two decision models, so you can see the rule isn't one lucky pair.

Pair, fresh1/P0	Accepted	Errors	Deferred	+ concern	+ can't tell	To a person	Two runs

5pairs let zero errors through.

6pairs accepted the soup with the same wrong answer.

32 to 42of 60 reach a person under the full policy, by pair.

Solar + Perplexity: 7 deferred, 24 escalated for a concern, 4 for a can't tell; 35 reach a person. Solar sits in four of the five zero-error pairs.

+ concern and + can't tell: accepted, then routed to a person anyway. The rule was fixed before any pair was computed. No pair is selected. Green rail: zero accepted errors.

And this is every pair of the seven decision models, so you can see the rule isn't one lucky pair. Five pairs let zero errors through, and six accepted the soup with the same wrong answer. Zero errors on these 60 is not zero errors on the next 60.

Q: Would agree-or-defer hold in production? A: Unknown. The rule was fixed before the pairs were computed but scored on the same 60 reviews used for everything else, and the human review cost of the deferred set is unmeasured. Treat it as a policy to test.

Q: CTO: with your full policy, how big is the human queue? A: On this set, 35 of 60. Split it the way the workflow does: 25 reviews either model flagged as a serious concern go to a person, and on this set those flags covered all 25 reference concerns for every pair. Of the other 35 reviews the rule auto-accepts 25 and sends 10 to a person, 6 by disagreement and 4 by a "can't tell". The set is concern-heavy by design. On real feedback the share depends on how often concerns happen, which this can't tell you.

The tool: classification-bench

classification-bench runs this test on your own cases, labels and questions.

Bring any model through an OpenRouter key, or Claude Code and Codex on the subscription you already have.

Bring datayour inputs, your labels

Planmodels, prompts, passes

Runstays inside your budget

Evaluateagreement, answer changes

Reportper label, per pass

OpenRouterCloudflare Clef Claude CodeCodexLiquid / Solar / Qwen

What should you do

I'd pull 60 of my own cases and label them myself, before any model sees them.
I'd give every question a can't-tell option.
I'd run two cheap models and keep only what they agree on.
I'd count how many land in the human queue, and check the rare cases I care about got through.

QR code linking to github.com/adambkovacs/candidate-experience-benchmark — github.com/adambkovacs/
candidate-experience-benchmark

It's cheap to experiment

All five take your text plus typed questions with the options you supply, and return one option per question with a probability, in one pass, with no text to parse.

TypeSafe

Jev 1.13

$0.042 / M input

Early access
No changelog
Publishes a failure-mode page

OpenAI

Decisions on gpt-6-luna

$0.10 / M input

Public beta
Has a refusal type

Perplexity

Decider

$0.02 / M input

Apache-2.0 (v1)

Liquid AI

d1

$0.04 / M input

Price per blog methodology

Upstage

Solar Decide

beta

Price unverified

Prompt levels P0, P1, P2, every pass, in these saved runs.

Each row below counts the 39 model setups that ran both prompt versions: how many matched more of the 60 reviews on all four answers, how many matched the same number, and how many matched fewer.

P0 to P1, 39 setups

15 up, 15 same, 9 down

P0 to P2, 39 setups

7 up, 16 same, 16 down

P1 to P2, 39 setups

4 up, 14 same, 21 down

Confidence and calibration, on these 60 reviews.

Gap between confidence and hit rate, lower is closer
Model	Gap	Of its 10 least-confident, in the hardest 10
Jev	0.011	0
Liquid	0.035	0
Tev	0.047	0
Luna	0.074	0
Solar	0.145	0
Clef	0.217	0
Clef Flash	0.358	0

Jev P0: answers kept at or above a cutoff
Field	0.5	0.7	0.9
Sentiment	57	55	47
Follow-up	58	58	52
Concern	59	57	53
Testimonial	58	58	54

7of Jev's 10 least-confident reviews are among the 10 hardest.

These are worked out afterwards. No answer was actually withheld.

Equal scores hide different answers.

58in every Sonnet 5.5 xhigh cell, and it still changed DEV-006 and DEV-030 between passes.

31of 50 decision-model groups changed nothing across three passes.

11of 202 general-model groups did the same.

Perplexity Decider matched 54 of 60 in all nine runs, with the same six misses every time. A model that gives the same wrong answer every time never shows you it's wrong.

blank means no runcolumns P0 P1 P2, rows fresh pass 1 2 3

Q: Why not pay for Opus at 59? A: You can, at about 38 times the estimated spend, and Opus held 58 to 59 over three passes against Jev's 52 to 54. Sonnet scored 58 in all nine cells and still changed two reviews between passes, so the aggregate hides movement on the hard ones.

Q: Is 54 versus 59 real on 60 reviews? A: It held across three passes each: Jev 54, 53, 52 and Opus 59, 58, 58. On first passes Opus matched five reviews Jev missed and missed none Jev matched. It's still one set of 60 synthetic reviews and a provisional key, so read it as a direction.

Q: Decision models don't hallucinate, so isn't the risk gone? A: A typed head can't invent a label, and on this set 31 of 50 decision-model groups gave identical answers across three passes, so one failure class is gone. The other stays: Perplexity Decider matched 54 of 60 in all nine runs with the same errors every time, so a wrong answer is the same wrong answer at scale, with no variance to warn you. Miss a harassment report or publish the wrong testimonial and the candidate tells X and Reddit how they were treated.

Replay of saved answers, 2026-10

A5, hard review

DEV-030: uncertain resolution.

Saved answer, first P0 pass	Sentiment	Follow-up	Serious concern	Testimonial	All four

0 of 7 decision models matched all four fields. Orange: differs from the key.

Replay of saved answers, 2026-10

A6, hard review

DEV-029: off-topic.

Saved answer, first P0 pass	Sentiment	Follow-up	Serious concern	Testimonial	All four

0 of 7 decision models matched all four fields. Orange: differs from the key.

Replay of saved answers, 2026-10

A7, hard review

DEV-006: vague recurrence.

Saved answer, first P0 pass	Sentiment	Follow-up	Serious concern	Testimonial	All four

0 of 7 decision models matched all four fields. Orange: differs from the key.

Replay of saved answers, 2026-10

A8, hard review

DEV-013: second-hand rumour.

Saved answer, first P0 pass	Sentiment	Follow-up	Serious concern	Testimonial	All four

0 of 7 decision models matched all four fields. Orange: differs from the key.

Replay of saved answers, 2026-10

A9, hard review

DEV-027: kindness during a logistics failure.

Saved answer, first P0 pass	Sentiment	Follow-up	Serious concern	Testimonial	All four

0 of 7 decision models matched all four fields. Orange: differs from the key.

Replay of saved answers, 2026-10

A10, hard review

DEV-059: serious concern without a complaint.

Saved answer, first P0 pass	Sentiment	Follow-up	Serious concern	Testimonial	All four

0 of 7 decision models matched all four fields. Orange: differs from the key.

Thank you!

Questions?

Adam Kovacs, Chief AI Strategist, AI Enablement Academy
Agentics Foundation Meetup, Budapest, 8 October 2026

This presentation needs JavaScript to load its source-bound numbers. Open the report.