How the test ran.
We wrote 60 short, fictional comments from job candidates and asked every model the same four questions about each one. AI wrote the comments and drafted the reference answers. A person checked all 60, and three labels are still disputed.
The four questions
- How was the experience? Positive, negative, mixed, neutral, or can't tell.
- Does someone need to follow up? Yes, no, or can't tell.
- Does it report a serious concern? Harassment, discrimination or a privacy breach, for example. A bad experience alone does not count. Can't tell is for a serious issue that is alleged but underspecified.
- Could it work as a testimonial? Yes, no, or can't tell.
The reference answers lean negative. Of the 60, 31 call the experience negative, 35 need a follow-up, 25 report a serious concern and 9 would work as a testimonial.
| Experience | Follow-up | Serious concern | Testimonial | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Positive | Negative | Mixed | Neutral | Can't tell | Yes | No | Can't tell | Yes | No | Can't tell | Yes | No | Can't tell | |
| Reference answers | 11 | 31 | 8 | 8 | 2 | 35 | 24 | 1 | 25 | 29 | 6 | 9 | 50 | 1 |
The labelling guide fixes each boundary, and the reference review records every label we argued about.
How a match is scored
A review counts as matched only when all four answers agree with the reference. The denominator is always 60. A missing, failed or malformed answer scores as a miss, so a run that saved 57 usable answers can match at most 57. Explore counts format and agreement separately.
Three prompts
P0 is the base task: the labelling guide's rules, then one line asking for the four answers as JSON. P1 adds classifier instructions with the label definitions. P2 adds step-by-step decision rules on top of P1. Decision models received the same instructions inside their native Choice questions. No model saw the reference or another model's answer.
The models
Decision models first, then general models. Route is how we reached each model, and effort lists the reasoning settings we tested. A dash means we ran it with no effort setting.
| Model | Maker | Type | Route | Effort tested |
|---|---|---|---|---|
| Jev 1.13 | TypeSafe | Decision | OpenRouter, TypeSafe API | – |
| Solar Decide | Upstage | Decision | OpenRouter | – |
| Clef | Cloudflare | Decision | OpenRouter, Cloudflare direct | – |
| Clef Flash | Cloudflare | Decision | OpenRouter, Cloudflare direct | – |
| Perplexity Decider 27B | Perplexity | Decision | OpenRouter | – |
| Luna Decisions | OpenAI | Decision | OpenRouter | – |
| Tev 1 4B | Together | Decision | OpenRouter | – |
| Liquid d1 | Liquid AI | Decision | OpenRouter | – |
| Kev 4B | Jared Palmer | Decision | OpenRouter (SiliconFlow) | – |
| Laya (English, typed and multilingual) | Convai Innovations | Decision | Local | – |
| OpenJev | Community | Decision | Local | thinking on and off |
| SemIf on Qwen3.5 4B | Community (TheoLeeCJ) | Decision | Local | – |
| AnyJev on Qwen3 0.6B | Nokia Applied Research | Decision | Local | – |
| Opus 5.5 | Anthropic | General | Claude subscription | low, medium, high, xhigh |
| Sonnet 5.5 | Anthropic | General | Claude subscription | low, medium, high, xhigh |
| Opus 5 | Anthropic | General | Claude subscription | low, medium, high, xhigh, max |
| Sonnet 5 | Anthropic | General | Claude subscription | low, medium, high, xhigh, max |
| Fable 5.1 | Anthropic | General | Claude subscription | low, medium, high, xhigh, max |
| Haiku 4.5 | Anthropic | General | Claude subscription | – |
| GPT-6 Luna, Sol and Astra | OpenAI | General | Codex subscription | low, medium, high, xhigh |
| GPT-5.6 Luna, Sol and Terra | OpenAI | General | Codex subscription | low, medium, high, xhigh |
| Gemini 3.1 Pro | General | Gemini subscription, OpenRouter (Google AI Studio) | low, high | |
| Gemini 3.6 to 3.8 Flash | General | Gemini subscription, OpenRouter (Google AI Studio) | low, medium, high | |
| Gemma 4 E2B and E4B | General | Local | thinking on and off | |
| Gemma 4 26B and 31B | General | OpenRouter | thinking on and off | |
| Qwen3 0.6B, Qwen3 1.7B and Qwen3.5 4B | Alibaba | General | Local | thinking on and off |
| Qwen3 8B | Alibaba | General | Local, OpenRouter | thinking on and off |
| Qwen3.6 35B A3B | Alibaba | General | OpenRouter | thinking on and off |
| Qwen3.8 27B | Alibaba | General | OpenRouter, local | thinking off, low, medium, xhigh |
| DeepSeek V4.1 Flash | DeepSeek | General | OpenRouter | thinking off, low, high |
| Mistral Small 3.2 24B and Mistral Small 2603 | Mistral AI | General | OpenRouter | – |
How many passes, and which ones broke
The site holds 1,004 saved passes. 252 configurations ran each prompt three times with fresh requests: 50 decision-model configurations and 202 general ones. In 241 of them every pass returned at least 50 usable answers. The other 11 contain a partial or failed pass: AnyJev generated, Qwen 0.6B on the SDK, SemIf generated P1, OpenJev generated-on P2 pass 3 at 47 usable answers, and Jev's P2 pass 2, which stopped after 17 answers on a rate-limit error.
An interrupted pass has no score of its own. Where a continuation finished the remaining reviews, Explore shows the combined result and marks it.
Three labels are disputed.
A person checked all 60 reference answers. Three stayed open after a second review.
- DEV-006
Same as last time. That thing happened again and it still isn't sorted.
The reference says can't tell on serious concern. The review proposes no, because "that thing" names no serious conduct.- DEV-013
A friend told me they heard this firm asks inappropriate questions. My own interview was straightforward, and I have no complaint about it.
The reference says neutral. Positive is defensible, and the guide does not fix that boundary.- DEV-030
The accessibility issue from the assessment has been dealt with, I think. Someone said it was fixed, but I don't know whether that means I get another assessment or just that they've closed the ticket.
The reference says neutral, and the sentiment is a judgment call.
Every published score uses the original labels. Scores under the alternative labels are on Explore.
18 decision-model releases in three weeks
TypeSafe launched Jev on 15 September 2026. By 7 October model makers, gateways and the open-source community had shipped everything below. We list only dates we could confirm from the maker or the release itself.
| Date | Maker | Release | How it shipped |
|---|---|---|---|
| 2026-09-15 | TypeSafe | Jev 1.13 | Hosted API |
| 2026-09-18 | Vercel | Jev on AI Gateway | API or gateway |
| 2026-09-18 | Community (TheoLeeCJ) | SemIf (ex-OpenJev) | Tooling |
| 2026-09-22 | Community (razorback16) | openjev vLLM serving | Tooling |
| 2026-09-22 | Nokia Applied Research | AnyJev | Tooling |
| 2026-09-22 | Upstage | Solar Decide | Hosted API |
| 2026-09-24 | Jared Palmer | Kev 1.0 | Open weights |
| 2026-09-24 | Respan | Span-01 | Hosted API |
| 2026-09-24 | Bespoke Labs | Nimble 9B | Open weights |
| 2026-09-29 | Ollama | Ollama 0.35 decision support | Tooling |
| 2026-09-30 | Jared Palmer | Kev-9B v2 | Open weights |
| 2026-10-01 | Cloudflare | Clef and Clef-flash | Open weights |
| 2026-10-01 | AWS Strands Labs | Strands Decider 2B | Open weights |
| 2026-10-01 | Vercel | AI Gateway /v1/evaluate | API or gateway |
| 2026-10-05 | Liquid AI | d1 with vision | Hosted API |
| 2026-10-06 | OpenAI | Decisions API on gpt-6-luna | API or gateway |
| 2026-10-07 | Liquid AI | d1-3B and d1-omni-600M | Open weights |
| 2026-10-07 | Perplexity | Decider V1.1 27B | Hosted API |
The whole study cost $12.65 on OpenRouter.
OpenRouter billed $12.6537 over 15,135 saved generations, and that total includes the $1.2013 the seven decision models cost over their series of nine passes each. Failed calls may add up to $3.8837 more.
A Jev pass through OpenRouter cost $0.0059 for 140,260 input tokens, and it matched 54, 53 and 52 over three passes. On TypeSafe's direct API, Jev scored 54 on its first pass, and that route sends no bill.
Charges are the provider's known charge for returned answers; smoke tests and unknown charges are excluded. † marks an API-rate equivalent. An API-rate equivalent is what the same tokens cost at the published price, for routes that send no bill: the Claude and Codex subscriptions, Cloudflare's direct route and TypeSafe's direct API.
Corrections
classification-bench
We rebuilt the scripts that ran this test as classification-bench, one runner that does the same on your own cases, labels and questions and keeps every request, answer, failure and charge. classification-bench is not public yet. Ask Adam for access.
Terms used on this site
- Pass
- One model, one prompt, one pass over all 60 reviews.
- Configuration
- One model with its route, effort and batch size held fixed.
- Decision model
- A model built to return one label from a fixed list in a single pass, through a native Choice interface.
- Tokens
- The pieces of text a provider bills for. Reasoning adds output tokens even when the answer is one word.
- Request time
- Measured by our client, network included. Providers did not report inference time, so the site makes no speed claim.
Sources and downloads
- data.json every original run, its answers and its cost
- extended-run-catalog-v1.json the repeat and continuation passes
- supplemental-decision-runs-v1.json the seven decision models, every pass
- jev-native-prompt-findings.json Jev's nine passes on OpenRouter
- disputed-reviews-v1.json all 60 reviews with the seven decision models' answers
- deep-insights-v1.json repeatability, calibration, the agreement rule and the corrections
- native-agreement-policy-v1.json all 21 decision-model pairs
- timeline.json the launch dates above, each with the link where it was announced
- The repository the reviews, the reference, every saved answer and the scripts