autotrust/JEV-27B: fast, calibrated decisions and full reasoning from one open model

Agents make a lot of small decisions. Is this email phishing? Which team should get this ticket? Does this diff break a code rule? How urgent is this request, from 0 to 5? These questions don't need a long chain of thought. They need a fast, calibrated answer you can put a threshold on. TypeSafe AI's hosted TypeSafe Jev 1.13 made this kind of "System 1" decision model popular, but it is closed.

Today we are introducing autotrust/JEV-27B, an open-weights (Apache-2.0) model that answers these questions with calibrated probabilities. In our runs it averages 0.22 percentage points higher than TypeSafe Jev 1.13 on six public decision benchmarks. The same weights also serve an untouched Qwen3.8-27B for ordinary generation and step-by-step reasoning (System 2), from one vLLM engine.

TL;DR

• Six public decision benchmarks, one protocol: JEV-27B averages 84.07, TypeSafe Jev 1.13 83.85 in our runs of both. JEV-27B is higher on 4 of the 6.
• Close to Jev at the level of whole probability distributions: mean KL ≈ 0.017 on 25,376 held-out questions labelled with Jev's own outputs. An observer needs about 60 sampled decisions to gather one nat of evidence about which model produced them.
• System 2 is untouched: HumanEval 78.0 % before and after; all 164 completions are byte-identical to the base model.
• Fast: a median 137 ms per decision on one B200, against 238–301 ms measured independently for the hosted API.
• Runs on one H100: full bf16 precision, no quantization, the same decisions as on B300; about 100 decisions per second and 64 concurrent chats.
• Cheap to build: 108.9 M trained parameters (0.4 % of the backbone), ≈ 9.2 hours on one B200.

A note on names. TypeSafe Jev 1.13 is the hosted, closed model made by TypeSafe AI. It is the teacher whose published output distributions we trained on. autotrust/JEV-27B is an independent student built by AutoTrust AI from the Apache-2.0 corpus SargeDev/jev-distill-corpus-v3. We are not affiliated with TypeSafe AI and share no weights or code with it.

What a System 1 decision looks like

JEV-27B answers three kinds of typed questions about any text or JSON state:

Each answer takes one forward pass: no decoding, no JSON parsing, no prompt engineering. What you get back is a probability distribution, so you can act on it: auto-approve above 0.9, escalate below.

A few examples from fresh Hacker News and V2EX posts (23–25 September 2026) and community use cases. We wrote the expected answers by hand before running the model:

Across four groups of such examples, with 96 decisions that have a hand-written expected answer, it got 95 right. Treat these as illustrations, not a benchmark.

Two systems, one set of weights: the Blocks of Experts recipe

The usual way to build a decision model is to fine-tune a whole LLM into a classifier. That costs you the LLM. We took a different route, which we call Blocks of Experts: keep a strong pretrained model frozen as one expert block, and add a small, detachable block trained for one capability.

Why not merge the LoRA into the backbone and ship one model? We measured this on our first-generation 9B model: merging dropped HumanEval from 70.7 % to 61.6 %, a 9-point loss, even though prose perplexity barely moved (3.15 → 3.30). Keeping the blocks separate avoids the trade-off: on JEV-27B, HumanEval is 78.0 % (128/164) with and without System 1, and all 164 completions are byte-identical to the base model.

In vLLM we re-express the decision head as a LoRA on lm_head in which only the 24 verbalizer rows change. A decision is then just a one-token completion constrained to the option tokens, read back as log-probabilities. Ordinary chat requests go through the untouched lm_head. Both kinds of request can share a batch.

How close is it to TypeSafe Jev 1.13?

"Close" can mean different things, so we measured it three ways.

1. Head to head on six public decision benchmarks

We ran JEV-27B and the hosted TypeSafe Jev 1.13 in full on six benchmark groups (26 September 2026). The open baselines are the values reported in the NeoHorse-Jev-4B evaluation; we did not re-run them.

JEV-27B's mean is 0.22 percentage points higher. JEV-27B is ahead on JevBench, OpenJev text, Nimble and MASSIVE-en; Jev is ahead on Kev (by 1.77) and VitaminC (by 1.00). JevBench here is the public 231-example set with its family-macro score, not the JevBench v1.4.2 leaderboard.

2. At the level of probability distributions

Accuracy only checks the top answer. A distilled model should also reproduce the teacher's probabilities. On the 25,376 held-out questions (53 domains) whose labels are TypeSafe Jev 1.13's own output distributions:

It is calibrated without any post-hoc fix: expected calibration error 0.0009, fitted temperatures ≈ 1.00. Among the open Jev reproductions we could find, the JEV models are the only ones that publish this distribution-level measure.

3. On data nobody trained on

gazelle93/decision-models-under-pressure is an independent benchmark with human gold labels (CLINC-150, MTOP, GoEmotions, DBpedia). It publishes Jev's own results, and none of its data is in our training set.

That is 96–98 % of Jev's accuracy. At 16 options, JEV-27B reaches 96 % of Jev's accuracy. Of the models the benchmark's author ran, the closest are Laya and zero-shot DeBERTa-v3-large, at 90 %.

It copies Jev's mistakes too

A faithful student inherits the teacher's blind spots. On a poker spot where a solver always checks, Jev shoves with 0.62 in a published test; JEV-27B shoves with 0.63. Both change about 7 % of their 16-option answers when only the option order changes. If Jev is wrong about something, assume JEV-27B is wrong about it too.

Speed

On vLLM, System 1 runs 141 decisions/s offline, and System 2 generation is about 48× faster than the plain PyTorch path. One caveat: our timings are measured on the serving host, while the hosted-API numbers include the network and depend on the client's concurrency. For production capacity on H100 and B300, see Where can you run it?

Try it

1. Start one server for both systems:

Deployment version: vLLM 0.30.0.

hf download autotrust/JEV-27B --local-dir JEV-27B          # ~54 GB
vllm serve JEV-27B --served-model-name autotrust/JEV-27B \
    --dtype bfloat16 \
    --enable-lora --max-lora-rank 32 --lora-modules jev-decision=JEV-27B/adapter_vllm \
    --logprobs-mode processed_logprobs --max-model-len 4096 \
    --reasoning-parser qwen3 --enable-auto-tool-choice --tool-call-parser qwen3_coder

2. System 1: a typed decision (only requests plus two small JSON files from the repo):

import json, math, requests
from huggingface_hub import hf_hub_download

REPO, URL = "autotrust/JEV-27B", "http://localhost:8000"
dh = json.load(open(hf_hub_download(REPO, "adapter_vllm/decision_head.json")))  # bias + verbalizer token ids
T  = json.load(open(hf_hub_download(REPO, "calibration.json")))["per_kind"]       # per-kind temperatures

def decide(kind, state, question, options=None):
    options = {"noul": ["false", "true"], "score": [str(i) for i in range(6)]}.get(kind, options)
    lines = options if kind != "choice" else [f"{'ABCDEFGHIJKLMNOP'[i]}) {o}" for i, o in enumerate(options)]
    prompt = f"[kind] {kind}\n[state] {state}\n[question] {question}\n[options]\n" + "\n".join(lines) + "\n[decision]:"
    s = dh["slots"]["ranges"][kind][0]
    ids = dh["verbalizer_ids"][s : s + len(options)]
    r = requests.post(f"{URL}/v1/completions", json={
        "model": "jev-decision", "prompt": prompt, "max_tokens": 1, "temperature": 1.0,
        "logprobs": len(options), "allowed_token_ids": ids,
        "add_special_tokens": False, "return_tokens_as_token_ids": True}).json()
    lp = {int(k.split(":")[1]): v for k, v in r["choices"][0]["logprobs"]["top_logprobs"][0].items()}
    z = [(lp.get(t, -1e9) + dh["bias"][s + i]) / T[kind] for i, t in enumerate(ids)]
    e = [math.exp(x - max(z)) for x in z]
    return {o: x / sum(e) for o, x in zip(options, e)}

print(decide("choice", "SKU AX-330 stock at 8% of safety level; supplier late twice this quarter.",
             "Supplier response for this scenario.", ["issue_warning", "renegotiate", "dual_source", "maintain"]))
# ≈ {'issue_warning': 0.25, 'renegotiate': 0.13, 'dual_source': 0.62, 'maintain': 0.001}

3. System 2: ordinary chat on the same server. Send a normal /v1/chat/completions request with "model": "autotrust/JEV-27B". Thinking is enabled by default. To disable it, pass "chat_template_kwargs": {"enable_thinking": false}.

A natural pattern: escalate when unsure. Let System 1 answer when it is confident, and hand the rest to System 2 in the same engine. The model card has a ready-made solve() function. This is a usage pattern we have not benchmarked, so choose the threshold on your own data.

Where can you run it?

One H100 80 GB, at full bf16 precision. We moved our JEV-27B deployment from a B300 to a single H100 80 GB in the original bf16 precision, with no quantization. The decisions match those of the B300 deployment, so the smaller GPU costs capacity, not quality. Capacity in our deployment:

An H100 delivers 42–54 % of a B300's capacity, depending on the metric. (The benchmark and latency numbers elsewhere in this post were measured on a B200.)

Not yet tested, estimated from memory arithmetic only (the bf16 weights are 50.1 GiB):

• Two RTX 5090s: tensor parallelism (TP=2) should fit the bf16 model with about 3 GiB of cache per card. The PCIe link between the cards will limit generation speed.
• One RTX 5090 (32 GB): needs FP8 or 4-bit weights and is tight. Quantized quality would have to be re-measured.

For smaller GPUs there is autotrust/JEV-9B (18 GB in bf16), our first generation, built with the same recipe. It is 2.6× faster than JEV-27B, with mean KL ≈ 0.019 to Jev and 90 % of Jev's accuracy at 16 options. JEV-27B is the better choice for long option lists, unfamiliar task families, code-rule checks, fraud screening, and whenever System 2 matters.

Why the recipe is so efficient

1. The model starts out close. We initialise the decision head from the backbone's own lm_head rows for the answer tokens, so before any training it already is the pretrained model's zero-shot answer. That starting point agrees with the test targets on 58 % of choice questions, with a noul AUROC of 0.88. Distillation raises these to 90 % and 0.996.
2. Soft labels carry a lot of signal. Every training row is Jev's full probability distribution, not a single label. Distilling it with KL transfers Jev's uncertainty along with its answers, and calibration comes for free.
3. The backbone stays frozen. Pretrained knowledge is intact, which is where transfer to unseen tasks comes from. Going from 9B to 27B changes in-distribution KL only a little (0.019 → 0.017), but it more than halves KL on unseen task families (0.234 → 0.104).

What "on par" does and doesn't mean

• "Indistinguishable by KL" holds inside the training distribution. The 25,376 Jev-labelled test rows come from the same 53 domains as the training data, and no Jev-labelled out-of-distribution set exists yet. Outside those domains, our best evidence is the independent benchmark (96–98 % of Jev's accuracy).
• Parity is on the average, not on every task. JEV-27B trails on Kev and VitaminC by 1.77 and 1.00 percentage points, respectively.
• Most comparisons are our own runs. Community leaderboard submissions (the Jev Decision Index and JevBench v1.4.2) are still to come.
• Matching Jev is not the same as being right. Jev's known weaknesses (multi-hop reasoning, arithmetic, counting, adversarial inputs) are inherited. Use confidence gating, and keep humans in the loop for high-stakes decisions.
• The two systems share weights, not knowledge. System 2 does not know what System 1 decided and was not trained to agree with it.

What's next

• Submit JEV-27B to the Jev Decision Index and the JevBench v1.4.2 leaderboard for third-party verification.
• Build a Jev-labelled test set from domains outside the training data, to test fidelity beyond the 53 training domains.
• Verify and document deployment on two RTX 5090s.

Links

• Model: autotrust/JEV-27B · first generation: autotrust/JEV-9B
• Training data: SargeDev/jev-distill-corpus-v3 (Apache-2.0; its Open-Jev stream is CC0)
• Independent benchmark: gazelle93/decision-models-under-pressure
• Full evaluation details, training settings and limitations are in the model card.

Thanks to the authors of the distillation corpus, the Open-Jev project, the independent benchmark and the NeoHorse evaluation for making this comparison possible. We look forward to your feedback, and to seeing what you build with JEV-27B.

AutoTrust AI is not affiliated with TypeSafe AI. TypeSafe Jev 1.13 is a closed model by TypeSafe AI, used here as the distillation teacher.