TypeSafe publishes evals, but the numbers that matter for a build decision are the ones measured on your task. A wave of independent benchmarks and field reports now covers enough ground to say what Jev is reliably good at — and where it breaks.
The broadest board
Jevals runs three boards of 300 questions each: PubMedQA for noul, Banking77 for choice, and HelpSteer2 for helpfulness scoring, for 31,500 scored decisions across seven models against human labels. Its headline finding is a price result as much as an accuracy one: on nouls, Jev is reported statistically tied with the best of six LLMs at 1/28 of the price. Jev scores 69.0 on PubMedQA, 67.8 on Banking77, and 9.2 on HelpSteer2 against GLM-5.3's 7.8.
Where question design dominates
The most useful independent result of the season is one judge call or twelve dimension scores. Across 25,174 Jev calls and 5,477 test rows for $1.43, a single direct question lost to a dimension model on every hard task: 0.837 versus 0.908 accuracy on Japanese NLI, 0.400 versus 0.911 on a 12-option ledger classification, and 1.5% versus 37.2% flag rates on hard-benign guardrail text. The same lesson shows up in amankumar's 16,000-call run, which found a whole-document read scoring 68.6% with no prompt fixing it, and in lindfors.no's early-access trial, where rewriting short questions with careful qualifiers cut agreement from 0.89 to 0.86 and raised calibration error from 0.040 to 0.116.
What the accuracy spread looks like
Independent runs converge on a familiar shape: strong on classification with a clean label space, weaker where categories overlap. The Jev Capability Atlas collects both sides, including a suite where a typo in an option produced a 0.90-confidence wrong answer that disappeared once the typo was fixed, and cited third-party work where an emotion task reported 0.819 mean confidence at 48% accuracy. amankumar has Jev leading Enron spam at 98.7, SST-2 at 95.7, and AG News at 91.3, while losing Banking77 at 76.0 against 81.7 for gpt-5.6-luna — the same benchmark where Jevals reports 67.8 for Jev.
Latency, cost, and tails
The operational numbers are consistent across reports. nearhere.events measured 0.58 s median and $0.043 per 1,000 decisions against $0.370 for Mistral Small 4 and $2.496 for Gemini Flash-Lite, matching 48 of 50 expected decisions and rejecting none of the 13 valid events. lindfors.no answered eleven questions across 24 documents for half a cent total at a 0.32 s median. And amankumar documents the tail: at 100 in-flight calls, a few per thousand took 10 to 35 seconds, so timeouts belong in the design.
Calibration behaves
Where it is measured carefully, the probabilities hold. lindfors.no found 0% yes in the 0.0–0.1 bin and 98% in the 0.9–1.0 bin across 192 judgments — slightly underconfident, which is the safe direction. Jevals and the Capability Atlas both note the failure mode to watch for: tasks where categories overlap can produce confident errors.
What to measure on your own task
- Fit first. Run a few hundred labelled rows before committing, as every one of these reports did.
- Design the question, not just the prompt. Dimension questions and careful qualifiers beat one broad question on hard tasks.
- Set thresholds from data. amankumar suggests putting the drop line at half the lowest yes-probability any known positive received, tuned on one fold and checked on another.
- Watch the tail, not the median. Budget for timeouts and retries at concurrency.
- Check calibration in bins, not on average. A mean confidence number hides the bins where it fails.
The practices category collects these studies alongside the pattern guides.
