After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x Show more
JevBench
Independent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.
- Category
- Practices & Patterns
- Published by
- Community
- Author
- fstandhartinger
- Added
- 2026-09-22
Tagscommunitypythonbenchmarksevaluationcalibration
Highlights
- Each complete entrant runs 534 frozen cases, including 220 hard decisions written by Claude Opus 5 and GPT-5.6 Sol and hashed before any system ran.
- The JevBench Score is the geometric mean of chance-corrected Intelligence, Calibration, Speed and Cost, 25% each.
- Jev 1.13.0 ranks second at 74.4 behind Hopper at 75.4 on the 66-row board, reading 950 input tokens per decision.
- Cost is dollars per 1,000 decisions, not tokens: Jev's $0.0399 comes from 950 tokens at $0.042 per million.
- Small models are option-order sensitive: reversing A and B order dropped open-alternative-jev from 72% to 21% on answer judging.
Watch out
MIT for the harness and 72 original public decisions; other artifacts keep their own licences. Not affiliated with TypeSafe AI, some self-hosted latency is adjusted by assumption, and held-out cases were exposed to the services measured.
More like this
67GitHub stars
An experiment comparing Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as evaluators of five frozen weather-agent runs, measuring pass-or-fail accuracy against human labels plus variance, cost, and latency.
Practices & Patterns#community#python#evaluation
Communityjev-as-a-judge
17GitHub stars
A probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Practices & Patterns#community#python#benchmarks
Communityjev-benchmarks
5GitHub stars
An independent calibration study of Jev over three public benchmarks and 900 rule-generated support tickets, publishing every raw Gateway response and the quantization limits of returned probabilities.
Practices & Patterns#community#python#calibration
Communityjev-ood-calibration
From the community
Posts from builders shipping with Jev right now.
The launch post
Trading bot, one decision per block
I built a trading bot with Jev! Jev decides if it should "buy" or "sell", given the price feed of an asset pair, and executes real trades. It uses Monad to place the orders on Kuru's on-chain order book in every 300ms block. Demo link → jev-trader.vercel.app
Classifying 1,500 real emails
this model is actually insane at email classification i tested it on 1500 of my own emails to see how well it works and I am blown away
@CompleteSkeptic
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x
Fast browser use with Stagehand
we built blazing fast computer/browser use with Jev + @Stagehanddev. this task cost $0.001 and executed at near instant speed (in a remote browser btw) the loop: observe the page, send a11y tree as state + actions as questions, Jev decides the next action, then Stagehand Show more
@CompleteSkeptic
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x
LLM-as-a-judge, sped up
Jev has spoken. It picked which model is AGI. 20–200x faster. 40–400x cheaper. This could make things like LLM-as-a-judge insanely fast and nearly free. (I tried a bunch of prompts and still didn’t burn through $0.10.)
@CompleteSkeptic
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x
Instant compaction with Jev
found the perfect use case for @typesafeai Jev: instant compaction in 2026, why is compaction still a summarization prompt? Jev can make it instant by scoring every tool call and dropping what’s irrelevant