Skip to content
JevDirectory.org
CommunityPractices & Patterns0 starsVerified 2026-09-22

jev-orderby-bench

Independent measurement of whether ORDER BY over a Jev probability is defensible, pre-registering gates on pairwise inversion, Score ordinality, calibration, and negation and paraphrase invariants.

Category
Practices & Patterns
Published by
Community
Author
yodablocks
Added
2026-09-22
Tagscommunitypythonbenchmarkscalibrationduckdb

Highlights

  • jev-1.13.0 passed all six pre-registered conditions on 360 human-labeled rows: boolean inversion 0.036 and Score inversion 0.143.
  • On Amazon ESCI's 306 graded pairs over 30 hard queries it failed four of six conditions, with jev_bool inversion 0.255.
  • Request shape changes the numbers: 40-row batching failed the ranking gate at 0.171 that one row per request passed.
  • Probabilities return at two decimals: 360 rows gave 45 distinct values and 53 rows tie at 0.99, so LIMIT k cuts inside a tie.
  • The full 360-row run cost about $0.013 at $0.042 per million input tokens, and cached responses allow free --analyze-only replays.

Quickstart

bash
python3 harness/run_calibration.py
python3 harness/run_calibration.py --analyze-only

Watch out

MIT-licensed for the harness only. Uses Python and a TypeSafe API key for fresh runs; the ESCI probe has one seed and 30 queries, and 20 Newsgroups carries no explicit license.

More like this

3.6kGitHub stars
An independent baseline that reads typed option probabilities straight from a frozen Qwen3.5-4B's logits in one forward pass, reproducing Jev's interface pattern with open models rather than Jev's undisclosed model or training.
Practices & Patterns#community#python#open-models
CommunitySemIf
3.2kGitHub stars
Apache-licensed, locally runnable Jev-style decision models (0.8B, 4B, 9B on Qwen3.5) with released weights, training code, a System One-compatible server, frozen eval suites, and a playground.
Practices & Patterns#community#python#open-models
Communitykev
296GitHub stars
An independent open reproduction of the System One model class: Qwen3.5-based 2B and 35B mixture-of-experts models that return typed Choice, Score, and Noul probabilities in one forward pass, with nothing distilled from Jev.
Practices & Patterns#community#python#open-models
Communitydecider
Back to all resources

From the community

Posts from builders shipping with Jev right now.

Follow @typesafeai

The open System One roundup

Jev 发布没几天,开源社区已经开始疯狂复刻了🔥 最值得推荐的五个模型: 1、Laya 421M:原生决策模型,支持 Mac 2、Decider-2B:最像 Jev,基于 Qwen3.5 3、NanoJev 0.6B:专门的 Decision Head 4、Reflex:Qwen3.5 + Direct Logits 5、System-One 4B:专门做概率校准 Show more

小墨同学
小墨同学
@xiaomovps

Jev 刚发布没几天,开源社区就出现了同款🔥 Decider-2B模型,是基于 Qwen3.5-2B 做了特殊调整 它和 Jev 模型是一样的 只做选择 评分和判断 不是文本类的 LLM 模型 但两者还是有几个明显区别: 1、模型 Jev:闭源 System One Model Decider:Qwen3.5-2B,约 1.9B 参数,Apache 2.0 开源 2、价格

Image
Reply

The launch post

Trading bot, one decision per block

Classifying 1,500 real emails

this model is actually insane at email classification i tested it on 1500 of my own emails to see how well it works and I am blown away

Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply

Fast browser use with Stagehand

we built blazing fast computer/browser use with Jev + @Stagehanddev. this task cost $0.001 and executed at near instant speed (in a remote browser btw) the loop: observe the page, send a11y tree as state + actions as questions, Jev decides the next action, then Stagehand Show more

Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply

LLM-as-a-judge, sped up

Jev has spoken. It picked which model is AGI. 20–200x faster. 40–400x cheaper. This could make things like LLM-as-a-judge insanely fast and nearly free. (I tried a bunch of prompts and still didn’t burn through $0.10.)

Image
Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply