when a designer gets access to Jev
Jevals — Independent Jev Benchmark for Models That Decide
An independent benchmark of hosted Jev against six LLMs on PubMedQA, Banking77, and HelpSteer2, scoring decisions against human labels and reporting per-board results for each model.
- Category
- Practices & Patterns
- Published by
- Community
- Author
- —
- Added
- 2026-09-22
Tagscommunitybenchmarksevaluationmodels
Highlights
- Three boards cover 300 questions each: PubMedQA noul, Banking77 choice, and HelpSteer2 helpfulness scoring.
- 31,500 scored decisions across 3 question types and 7 models, measured against human labels.
- On nouls, Jev is reported statistically tied with the best of six LLMs at 1/28 of the price.
- Jev scores 69.0 on PubMedQA noul, 67.8 on Banking77 choice, and 9.2 on HelpSteer2 against GLM-5.3's 7.8.
- Decision Score sets 100 as perfect and 0 as guessing label base rates; every row is run at list price.
Watch out
Independent and unaffiliated with TypeSafe, though several advertised model rows such as GPT-6 Astra are still listed as coming next rather than scored.
More like this
208GitHub stars
A self-hosted implementation of TypeSafe's Jev System One API powered by the 400M-parameter GLiFormer encoder: it serves choice, score, and noul and drops into the official typesafe-sdk via TYPESAFE_BASE_URL, but trails Jev on reasoning-heavy tasks.
Practices & Patterns#community#python#open-models
Communityjeff
104GitHub stars
A personal-assistant agent with 100 mocked tools that measures how many steps a task takes when the LLM picks the tool versus when Jev picks it before every model step.
Practices & Patterns#community#typescript#agents
Communityjev-eval-agent
From the community
Posts from builders shipping with Jev right now.
When a designer gets Jev
Full Jev video tutorial
Full Jev Tutorial What it is, how you can build with it and what new applications it can unlock → 0:00 Intro → 0:34 Jev explained → 4:06 API setup → 5:59 Demo 1: Voice-controlled browser → 11:33 Demo 2: AI memory → 17:27 Demo 3: YouTube predictor
The case against Jev-scored compaction
This is a terrible compaction strategy that fundamentally doesn't understand how compaction and context management work. Seems like a lot of people are confused so let's break this down. 1. Compaction isn't a filter The role of compaction is to clean up history to keep the Show more
found the perfect use case for @typesafeai Jev: instant compaction in 2026, why is compaction still a summarization prompt? Jev can make it instant by scoring every tool call and dropping what’s irrelevant
Classifying rows in DuckDB
I made a DuckDB extension where you can use @typesafeai 's Jev to do quick classification of rows in any csv/parquet file or duckdb table about 10sec for 1k rows ~ better than using an LLM, way more ergonomic than a classifier game-changing for data analysis!
A playable 16-judgment demo
typesafe's jev is fun! live demo you can play with: typesafe-demo.val.run
AI multiple choice, not essay writing
WTF is Jev by @typesafeai? Here’s the tl;dr ELI5: Think AI multiple choice, not AI essay writing. It doesn’t chat. It makes decisions your software can act on: “Spam or not?” “Which tool should this agent use?” “Does this need a human?” The exciting part: roughly 200x faster Show more