Skip to content
JevDirectory.org
CommunityPractices & Patterns91 starsVerified 2026-09-22

Jev Arena

A comparison arena that runs the same batch of review comments through Jev and DeepSeek, showing processing time, cost, and per-label results with CSV/Excel import, replay, and offline reports.

Category
Practices & Patterns
Published by
Community
Author
NanmiCoder
Added
2026-09-22
Tagscommunityjavascriptbenchmarksevaluationdataopenrouter

Highlights

  • On 10,000 reviews Jev took 203.2 s and about $0.84 versus DeepSeek Flash at 823.5 s and an estimated $1.50.
  • Under GPT-6 Astra review, all three labels (relevance, sentiment, intent) were correct together for 62.69% of Jev rows versus 67.26% for DeepSeek.
  • The README states the accuracy check used an AI reference rather than a human gold standard, and that strict first-choice scoring gives 50.58% and 55.45%.
  • Runs need Node.js 22+ and OpenRouter plus DeepSeek keys; replays call no models and reports can be generated offline.

Quickstart

bash
git clone https://github.com/NanmiCoder/jev-arena.git
cd jev-arena
npm ci
npm start

Watch out

MIT-licensed for the code, while demo comments belong to their original authors and platforms. Needs Node.js 22+ and OpenRouter and DeepSeek keys; results describe this one dataset and configuration.

More like this

104GitHub stars
A personal-assistant agent with 100 mocked tools that measures how many steps a task takes when the LLM picks the tool versus when Jev picks it before every model step.
Practices & Patterns#community#typescript#agents
208GitHub stars
A self-hosted implementation of TypeSafe's Jev System One API powered by the 400M-parameter GLiFormer encoder: it serves choice, score, and noul and drops into the official typesafe-sdk via TYPESAFE_BASE_URL, but trails Jev on reasoning-heavy tasks.
Practices & Patterns#community#python#open-models
Communityjeff
74GitHub stars
Independent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.
Practices & Patterns#community#python#benchmarks
Communityjevbench
Back to all resources

From the community

Posts from builders shipping with Jev right now.

Follow @typesafeai

Full Jev video tutorial

The case against Jev-scored compaction

This is a terrible compaction strategy that fundamentally doesn't understand how compaction and context management work. Seems like a lot of people are confused so let's break this down. 1. Compaction isn't a filter The role of compaction is to clean up history to keep the Show more

tamara
tamara
@tamarajtran

found the perfect use case for @typesafeai Jev: instant compaction in 2026, why is compaction still a summarization prompt? Jev can make it instant by scoring every tool call and dropping what’s irrelevant

Reply

Classifying rows in DuckDB

A playable 16-judgment demo

AI multiple choice, not essay writing

Screening agent actions with Jev

Tested TypeSafe’s Jev (no-text, probability-only model) as an AI agent safety monitor. Checking each action first worked well caught most attacks with almost no false blocks, and much faster than Gemini.

Image
Image
Image
Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply