Skip to content
JevDirectory.org
CommunityPractices & Patterns2 starsVerified 2026-09-22

Janus

An independent calibration study of Jev against a frontier fallback on Banking77 and Web of Science, packaged as the janus-decide tool that measures a routing threshold on your own data instead of shipping a default one.

Category
Practices & Patterns
Published by
Community
Author
FirasSX914
Added
2026-09-22
Tagscommunitypythonroutingcalibrationevaluationbenchmarks

Highlights

  • On Banking77, routing at threshold 0.67 reaches 80.2% accuracy for $0.1033 and a 302 ms median, against $0.2207 and 2269 ms for the fallback alone.
  • On Web of Science no threshold beats the better single model, so the verdict is DO NOT ROUTE.
  • The protocol was frozen before any result; the raw JSONL and figures are committed, with 500 labelled examples per dataset.
  • No routing parameter transferred between the two datasets: the optimal threshold, the sign of the accuracy gap, and whether routing paid for itself all changed.
  • It measures from a labelled dataset or from a log of past decisions, and refuses to print ground truth for a log.

Quickstart

bash
pip install "janus-decide[typesafe,deepseek]"
janus measure \
  --dataset mydata.jsonl --labels mylabels.json \
  --primary typesafe:jev-latest \
  --fallback deepseek:deepseek-v4-pro \
  --out janus.json

Watch out

MIT-licensed; the Web of Science labels come from publication metadata rather than per-document annotation, so part of the error measured there is label ambiguity, and the bare package depends only on numpy with each backend as an extra.

More like this

74GitHub stars
Independent cross-model benchmark for Jev-class decision models, running 534 frozen cases per complete entrant with scoring code and a four-axis score of accuracy, calibration, latency and cost.
Practices & Patterns#community#python#benchmarks
Communityjevbench
An experiment comparing Jev with GPT-5.6 Luna, GPT-5.6 Terra, and Claude Sonnet 4.6 as evaluators of five frozen weather-agent runs, measuring pass-or-fail accuracy against human labels plus variance, cost, and latency.
Practices & Patterns#community#python#evaluation
17GitHub stars
A probability-aware evaluation harness that compares TypeSafe Jev with GLiNER2.5 on zero-shot single-label text classification, measuring calibration, coverage at a fixed error budget, latency, and token cost.
Practices & Patterns#community#python#benchmarks
Back to all resources

From the community

Posts from builders shipping with Jev right now.

Follow @typesafeai

Vercel's fx safety reviewer, 18x faster

We're seeing extraordinary results from @typesafeai. Default mode in 𝚏𝚡 is auto, with a safety reviewer analyzing every command. That reviewer runs on GPT Luna today. Jev is up to 18x faster (p95) *and* more accurate. It's coming to @vercel AI Gateway and likely new default.

Pranit
Pranit
Vercel
@fazxes

We benchmarked fx auto mode (safety) classifier with @typesafeai's Jev. tl;dr: ~5-18x faster and more accurate than 𝚐𝚙𝚝-𝟻.𝟼-𝚕𝚞𝚗𝚊, our current top choice

Image
Reply

Jev lands on OpenRouter

700 leads scored for $0.09

Beating Gemini Flash Lite on an eval

Browser Use Ultrafast, powered by Jev

A really smart switch statement

hype-free explanation of jev: jev does not replace gpt / claude jev is just a *really* smart switch statement like if 2016 ml classifiers got 2026 levels of intelligence it's a new* type of tool that will make a lot of workloads insanely fast, cheap, and accurate * = and by Show more

Diogo Almeida
Diogo Almeida
TypeSafe AI
@CompleteSkeptic

After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI? I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev • 20-200x faster • 40-400x

Reply