Skip to content
JevDirectory.org
CommunityPractices & Patterns5 starsVerified 2026-09-22

jev-rerank-bench

Reranking benchmark that gave Jev, Cohere Rerank 4 Pro, ZeroEntropy zerank-2, and DeepSeek the same thirty BM25 candidates across eight English datasets, publishing saved responses, scoring code, and paired-bootstrap intervals. Jev's rubric scored 0.692 nDCG@10 against Cohere Pro's 0.691.

Category
Practices & Patterns
Published by
Community
Author
anessbelbati
Added
2026-09-22
Tagscommunitypythonsearchrerankingbenchmarks

Highlights

  • Gave Jev, Cohere, ZeroEntropy, and DeepSeek the same thirty BM25 candidates across eight English datasets.
  • Jev's 4-level rubric scored 0.692 nDCG@10, Cohere Rerank 4 Pro 0.691, with a 95% interval of -0.009 to +0.012.
  • Jev did better on negation: 71% of NevIR pairs right versus Cohere's 67%.
  • Jev's rubric cost $0.45 per 1,000 queries at 422 ms mean, against Cohere Pro's $2.51 and 844 ms.
  • Saves every ranking response, score, latency, usage, and cost; recorded API usage was about $61.

Watch out

MIT for the code, while dataset content keeps its source licenses (BEIR, BRIGHT, CodeSearchNet, NevIR, MIRACL). Reproduction needs TypeSafe, OpenRouter, and ZeroEntropy keys, and the paired intervals are exploratory without multiple-comparison adjustment.

More like this

A graded relevance evaluation of Jev as a reranker: 9,831 labelled pairs from 164 queries over a 33,047-item skills catalog, comparing Jev score reranks with BM25, bge-m3, and rank fusion.
Practices & Patterns#community#python#search
A measured search reranking run over 33,047 catalog entries, 164 real queries, and 9,831 labelled pairs, reporting how Jev compares with BM25 and bge-m3.
Practices & PatternsPost#community#x#search
Community
3.6kGitHub stars
An independent baseline that reads typed option probabilities straight from a frozen Qwen3.5-4B's logits in one forward pass, reproducing Jev's interface pattern with open models rather than Jev's undisclosed model or training.
Practices & Patterns#community#python#open-models
CommunitySemIf
Back to all resources

From the community

Posts from builders shipping with Jev right now.

Follow @typesafeai

A 706K-parameter form filler

cua open sourced a 706k param model that fills a whole form in one 50ms pass the llm agent doing the same form took 23 turns and 39.6 seconds the specialists are going to eat the generalists from the bottom

Cua
Cua
@trycua

1/ Introducing CUA-S1: a family of System One Models, small, specialized, and built for computer use. Today we're open-sourcing CUA-S1-FORMS, the first in the family: github.com/trycua/cua

Image
Reply

Navigating Neo4j with Jev

Jev 这个 waitlist 还是很给力的,昨天申请,今天就能用上。 给已经拿到 API、但还不知道怎么玩的人整理了一份 Awesome Jev,目前我能确认到的 Jev 项目基本都在这里: 1. jev-ultrafast Browser Use 做的高速浏览器 Agent。Jev Show more

Image
思维怪怪
思维怪怪
@0xLogicrw

前 OpenAI 研究员 Diogo Almeida 创办的 TypeSafe AI 推出新模型 Jev。它有点像一个能读懂自然语言的超级分类器,不生成文本,只返回选项、分数和概率,专门给软件做判断。 普通大模型需要一个 token 一个 token 往外生成,Jev 则可以并行给出多个结果。TypeSafe 还用新的 RLCD

Reply

Reranking 33,047 catalog entries

拿 Jev 做搜索重排,我先泼一盆冷水:单独用,它没打赢向量检索 TypeSafe 的 Jev 这阵子很火,一堆项目拿它做重排。我们在 Agent Skills Hub 的 33,047 条目录上认真测了一次,164 条中英文真实查询,9,831 对分级标注,整套只花了 2.6 美元 三个结论 01|单独重排,约等于没赢 Jev 重排 bge-m3 Show more

Jason Zhu
Jason Zhu
@GoSailGlobal

有美团、阿里的老哥嘛? 试试加一路召回、重排(离线、近实时实现),我觉得有奇效 他在文本理解上 跟之前机器学习、llm很不一样 还能自动打标签做特征

Reply

Six uses that stuck after 60 days

Security decisions that fit Jev

This made me rethink where AI actually fits into security engineering. For purely engineering work, forget about ChatGPT or Claude. TypeSafe AI just released Jev, and I think it’s going to change how we build AI into security workflows. Instead of asking an LLM to “investigate Show more

TypeSafe AI
TypeSafe AI
@typesafeai

we are officially out of stealth! join the frontier and get access to Jev on our website (link on profile)

Reply

A million judged questions