Hacker News: florianstandhar

New comment by florianstandhar in "Show HN: JevBench, a reproducible benchmark for typed decision models"

florianstandhar — Wed, 23 Sep 2026 11:32:15 +0000

Happy to — please open an issue at https://github.com/fstandhartinger/jevbench/issues with an endpoint or runnable code, the model and licence, and whether it was trained on the public items. Every entrant runs through the same harness, including the sealed set. I definitely prefer open models I can run locally, because sending our private held-out set of tasks to an external API produces some headaches on ur side and basically means we have to rotate the test set a lot, to avoid contaimination. We can call API hosted models though, we'll just flag the leaderboard entries appropriately.

New comment by florianstandhar in "Show HN: JevBench, a reproducible benchmark for typed decision models"

florianstandhar — Wed, 23 Sep 2026 11:29:57 +0000

Thanks! Yeah, the code is open: https://github.com/fstandhartinger/who-is-right. Email classifying is definitely a good Jev usecase, I agree

New comment by florianstandhar in "Show HN: JevBench, a reproducible benchmark for typed decision models"

florianstandhar — Wed, 23 Sep 2026 11:26:11 +0000

Good call... BART-large-MNLI is the classic zero-shot baseline and it should be there. Adding it to the next run.

New comment by florianstandhar in "OpenAI is about to eat Jev's lunch – Arcturus Labs"

florianstandhar — Tue, 22 Sep 2026 15:09:01 +0000

maybe open source even eats Jevs lunch first

see here: https://news.ycombinator.com/item?id=49800574

Show HN: JevBench, a reproducible benchmark for typed decision models

florianstandhar — Tue, 22 Sep 2026 13:01:03 +0000

Hi HN! I built JevBench because Jev kicks ass, and the world deserves to know how the serious open source and fake lookalike projects really perform in comparison.

Jev-class models return bounded choices and probabilities instead of text, and are disruptively faster and cheaper than LLMs, while being similarly intelligent on the text input they operate on.

JevBench allows looking at accuracy, latency and price all at once, in a weighted way - you can even configure the weighting.

A full run asks 534 English decisions. The v1.3 score combines chance-corrected Intelligence, Calibration, Speed and Cost.

Leaderboard right now:

  #1 - Jev            74.4
  #2 - SemIf          73.1
  #3 - djev           73.0
  #4 - Winnow-12B Q8  71.2
  #5 reflex 4B        70.3.

MIT harness, public items, frozen artifacts, scoring code and public per-task outcomes:

https://github.com/fstandhartinger/jevbench

Two no-signup demos:

https://who-is-right.app.mintapis.com

https://is-it-ai-slop.app.mintapis.com

Limitations: English-only; latency from one German server; local/demo latency gets a disclosed ×2 adjustment (+150 ms on my servers) which is an informed assumption; held-out prompts still reach evaluated services; ~1-point gaps can be noise.

Wdyt?

Comments URL: https://news.ycombinator.com/item?id=49800574

Points: 139

# Comments: 36