Hacker News: fmaccomberhttps://news.ycombinator.com/user?id=fmaccomberHacker News RSShttps://hnrss.org/hnrss v2.1.1Tue, 09 Jun 2026 00:12:57 +0000<![CDATA[New comment by fmaccomber in "Benchmark accuracy retention is the wrong metric"]]>Whether model routing works is an empirical problem. Existing empirical efforts rely on benchmark accuracy retention, i.e. how does a model routing system score compared to a sophisticated model like Opus 4.7 on a complex task benchmark like Terminal-Bench 2.0.

However, that metric is completely divorced from what we care about. The better metric is utility retention, which takes into account task importance.

]]>Mon, 08 Jun 2026 15:57:05 +0000https://news.ycombinator.com/item?id=48447033fmaccomberhttps://news.ycombinator.com/item?id=48447033https://news.ycombinator.com/item?id=48447033<![CDATA[Benchmark accuracy retention is the wrong metric]]>Article URL: https://www.joshuahedtke.com/writing/benchmark-retention-is-not-utility-retention

Comments URL: https://news.ycombinator.com/item?id=48447032

Points: 1

# Comments: 1

]]>Mon, 08 Jun 2026 15:57:05 +0000https://www.joshuahedtke.com/writing/benchmark-retention-is-not-utility-retentionfmaccomberhttps://news.ycombinator.com/item?id=48447032https://news.ycombinator.com/item?id=48447032