Hacker News: lorey

New comment by lorey in "Pareto Front"

lorey — Thu, 06 Aug 2026 12:58:16 +0000

Found this to display the optimal LLM choice while building evalry. It's such a useful tool, not only for thinking about it, but for visualization, too.

Example: Which LLM gives me the best ELI5 explanations for a given price. https://evalry.com/benchmarks/explain-like-i-m-5-321

New comment by lorey in "Interview with Boris Cherny [video]"

lorey — Wed, 29 Jul 2026 10:05:16 +0000

What I read from that is that there's a chance claude gets worse using a claude.md and that's a usability issue on their side.

New comment by lorey in "Hey Nico, you didn't vibe code your data room but stole it from Papermark"

lorey — Thu, 25 Jun 2026 15:45:12 +0000

Their response:

> The team that made dataroom has stated that they did not use any of papermark’s code and that dataroom was made from scratch with inspiration from existing document sharing softwares, and that this post’s allegations of us stealing code are false. [...]

The screenshots clearly show they copied whole pages verbatim, both design and texts. The founder, Nico Laqua, basically responding with "we didn't copy _code_" and not taking any responsibility says a lot about his and his company's moral code. It might not be enough to get sued. That doesn't make it right.

https://x.com/nico_laqua/status/2070158170937581951

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Thu, 22 Jan 2026 13:00:40 +0000

That is not what the article argues.

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Thu, 22 Jan 2026 13:00:00 +0000

Haha, very true. Exactly as described in the article.

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Thu, 22 Jan 2026 12:58:59 +0000

This is true with one caveat.

In most cases, e.g. with regular ML, evals are easy and not doing them results in inferior performance. With LLMs, especially frontier LLMs, this has flipped. Not doing them will likely give you alight performance and at the same time proper benchmarks are tricky to implement.

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Thu, 22 Jan 2026 12:55:01 +0000

This is a very good point. When I came in, the founder did a lot of evaluation based on a few prompts and with manual evaluation, exactly as described. Showing the results helped me underline the fact that "works for me" (tm) does not match the actual data in many cases.

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Wed, 21 Jan 2026 21:35:56 +0000

Doesn't this depend a lot on private vs company usage? There's no way I could spend more than a few hundreds alone, but when you run prompts on 1M entities in some corporate use case, this will incur costs, no matter how cheap the model usage.

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Wed, 21 Jan 2026 18:48:39 +0000

It's not you, it's the HN hug of death. There's so much load on the server, I'm barely able to download the redis image I need for caching...

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Wed, 21 Jan 2026 18:44:17 +0000

Thanks. Will take a look.

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Wed, 21 Jan 2026 18:39:19 +0000

Depends on your remaining budget ;)

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Wed, 21 Jan 2026 18:24:08 +0000

I've skipped that in the article, but absolutely!

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Wed, 21 Jan 2026 18:06:42 +0000

Fixed, thanks. Not a native speaker.

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Tue, 20 Jan 2026 22:16:54 +0000

That's interesting. Similarly, we found out that for very simple tasks the older Haiku models are interesting as they're cheaper than the latest Haiku models and often perform equally well.

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Tue, 20 Jan 2026 22:02:40 +0000

Pushed a fix. Could you check, please?

Any resources you can recommend to properly tackle this going forward?

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Tue, 20 Jan 2026 21:43:48 +0000

Will fix, thanks :)

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Tue, 20 Jan 2026 21:36:40 +0000

Totally agree with your point. While I can't say specifically, it's a traditional (German) business he's doing vertically integrated with AI. Customer support is really bad in this traditional niche and by leveraging AI on top of doing the support himself 24/7, he was able to make it his competitive edge.

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Tue, 20 Jan 2026 21:34:06 +0000

This went straight to prod, even earlier than I'd opted for. What do you mean?

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Tue, 20 Jan 2026 21:32:15 +0000

Appreciate the feedback, will work on that.

New comment by lorey in "Without benchmarking LLMs, you're likely overpaying"

lorey — Tue, 20 Jan 2026 21:30:54 +0000

You're right. We did a few use cases and I have to admit that while customer service is easiest to explain, its where I'd also not choose the cheapest model for said reasons.