> Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.

And the savings listed are coming from a benchmark harness that implements different OSS bugs one time with and one without lumen - in those cases the % saved are reproducible (caveat: it was on older models, Opus 4.6 I believe).

Also I explain WHY it saves tokens - because the model doesn’t have to brute force different terms until it finds the match it needs, but uses semantic „distance“ so the embedding does it for the model.

What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly. Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.

Why does it work you may ask? Well, LLMs basically brute force words/phrases and pipe that into find/grep/pgrep/whatever (or as recently discussed here write a python script for it - https://news.ycombinator.com/item?id=49654229). Semantic search looks for similarities so you have to do less brute forcing. Comes of course at the cost of indexing everything first.

You can find the project here: https://github.com/ory/lumen

I can’t speak for Vercel‘s goals or pricing - but Ory is evidently still open source while many others went other routes!

Open source development needs to be paid by someone - most of the time people complaining about paying for software are working themselves (for money!) in some company making huge bucks, or looking up to "successful (as in money) tech leaders".

For Ory, B2B login is a good value differentiator, because it's required by companies selling to other companies meaning they can spend some money on licenses to further develop software.

Ory powers the largest technology providers, and super small solo projects. It's robust, stable, Apache2 licensed. It's the best CIAM tech out there that's free (!!).

In the end, everyone is entitled to their opinion but the "open source can't make money" train is honestly a bottom tier opinion and I'm tired of reading it on HN, probably written by people making $100K+ a year for writing software and using open source daily (without paying a dime).

It's like the people complaining that Wikipedia is collecting too many donations, while they cheer on Apple or Anthropic or whoever raking in billions of dollars.

Somehow, only if it's open source / non profit it's bad to make money. If it's proprietary nobody gives a damn. Says a lot about society.

If you serve 900m weekly active users, you need this type of distributed database architecture that is expensive to run. But at that point the cost of running it is a fraction of overall infra spend. No start up really needs this level of scale, only Enterprises (hence it‘s gated). Making Cockroach work is more work than just wiring up the SQL, you actually need to deal with it like dynamodb under the hood and use primary keys efficiently, avoid hotspots, and all that jazz.

Most companies (like Cloudflare!) do just fine with Postgres and one of our services. Ory Hydra is written in Go, doesn’t need JVM, very little RAM, doesn’t need caches or start up time due to cold starts. The architecture is different and that makes it cheap and fast to run. From the blog post - they run Hydra on 0.6 vCPU and 200MB of RAM. That’s probably as cheap as it gets!

It‘s a different tool for a different problem than KeyCloak - both have their place.

Yes we have an commercial version because how else can one finance world class open source powering the biggest software names on the planet? It‘s a good thing that Ory has a business model that works, not a bad thing. And by the way, IBM finds ways to charge you for KeyCloak too ;)

For API Keys we just launched Ory Talos (https://github.com/ory/talos) - a perfect alternative for when OAuth2 is too much for the use case.

There are use cases and security concerns that legitimize using OAuth2 - with specs like DPoP you can make these flows more secure. In my view the use cases presented here is a good one for OAuth2, but it certainly doesn’t make sense everywhere - complexity makes system harder to secure.

If anyone here is interested in providing their own oauth, IAM, rebac permissions, API keys, agent security - check out our open source & commercial products at https://github.com/ory and https://www.ory.com/

For AI Agents we have added token derivation to Ory Talos which allows you to exchange a static API key for a ephemeral, short lived, and restricted token. It can be both a JWT and a Macaroon (super interesting for caveats)!

However this would require GitHub to use Ory Talos and it‘s not a solution for third party credentials really.

So your project solves that need quite nicely, and I‘ll check it out in more detail today :)

If you have any questions, please shoot :)

Anthropic is not incentivized to reduce token use, only to increase it, which is what we are seeing with Opus 4.6 and now they are putting the screws on

Comments URL: https://news.ycombinator.com/item?id=47620090

Points: 7

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=47618822

Points: 2

# Comments: 0