We are exposing the routing decision information already, scroll over the response in the Dashboard and it'll tell you what models it considered and what it chose.

We think the results are a work in progress. Routing is convincing for quality for the following reason. Take your favorite benchmark, and on each task, run both the top GPT and top Claude model, and call the task a success if at least one of them succeeds. You'll find the accuracy is SIGNIFICANTLY than just max{GPT, Claude}. And the more models you add the better the effect.

Hence, if we can make the router approach this "oracle," the quality could be really good (but also, with turn-by-turn routing, there's no reason we couldn't exceed the oracle)

We will likely do some ablations. Our approach was SOTA back when we first applied to YC with it, we just neglected to post it (we had only done stuff with tiny Qwen models at that point).

We also think there are tons of people working on "context management"--e.g. retrieval systems, prompt compression, log compression, etc. We want to work harder on the "decode" side as we think there are lots of savings to be made

A turn here is a tool call/user input, anything that causes the model to get some new input. We're working on adding Minimax M3 and other models. We think that people have some intuitions about which models are good when--we seek to quantify them scientifically.

The largest models are very well known for having large reasoning traces. The thing that cracks down on overthinking is reasoning-effort training, which is largely a function of how the model is trained (certain RL objectives/teachers) rather than size of the model. The Kimi K3 technical report has some excellent work on this: https://arxiv.org/pdf/2607.

That being said, whatever models we select to race at any point will ALL have their cache updated, so if they're chosen on the next race they'll be a lot cheaper.

Not exactly. It can also make sense if the cache is hot to switch models. For example, we might currently be on Deepseek and the next task is SO HARD that it doesn't make sense to not use frontier (1 turn to crack versus 100 turns for Deepseek to crack). Likewise, the router may judge that it's likely that the next few set of tool-calls will be ridiculously easy, and so switching to Deepseek will save over time.

Remember that Deepseek can be something like 30x cheaper even cold.

There is a more fine-grained view of caching here, because each model can have a different amount of the prefix already cached as well (since we were switching a bunch) so even if the cache is somewhat warm it may make sense to switch.

The cost of AI tokens is top-of-mind for many. Companies like Uber and Salesforce have been complaining about blowing their yearly AI spend faster than expected.

Frontier models are amazing for dev work, but are so expensive. Open-source models are cheap and rapidly improving, closing the gap with frontier models, but aren’t quite there yet.

Tokenless gets you the best of both worlds–routing harder turns to smarter models only when needed, which keeps costs low.

Before Tokenless, I was doing a PhD at Princeton. While using coding/other agents, I constantly agonized over model choice, to make sure my AI spend was going as far as possible on my academic Cursor account.

At the same time, I was doing LLM research, and a small technique I developed while in recovery from NeurIPS submission season seemed to hit SOTA pretty fast. I was surprised that such simple ideas could do routing well.

We’ve been able to develop a version of the router that matches the performance of Claude Fable 5 at half the cost. The blog post on our website explores the technical details on how we did this (https://usetokenless.com/blog/building-tokenless/).

Highlights: - Our approach queries multiple models at once and uses their progress to make decisions (this technique is novel AFAIK, let us know if you know anyone else doing this). - Switching models doesn’t destroy the cache if the routing algorithm is aware of when the cache is hot/cold.

To come: - Adding Kimi K3, all other GPT efforts and more to the router

Go ahead and sign up on usetokenless.com and try using Tokenless with your agent, you’ll get $20 of free credit. Here’s a demo on how to use it: https://youtu.be/sjZWriclcls

Tokenless provides frontier-level intelligence for cheaper, so we’d love some feedback on how it feels to use, any corner cases that the router routes incorrectly, and whether you find the routing problem interesting!

Comments URL: https://news.ycombinator.com/item?id=49099143

Points: 69

# Comments: 59