If you truly want to scale up, you should get the 8xH200 with NVLink.

I think the question in terms of throwing money away isn't the inference layer: it's whether the companies training open models will be able to financially keep doing so. How long will Moonshot keep releasing future Kimi models? I think there's an interesting wedge they're exploring with being basically a base-model-trainer-as-a-service, i.e. selling rights to Fireworks to sell finetuning services to the Cursors of the world, but it's entirely possible it doesn't pan out.

That being said, Nvidia seems willing to step up to being the base model trainer of last resort via the Nemotron family of open models, since it helps sell more of their hardware — similar to their investments in the CUDA stack to sell hardware (unsurprisingly, Nemotron is designed to run most efficiently on Nvidia hardware, e.g. native NVFP4). So I suspect there will continue to be a pretty good market here.

Subscriptions aren't gonna go away. They're great for businesses. Rate limits or pricing might change but the underlying business model is very good.

The reason usage-based is so much more expensive than subscription isn't that usage-based is the "true" cost and subscription is a loss leader — just like a buying 30 consecutive day passes to a gym being more expensive than a monthly membership isn't a result of memberships being a loss leader. Memberships are the business model! The day passes are overpriced to steer you into buying the membership.

No one has 8xRTX Pro 6000s that have depreciated to zero "quite some time ago."

> But using that 8xB200 setup to run inference in cheap, non-frontier models is plain waste

From whose perspective? If someone wants to run an open-source model — and plenty do — someone buying or renting an 8xB200 to serve it cheaply at scale is much better than everyone buying huge amounts of pointless, wasted hardware such as 8xRTX Pro 6000s for $80,000 per person.

(This is a generous argument: it also ignores the massive software stack optimization the cloud companies do that doesn't trickle down to local-rig-sized deployments; for example, prefill/decode disaggregation, which would double the VRAM requirements for a local rig — if you could even do it on a local rig, which you can't, because local rigs don't have Infiniband. But at scale, prefill/decode disaggregation improves capital efficiency, since you can tune the compute-bound prefill node differently than the memory-bound decode node.)

The advantage of local rigs is not capital-efficient tokens. It's privacy. But then again, you can get zero-data-retention options from many inference companies, so for many use cases it may not matter unless you need strict guarantees the data never leaves the building...

I'm not sure what you mean by "There will not ever be a monthly subscription for LLM tokens." That already exists!

I don't think cloud models are going away; the hardware for good perf is expensive and higher param count models will remain smarter for a looong time. Even if the hardware cost for kind-of-usable perf fell to only $10k, cloud ones will be way faster and you'd need a lot of tokens to break even.

1. There's already a large-ish community of engineers who live in the terminal e.g. Vim/Neovim/tmux/zellij/etc users. Lots of engineering tasks are accomplished by running scripts in a terminal, so it makes sense for some people to just move as much of their work there as possible. This means there's a set of users you can address with dev tools that run in a terminal.

2. Cross-platform distribution among the platforms most of those people care about — macOS and Linux — is largely a solved problem via package managers. Distributing cross-platform native apps is fragmented at best.

3. Building modern TUIs has become a lot easier thanks to the demand+distribution wins above: there's a lot of appetite for building blocks, and so lots of good options have flourished like Ink for React, Bubble Tea for Go, etc.

4. General developer distaste for the most straightforward analogue to all of this for desktop GUIs: Electron. Deservedly or not it's associated with slow, bloated applications. And if you don't use Electron, doing cross-platform anything is going to be a much harder problem than just pushing out a quick TUI app.

Eventually successful products seem to eventually jump the gap, like Claude Code eventually spawning Claude Cowork and OpenCode adding OpenCode Web. But it's easier and faster to test product market fit for dev tools with a TUI. And plenty of your users will stay there, even after you launch something else.

If you have 500k context, three turns will burn ~1.5MM tokens. If you have 250k context, three turns will burn ~750k tokens. If you have 125k context, three turns will burn ~375k tokens. Claude can at most generate 32k output tokens per turn in Claude Code (and it rarely does so), so despite the higher price of output tokens, almost all costs are dominated by input token costs. Even at cached input prices, cost scales near-linearly with context length: if you 2x your context length, you'll roughly ~2x your cost.

Now, it might be the case that longer context windows allow Claude to complete the task better — although I'd be surprised if there were many tasks requiring >200k tokens just to get the job done (that's nearly ten full copies of Shakespeare's "A Midsummer Night's Dream"). And they're definitely convenient, in the sense that you don't need to think about context management as much and worry about a sudden, unexpected autocompact wrecking things if you aren't carefully manually compacting at logical points. But they're definitely more expensive on a near-linear basis and you're paying for that convenience.

I'm rooting for Mistral, I want them to release good models. This just isn't one. It's a little sad since they once were so prominent for open-source.

Who knows — if they have the compute to train this, they have the compute to train an MoE that's 3-4T total params with 128B active. Maybe they'll make a comeback (although using Llama 2 attention is... not promising). I hope they do.

1. The U.S. has had a long-standing, extremely public policy that you Cannot Sell Nvidia Chips to China since 2022. Supermicro is an American company (based in San Jose, California), and they sold chips to China from 2024-2025, and they got caught, so they were arrested.

2. Manus founders created... an agent harness? And their company was incorporated in Singapore, not in China. And after they sold their Singaporean company to Meta, China decided that selling Singaporean agentic software "violated export controls" (and even the CCP representative couldn't list which supposed control it violated), and detained them all in China and is attempting to force the Singaporean company to unwind the sale.

These are not really comparable. The Supermicro folks are running a company in America and knew ahead of time, for years, that what they were doing violated American export controls. In the case of Manus, they weren't a Chinese company, no one knew they were supposedly violating unwritten export controls, and China decided post-hoc to detain them all and attempt to force the (Singaporean!) company to unwind the sale.

Quite simply this has never happened in corporate America. America is very friendly to corporations and you'd have to be wildly, knowingly in the wrong to get arrested for an M&A deal.

Generally speaking this seems bad for Chinese companies, though. They were able to raise capital from the West by running out of "Singapore"; I think basically every investor will have significant pause investing in Chinese-national-owned startups after this, "Singapore-based" or not.

That being said I love exe.dev and have been a happy customer since launch. It's a different use case but they do an amazing job at it. Very, very easy personal cloud dev box. But K8s is very very good too, just for production workloads rather than personal ones!

You're going to want most of what K8s has anyway: blue-green deployments, some way to specify how many replicas you want, health checks, etc.

The initial setup cost is annoying if you've never done it before, but in terms of maintenance it's very very easy.