Tokens are not normal software, because they have marginal cost, and I think people who are used to software economics really struggle with this. With token generation there really can be manufacturing cost efficiencies where one producer is just straight up better at serving product at a lower marginal cost.

I've previously found flash (for all the hate it gets) to be good for these kinds of things. Haiku was fine but it's ancient.

But user->AI calls are very much the rare case now, the more agentic the workload. Most of them will be tool->result->tool without the user involved. And token burn is highest with these long running agentic chains, but that's precisely where routing doesn't work because of the KV cache.

How do you deal with that?

If you think it should be different then you have to make a cogent argument why the public should get to interfere with a settlement the two sides mutually agree on.

If the idea is that there's no point in encrypting anything on a USB stick, and you just hope they don't look at it at all, sure.

But there is no threat model here that includes "check the USB stick" and does not automatically lead to finding and breaking "hidden" content.

Use the right tool for the job. It's like asking why a screwdriver isn't good at sawing wood, or calling C a terrible language because it's hard to make CRUD apps with it.

The labs are not interested in the small, fast, single purpose end of the market. Google increased their pricing on Flash so much that it stopped becoming a cheap model; instead, they released Gemma 4 open source, which is actually easier to use from a third-party inference provider than from Google.

From a total token volume perspective, these "utility" models (classifiers, simple summarizers, small OCR models) will absolutely drive enormous volumes of tokens, at low prices and margin and modest overall market size. Because the models are small and the performance requirements are modest, and because their use cases are specialized rather than general, there are poor economies of scale: they can run cost effectively on rented small GPUs, and a big player doesn't get a structural cost advantage. These models are usually 1b - 30b in size, and can run on a rented 5090. I've productized these myself: I run millions of pages through a fine tuned 1b OCR language model that runs on 5090s at a cost far lower than commercial providers.

But that's not the segment of the market where GLM 5.2, Kimi 3, etc., play. They compete with frontier capabilities, and they are not particularly cheaper than OpenAI models at a cost per task. (I do actually think they compete well with Anthropic, because Anthropic's model efficiencies are poor compared to OpenAI.) And although this part of the market may not be the bulk of the token volume, it is the bulk of the market value.

That's because a lot of human knowledge work is too generalized and fuzzy for dedicated, fine-tuned models, so they are almost entirely different markets that don't particularly compete with each other. (Though if SaaS companies successfully build around verticals that can use small models applied against well-defined jobs, there may be opportunity to push the small/big capability boundary to subsume marginally more valuable tasks that today would require mid-grade reasoning.)

  - Commodity market profitability is determined by marginal cost of production. LLMs have marginal cost; traditional software does not.

  - Models are not free. Downloading them is free. Running them is not. This has manufacturing economics, not software economics; the idea that they are "free" is an economic category error as it relates to their actual use

  - The highest tier Chinese models are not more economical than US frontier models. Try GLM 5.2 and see how much it costs to do real work. I did, and it was more expensive than GPT 5.6.

  - This is because US labs are leading on cost efficacy of inference ($/task)

  - Training will decline as a percentage of costs as inference expands compute share due to agentic workloads. A big part of training now is optimizing token efficiency. It's hard to distill token efficiency; that is perhaps why Chinese LLMs are so inefficient.

  - With increasing inference as % of total compute, if labs create efficient models -- which they can, because they can create highly optimized models amortized over very high inference loads -- they can be low cost producers, and be competitive at $/task rates

Wouldn't it be interesting to learn an influential work that changed how health care professionals run hospitals? Or a document that changed how mining works? A paper on Wright's theory of manufacturing scaling that explains the solar revolution? A thesis on how the world's factories moved from pneumatics to servo systems and why? How a policy thesis changed federal regulators' approach to approving rare disease drugs? Maybe Hayek's views on socialism and information theory, or perhaps an influential thesis on how antitrust monopoly regulation should work?

But instead it's a bunch of crypto and AI stuff that everyone in tech already knows about, rewarmed again for its hundredth serving.

Good communicators learn to use the written word. Bad ones rely on mental crutches.

Good communicators get an audience, and bad ones won't.

You think it's a lost cause, but it's not, because people don't like this junk, because it is low quality and, on average, lacks substance.

The best minds in AI that I've seen all write their own words. They use AI to help them research or ideate, but what they write is their own.

Before assuming this is a "lost cause," consider why the smartest people in the room don't do it.

The same problem exists in the airline market. Airline ticket prices are historically very low, but people complain about seats, fees, and so on. But then they keep buying the absolute cheapest ticket.

What consumers say they care about, and what they actually care about, are not the same. Otherwise they'd pay more for the less irritating product.

With the SWIFT incident I saw, there was a rule that no payment can go to a vendor's bank that isn't a current, approved vendor. But the rule was not enforced in software: it was an internal accounting rule that humans were supposed to follow. The AP person "thought it had been approved" because there was a similar transaction with a different company that was a new vendor at a similar time. The other transaction was legitimate, the fraudulent spoofer wasn't. The wire got sent to a party in China.

With AI agents, if you approach it from the perspective that it will be gullible and trickable by fraudsters, you build in these hard guardrails. With humans, it's much easier to believe that "we trained Lucy on this procedure" will work in all circumstances, even if Lucy still has the technical ability to bypass the official procedure.

In these cases, it starts looking a lot more like traditional software, with your little AI chaos monkeys constrained in little boxes within the software chain.

That's why we have purchase orders that can only be entered by buyers. Product is received and approved by buyer. Invoice goes to accounting, who can't approve it unless there's a matching purchase order and receiver.

Yes, letting agents do whatever they want leads to disaster. But humans are gullible stochastic token generators as well. And that's why the problem is already solved.

https://github.com/orgs/chattocorp/projects/1?pane=issue&ite...