However, this is not something that is inherently part of models or inference engine, but part of the harness.

Harnesses are very hit and miss, and are not integrated into the stack, and I think that will have to happen eventually. Like, conceptually similar to an LLM performing a tool call that just calls itself recursively, I think this would go a long way to making LLMs more viable for being an actual product people could conceivably want.

Models that use, for example, sparse attention mechanisms are just trying to make the bad situation slightly less bad, such as using less RAM for context (thus requiring less context quantization) or using less bandwidth (thus running faster).

If people keep using temp, top-k, top-p, and min-p, and nothing else for samplers, we're ignoring ~3 years of sampling research that virtually eliminates the worst of context rot issues.

RAG was supposed to be the way out on that, and ended up being mostly abandoned.

At this point in time, due to how most people and companies run their inference engine, regardless of the model (yes, this includes the newest from OpenAI and Anthropic and the Chinese Tigers and Dragons), you run out of useful context that the model can accurately attend to around the 250k mark no matter how much they advertise their context size is.

You need to cut your prompt up. If you believe LLMs work, have the LLM help you shape the overall plan, and then have multiple sessions run each step in the plan without being bloated with the context of previous successful steps.

I don't see LLMs being production-ready until the context rot and sampling problem is fixed forever. This has not occurred, and the big inference providers aren't even bothering to integrate any of the research on that subject.

If anything, many of the bigger companies are actively making inference quality worse just to extend their runway a tiny bit farther before they go bankrupt.

The only thing the article gets right is this: if you're serious about LLMs, abandon Big AI and infer locally only. This is the only way you have control over the quality of the output.

Even though Guerilla Games is currently owned by Playstation Studios, they are still effectively an independent studio.

Microsoft, otoh, owns like half a dozen engines, uses none of them, and licenses none of them. The only way I'd see a engine swap happening is if Kojima somehow swung a deal to license the newest version of the Doom engine.

Aaron Swartz was murdered by a company that prints scientific journals that paywall papers paid for with US government grants; his "crime" was using the access granted to him by MIT, legally. At no point did he break any laws nor the license granted to him by the paywall service.

OpenAI, Anthropic, and Meta all _knowingly_ pirated Copyrighted works en masse, knowing they did not have a license to do so, and then distributed those works, again, knowing they did not have a license to do so.

I do not think anyone should be murdered by the state, but the difference between Aaron Swartz and Sam Altman is one of them did it to fix a great injustice, the other did it as a fly-by-night get rich scheme.

Notice the one that did things legally got murdered, while the criminal continues to walk the streets.

Kind of insane harmless things like THC are Schedule I, but Nicotine isn't.

(Sorry Dan, but this needs to change)

Just use Vulkan for compute.

This is what I have done on mine, and mine as well also runs it when its off.

If your monitor is a Samsung OLED panel, yours is a near identical sibling of mine.

Early OLEDs really did need to be pixel cleaned every 8 hours according to manufacturer estimates, the choice isn't have warning or not, its have a lifespan or not.

I don't want to blame early adopters for being early adopters, but they early adopted, and this is the early adoption problem.

The DOJ's clever hook won them the case.