The important part of "actually wait, I really need to XYZ" is just "XYZ".

The model can attend to just "do XYZ" and produce almost the same vector modifications as full verbose "reasoning".

Nothing is good enough on its own.

Geoblocking, fail2ban, port obscurity, SSH keys, limiting logins to specific usernames, not using your public internet nickname, putting things behind CloudFlare tunnels or WireGuard, wildcard DNS obscurity, 2FA... There are many options.

Defense in depth is the only way to put services on the internet.

Top-P: example setting 0.9. Select tokens whose probably accumulates to this number. So say you have tokens with 0.7 then 0.2 then 0.1, the last will not be selected because the first two tokens already accumulated to >=0.9.

Min-P: example setting 0.05. Don't select tokens less probable than this value. So a token with 0.1 would be considered, a token with 0.01 would not.

The purpose of all of these is to exclude very unlikely next tokens.

You might understand this as "The capital of France is..." and the model isn't always going to select "Paris". Sometimes it will start a descriptive sentence or even get the answer wrong.

That selection of the next token is what these settings control, and lots of sub-optimal selections compound over time to produce a junk response.

One really would think these companies (including Google) who spend many millions of dollars on compute could write a few hundred lines of Jinja correctly, so their investment works optimally or at all.

But they don't.

Then a couple of individuals on HuggingFace fix it, either a 2-person startup like Unsloth or a volunteer like froggeric.

I also don't understand how this repeatedly happens.

It's true they make architecture-specific changes like keeping certain layers at F16 but it's also more than that.

    #define RN (((seed = seed*11109+13849) & 0x7fff) >> 1)

I would not be surprised if other contemporaries like glibc, Borland, Watcom, Lattice, etc all did the same. Or did they do something else/better?

Yeah maybe Claude/OpenCode/KiloCode/Hermes/whatever are not as minimal as Pi but they also work right now.

https://ogrebattlesaga.fandom.com/wiki/WORLD_System

https://www.wiki.ultimacodex.com/wiki/Runic_magic/Ultima_Und...

https://www.wiki.ultimacodex.com/wiki/Rune_stones#Meanings

I remember an undocumented spell "Vas Kal Corp" which meant "Great Summon Death" and indeed it does. It ends the entire world and results in a game over.

https://en.wikipedia.org/wiki/Cirth#Runes_from_The_Hobbit

I got to the point I could read and write without a reference. Not as fast as English but not slow either.

Many years later I found a notebook with many pages of things I'd written but had since lost the ability to read again.

If all the research went away tomorrow, people are still going to sell inference at current prices or higher. There are already open weights competitive with proprietary models. There will be no lost capability.

If businesses find inference useful today then it doesn't have to drastically improve in a short timeframe anymore. History is full of inventions which became "good enough" and didn't improve much or at all for a long time.

eg: Western society runs on radial tyres which have seen only marginal improvements for the last 50 years.

(yes there have been some small improvements in compounds, tread patterns, TPMS, etc. hardly drastic revolutionary changes to the tyre industry)

If you have a pair of 3090s and run Qwen 27B, or an old Threadripper with heaps of system RAM and Deepseek or MiniMax or Kimi, no it won't be as fast as Claude.

Most local LLM nerds are not running locally for superior speed, we're doing it for sovereignty and/or privacy, or maybe just because it's fun which accidentally became useful this year.

https://materiasiis.uson.mx/docs/control_de_calidad/1.CALIDA...

Notably, when the Lucky Guess method doesn't work, don't keep guessing, troubleshoot properly.