Entropy is a metric targeted in LLM training which is likely why LLMs are overfitting less and less as they improve and why they subjectively seem to obey Occam's razor.

There's lesson for software engineering in general behind that kind of simplicity. https://benoitessiambre.com/entropy.html

https://benoitessiambre.com/entropy.html

Refactoring towards good abstractions is more powerful than people realize. There's information theoretic bayesian math to back this up.

It's a bit of a divine coincidence that software that is more economically and energy efficient to process and run tends to also be more correct.

It's all about reducing the entropy of your code. https://benoitessiambre.com/entropy.html

If you look at how simulated annealing (https://en.wikipedia.org/wiki/Simulated_annealing) is done, while the average size of jumps shrinks in the wikipedia annealing animation, there's always _some_ probability of large jumps in the optimized metropolis-hasting process as the jumps are still normally distributed just with shrinking variance over time.

"[I split a large PR into multiple] But also, I could not have developed the quotas feature in real life in that artificial order. The grants structure evolved as my understanding of pricing and quota enforcement evolved. The original quota semantics sucked, so I rewound back to the data structures, which affected how the pricing got imported, which changed how the quotas were stored. The code reviewers didn't have to worry about that but I did."

This is also one way LLMs are fundamentally different from prior language models which worked by searching over parse trees top-down or bottom up trying to fit independently evolved pieces. LLMs lay everything out in a large matrix of randomized weights and try to slide everything into place jointly.

This means organizing all the pieces well into a single context window unlocks a special AI power: to efficiently jointly converge these pieces to fit better with each other (like a smart human having loaded up on context would do). Splitting the work into multiple PRs or contexts might stymie this powerful aspect of AI.

It is a challenge and somewhat of an art to pack and organize the information in a context window to exploit this type of reasoning LLMs are made for.

Comments URL: https://news.ycombinator.com/item?id=48351575

Points: 4

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=47433828

Points: 2

# Comments: 0

This means that the related parts in the inputs and outputs are recursively as close together as possible.

There's a reason humans don't type and read http/json on the command line. It's hard to read and reason over that type of syntax. json is made to be easy to parse for simple algorithms, not meant to organise info in an easy to reason about layout.

AIs benefit from the easy to reason about layout. It's not just about being able to fit in the context window but about the contents of that context window being organized such that the attention mechanism doesn't have to stretch itself out trying to connect far apart related pieces. It doesn't have to try to match brackets to disambiguate information. CLIs tend to use formats that are obvious at a glance for humans and LLMs alike.

It's about the entropy of the formats. https://benoitessiambre.com/entropy.html

The probabilistic version of "Do No Harm" is "Do not take excessive risk of harm".

This should work as AIs become smarter because intelligence implies becoming better bayesians which implies being great at calibrating confidence intervals of their interpretations and their reasoning and basically gaining a superhuman ability for evaluating the bounds of ambiguity and risk.

Now this doesn't mean that AIs won't be misaligned, only that it should be possible to align them. Not every AI maker will necessarily bother to align them properly, especially in adversarial, military applications.

Open source solutions like https://en.wikipedia.org/wiki/GendBuntu could become more prominent. There's even interesting non us hardware options like https://starlabs.systems/

The US has had an unfair advantage in tech, defense, science and finance because it hosted the global hubs of the free world. This attracted eye-watering amounts of money to places like SF and NY. With the newfound isolationism, tariffs, threats etc. reducing the viability of hosting the global hubs, there's massive opportunities opening in europe and elsewhere, especially if governments can help bootstrap these sectors with efforts like these.

Comments URL: https://news.ycombinator.com/item?id=46747596

Points: 4

# Comments: 0

Open source is the global alternative you're looking for. There's even interesting hardware options like https://starlabs.systems/

The US also has had an unfair advantage in tech/defense and finance because it hosted the global hubs of the free world. This attracted eye-watering amounts of money to places like SF and NY. With this newfound isolationism, tariffs etc. reducing the viability of hosting the global hubs, there's massive opportunities opening in europe and elsewhere.