The only way to characterize whether a choice is 'right' is to characterize the output distribution (i.e. evals)! Changing the underlying weights necessarily invalidates whatever characterization may have been done. One may assert that one's harness regularizes outputs back toward the desirable distribution, or one may hope the different weights induce a sufficiently similar output distribution.

But no, one should not be completely agnostic to the choice of weights just because there's some nondeterminism.

Seems like an Amdahl’s law of inference economics? there’s so much compute relative to SRAM on the chip and shoreline bandwidth onto the chip that caching buys ~nothing? The contended resource is SRAM and a given token of context needs just as much as another.

contrary, this is one TLD _ceasing_ its idiosyncratic mode of operation.

This is more a culinary and ethical argument than efficiency or safety. One should use the whole animal, in all its varieties, because they're delicious. Grinding otherwise edible parts of an animal killed on one's behalf because one disfavors (or is repulsed by) them seems to me much worse than the killing. Eat - and enjoy! - all of it or don't kill it.

There's your problem. Ground products are completely dispensable.

This is going to end up like the Law of Headlines, isn't it? "Do x, y, z Cure All That Ails You?" ... no but we got you to read the article. "LLMs _could_ x, y, z" ... but they don't because they're programs, not magic.

What’s next? $20m series B in ByBy, a horizontally integrated trash disposal, concrete shoe manufacture, lye bath, and pig farm operation?

[0]: https://github.com/ggml-org/llama.cpp/blob/master/tools/quan...

Comments URL: https://news.ycombinator.com/item?id=49341009

Points: 73

# Comments: 45

OTOH if LLMs are to write, but not supplant, much of software, then boundaries, delegation to deterministic layers, good compilers to bonk miscreant models on the head with error message seem essential.

At one point it would have been shocking to assert that the compiler would live in-band with the program too. and yet JS eats the world. It seems shocking today that we could have a universal prior over the world operating in the ms/us nJ/pJ range required. And yet … ?

Which itself released yesterday? You're writing, reading, and evaluating enough software in a ~36 hour period to form, reject, and form another opinion about which model makes better _architectural_ choices?