How much would it cost to find those things, if they exist? For using the blog post's methods to label every neuron in modern arch, Claude and Gemini produce similar back of the envelope math that looks something like, within an order of magnitude of the resources it takes to pretrain the model in the first place. If you have a better way of narrowing down the set of candidates, probably a lot cheaper.

Would that be reliable and useful? Well.

The pretty 3d stuff you see are agents controlling a Blender MCP in a loop where they look at the scene, think, adjust the scene, repeat.

You put yourself in a situation where nobody understands you in your native language, nothing helpful for your daily life is written in your native language, and you must constantly try to comprehend and produce the target language if you want to accomplish even basic tasks.

If you can't up and move, then studying a language with a lot of cultural export helps a bit, if you can get into the habit of switching all of your media consumption to the target language. It's not the same, though.

I'd say it's much closer to the concept of continual learning, but I'm only a few pages deep and haven't groqued it fully yet.

Smolensky's latest paper posted here the other day has some thoughts on how modern neural networks might beconsidered neurosymbolic, or rather "gradient symbolic processing," from another perspective entirely.

I wouldn't say the bitter lesson has given out! If you haven't noticed, these things keep getting bigger and bigger.

For better or worse, it looks more like that we're gonna tile the whole damn planet in data centers, at a faster and faster clip.

He's the axis of this particular group of researchers, being the most senior at the place where they all met, Johns Hopkins.

So this is less a straw man and more a quick reminder to his peers: "Right, so, remember this particular thread we've spent the last 40 years hashing out, here we've got another contribution to that particular conversation."

Any act of making can be treated as product, as craft, or as art. You'll find yourself taking all three stances at different points. Any engineer or artisan or artist can choose to pour their attention and love into any layer of the production process.

I like ceramics. I like wheel throwing. Once upon a time, the pottery wheel was a newfangled technique for rapid and regular production of commodity goods. Same with coil building, same with slip casting, same with standardization of glaze recipes, gas and electric kilns, all of it.

But I know people who choose to quarry and purify their own clay, or to mix their own glazes, or to fire their pieces in a hole in the ground dug by hand with a fire they built by hand. I know people who make dozens of identical copies of just a few forms, after they spend months iterating on carving intricate molds. I know people who will spend a whole season working on a single hand-built bust.

And none of these people have ever, to my knowledge, expressed scorn for any of the others, for focusing on a different step in the process.

Some tasks within the benchmark are much easier than others. The hardest several tasks often have vastly different difficulty levels. Often, the hardest few tasks are literally impossible; malformed problems due to poor curation, often.

Imagine you've got a basketball robot, and one way you test it is on the Three Pointer benchmark. It tests the robot's ability to shoot a three pointer from 20 feet, 25 feet, 30 feet, 40 feet, 50 feet, 60 fee, 75 feet, 100 feet, 200 feet, and 182 miles.

Is a robot that scores 90% on this benchmark 90% as capable as one that scores 100%?

We are perfectly capable of running LLMs in a way that does a backward pass to update some or all of its weights after every user message. But, naively implemented, you only get partial, fragmentary absorption of the info in those messages, it costs three times as much compute, and you lose out on the ability to implement a ton of optimizations that making modern LLM serving economical.

If you want to do it, though, ask your friendly neighborhood robot to get it working with a tiny model (whose full precision weights fit several-times-over on your machine's resources).