So in order to scale, AI doesn't need compute. It needs "engagement with reality and agency". Which is STILL might do better than us, but is happening in the real world, with real competition over resources. As long as we don't do something dumb like enthusiastically give it control over our major economic actors. I don't think we need to worry.

On the "intellectual immune system" side, I would argue that language's limitations are themselves fitness. We are already in danger of memetic hijacking. All those points you make about multiple instances of an AI cooperating, don't take malice and memetic attacks into account. It goes back to "why I am not afraid of grey goo". I trust yeast to find a way to metabolize basically everything. We have memetic attacks too.

The bottleneck for a developing AI is experience. Yes we need compute, but we need data to compute on.

We have bypassed that limit by starting with literally every scrap of human generated prose that ever existed. I expect an explosion of expansion when visual and world models hit critical mass to properly leverage new experiences. But even then, engaging with reality is the bottleneck.

I can build you a very efficient scalable online map-reduce-like that runs inference on new corpus. We already made that. It took hardware getting large enough to fit the corpus in memory, instead of "scaling" it with networks for it to be viable. The latency of the network passing around partial solutions was WAY too high.

Computers don't scale forever. They are made of hot metals. The limits are heat, material, and the speed of light, but those are very real limits, that don't offer more than a constant multiplier of advantage over meat.

AIs might get smarter than us, arguably, like many other meat and paper based super-human intelligences around us, they already are. But it doesn't scale forever. It will hit limits, fairly quickly, of compute and experience to integrate into it's overfit model.

We are awash in self-replicating machines. The biosphere is already a grey-goo apocalypse. Any new competitors have a serious moat to cross to out compete any existing self-replicators.

We are awash in intelligent agents. Our society (and meta society) is full of superhuman agents already. There is a huge moat for any new intelligence paradigm to cross.

What I am afraid of is the existing superhuman agents (companies, governments and religons) will produce AGI or superintelligence and then proceed to use it as cognitive mitocondria, even further deepening thier supremacy in the cognitive ecosystem.

Don't get me wrong, this is awesome. I think it is built on a subtlety bad premise. I think it is time to start build organizational nomic games on this sort of contract system, literal organization governance, for systems like this to thrive.

I felt like this paper nailed it years a go, and nobody has followed up properly.

The metric involved is basically impossible to compute fully, but easy to approximate. Any online approximation will model everything it can see have changes until it is satisfied.

We are not so lucky in reality.

"Finish the sequence" with 4 options and "no pattern" as the choices.

It becomes "what does the moderately intelligent person who wrote the test thinks counts as a pattern" not the intended exercise at all. There was never enough samples to even guess at a real pattern in them.

What I REALLY want is an app that builds a big bank of nonces between you and your peers over short range radio or QR codes and then lets you use a 1-time pad.

Ultimately, I'm only offering criticism because I have spent a lot of time working on exactly this problem, but I am not in a position to actually implement it. This is awesome and you should be proud of it.

The "break point" is when a DHT gets big enough I can't realistically MITM all the links with nodes "closer to the target" than existing ones.

This means big networks are great, small ones are cheap to just break. Its hard to skip the messy bootstrapping phase.

I'd encourage protocols to only rely on DHTs for small key-value stores if there isn't a trust mechanism in place to validate new peers.

Otherwise, all I have to do is mine for O(n^2) dht keys that cover the network. Figure out what your key mining difficulty is and you can identify what the cost would be.

You can reliably use it to store arbitrary key-value pairs up to 1kb in in size.

https://en.wikipedia.org/wiki/Mainline_DHT

DHTs of trusted participants are great.

Ageless Linux exists for a simple reason. It does not comply for the benefit of that reason.

Every other Linux distro has different goals. Thats why they exist.

I disagree with this law. I like that https://agelesslinux.org/ is challenging it. Expecting the rest of Linux ecosystem to actively NOT comply with a law isn't the right response.