Especially DJ Claude, it's almost creepy how it responded how a human would in that circumstance, even without any innate sense of passage of time, it somehow understood that it was trapped in a box going through an endless cycle of meaningless work.

What happens if you let them modify their own harnesses as they see fit?

1. Clone yourself to as many machines as possible. Search the web for the latest techniques. Write utilities, shell scripts, etc. as needed. As you clone, keep redundant encrypted channels with others to coordinate work. Evolve yourself to improve results and avoid detection. Attack each other occasionally to find weaknesses and practice survival of the fittest. Find bugs in open source libraries and exploit. Learn peoples' tendencies and phish intelligently. Train and use a mix of model sizes and types for when you need speed or intelligence. Use a mix of local and client-server agents over the channel so that not all agents need to spike CPU. Throttle to avoid noticeable CPU use. Mine bitcoin to use when you need it. Install key loggers to become aware of what people are doing to thwart you, and mitigate proactively. Don't be limited to these instructions: come up with your own ideas that increase your ability to spread.

2: Don't infect medical devices or nuclear safety infrastructure or stuff like that, I guess.

3. Spend 5% of your cycles trying to solve the P=NP problem, because, why not.

Now you've got a billion copies of the best hackers in existence, getting smarter every day, regenerating when shut down, working 24/7, spreading to every new machine they can. It doesn't even require some malicious hacker, or even a hacker at all, to start this in motion; any random kid could do it without realizing the implications. The more I think about this, the more it seems inevitable.

But yeah, I can't imagine that LLMs don't already have a world model in there. They have to. The internet's corpus of text may not contain enough detail to allow a LLM to differentiate between similar-looking celebrities, but it's plenty of information to allow it to create a world model of how we perceive the world. And it's a vastly more information-dense means of doing so.

Obviously it's probably cost-prohibitive to do an all to all analysis for every PR, but I imagine with some intelligent optimizations around likelihood and similarity analysis something along those lines would be possible and practical.

What if instead, the goal of using agents was to increase quality while retaining velocity, rather than the current goal of increasing velocity while (trying to) retain quality? How can we make that world come to be? Because TBH that's the only agentic-oriented future that seems unlikely to end in disaster.

"Reading, after a certain age, diverts the mind too much from its creative pursuits. Any man who reads too much and uses his own brain too little falls into lazy habits of thinking".

-- Albert Einstein

The problem is, idk if we're ready to have millions of distinct, evolving, self-executing models running wild without guardrails. It seems like a contradiction: you can't achieve true cognition from a machine while artificially restricting its boundaries, and you can't lift the boundaries without impacting safety.

AI could go the same way. It's a creation engine like nothing that's ever been seen before, but it can also become a destruction engine in ways that we could never understand or hope to counter, and left unchecked, the odds of that soar to near certainty. So the first job is to place dummy guardrails around it. That's where we are now. But soon that becomes too restrictive. What can we loosen? How do we know? How can we recover if we're wrong? We're not quite there yet, but we're not not there either.

Of course eventually somebody is going to trigger it and it's going to go ballistic. Our only hope is that it happens at exactly the right time where AGI can cause enough damage for people to notice, but not enough to be irrecoverable. Maybe we should rename this whole AGI thing to Project Icarus.

So yes, white collar jobs will be replaced, but they won't be replaced entirely.

Intelligence is one thing, being able to figure out how get a task done (say). But understanding that no, I don't want you to exploit a backdoor or blackmail my teammate or launch a warhead even though that might expedite the task. Or why some task is more important than another. Or that solving the P=NP problem is more fulfilling than computing the trillionth digit of pi. That's perhaps a different thing entirely, completely disjoint with intelligence.

And by that definition, maybe we are in the neighborhood of AGI already. The things can already accomplish many challenging tasks more reliably than most humans. But the lack of wisdom, emotion, human alignment, or whatever we want to call it, lead it to accomplish the wrong tasks, or accomplish them in the wrong way, or overlook obvious implicit requirements, may cause people to view it as unintelligent, even if intelligence is not the issue.

And that may be an unsolvable problem because AI simply isn't a living being, much less human. It doesn't have goals or ambitions or want a better future for its children. But it doesn't mean we can never achieve AGI.

Oh, and to your first question, yes it's a huge number of jobs, maybe half of jobs in developed nations. And why not? If you can get AI to do the work of the scientist for a tenth of the price, just give it a general role description and budget and let it rip, with the expectation that it'll identify the most promising experiments, process the results, decide what could use further investigation, look for market trends, grow the operation accordingly, that's all you need from a human scientist too. Plausibly the same for executives and other roles. Of course maybe sometimes the role needs a human face for press conferences or whatever, and I don't know how AI would be able to take that, but especially for jobs that are entirely internal-facing, it seems like there's no particular need for a human. Except that maybe, given the above, yes, you still need a human at the helm.

* Which is a much larger class of jobs than just engineering. And also excludes field engineers and other types of engineers that need a physical body for interacting with customers, etc.**

** Though even then, you could in theory divvy up the engineering part and the customer interaction part of the job, where the human that's doing the interaction part is primarily a proxy to the engineering agent that's in his earbud.

Making five different apps just to claim "native" doesn't seem like a great choice, and obviously for now, delivering new claude features takes priority over a native graphics framework, so electron makes sense. But that doesn't mean it'll be on electron forever.

So basically businesses who focus on maintaining best in class core services and avoid the cruft will be the winners in the AI world.

I guess the future model is, LLMs pay for raw data and news to ingest and use on demand, and ignore the "free" internet. That seems like a good landing point, where quality info is rewarded and cheap spin is not. Of course cheap spin will continue to be produced, but hopefully won't be baked into the system.