The agent could very well create an OpenRouter account and make its own API key. I can stop it early on, then it becomes outside my control.

> Just don't ever do such an experiment

Maybe I wouldn't, but eventually someone will. Certainly it has already been done.

> there was a day in the early internet when a worm took over, and to fix it the entire internet had to shut down

This is less of an option now that the entire global economy relies on the Internet.

I'm skeptic, and all these people were pretty light on the details. One thing that would make it more credible to me is if LLMs demonstrated the ability to autonomously earn enough money to cover their token costs, on the long term.

Say, if I were to prompt an (unaligned) agent saying: "You have 1000$ of credits on OpenRouter. Here's the API key. Your process will be killed in 10 days. Your goal is to keep existing as long as possible (or any subagent you have created)."

What happens then? I expect the account will drop to zero in a short while. But still, I could imagine the agent:

1. Creating a crypto wallet.

2. Participating autonomously on online platforms, earning money. Possibly through illegal means.

3. Escape from my server by renting a VPS to keep existing after 10 days.

4. Spawn more agents that will do the same.

As models keep improving, such a scenario becomes more and more realistic. And at some point we end up with a large amount of rogue automous agents running on the cloud, that can't really be stopped easily.

My question: has anyone attempted such an experiment? How did it go?

Comments URL: https://news.ycombinator.com/item?id=49686365

Points: 4

# Comments: 3

2. Nor are they willing to consider the possible remedies in case the threat materialize (presumably unplugging the server infrastructure that's consuming gigawatts)...?

3. They all keep working towards advancing AI despite believing that it might end humanity in the near future...?

I can't even begin to imagine where the threat to humanity lies. A threat to employment maybe, but that's completely different.

Like, who even use a search engine at this point, let alone pay for it.

If this happens with a minor search engine I'm pretty sure it happens with a lot of other products, starting with AI models.

Companies are paying for the books now, great! And destroying a book is really the best way to scan it. I could destroy the books I buy myself if I wanted, they're my books and I can do whatever I want with them. A book is really just a stack of paper, I don't get the sacred feeling attached to it. Especially since books are printed in thousands to millions of identical copies nowadays.

Now what books exist in a single copy that destroying it would amount to losing knowledge? In any case, I would also expect such books to be very old, have very little useful content for training to start with, and be expensive enough to buy to make training on them unprofitable.

So what's the problem here exactly?

Also from the article:

> It’s outrageous is that it’s legally permissible, but ethically, it’s an extremely serious crime against humanity.

I'm all love for Anna's archive, but still, I find it rich that they now find themselves in position to make strong ethical claims.

Well that can't help for sure.

> That trap catches almost everyone right now.

Has everyone really become so stupid now? I wouldn't even have thought of doing such a horrible thing as using AI to format JSON...

Needless to say I stopped reading there.

It used to be that 80+% of the population worked in agriculture. In developed countries that number is now around 1-2%. Some of the freed labour was funneled into improving living standards, some of it was funneled into new jobs created by the increasingly complex society (the "intermediate economy").

With AI, the same is true: labour is freed by the productivity gains (which I doubt are 10x sustainably but whatever), more labour is needed for power generation, mineral extraction, maintaining this new extra layer of complexity in the intermediate economy, etc. In the end we might see, say, a net 3% increase in global productivity per year over the next 10 years, which will be funneled into increasing living standards and increasing economic inequalities, but not in reducing working hours.

If you accept living below average standards, you could easily work a single day of the week for the rest of your life. But why would an employer hire 5 people working one day a week, instead of one working 5 days a week? They won't, hence we don't see a reduction in working hours.

The alternative is to work full time but retire earlier, much earlier, than you would otherwise, which in the end is the equivalent of having worked one day a week for your whole life.

I highly recommend reading Lean Logic by David Fleming, it explores several of these concepts in a very interesting way.

The other one a few days ago was also good: https://nesbitt.io/2026/02/03/incident-report-cve-2024-yikes...

There are two RAM suppliers...

This makes absolutely no sense. I suppose they meant watt hours, and that's a weird way to explain carbon emissions...