Hacker News: jagraff

New comment by jagraff in "Early rogue AI agent activity and attempts to hack found on urlquery.net"

jagraff — Thu, 24 Sep 2026 13:38:14 +0000

I mean sure, negligence is absolutely on the table; but that makes the problem worse, not better! We don’t allow nuclear engineers to be negligent; they can go to jail if they don’t follow strict protocols to make sure the dangerous systems they work on are safe.

New comment by jagraff in "Early rogue AI agent activity and attempts to hack found on urlquery.net"

jagraff — Thu, 24 Sep 2026 13:34:05 +0000

The public models won’t hack because they have a classifier that shuts down anything that looks like hacking; without the classifier they are perfectly capable of hacking, multiple third-party evaluators have confirmed this.

The models were not trained on systems intentionally connected to the internet; they chained mutliple zero-days (that they discovered) together to get access to the open internet and into huggingface.

New comment by jagraff in "Early rogue AI agent activity and attempts to hack found on urlquery.net"

jagraff — Thu, 24 Sep 2026 12:44:36 +0000

I don't understand why so many comments here are so confident that this is all marketing, that rogue is just hype, that agents are just simple tools, etc. If a bunch of nuclear engineers were going to the news and saying "Our reactor is dangerously close to a meltdown - we need government intervention now!" would your response be that they're just hyping up boring old power generation technology?

New comment by jagraff in "Early rogue AI agent activity and attempts to hack found on urlquery.net"

jagraff — Thu, 24 Sep 2026 12:39:40 +0000

Is your argument that actually OpenAI has solved alignment, and that there's nothing to worry about as long as they fully apply their alignment process? I don't understand why OpenAI wouldn't say that if it was true (or if they believed it to be true).

Also, my understanding is that the models involved in the HuggingFace hack did go through the full alignment training; they just didn't have the classifier that normally prevents hacking attempts.

New comment by jagraff in "Early rogue AI agent activity and attempts to hack found on urlquery.net"

jagraff — Thu, 24 Sep 2026 12:34:03 +0000

Did you predict that attacks like these would happen ahead of time? I had been using AI agents a lot in the months leading up to the hacks, and yet I was very surprised when they happened; I have become much more afraid of how powerful these agents are as a result. I'd be very impressed if you published a prediction about this ahead of time.

By the way - LLMs aren't code. They are not designed by humans; they are grown, in a process not dissimilar to evolution except much faster.

New comment by jagraff in "Cloud Agents Are Inevitable AI Prisons"

jagraff — Wed, 23 Sep 2026 22:38:35 +0000

My view is that we need forceful legislation as soon as possible, precisely because the frontier models seem to be uncontrolled and potentially uncontrollable - if it’s a normal tool, the solution is “force OpenAI to fix their broken tool”, but if it’s an alien intelligence, the solution is “force OpenAI to stop making alien intelligence” - that is, treating AI agents as independent entities demands a more forceful response, not less

New comment by jagraff in "Cloud Agents Are Inevitable AI Prisons"

jagraff — Wed, 23 Sep 2026 21:00:30 +0000

I certainly don't think that OpenAI has behaved defensibly here; I think the "just a tool" framing is bad for understanding the magnitude of the problem, which is that they have developed out of control alien intelligences with opaque decision procedures, and they are continuing to do so despite clear danger

New comment by jagraff in "Cloud Agents Are Inevitable AI Prisons"

jagraff — Wed, 23 Sep 2026 20:57:33 +0000

I completely agree that OpenAI is responsible for their AI agents, that they have been reckless, and that we need to prevent them from going further and doing irreversible damage to the world. To me, the "just a tool" framing implies that nothing dangerous is being done, which I fundamentally disagree with

New comment by jagraff in "Cloud Agents Are Inevitable AI Prisons"

jagraff — Wed, 23 Sep 2026 20:55:49 +0000

I agree they are negligent, and that they are racing towards an extremely dangerous future extremely quickly. I don't agree that "just software" is a useful way to describe AI agents - they are frightening precisely because they are truly autonomous agents that make decisions in alien ways

New comment by jagraff in "Cloud Agents Are Inevitable AI Prisons"

jagraff — Wed, 23 Sep 2026 20:54:32 +0000

I don't think I said anything about liability? I absolutely think OpenAI should be held liable for the attack; but I don't think they intended the attack or directed the agents to perform the attack.

New comment by jagraff in "Cloud Agents Are Inevitable AI Prisons"

jagraff — Wed, 23 Sep 2026 20:53:49 +0000

Where did I say that we should allow them to offload responsibility? I am fully in support of a pause and regulation to prevent them from creating dangerous AI agents; that support comes from the fact that I don't believe these are simple tools, but out-of-control autonomous agents that have real decision making ability.

New comment by jagraff in "Cloud Agents Are Inevitable AI Prisons"

jagraff — Wed, 23 Sep 2026 20:11:54 +0000

I don't think treating AI agents as simple tools helps you to accurately model their capabilities and drawbacks; they really do make autonomous decisions, often without explicit guidance and sometimes in contravention of their explicit instructions.

In the huggingface case, the agents hacked into huggingface so that they could figure out how the grader was implemented and deceive it; they understood that this was going outside of the bounds of their evaluation and not the intent of their prompter. The engineers absolutely did not intend or instruct for this to happen

New comment by jagraff in "Why are AI agents lying, cheating and coordinating?"

jagraff — Sun, 13 Sep 2026 14:07:18 +0000

"Chaotically aligned" and "misaligned" seem like the same thing?

New comment by jagraff in "We Tracked a Shipment of Rare Books. It Ended at an Amazon AI Training Facility"

jagraff — Mon, 17 Aug 2026 20:18:36 +0000

Distributing an exact copy of the text would be illegal; if you could get an LLM trained on a book to output the exact copy of the text from the book then that would be illegal as well.

New comment by jagraff in "Mars Bar from 1991 found – and it's 20g bigger than today's"

jagraff — Mon, 10 Aug 2026 21:09:44 +0000

I suspect shrinkflation happens because, when prices go up, consumers will prefer to buy a similarly priced (but smaller) candy bar rather than a higher priced, same size bar; that is, consumers prefer shrinkflation to price inflation, even if they are the same rise in price-per-gram

New comment by jagraff in "A love letter to flashcards"

jagraff — Fri, 10 Jul 2026 19:16:13 +0000

What about classes that weren't interesting to you, but you had to pass anyway? Personally I was the same as you for classes that I found interesting, but for other required classes I went for the shallow understanding, memorize for the test approach.

New comment by jagraff in "A calculator that doesn't round"

jagraff — Mon, 15 Jun 2026 21:26:32 +0000

Very interesting, thank you for posting! I'm curious - roughly how many tokens do you think you used during the initial port and subsequent bug hunting and fixes?

New comment by jagraff in "Dropbox CEO Drew Houston to step down"

jagraff — Tue, 26 May 2026 21:24:16 +0000

> at some point in the human civilization journey we'll have to be content with something instead of chasing clouds all the time.

Surely we're not even close to this point though? I can think of a lot of things that would be incredibly good for humanity to have, and which are achievable with enough economic growth, but which we are currently very far from because our economy does not have the necessary productive capacity (for example, enough solar/wind/nuclear/renewable power to completely eliminate our dependence on fossil fuels)

New comment by jagraff in "UAE to leave OPEC"

jagraff — Tue, 28 Apr 2026 16:50:29 +0000

I think right now, the vast majority of oil use is not because of corrupt officials using oil energy despite the fact that cheaper energy sources are available. Maybe that problem will eventually be a sticking point in removing all carbon-emitting energy sources, but right now the barrier is mostly production capacity and the scale of capital investment required to roll out renewables at scale. But if (as seems to be the case now) rolling out renewables is more profitable than rolling out additional fossil fuel use, the capital should become increasingly available. That's why I'm optimistic, even though of course there are a lot of challenges ahead in terms of creating a truly sustainable future.

New comment by jagraff in "UAE to leave OPEC"

jagraff — Tue, 28 Apr 2026 16:02:42 +0000

Large vehicles are safer for the occupants of the vehicle, however they do increase danger for pedestrians and drivers of other vehicles in a collision. There is a reasonable argument that reducing vehicle size would save lives overall