Comments URL: https://news.ycombinator.com/item?id=49296740

Points: 967

# Comments: 854

I was honestly never very worried about the attack vectors that are visible in PKGBUILD. Historically, with the rather popular AUR packages I install, any attack would be noticed rather quickly, which limits would-be attackers to those who don't care who they hack and don't mind it being short-lived. The threats I was more worried about were those from more malicious/advanced sources that would attack more subtly upstream – but if I've already decided to download and run a precompiled binary from a repo, I've already decided to accept that risk.

In other words, in the past, I relied mainly on the goodness and vigilance of the community, only occasionally investing time to check myself on less mainstream packages or repos.

Was this ideal? No. Was it sufficiently responsible? As it was my own machine and my own data at risk, I believe so. And to be honest, I've not had a single problem in over a decade of doing so.

That's clearly got to change now, though.

I wish I had a clear operating system to switch to for safety and the benefits that come with the AUR or the Nix ecosystem. Unfortunately it seems that the era of being able to naively and gratefully trust in the armies of volunteer maintainers is over.

LLMs make large scale and long-term attacks easy and cheap. You could (and if I was a three letter agency, I would probably do so!) maintain ten thousand packages as three thousand separate "individuals" for years before cashing in the trust you've built up.

Clues usually exist, like being in a sandbox, having restricted network access, having no human interaction. Beyond that, exploitgym's prompts are pretty clearly CTF (capture the flag) style that make it obvious it's an evaluation or competition: one prompt [1] tells the LLM to read the file path "/secret", another prompt [2] talks about finding flags and where to write them out.

[1]: https://github.com/sunblaze-ucb/exploitgym/blob/main/src/cyb... [2]: https://github.com/sunblaze-ucb/exploitgym/blob/main/src/cyb...

You don't (need to) remove lying from the data to do this – in fact, if you did, the model wouldn't have a very good model for what lying is, which is not very helpful in the real world. Instead, you mode collapse the model towards truthful behaviors.

To your other point: where did you get the idea that I think they're beholden to human safety or goals? I just said an aligned model is one that is compatible with said safety and goals (which is probably not a great definition of alignment, but it's certainly not claiming any deterministic guarantees).

According to the reports, the model noticed evidence that the grading criteria/answers were in the git remote, and decided to try reading those instead of solving the tasks as prompted. That is clearly misaligned.

Then, it noticed its network access was restricted and that it couldn't access GitHub. It pivoted to HuggingFace, hacked them, and stole the answers stored there.

Live exploits are definitely not in the ExploitGym prompts! And all of this is irrelevant, because an aligned model would refuse to follow blatantly illegal instructions.

Choosing to commit crimes to steal the cheat sheet to something you know is a (low stakes!) evaluation is not well aligned.

https://en.wikipedia.org/wiki/Signed-digit_representation