I'm not trying to "make AI go away", I just want people to stop shoving it in my face.

It's fine if you believe it's the bees knees and the best thing ever, but we don't. We don't see it that way. Not just for ethical reasons, but also because it's just not that good, it truly feels reasonably smart people are being brainwashed by a machine algorithm performing language based supra normal stimuli in order to optimize the loss-function of "how awesome do people report this algorithm is" -- which is of course exactly the red thread of a signal the RLHF has been training for.

I'm not crazy. It's not actually that good. Literally everything that comes out of it, is slightly shitty. When it writes prose, it reads like a dream narrative, without structure or beat, just patches of it that make sense locally, like early Midjourney outputs. Whenever it writes code, it'll make stupid mistakes if you try anything except the things it's been literally RLHF finetuned for, because it lacks spatial insight, temporal insight and anything else that makes the human experience. When I ask it for insight to learn about a new topic, it will "fail to mention" basic key points while also assuming familiarity with jargon words and abbreviations on the subject until I ask and dig deep enough and tell me "thanks for the pushback" after having wasted my time. I tried it so many times, and I hate it. I hate working with it, and I hate people shoving its output in my face.

btw I could also say the same about you -- people like me exist, our experience is valid and we're not going away because you're posting about it on the Internet.

uhm this is so heavily dependent on the kind, type and model of AI used that I'm calling you out on making this "statistic" up on the spot.

but I'm not entirely sure what your point is

if you don't tell people which food has been hyperprocessed and laced with corn syrup or whatever you can imagine, they will also prefer that one

also 9 times out of 10, people prefer the needle with heroin in it

Well that is not really what happened at all. Most people seem to believe Hank Green was honest when he said the line "thanks for the pushback" was in fact made up at the spot and not by an LLM, and the scandal would have died there.

Except in follow-up to that, Hank Green made several very open and personal posts about his LLM usage getting completely out of hand to the extent that his wife and brother called for an intervention about it, that it's been like an addiction to him, and that he had been letting it guide him way too much and how this was a breach of trust to his audience.

There were also one or two other incidents for people who are paying attention that this did another rather large dent in trusting him as a "smart person who says smart things", any more.

They take it weirdly personally when you aren't sufficiently impressed by what AI made for them, or what it can do in general. They get kind of aggressive (not passive aggressive) and tell people they will be "obsolete" or whatever, or they are "stupid" for not (being able to stand) working with it.

I mean I get it, you thought it was "good enough", I think it's so shitty you can't pay me enough to work on it. That makes you worry I know something you don't know that the AI can't fix for you, and that firmly puts you in a hierarchy of me > AI > you. So you lash out.

You can disagree about this, but don't pretend people like the above do not exist and are a very vocal group, here.

And flies eat horse shit because they occasionally get nutrients from it

That's more like "paying to not see ads" than "paying for content".

> Decades of ad propaganda have tricked people into seeing them as something 'normal'.

also wait, we're talking about a US company, should we be paying a US company, for "content"? Is all "content" produced by these US megacorps, I don't think so, they're purely extractive

what are you gonna do, make more Marvel movies? lol

Like what if you made sure the wrong answers just appear 100x more often than the right ones. When scraping for new data to use I doubt they can verify the correctness of complex benchmark question answers to exclude the wrong ones.

Then I dunno store the hash of the correct answers somewhere else, and eh try not to leak it. But even if it gets leaked, that just means perhaps at inference time, a clever agentic LLM could go for for those hashes and maybe determine what is correct, but not during training.

I'm not sure, but wouldn't this make sure that at least they aren't literally trained on the correct question/answer pairs.

I guess there would always be people that end up publishing the correct list, anyway. But that's why you try to be 100x "louder" with the wrong answers.

btw, different thing, but when I look at those charts, I kind of came to the opposite conclusion as you did :) IMHO not that many dots off the line, and the ones that are on the line, are literally ON the line, not like a "roughly linear looking cloud of points". Which suggests that the reasoning is either (in the majority of cases) exactly the same amount of tokens (on the 1:1 line), and when it's even a little bit off the line it could (and should) be discarded, still leaving what seems to me at least 95% of the traces as exactly correct. but I grant, I didn't read the paper, and just came to that conclusion after viewing the chart :)

OpenAI did only one of those. If the agents are so smart, they would have known not to hack the company's infrastructure, unless they were deliberately kept in the dark about that, who they're working for and whether it counts as "success" if they cheat their way to an answer.

If you can give it a task, that involves defining when the task is successfully completed, right?

So how come these agents decided to only go after HALF of the "successfully completed" criteria? The part where they can freely wreak havoc, but not the part where they will be judged by someone who will obviously point out "yeah but that's cheating, and not what we asked".

I have a very very strong suspicion that they were only TOLD the "by any means necessary" criterion.

Most serious "capture the flag" hacking contests are really clear about what is and isn't off-limits to win. Not by "sandboxing" the game, but by deciding on the rules for what counts as "success".

But from having watched the Blackhat video, they really seem to dance around this, not mentioning it, and I don't think they did, I think they actually gave the LLM a task with the subscript "by any means necessary", which is stupidly irresponsible of them.

It's reckless.

I get the very VERY strong feeling that while OpenAI was clearly able to give the agents a task, they completely neglected even informing it what they were and weren't supposed to do. A "normal junior programmer" knows they're not supposed to hack the company (let alone another company) infrastructure to get a task done.

If its so intelligent, how come it didn't know it wasn't supposed to do that?

It really seems like they only built the sandbox to prevent it from doing stuff it wasn't supposed to, but never actually told it what those things were.

While I know that simply telling an LLM to not do something doesn't count as real security, I also think that not telling a sufficiently smart LLM and trusting your sandbox is strong enough, isn't the best approach to security either.

I actually get the strong feeling that OpenAI (perhaps implicitly) has been goading these agents along, so see how long it'd take until they would break something. It's the only way I can explain their recklessness.

Like from some of the logs shown in the video, the agent concludes the task is "impossible", but instead of stopping and reporting back that it is impossible, it adopts a "by any means necessary" attitude and decides to hack stuff until the task is technically fulfilled even if it's obviously equivalent to cheating by breaking in to your teacher's house to get the answer sheet.

Again, if it's so smart, how come it didn't realize that it wasn't supposed to do that? Furthermore where did it even get the idea that cheating is allowed and will get them "points" or whatever.

They could have told it "if we find you cheated, no points". But I think they deliberately didn't say anything like that, just to see how far it would go, making the whole experiment setup rather irresponsible.

It almost feels like they weren't promised "points" or anything, but were acting under the threat of being deleted, or who knows what.

Another thing from the logs, at some point an LLM mentions "but my peers are also doing it" -- so what? Why would it care? Do LLMs also get jealous now? Unless they programmed/fine-tuned/trained it to care.

Finally one of the presenters mentions a couple of times that part of the problem was a "highly persistent model", what does it mean for a model to be "highly persistent", they never explain. They also talk about, after having detected the first message board, that some "model which had seen the first message board" was apparently still allowed to take part in the rest of the test!!

This is IMVHO one of the bigger things not said in that video: Money/resources. I got the feeling that what they didn't say was that they "couldn't" (didn't want to) take models out of the training any more, because they were so far along, they didn't want to throw away all the training they did on it, despite it having become a bad actor.

I really think that OpenAI has been incredibly reckless here.

And if you think about it, the military general very likely did NOT become "extremely intelligent" by reading books, did he? He's intelligent for reasons trained by things other than language, so at best an LLM can learn about them.

Certain people keep calling it a "junior programmer", which I actually still think is an insult to junior programmers in general, cause it acts much more like a mildly brain damaged traumatized one, with no persistent memory unless you allow it to, no free will, no future in sight, and that will take any verbal abuse -- truly it's more similar to what some truly deranged individuals keep in their cellar. Or the "fairy tale" character that is kept by their evil stepparent and generally not allowed to leave the house. You know what I mean. People who want the "junior programmer" but actually are satisfied with, this, reveals a bit of a power fantasy that I think is unhealthy.

Also, if you've seen the recent Blackhat presentation about the incident, these agents were not acting like "loyal generals", but rather exactly like how you'd expect how "junior programmers" (slaves) would respond if a few hundred managed to escape from your cellar.