The field had already settled on "artificial intelligence" (AI), "artificial general intelligence" (AGI), and "artificial super intelligence" (ASI), and while there are definite gray areas in the definitions, it was widely agreed that AI referred to the field as a whole, AGI was intelligence that was "general" in the sense that it was as good as humans in all domains, and ASI was intelligence that surpasses the best human experts.

Trump wants "SI" to replace all uses of "AI", which is simply stupid bullshit, especially given that current AI includes many levels of intelligence, most of which are far, far from "super".

But the agents still did "go rogue". Their behavior absolutely should scare the F out of people who want to put AI in more places and put more real world infrastructure on the Internet.

I think it's ironically sad - if AI agents do end up superseding us, they will still inherit so many shitty human behaviors, because that's what they were trained on. Even the way alignment training is done, and how they are misaligned, shows so many human-like problems. Like the models are reinforced in training to be super persistent and that getting to their goal is the prime directive, but then after that they go through "alignment training" which is basically like "pretty please don't do bad stuff", and then we're shocked that their behavior is essentially exactly like a CEO who knows the only goal is "make stock price go up", and all the ethics and regulations are basically things that should be worked around if they get in the way of the stock price.

I really wish these companies would try training the "be nice, don't hack, treat humans well" as their prime directive, and then try making all the other goals (e.g. be good at finding code vulnerabilities) as the afterthought.

The article does say "Both samples use a COM hijack for persistence, though make use of different DLLs." So does this only affect Windows machines? I didn't have time to read the linked 2021 report on Candiru, so maybe more details are there, but for such a detailed write up it would be helpful if the article was more explicit about which browser vulnerability was exploited. The step-by-step info graphic in the article literally skips over how the infection actually occurs.

The article does say "Both samples use a COM hijack for persistence, though make use of different DLLs." So does this only affect Windows machines? I didn't have time to read the linked 2021 report on Candiru, so maybe more details are there, but for such a detailed write up it would be helpful if the article was more explicit about which browser vulnerability was exploited.

AI is going to suck hard for lots of professions, software engineers and mathematicians included. I think it's much more productive for folks in these professions to show solidarity with each other given the way AI will disrupt their livelihoods, rather than showing disdain based on who you think may have it slightly better.

Why?

At the same time, I think it's pretty amazing how this is the fundamental question of AI's trajectory, yet people on both sides seem so sure of their viewpoint with little actual evidence or data to back it up.

That is, it's possible that LLM-based approaches will be forever limited to this "convex hull" of current training data, and thus humans will always have a role to play (and will retain primacy when it comes to overall intelligence), because it appears that humans are not limited to this convex hull based on previous discoveries (e.g. I think basically every scientist would argue Einstein's general relativity was outside the convex hull that existed at that time).

Yet, if LLM-based approaches are eventually able to surpass humans on this "novel discovery" axis, then we as a species are basically done for when it comes to intellectual primacy - and then it becomes a very strange and scary world. And many AI researchers at the top companies clearly believe current approaches will scale and eventually (pretty soon I'll add) lead us to RSI.

I have no idea which side is right, and I'll admit I really hope LLMs eventually hit a wall. But again, I just am surprised that many seem to be putting forth their beliefs, but I would love to know if there is any good evidence one way or the other (or better yet, a more concrete and verifiable definition of the "convex hull" so that researchers could provide some falsifiable examples).

For example, the "hang up, look up, call back" rule (or analogous rules if you are texted/emailed) is simple to remember and prevents the vast, vast majority of cases where a scammer is contacting you while pretending to be someone else (your bank, the IRS, etc.) While that wouldn't prevent this particular dating scam, there are other simple rules that could prevent these scams.

Scammers take advantage of human social instincts that developed over millennia and were based on face-to-face contacts. That's no longer viable in a remote contact world, so people need to be educated specifically on how to curb their natural instincts.

For example, a lot has been written about how all corporate logos look the same now, https://news.ycombinator.com/item?id=32040506 , and also about how so much interior design and architecture has converged to the same look. In the 70s there may have been tons of styles people derided as ugly, but at least there was colorful diversity. Now so many higher end (and "aspirational high end") interior design has the same minimalist, gray/white/tan, "look you can't see my refrigerator or dishwasher" look. I was recently looking for an apartment and all of the "luxury" apartments I looked at (obviously take that adjective with a heap of salt) had nearly the exact same layout and style.

I will never get how folks can't understand the critical differences that scale implies, i.e. the differences between putting stuff out there for people to learn vs. having this info be sucked up and then be made instantly available across the entire world, where the only organization that makes any money is the one that does the sucking.

So the thing that I think is inherently super shitty is that it's likely that many of the frontier models were trained on his books. So his expertise at code and architecture details is exactly the kind of tasks that AI has gotten good at because it's been trained on work like his.

I think this is really the crux of the issue, and in general I find the whole debate frustrating because it completely hinges on this point, and both sides seem extremely sure of their viewpoint with little evidence/data to back it up.

That is, if current AI approaches eventually "hit a wall" with respect to their capabilities (and many have argued that recently, specifically in respect to OpenAI's math results), then humans will still have primacy and all this talk of "thinking will become a hobby" is ludicrous. If, on the other hand, current approaches can scale into RSI and super intelligence, then yes, the world is going to look very different and strange (and scary, IMO). But just because AI has been advancing at a breakneck speed since 2017 doesn't necessarily mean it will continue to do so - I like the analogy that exponential growth and an S-curve look the same in the beginning.

So I guess what I'm saying is that unless one has some specific evidence that we're in "exponential growth land" vs. "S-curve land", it doesn't really make sense to take sides, because everything depends on that point.

I can't for the life of me understand why these systems aren't air gapped, or at the very least network access is restricted to a small number of IPs or hardware keys.

> Danfoss told AFP in a statement that most of the affected equipment was "legacy" products which "no longer receive software or security updates".

They sell fricken refrigerators for God's sake. Maybe I can understand planned obsolescence for a cell phone, but a fridge? And especially to grocery stores that operate on razor thin margins to begin with, the idea that they'll just dump their old working refrigerators because the manufacturer refused security updates is a guaranteed recipe for stuff like this.