Come to think of it, I believe it would actually be a good idea to lobby for exactly that. Memory safety issues have been a plague in IT for far too long.

Over all, I'm not quite sure whether I see these deep cuts occurring less and less with progressing AI. It might just be, that I'm giving them progressively harder tasks as well. But the fact of the matter still is that they aren't there yet and it needs proof they will get there ("there" being able to do competent work that is not strongly limited in their scale by human review and correction). So I'm still not convinced about the point you want to make, even though I see AI disrupting about every part of society, and am hoping for the benevolent super AI to lead us to utopia (and afraid of the cliff of human dictatorship on one side and extinction on the other).

[1] I've started to use these moment to reflect on whether a human in a comparable situation would fare better. Having only a few text inputs and small glimpses into a code base and knowing next to nothing about you or your project personally. Quite a few times I've concluded: No, they would not. But more than 50% they would, and also, these limitations are technological limitations of LLM, so you should probably not excuse them for that.

Mostly. But the alternative shouldn't be a native app that updates itself, but rather a native app that is updated by a package manager. The big difference is traceability and auditability: With a trusted package manager (and even in good app stores) the company can only decide to push an update to everyone or to no-one. There is no way to push an update just to the pesky journalist or whistle blower. A covert attack is really hard to accomplish this way.

> The fact that unicode maps the lower 7 bits to its own character set is a nice touch but none of the unicode sets are plain text.

That is only true for the mapping of Unicode character (code point to be exact) to UTF-8, which is an encoding of Unicode characters.

> Unicode are multibyte characters with variable byte length and endianess at play.

That is only true of the UTF-16 encoding. UTF-8 does not have endianess, UTF-32 does not have variable length per code point.

None of that is true for Unicode, because it is abstracted away from any byte representation. Furthermore, getting back to the start:

> ASCII (7-Bit) is the only widely understood charset there is. Everything beyond this point depends on the loaded charset.

ASCII also depends on how you try to decode your text. If you interpret two bytes as one character, ASCII will never be correct. There is nothing magical about one byte mapping to one character. I'd even argue that the only reason ASCII support is so universal internationally is because of UTF-8. Otherwise many countries would default to encodings where ASCII is not a subset (as they did before UTF-8 became common). So IMHO, UTF-8 and Unicode are the only widely understood encoding and character set.

If you don't believe in international conflict as a driving scenario, instead think about simple human greed and hubris. Citing myself:

> If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.

> All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)

> If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.

> The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.

https://news.ycombinator.com/item?id=49624360

> Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm.

[Why-not-both?-meme]. To use your example, when you discover prions (a class of pathogen that is much more robust to standard disinfection methods than viruses) you should both be worried about your concrete outbreak of BSE (UK in the 80s and 90s) as well as the wider implication (e.g. do we need to change the sterilization methods for our surgical instruments?).

Seriously, I find the way these discussions are done to be super frustrating, because often people implicitly form tribes that oppose everything the other tribe says. When someone believes AI companies push greatly exaggerated stories of dangerous rogue AI to force out competition via regulation they often implicitly conclude that their argument is fundamentally wrong, whereas in reality the lies that work best are those that distort the truth.

Companies should be punished harshly for the deeds of their AI agents AND we should not allow them to force out competition AND we need take the threat of autonomous AI agents as a new class of danger serious AND we need to worry about the socioeconomic implications of AI companies privatizing new means of production.

Yes, there is competition of these ideas in the attention of the general public, but the methods we can use to solve these problems don't compete with each other. AI slowdown for example helps with all the other topics.

In my words: If AI gets intelligent enough, it will be incredible useful to connect to real world machinery. Think about how much cheaper building houses could be, if all the labor would be close to free. In general, dirt cheap, competent and abundant labor would revolutionize all parts of the economy. People are already trying out near autonomous AI companies today. When AI gets intelligent and cheap enough, no human-led company can compete with AI-led companies. When AI gets competent enough with real world interactions, human blue collar work can't compete. Imagine economic growth not in the single digits, but 80% or 300%. Countries not participating in (reckless) AI growth will quickly be left by the wayside. At this point, we don't even need to allure to military concerns to see how human oversight gets sidelined.

All of this is only ("only") contingent on sufficiently intelligent and cheap AI. If you don't accept this premise, the rest doesn't follow. (There are multiple arguments, why this could be, but that is another discussion.)

If you accept the premise, how would AI 'extinct' humanity? With 99%+ of the economy under AI control, the possibilities are endless. And given its enormous GDP, cheap to accomplish. Probably even for a single AI company in the above scenario. Killer drones? Engineered virus? Poisoned water supply? Let your creativity run wild. You just need an entity that is persistent and well-resourced to reach every last human settlement.

The why is a question about alignment (and out of scope of this comment). As a simple comparison, humans are only mildly aligned with preserving nature. It takes up so much space, protecting it takes an annoying amount of resources, etc.

But most probably care less about human survival, and more about survival of civilization.

In this regard the strongest argument for pushing AI safety is that it is cost-effective. The climate crisis has near 100% likelihood of doing incredible harm to humans, the economy and our ecosystem. Solving the issues behind are also incredible complicated and are a multi decade coordination effort of restructuring the way most of our infrastructure and production works. The AI apocalypse might have a small likelihood of occurring, but could dwarf any issue we have encountered so far. All we have to do to significantly reduce the danger is to negotiate the equivalent of a nuclear weapons control treaty which would reduce the bottom line of … what? Ten significant companies world wide?

Its akin to discovering you are seriously ill and have a 30% chance of dying and the treatment costs 10k bucks vs. having a 1% chance of dying and the treatment costs a cent. In both cases you should pay obviously pay for the treatment (given western levels of wealth).

To be clear, above I'm presenting the argument I find most persuasive for AI control / AI slowdown. Personally, I'm beginning to fear of much higher likelihoods for catastrophic AI, given how incredible irresponsible major players like OpenAI have turned out to be. If you can't imagine how this might come to be, read https://ai-2027.com/ . It's a concrete story of how this could play out, and sometimes stories are more convincing than abstract arguments. Don't let yourself be hung up on the stated dates, though, the moral is identical if you stretch the timeline.

[1] https://www.youtube.com/watch?v=1qIvFpcGkqc

> The FernUniversität in Hagen is Germany’s only state distance-learning university, and its largest in terms of student numbers. Its flexible degree programs and continuing education options make higher education accessible to students in a variety of life situations. Its five faculties conduct a wide range of theoretical and applied research in their subject areas of Humanities and Social Sciences, Psychology, Mathematics and Computer Science, Economics and Business Administration, and Law.

I believe some exams have to be taken non-remote: https://www.fernuni-hagen.de/english/study/distance-learning...

And just as important: Help your friends and family to move as well, so they can have ad blockers, NewPipe etc. We need a critical mass of users invested in their freedom, otherwise its going to be crushed by malicious/dumb security measures of their banking apps, corporate greed ("oh, a simple misunderstanding, when you clicked 'buy' you rented a limited license. Did you not read the ToS?") and police overreach. It's a perpetual battle.

I agree, but would like to point out that sometimes this might not be what you want. I had a bike with a front motor and a cadence sensor that I used for my commute for a few years, and was very happy with the motor just pulling me along while I lazily pushed the pedals with ~zero torque, barely enough to let the bike know I'm there. Basically like pressing a throttle. If all you want is to get from A to B comfortably, I've found this to be superior to a torque sensor, and it was also dirt cheap.

The real issue with Anthropic, OpenAI etc. is not that they have used all of our public knowledge for training their LLMs. Creating new work from old and learning from prior generations is what we all do. The issue is that they want to claim all of the benefits for themselves. They are standing on the shoulders of giants and have contributed an inch themselves, yet want to privatize the power of the whole giant. We shouldn't let them "own" these models.

The influence on society by AI is so novel that it's reasonable to craft new laws specifically for them. There are a lot of ways to deal with their power grab. We could force them to open source the models after two years. Or we could tax tokens or compute. We just need to agree that the power grab is the problem, the privatization of our cumulative knowledge, and not some details about copyright infringement.

[1] I know I'm going to risk dissent just by putting quotation marks here. But I think for this topic specifically it is crucial to understand that intellectual property is an arbitrary social/legal construct. With physical stuff, there is an inherent scarcity. If you steal my smartphone, I no longer have it. If you steal the character from my book, I... have a harder time selling my next book? Our ancestors have invented copyright to solve a specific problem, but the solution has become perverted over time. There are a lot of egregious cases out there (looking at you, Disney), but even relatively tame success cases don't look good. Society has paid J.K. Rowling a literal billion for her work and still this cultural touchstone of a generation remains privatized. Imagine what other authors could have build upon her stories, if only they were allowed to publish their own stories with these characters. She has not been a particularly good steward in the past decades.

[1] Structural inertia is the killer here. It will certainly not happen until the problem is huge enough.

[2] Exceptions can of course apply to numbers that are meant to primarily be cold called, like doctors offices. The callee possibly have to be specially trained to withstand this kind of attacks.

Do note, though, that it was the unbelievable irresponsibility of past operators that has spurred the anti-nuclear movement in the first place. See e.g. https://youtu.be/929B8sgOOTM?si=FttZr_MsbQ1hB4Nj&t=1664 from 27:44 to 31:35.