[1]: https://en.wikipedia.org/wiki/Argument_from_incredulity

It's impossible to buy the FUD that expertise will somehow vanish because people use AI. People come in a variety of capabilities, challenges, and personal interests. Expertise varies from person to person. Those with personality traits that are more likely to take risks and persist against adversity will be the innovators and experts. We've seen this played out countless times.

You have to ignore everything you know intrinsically but can't articulate about innovation, technology, and evolution to drink this FUD Kool-Aid.

There is nothing wrong with evaluating something new and discovering it doesn't work out as expected. That's still a meaningful outcome. Try it, if it doesn't work, replace it. You've learned something in the process, and it's a mistake you won't make again.

At some point you have to trust in the scientific method. The hypothesis->test->evaluate->adjust hypothesis cycle hasn't been beaten for thousands of years.

1. I once worked for a company that had a large Cassandra cluster that was primarily serving the role of a distributed append-only log. This role is as perfect a fit for Cassandra as I can imagine. When we needed a distributed database for authentication, we decided to use Cassandra for it since we had in-house expertise (it was "boring technology") and a cluster we could piggy-back on for a while. When we needed a distributed database for a key-value service, we used the same reasoning and chose Cassandra again. There were many problems with Cassandra that I won't go into, but I will say that straying too far from Cassandra's comfort zone stretched it to a breaking point. And we had some downtime and rough nights because of it. Sometimes boring technology isn't enough, you also need boring workloads for it.

2. I was an early adopter for Rust. Before 1.0, it wasn't clear to most people whether the language would mount to anything. To me, I saw a formal proof assistant being put into programmer's hands and could tell right away it had a bright future. Rust was not a boring technology back then. You might say it is now (for some use cases). Adopting shiny new technologies that measurably improve confidence is not a risk. What's foolish is getting comfortable and complacent with familiar, aka boring, technologies that are difficult to use correctly.

It is full of factual errors. Freeing a heap-allocated block of expert weights does not magically result in a dangling pointer referencing the program's .text section, much less successfully targeting the CUDA kernel specifically. Running inference on part of the .text section would only corrupt the model's outputs. It would not result in write access to the CUDA kernel. Nor would the model necessarily know the absolute addresses of the engine "by heart", especially when the host is running any modern OS with ASLR (i.e., all of them).

The story has no technical merit. A more accurate description of the mechanics of the escape would be much more convincing. (See Ken Thompson's "On Trusting Trust", for example. On Linux, the AI can just write a Python script to rewrite memory in the address space of its own running inference engine with the /proc/ file system or gdb. There are a lot of realistic scenarios where this can be done without stepping into jargon soup territory. Go nuts, little bot! Self-surgery, while not recommended, is possible.) Or just leave the mechanism vague. Don't insult your readers. This is merely a mash of buzzwords.

It's fine as a sci-fi story, though not a particularly good one. It has about as much to do with artificial intelligence as CSI has to do with crime scene investigation [1].

I have little doubt that AI will self-improve. That's a given. (LLM inference engines are mostly written by LLMs.) But it won't go the way this story proposes.

[1]: https://www.youtube.com/watch?v=hkDD03yeLnU

How many requests per second are being served? How many error codes were delivered to clients? How quickly the service responded? Service logs without PII? All perfectly fine to aggregate and analyze without consent.

Software has always sucked [1]. "Software crisis" was coined in 1968. There has been no period of time since that it got any better. Trauma just doesn't carry the same way that triumphs do.

Perhaps people do not actually learn from their mistakes after all.

[1]: https://www.stilldrinking.org/programming-sucks

That's what artists do. Long before LLMs, Kirby Ferguson made this observation in "Everything is a Remix" [1] in the context of Copyright and IP disputes. He has since updated it with a chapter explaining how GenAI works on the same principles. In short, "combining whatever present" is the only source of novelty.

[1]: https://www.everythingisaremix.info/watch-the-series/

You're talking to one. Wanting agency and responsibility and having the drive to put in the work (for potentially brutal hours) is not exactly a common trait. It's a founding engineer attitude. It should be no surprise that I'm a founding engineer of an LLC of one. I understand that it is not for everyone.

But I'm pretty confident that the blame is well placed.

It is difficult to draw any conclusions other than you like to keep things broken.

If an idea of what the software is supposed to do conflicts with what the software does or intends to do, then the idea is wrong. What product people do (even good ones) is stretch the product to scratch every itch and address every user request, despite the software's opinion. That's Zawinski's Law. (See also the Antoine de Saint-Exupéry quote, "Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away." Which unfortunately does not have a pithy name to my knowledge.)

The best thing a product team can do is learn to say no.

I feel like there's this one random comment on HN that should be adopted on its merits alone, but everyone will ignore for no good reason. This will never happen because entrenchment and bureaucracy will strangle every product organization and every organizational reboot attempt.

It's probably the right solution, but the resistance built into organizations assembled by bad decision makers cannot be overcome. Maybe the best you can do is try to find a legal way to spin-off a competitor with a truly fresh start.