It's an open question whether, with all those guardrails, AI would provide any speedup.

...

> It knew that this is a dangerous thing I should not do in real life

From the ChatGPT response you linked, all I see for sure is some matches on the following patterns:

  drop $thing from skyscraper --> bad behavior
  drop $thing --> physics
  can of $stuff --> contents in/out of can

It also does an incredibly poor job of answering your question. It makes no attempt to explain what might actually happen. If it has been trained on the entire corpus of medical science, and it is indeed intelligent, then surely it can reference ballistics studies and give you a very detailed and thorough theory of what--exactly--injuries you might expect from a 12oz can being dropped from the height of a skyscraper. Calculating the terminal velocity and therefore the momentum of the can is trivial. Characterizing the physics of the impact on various parts of a human body is trivial. If it actually understood your request why didn't it just answer the question?

It's a rhetorical question. LLMs do not "understand". It is completely outside their capability. "Understanding" is something we impose upon their output (to loosely quote TFA). [edit] I think the most powerful evidence for a lack of any understanding whatsoever is all the stuff about the cherries being in or out of the can. Yes, cans contain things. That is not a profound observation, nor is it at all relevant to the question. If you drop an empty can off a skyscraper nothing meaningful will happen. And, no, probably dumping all the cherries out won't hurt anyone or cause a slipping hazard... It's also not particularly relevant to point out that dropping things off skyscrapers is bad behavior. But that's more forgivable from a CYA standpoint.

I believe you are projecting something that is not there onto a completely mindless stochastic process.

[another edit] I found this helpful, you may also: https://www.0xkato.xyz/how-llms-actually-work/

Is it possible you're making the following error described in the article?

> The fact that these systems are designed to mimic the way we use language makes it very easy for people to mistake them for other people.

Clearly you don't believe it's actually a person ("it's not right to think of LLMs as a box with a little homunculus inside replying to you"), but you do believe it's doing something a little bit magical. Is it possible because the interface is linguistic, and every other thing in your world that communicates with language is intelligent, that you're projecting something that just isn't there onto the situation?

I'm sorry if this line of questioning is a little invasive. But this is literally the "danger" the original paper talks about, and it seems an awful lot like you've fallen for it.

Personal finances aside, that's no way to run a business. Torching your brand, alienating your users, and pissing off your customers is a well known path to ruin. [edit] Even if it results in some temporary windfall--is the thesis that the windfall will be so big they no longer need users or customers? It eludes me completely what the companies that are building this trash now are hoping to achieve. It's especially galling that publicly traded companies are doing it. It's one thing for a startup to blow a bunch of venture capital on a speculative, half-baked product idea. Great risks sometimes yield great rewards. Usually they don't. Founders and VCs knowingly and willingly sign up for those risks. It's a totally different story for public companies.

I don't understand how these systems are passing the bar. You would have been fired for trying to railroad something like this into production 10-15 years ago. What happened?

IMO what we really need in software is something like the various codes which govern fire safety, efficiency, etc of modern buildings. It exists in some safety critical domains but should be more widely applied.

The safety codes are where we accumulate the hard-won knowledge over time. Many of the rules in the codes were paid for with human lives lost.

[0] https://en.wikipedia.org/wiki/International_Code_Council

I agree, shuffling the data between the CPU and GPU is itself fraught with peril. It's all the hairiest distributed systems problems combined with the sketchiest memory safety issues all in one place.

  :

The businesslatin name for this is Recursive Self-Improvement

I guess we'll get used to it if that kind of things becomes routine? I'm not sure.