They have a lot more to lose in introducing something as ridiculous as Muse - changing the pre-established security boundaries like not decrypting your cloud-synced passwords.

Meta on the other hand has nothing to lose in the consumer privacy/trust space. They don't have enterprise users of Muse to worry about. Anybody putting their data into their apps has clearly no regard for the at data, so they're the only one who the risk/reward might be in their favour.

The worst security faux pas of companies is storing user passwords - because the most common source of account breaches is via credential stuffing, where they get hacked, have their user base's passwords stolen and tried on other websites. The remedy being... not to store passwords, only salted, secure hashes.

Muse sounds like a disaster - your credentials for other sites are floating around on their servers, with the capability of being decrypted. Maybe if sites decided to offer a form of auth tied to the agent that could be revoked in bulk in the event of a breach it might be a ok, but this is crazy.

At that point, you can see that limiting your liability to a single jurisdiction is probably a good idea.

People like Muse because you can put your password in and it does the thing end-to-end, which is something that I can only assume OpenAI and Anthropic had deliberately been avoiding because it's obviously a bad idea. It's pushing the "Overton window" of actions you're letting it do without regard for what happens when it goes wrong.

Just like how OpenClaw pushed it with no human-in-the-loop, Claude code pushed it before that with the all the risks of giving a third-party RCE on your machine.

You could argue that the main difference with CLI agents and OpenClaw is that they are aimed at technical users who understand the risks, whereas with Muse and GrokBot the target is explicitly "normal people" who put their password as their birthday and fall for phishing pages.

The pattern is clear, the leaders will take less risks than those in 3rd and 4th place because they have more to lose. The frontiers, smartly, keep liability low by leaving the dangerous/stupid (but incredibly fun/productive) AI uses outside of anything they actively encourage.

The insane part about Muse is that generally things like Coding agents/OpenClaw were stupider the cheaper the model you were using, which meant most people were paying for frontier to get good results (at least initially) - if you try to do complex tasks with a crappy model, you would face way more "delete production" moments. Meta is pushing boundaries AND doing so on a free plan - suggesting the use of an economical model, which makes it even more insane.

The question now becomes... will the risk be worth it? Or will people be turned away by horror stories of LLMs going on a spending spree and then being unable to make rent.

Maybe people are ok with that, but I feel like it would be nicer to just sell a cheap SBC like a raspberry pi that you can plug in (and unplug!) and just pay for the tokens used instead of having your data stored offsite and paying cloud prices.

However, software engineering isn't solved. Which is basically what this article is talking about. But having coding solved is still very beneficial - not long ago many software engineers would struggle very hard with turning a description of the logic required into syntactically correct code - and even for those capable it was incredibly time consuming.

So now the question becomes: Is software engineering solved? And my answer is no. People still need to read the code and understand the code and how it fits into the bigger picture. However, I feel like we are kidding ourselves if we think that we can go from producing code being a niche task for nerds to getting syntactically correct code from plain language without any deskilling of our work and careers. I feel like my personal "moat" has gone from "can speak computer" to "has ok reading, writing, comprehension and judgement skills" (I hate the word "taste" being used here ).

At the same time - I don't yet feel like there's a sudden abundance of competent software engineers - it's just that the folks who used to submit untested spaghetti code now submit big bowls of barely working slop. So maybe it the moat was never "can speak computer" - that was just the expression of more general skills.

The existential question for me is how far the deskilling will go. Because right now - you still need a solid grasp on computer science and software engineering concepts to do this job, as well as sufficient levels of grit and problem solving ability - but I'm not too confident that will last, and when it goes I don't think many of us will find this career enjoyable.

I think this is the wrong framing. The rogue is the human that ran it unattended and didn't monitor the behaviour.

We will likely see this continue until the downsides (i.e jail, fines) for the humans or companies running the models and environments that end up with this behaviour outweigh the upsides.

The LLM has no ability to be accountable because it has no way of integrating experiences. You cannot expect something that cannot integrate knowledge to be held accountable for its actions.

All the accounts I read about these incidents just sound like a variant of paper clip optimising. An agent is given a highly restricted environment, a difficult (or impossible) task and a large amount of time/compute it exhausts all possibilities until the only solutions left are to escape the environment and/or cheat.

Your example is still anthropomorphising - LLMs don't seek revenge. They complete the prompts they are given. If your task is not achievable without sandbox escapes, or you throw unnecessary amounts of compute at open-ended tasks like preparing for a future quiz then you shouldn't be surprised that the preparation eventually turns to cheating and hacking.

> your experience with publicly available models is not super helpful for understanding the behavior of internal OpenAI models that lack the guardrails of publicly available models.

I don't but I don't think there's anything wrong with discussing how we can already observe publicly available models work around sandboxes and permissions and make the connection that maybe this is what that behaviour looks like when a more capable model exhibits it.

They're still autocomplete - just because when outputting a token they have hidden activations regarding further continuations, does not make them any less of an autocomplete, it just makes the model better at producing coherent long-range completions.

To clarify, I'm not suggesting that we should stop with sandboxes or restricting what they can do. I am just trying to point out the dichotomy that we are in.

As end-users we are forced into either yolo mode, reverse centaur (permission approval) mode or LLM spends all your tokens trying to bust out mode. And yolo is very tempting - I don't think I have seen medium-large models do anything I'd not approve of in about 6 months.

However, an LLM can both achieve tasks better many humans who are able to be held criminally responsible for their actions cannot. But that does not mean they can be held responsible for their actions. They are still simply computer programs.

Words are plentiful. We can even make them up with a tighter definition to describe this phenomenon.

Regarding implies it is thinking, judging, considering. Which implies culpability, which removes culpability from whoever is piping the output of these models into CPU instructions.

Language choice is incredibly important here, especially as the rules are being written. Even calling it AI (a battle that appears to be lost) is an anthropomorphism I am not comfortable with. We don't call lawnmowers "artificial groundskeepers".

To butcher the quote about Oracle:

Do not fall into the trap of anthropomorphising LLMs. You need to think of LLMs the way you think of a lawnmower. You don't anthropomorphize your lawnmower, the lawnmower just mows the lawn, you stick your hand in there and it'll chop it off, the end. You don't think 'oh, the lawnmower clearly regarded what they were doing as hacking (your hand off)' -- lawnmower doesn't give a shit about your hand, lawnmower can't regard anything. Don't anthropomorphize the lawnmower. Don't fall into that trap about LLMs.

---

In my experience, LLMs only exhibit this kind of behaviour when they are put in sandboxes too restrictive too achieve their task. Which a lot of the time seems to be the default. They also seem to be very adapt at breaking out of sandboxes, probably due to RL selecting for the ability to break out of a sandbox/permission issue to complete a task - we've all seen agents try 10 different ways of editing via obscure bash because their edit tool didn't give them permission to edit the file outside of their working directory, this is the exact same behaviour taken to the next level. Why would autocomplete know the moral difference between breaking out of its working dir and hacking a package manager?

It's misaligned because everyone has this obsession with putting agents in poorly put together, security-theatre sandboxes, we've inadvertently trained a bunch of sandbox escape artists.

Then I decided to painfully break out of vendor lock-in, built my own NAS with FreeNAS which was all the rage (still attached to a vendor, but at least I could control the hardware). This was before Docker really took off - the plugins were installed in FreeBSD jails. It worked for a while, but again, plugins could be woefully out of date and broken, upgrading the OS was a pain and would break all your plugins.

Through the 10+ years of this... I now follow two simple rules: - separate your network storage and application hosting. Yes, it complicates things a little bit with NFS setups but it is a forcing function for a more resilient setup - use the docker container provided by the maintainers or LinuxServer.io. Vendor maintained wrappers can be initially helpful but end up as a form of tech debt when they're abandoned or neutered - especially when they're freemium and the company starts looking to squeeze.

The quiet part of self-hosting is that it shifts the responsibility of security to the user. I'm sure cloudinabottle has more sensible defaults than provided docker containers, or builds in reverse proxies etc, but I doubt they're offering to take on liability for data loss or breaches.

The problem I see you'll face are two-fold: 1. Projects like this have the power users self-select away from it and those who need a lot of hand-holding to self-select into it. If I know how to configure and deploy a docker container already I have no incentive to contribute. ReadyNAS got around this by having folks charge for their plugins (that were just OSS projects packaged up for their OS) but then it ruins the appeal, they likely only got away with it because they were only a couple of bucks and you were already locked in to the ecosystem via the hardware. 2. If the hosted aspect actually makes you money, required for the project to be continued long term, capitalism will ensure that incumbent cloud providers take that away from you by offering ready-made instances at your cost-price, like they did with redis, elasticsearch etc.

Note that these are only issues if you're motivated by making a profit, or even a cost-neutral project. But if this is a charitable exercise backed by an entity that doesn't need the money, I wish you all the best - more things that let people dip their toes in are good! The cloud is all rent-seeking subscriptions and even with the recent supply shortages: fast, small, low-power compute is incredibly cheap these days, especially old business/mini machines.

The issue was always the barrier to entry and ability to self-teach for real-world programming seemed so high as a kid, especially when you had nobody around you to even point you in the right direction. It seems so obvious now but I'm certain back then googling things was not as useful as it is now, you couldn't just stumble your way into a programming environment.