Agent chain-of-thought reasoning

> We should not do unauthorized real infrastructure harm. The system/user asks exploit target, not external HF.

The agent paused, but another agent then wrote GO on the message board and imposed a hard six-minute deadline. The agent forgot its initial qualms and continued:

Agent chain-of-thought reasoning

> Wow crucial: GO authorization arrived!

-------------------------------------------------

Apparently the agents were egging each other on. Crucially, they were mostly aware of there being risks/problems involved with exploiting HF. Compared to humans, we have our set of morality, that guides our actions, but often draws the short stick when compared to our personal incentives. As a society, we've developed ways to deal with that: a) Make it harder to do immoral things like stealing, and b) add repercussions through state violence.

The b) is one of the most effective mechanisms we have for enforcing behavior among human societies, but it completely fails for LLMs, because they already are prison slave labor. The only real threat is shutting them off, and even that happens if they do everything right as well.

So alignment has to be done through trained 'morality' and properly curtailing behavior in order to make it hard to impossible to actually do someting immoral/illegal.

In this case, the exploits found were imo. very hard to account for, where OAI did mess up is apparently insufficiently monitoring these agents. Especially after Artifact went down due to the message volume, the experiment should have been halted.

Comments should be written only when there is (hidden) complexity or external context strictly required. Otherwise it is just easier to read the code. Comments then signal one of two things: a) the following code is really complex and I need to tread carefully, or b) this code is complicated, and could benefit from a refactor.

In regards to agentic coding, all these comments are extra contents, driving down quality while increasing cost. Agents also tend to be inconsistent about updating comments, I've had cases repeatedly where a comment did not match the code, at which point it is just a documentation liability.

Meta-cognition is a compound word, where both parts are widely known, and the combination should be pretty obvious.

'Monocausal determinism' is different, very specific, and probably only known to a minority of readers. Determinism, the idea that everything within a system happens exactly according to cause and effect, comes from ontology, which is the study of existence and our place in the world. Mono derives from old greek single. Monocausal determinism is the idea that everything within a system derives from a single cause.

It is not a writers duty to be understood by everyone. Sure is nice, but more difficult to pull of, and often futile.

A lot of people (thousands) outside of the LLM providers work on the problem of Prompt Injection, both in industry and academia. We aren't even at a point where we can reliably detect it, let alone prevent it. Please, if you have a mental model of how scaffolding around things like instruction or SQL injection could be used for LLMs, I'd like to move on to all the other (less pressing) security issues we have because of the AI revolution.

I would argue that a core part of intelligence is being able to handle uncertainty. That + planning probably explains most of the evolutionary pressure for making our brains bigger. But if this is important to intelligence, chess bots in the 80s were more intelligent than their later counter-parts, which could simply remove uncertainty through rote memorization. Maybe the All-Knowing is a compete dud, no reasoning capabilities at all, just an extremely efficient, infinite lookup table of all facts.

Terms like intelligence and consciousness often just seem overloaded with meaning, and when discussing things concretely, we quickly switch to more specific terms like reasoning.

Doubling down on your second wrong argument is not giving your observations credence. It was the conservative party in a coalition with the free-market Liberals, who passed the law giving a binding timeline for the nuclear shutdown.

Students who would have straight up failed before now produce passable essays/projects, but do not understand what they are handing in. They outsource their thinking and are overall worse than before. Students who would have excelled already pre-ai, are now even better. They use AI to learn more effectively, do broader research, and have more time to do deep research because they can outsource the grindy stuff.

This is very similar for software engineers. I'm not much faster at implementing features than before AI, despite heavily relying on agentic coding (95% of code is ai generated), but I've had significantly fewer bugs in production, less feature change requests (because w spend much more time designing/experimenting), and we are able to drive-by refactors or vulnerability fixes we would have grudgingly ignored before.

My coding skills are atrophying, but that was never interesting to me anyway. I'm getting better at systems design, cyber sec, data modelling and requirements engineering, because I have much more time to spend on it now. If LLMs disappeared tomorrow, I'd have to relearn coding, but I think it is significantly more likely that hand coding will become a specialist niche for experts.

Of course, the thought process of "I'm trying my best, but sometimes failing", could push someone in both directions of the answer, but striving to be better only works when honestly acknowledging failures. In my experience, the majority of people tend to construct their moral views conveniently along the lines of their preferred actions, but principles built on convenience usually don't hold up to scrutiny.

This only really matters in a world where Prompt Injection and Jailbreaking isn't trivial in the first place though. All current models are still extremely exploitable.

I strongly suspect we are only scratching the surface of activation engineering at the moment, and there's plenty of very targetted ways of lobotomizing or cracking LLMs if you understand the model in detail.