Hacker News: Turn_Trout

New comment by Turn_Trout in "System Card: Claude Mythos Preview [pdf]"

Turn_Trout — Wed, 08 Apr 2026 15:31:52 +0000

I agree that they called many things remarkably well! That doesn't change the fact that AI 2027 is not a thing which happened, so it isn't valid to point out "this killed us in AI 2027." There are many reasons to want to preserve CoT monitorability. Instead of AI 2027, I'd point to https://arxiv.org/html/2507.11473.

New comment by Turn_Trout in "System Card: Claude Mythos Preview [pdf]"

Turn_Trout — Wed, 08 Apr 2026 01:27:13 +0000

AI 2027 is not a real thing which happened. At best, it is informed speculation.

Automatic Alt Text Generation

Turn_Trout — Sun, 23 Nov 2025 00:23:32 +0000

Article URL: https://github.com/alexander-turner/alt-text-llm

Comments URL: https://news.ycombinator.com/item?id=46019580

Points: 1

# Comments: 0

An Opinionated Guide to Privacy Despite Authoritarianism

Turn_Trout — Wed, 29 Oct 2025 18:14:31 +0000

Article URL: https://turntrout.com/privacy-despite-authoritarianism

Comments URL: https://news.ycombinator.com/item?id=45750849

Points: 14

# Comments: 0

New comment by Turn_Trout in "Persona vectors: Monitoring and controlling character traits in language models"

Turn_Trout — Mon, 04 Aug 2025 22:04:26 +0000

No one has empirically validated the so-called "most forbidden" descriptor. It's a theoretical worry which may or may not be correct. We should run experiments to find out.

English Writes Numbers Backwards

Turn_Trout — Fri, 25 Jul 2025 23:04:52 +0000

Article URL: https://turntrout.com/english-numbers-are-backwards

Comments URL: https://news.ycombinator.com/item?id=44689547

Points: 3

# Comments: 0

New comment by Turn_Trout in "Chain of thought monitorability: A new and fragile opportunity for AI safety"

Turn_Trout — Fri, 18 Jul 2025 18:00:05 +0000

As someone who did their PhD in RL and alignment, it was not obvious to me a priori if, or when, or how badly obfuscation would be a problem. Yes, it's been predicted (and was predicted significantly before that Zvi post). But many other alignment fears have been _predicted_, and those didn't actually happen.

I don't think the existence of specification gaming in unrelated settings was strong evidence that obfuscation would occur in modern CoT supervision. Speculatively, I think CoT obfuscation happens due to the internal structure of LLMs and it being inductively "easier" to reweight model circuits to not admit wrongthink, rather than to rewire circuits to solve problems in entirely different ways.

New comment by Turn_Trout in "Guess I'm a rationalist now"

Turn_Trout — Wed, 25 Jun 2025 19:20:45 +0000

> The #1 comment says that the rationality community is about "trying to reason about things from first principle", when if fact it is the opposite.

Oh? Eliezer Yudkowsky (the most prominent Rationalist) bragged about how he was able to figure out AI was dangerous (the most stark Rationalist claim) from "the null string as input."[1]

[1] https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a...

Self-Fulfilling Misalignment Data Might Be Poisoning Our AI Models

Turn_Trout — Wed, 05 Mar 2025 18:18:04 +0000

Article URL: https://turntrout.com/self-fulfilling-misalignment

Comments URL: https://news.ycombinator.com/item?id=43270214

Points: 1

# Comments: 0

New comment by Turn_Trout in "Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs [pdf]"

Turn_Trout — Tue, 25 Feb 2025 22:49:55 +0000

They ran (at least) two control conditions. In one, they finetuned on secure code instead of insecure code -- no misaligned behavior. In the other, they finetuned on the same insecure code, but added a request for insecure code to the training prompts. Also no misaligned behavior.

So it isn't catastrophic forgetting due to training on 6K examples.

New comment by Turn_Trout in "Representation Engineering: Mistral-7B on Acid"

Turn_Trout — Mon, 19 Feb 2024 23:41:30 +0000

I'm the author of the GPT-2 work. This is a nice post, thanks for making it more available. :)

Li et al[1] and I independently derived this technique last spring, and also someone else independently derived it last fall. Something is in the air.

Regarding your footnote 2 re capabilities: I considered these kinds of uses before releasing the technique. Ultimately, practically successful real-world alignment techniques will let you do new things (which is generally good IMO). The technique so far seems to be delivering the new things I was hoping for.

[1] https://openreview.net/forum?id=aLLuYpn83y

New comment by Turn_Trout in "Optimal AI agents tend to seek power"

Turn_Trout — Tue, 07 Dec 2021 19:00:14 +0000

First author here. Thanks for your comment!

> there's a lot hidden in the "if physically possible" part of the quote from the paper: "Average-optimal agents would generally stop us from deactivating them, if physically possible".

Let me check that I'm understanding correctly. Your main objection is that even optimal agents wouldn't be able to find plans which screw us over, as long as they don't start off with much power. Is that roughly correct?

> Theories on optimal policies have no bearing if

See my followup work [1] extending this to learned policies and suboptimal decision-making procedures. Optimality is not a necessary criterion, just a sufficient one.

> if as we start understanding ML models better, we can do things like hardware-block policies that lead to certain predicted outcome sequences (blocking an off switch, harming a human, etc.)

I'm a big fan of interpretability research. I don't think we'll scale it far enough for it to give us this capability, and even if it did, I think there are some very, very alignment-theoretically difficult problems with robustly blocking certain bad outcomes.

My other line of PhD work has been on negative side effect avoidance. [2] In my opinion, it's hard and probably doesn't admit a good enough solution for us to say "and now we've blocked the bad thing!" and be confident we succeeded.

[1] https://www.alignmentforum.org/posts/nZY8Np759HYFawdjH/satis...

[2] https://avoiding-side-effects.github.io/

New comment by Turn_Trout in "Optimal AI agents tend to seek power"

Turn_Trout — Tue, 07 Dec 2021 06:55:33 +0000

Maybe you should read the paper, and/or the reviewer threads (as we discussed the nomenclature, and eventually agreed that "power" was accurate). We straightforwardly formalize a mainstream definition of power and show how it's more intuitive than the current standard measurement (information-theoretic empowerment).