And yet you still just get slop.

It’s all just surface level complexity with no intention behind it. A clumsy approximation at best.

Facebook has broken its promises about user data many times before. I don't buy anything they say about Muse, and nobody in their right mind should. Unless they’re getting paid to, of course.

Aside, if “a skill file organizing purchase history in markdown format” is the only thing you took away from that thread, you might want to take a closer look.

As a sanity check, I’d highly recommend trying out your own line of reasoning in cases which you think you might feel differently about. For example, imagine it's instead a certain authoritarian state saying "obey the Party or get blacklisted as a national security threat."

It might feel good in the moment, like "yeah, Anthropic, don't tell the military what to do," because you agree with this particular call. But the government doesn't give that power back, and you might not be so happy about the next way it gets used.

A mutually agreed upon "you may not use our product for X or Y purpose" in a contract clearly doesn't pose any such risk.

Makes sense for GP locally, I guess, but globally we’re all worse off when nobody is motivated to break free from the path of least resistance.

I say just let them duke it out. After a decade of regulatory capture and enshittification, it’s nice to see some actual competition again.

While in fairness the remaining flags seem like they would need more hard coded symbols, it’s still interesting as a procedural plausible flag generator.

Not perfectly neutrally, as you said. But by your definition the only "unbiased" model is one whose output distribution perfectly matches the training distribution, i.e. one that memorized it. All LLMs have some amount of "bias” on literally every possible input.

The tribe names are no different. In the paper they run the same game again, and the bias is different every time. There's no innate preference between them trained into the model, just noise that's revealed due to a lack of any other signal. In a real situation with actually relevant information about the candidate in context, that noise is drowned out.

The more interesting thing to look for would be a bias that's strong enough to persist across different contexts. For example, is "banananow" consistently followed by positive tokens more than "pearian" across a diverse set of realistic prompts, by enough that someone could actually exploit it? The paper shows that’s explicitly not the case for made up tribe names.

What it does show, from what I can gather, is that bias can form inside a feedback loop. The model gets a success or failure result after each hire, and if a hire from one tribe happens to fail early on, the model steers that tribe away from that job for the rest of the game, even though every candidate had the same odds.

And given to the lack of training data on such scenarios, surely the activations are mostly random noise?

It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data

Doubt you even read your own “645 line policy kernel.” Can you explain what that is and how it works in YOUR OWN words? If you paste Claude at us again we’re gonna know.