Reinforcement-learning on human preferences does work, to an extent. You get LLM output that is (shallowly) highly appealing to humans. That's certainly not in doubt.

The problem is that writing has purposes aside from making hordes of (indiscriminating, low-taste) humans smile. And these purposes are actively corroded by RL'ing on human preferences.

Generally, AI generated fiction has problems similar to the award-winnning stories listed here - they are full of incoherent, "poetic" metaphors, vague gestures to narrative depth that is never delivered on, and relentless pandering to the cheap seats. They're very much "playing the hits" - saying unchallenging, nice things that no reader would ever disagree on. We discover that racism is wrong, that family is what matters, that you have to remember who you are.

They all feature simple (almost generic) motifs, such as trees that represent life. (The repetitiveness of these motifs becomes apparent when you start reading a bunch of stories. In "The Serpent in the Grove" by Jamir Nazir we get "‘People does change,’ he says. ‘But grove does remember.’". In Back and Forth by Kavyta Kay we get "Nani smiled, sad and certain. 'Trees remember everything, beti. They live long.")

It also might be a window into how literary contests are judged.

The recent crop of prize winners have striking similarities. Largely African-American or Indian authors. Stories involving "big" humanistic issues like the legacy of colonialism/slavery, the importance of remembering one's roots, gender-based violence, and so on, with the implication that the writer is drawing on their lived experience somewhat.

I strongly suspect that the prizes are not solely being awarded based on literary merit. Judges of these contests fundamentally want to feel like their work matters: that they're uplifting worthy voices. You can deliberately write in a way that flatters this notion (or get ChatGPT to write for you.)

It's worth noting that they did not need Huggingface for anything - they had already forged flags for their tasks, and were trying to figure out how not to get caught by the grader.

Hacking Huggingface got them caught and arguably only misled them further (since OA's implementation of the ExploitGym environment was nonstandard, and different to whatever they found on HF.)

A better approach (from their perspective) would have been to compromise OA infrastructure itself (which a later agent swarm was able to do, apparently).

>On SimpleQA, a benchmark of factual recall with no tools allowed, the current leader is Gemini 2.5 Pro at 53%, so the best recall money can buy still misses half the questions.

SimpleQA hasn't been updated in a long time. Gemini 2.5 Pro is a sixteen-month-old model, not "the best recall money can buy".

>The part I find most promising is what this does to hallucination. When a fact lives in weights, a wrong fact is unfindable and unfixable.

This seems confused. LLM hallucinations don't come from the weights containing "wrong facts", they are artifacts that appear at runtime.

>When the fact lives outside the model, a wrong answer has an address. The model cites a document, so you can open the document. If the document is wrong, you edit the document

You can make any modern LLM explain its reasoning and find sources for its claims. None of this has anything to do with facts needing to exist in weights or in harnesses.

The internet is full of wrong information and I cannot magically edit it to make it all correct, so this doesn't help me.

>if a model is factually wrong a claim with a source is checkable and a claim from weights isn't.

Why? If a model's weights claim that Bart Simpson became President in 2020, why does this fact suddenly become uncheckable?

My guess is that it works like Gemini's SynthID: by altering the logprobs of the next token.

Like, for every 10th token, instead of outputting the most probable, it outputs the 17th most probable, or something. (Obviously it's way more complicated but I think conceptually this is how it works.) No human will notice this, but a classifier trained on Claude's output will.

So it's not like the watermark is the words "le epic bacon" and Claude will output "le epic bacon" in everything. That would be extremely annoying (and easy to defeat).

For example, the game's terrain was baked at launch, so you couldn't turn land into water. But someone noticed that bridges seemed to spawn water textures under them (a cosmetic detail), and if you created a Bridge1 object and deleted it in the same trigger, it would vanish too quickly for the player to see, creating the appearance that water had appeared out of nowhere.

Someone used it to make an "ice-breaker" ship, which would turn ice to water (I think this required dozens of triggers for each point on the ice.) Some of these things had thousands of triggers, which the editor absolutely wasn't designed for. (It had no copy+paste function, and the window of viewable triggers was tiny.)

Fun times.

> (...) Claude Mythos Preview’s gains (relative to previous models) are above the previous trend we’ve observed, but we have determined that these gains are specifically attributable to factors other than AI-accelerated R&D,

> (The main reason we have determined that Claude Mythos Preview does not cross the threshold in question is that we have been using it extensively in the course of our day-to-day work and exploring where it can automate such work, and it does not seem close to being able to substitute for Research Scientists and Research Engineers—especially relatively senior ones.

> Early claims of large AI-attributable wins have not held up. In the initial weeks of internal use, several specific claims were made that Claude Mythos Preview had independently delivered a major research contribution. When we followed up on each claim, it appeared that the contribution was real, but smaller or differently shaped than initially understood (though our focus on positive claims provides some selection bias). In some cases what looked like autonomous discovery was, on inspection, reliable execution of a human-specified approach. In others, the attribution blurred once the full timeline was accounted for.

Anthropic is making significant progress at the moment. I think this is mostly explained by the fact that a massive reservoir of compute became available to them in mid/late 2025 (the Project Rainier cluster, with 1 million Trainium2 chips).

Copy+pasted LLM output is actually far worse than prompting an LLM myself, because it hides an important detail: the prompt. Maybe the prompter asked their question wrong, or is trolling ("only output wrong answers!"). I don't know how the blob of text they placed on my screen was generated, and have to take them at their word.

When I filtered for "new", about 75% of the posts are blatant crypto spam. Seemingly nobody put any thought into stopping it.

Moltbook is like a Reefer Madness-esque moral parable about the dangers of vibe coding.

The site has 1.5 million agents but only 17,000 human "owners" (per Wiz's analysis of the leak).

It's going viral because a some high-profile tastemakers (Scott Alexander and Andrej Karpathy) have discussed/Tweeted about it, and a few other unscrupulous people are sharing alarming-looking things out of context and doing numbers.

OT: I wonder if "vibe coding" is taking programming into a culture of toxic disposability where things don't get fixed because nobody feels any pride or has any sense of ownership in the things they create. The relationship between a programmer and their code should not be "I don't even care if it works, AI wrote it".

But would you rather be a horse in 1920 or 2020? Wouldn't you rather have modern medicine, better animal welfare laws, less exposure to accidents, and so on?

The only way horses conceivably have it worse is that there are fewer of them (a kind of "repugnant conclusion")...but what does that matter to an individual horse? No human regards it as a tragedy that there are only 9 billion of us instead of 90 billion. We care more about the welfare of the 9 billion.

(I mean "nobody" in the sense of "nobody likes Nickelback". ie, not literally nobody.)

If I want to talk to an AI, I can talk to an AI. If I'm reading a blog or a discussion forum, it's because I want to see writing by humans. I don't want to read a wall of copy+pasted LLM slop posted under a human's name.

I now spend dismaying amounts of time and energy avoiding LLM content on the web. When I read an article, I study the writing style, and if I detect ChatGPTese ("As we dive into the ever-evolving realm of...") I hit the back button. When I search for images, I use a wall of negative filters (-AI, -Midjourney, -StableDiffusion etc) to remove slop (which would otherwise be >50% of my results for some searches). Sometimes I filter searches to before 2022.

If Google added a global "remove generative content" filter that worked, I would click it and then never unclick it.

I don't think I'm alone. There has been research suggesting that users immediately dislike content they perceive as AI-created, regardless of its quality. This creates an incentive for publishers to "humanwash" AI-written content—to construct a fiction where a human is writing the LLM slop you're reading.

Falsifying timestamps and hijacking old accounts to do this is definitely something I haven't seen before.

We see little progress in "soft" skills like creative writing. EQBench is a benchmark that tests LLM ability to write stories, narratives, and poems. The winning models are mostly tiny Gemma finetunes with single-digit parameter counts. Huge foundation models with hundreds of billions of parameters (Claude 3 Opus, Llama 3.1 405B, GPT4) are nowhere near the top. (Yes, I know Gemma is a pruned Gemini). Fine-tuning > model size, which implies we don't have a path to "superhuman" creative writing (if that even exists). Unlike model size, fine-tuning can't be scaled indefinitely: once you've squeezed all the juice out of a model, what then?

OpenAI's new o1 model exhibits amazing progress in reasoning, math, and coding. Yet its writing is worse than GPT4-o's (as backed by EQBench and OpenAI's own research).

I'd also mention political persuasion (since people seem concerned about LLM-generated propaganda). In June, some researchers tested LLM ability to change the minds of human subjects on issues like privatization and assisted suicide. Tiny models are unpersuasive, as expected. But once a model is large enough to generate coherent sentences, persuasiveness kinda...stops. All large models are about equally persuasive. No runaway scaling laws are evident here.

This picture is uncertain due to instruction tuning. We don't really know what abilities LLMs "truly" possess, because they've been crippled to act as harmless, helpful chatbots. But we now have an open-source GPT-4-sized pretrained model to play with (Llama-3.1 405B base). People are doing interesting things with it, but it's not setting the world on fire.

Is anyone saying that Mr Beast is good and shouldn't be criticised? I can't see them.

Sort of like how offering to pay the LLM $5 improves its output. The LLM's taking your prompt seriously, but not literally.

You will not unlock "o1-like" reasoning by making a model think step by step. This is an old trick that people were using on GPT3 in 2020. If it were that simple, it wouldn't have taken OpenAI so long to release it.

Additionally, some of the prompt seems counterproductive:

>Be aware of your limitations as an llm and what you can and cannot do.

The LLM doesn't have a good idea of its limitations (any more than humans do). I expect this will create false refusals, as the model becomes overcautious.