They’re not saying goodbye to the LHC, they’re upgrading it to have 10x the power.

Low and maybe medium will save money on simpler tasks, but after that it just isn’t worth it compared to Opus.

I wish they would have explained in the blog post why they think anybody would ever want to use this above medium.

Maybe it works well on things that aren’t clear in the benchmarks.

I’m wondering the same thing. You keep talking of some grand poisoning problem but can’t point to any specific public information except an article saying that it’s possible. As if that was ever in doubt.

Guess we’ll just have to agree to disagree.

And do you even realize how much data 0.001% of the training data for a frontier models is? They’re trained on 10s of trillions of tokens, meaning you’d need hundreds of millions of tokens of poisoned data.

Some of these problems you mention could become real barriers to models improvements, though there are plenty of countermeasures, such as by focusing on high quality data sources like I mentioned before.

We’ve already probably gotten as much as we’re ever going to get from simply scraping more and more unstructured text from the web as a way to improve model performance.

The type of training being done now is around tool use and solving specific types of problems better, which is the type of training data you simply don’t find lying around on the web.

What I’m saying is that the AI labs are handling this not by fixing the “garbage out” part, but by minimizing the “garbage in” part.

The fact that all you could come up with was research (not an actual example of poisoning a real training set) from 2025 kind of proves that this isn’t some kind of widespread, unsolvable problem like you seem to be claiming.

Your original claim is that this will be enough of a problem to prevent models from improving in expert level knowledge. I completely disagree with this premise.

If the models fail to improve, it will likely be due to limitations in the transformer architecture rather than poisoned training data.

And even then, I doubt that the transformer is the best architecture we will ever come up with.

Clearly it doesn’t learn or think like a human does, since humans don’t need many gigabytes of text samples to learn to talk, so there is some room for improvement.

Peer reviewed journals, textbooks, in-house teams of experts, trusted news publications, etc.

The whole idea of scraping large swaths of the internet for training data has always been pretty dubious due to the variable data quality.

I mean, just look at the early Google models that told people to put glue in their pizza due to a joke in the training set. Garbage in, garbage out.

This is one of the first and most obvious problems all of these labs have run into, and countermeasures are only going to improve.

For one, if your website/book is poisoned, who is going to trust it for anything at all, much less for training models?

For two, all the major AI labs hire or contract for subject matter experts to create curated data sets, evaluate model performance, etc.

Unless they hire malicious experts, this will provide a growing, high quality data set that should drown out any poisoned pretraining data.

The current admin flies by the seat of their pants and at least creates the perception of political decision making.

I guess you could find somebody online and buy their account, but surely this would be a slow and unreliable process.

Of course Kubernetes is going to be way less fun to use. The problem of managing servers and distributed applications at scale is inherently not fun once you get into the nitty gritty details.

Of course, with how mediocrely those side projects have been going, I’m not surprised Meta is turning to layoffs. They seriously over hired and never really found a good use for all those engineers.