My understanding of the statement is trying to use it in production for the first time. Once you start to actually use decision models beyond something simple you start to see the limitations.

I believe the normal term is "gaming a benchmark test".

Nothing in what you quoted is related to that. It's a normal way to train and test a classifier.

The only way to determine if it's brittle is to test on a different dataset and look at the sensitivity and specificity.

Even so, there is nothing wrong with a model that is trained to a narrow domain when it doesn't need to deal with "wild" images.

I think its a combination of being new to classification models and not using in anger.

LLM Decision models are still relatively new and suffer from a number of issues. Order of choices impacts probability, it starts to break down over 20 classifications and even how you word things impacts probability.

Why not invest your time on better services.

It has the advantage of speed and the confidence not being hallucinated.

But LLMs start to generalise on the pattern, rather than the classification that you want the more examples you have to train on.

LLMs start to break down as well the more classifications you have. Laya (Open source paper Jev is based on) even mentions that over 20 classifications and it starts to fail rapidly.

20 is around the level of sentiment analysis or minor intent routing. There are cheaper, smaller and easier ML models for that level of classification.

I can't see any benefits that a typical ML classifier would not be better at.

To me it’s like a solution looking for a problem that is already solved.

https://www.csszengarden.com

Does Jev solve this?

That was the media hype about it. There were two incidents.

1. Using a RL to train a model, it found that it got rewarded for certain garbage phases, so continued to talk that way.

2. Certain Latin words for fish/birds were used instead of "fish" or "bird". Just a token issue.

As I understand the paper they are saying the reasoning/thinking you see is actually a translation of what is actually going on, and stuff can be lost in the translation. Similar to what was observed in j-space.

Your analogy is flawed because you are saying the information was free to begin with.

That analogy works for IBM granite models because the information they trained on is free to use.

The major LLMs did not do that.

That would be on what the AI model generates, not what it is trained on.

The latter is where the contention is, and it's a valid argument. So much so that some companies are not using stolen information to build their models.

IBM for example indemnifies its models for its customers and has detailed information on where the sources came from to train them.

Black Forest Labs for example has a $4B valuation with half a billion raised so far.

I feel people are too focused on US/China and don't pay attention to what is going on in the world.

ASML in the Netherlands for example was the only company in the world that makes EUV lithography machines, which all the major chip companies depend on. China recently reverse engineered their work to create machines since late 2025, but not sold commercially.

Ireland has chip production facilities.

EU might not be in the top 2, but it is not out of the running at all.

The main one (IIRC) is Spains ALIA.

https://alia.gob.es/eng

There is also OpenEuroLLM and EuroLLM.

https://www.openeurollm.eu