I suspect if you feed unrelated or even garbage questions into the eval set, it would reduce the performance.

So I believe that, at least in the short run, we might be seeing breakthroughs in hard open problems or in low hanging problems which are not that interesting to spend time on.

I may be wrong, if some research labs have private contracted access to the models

Maybe the Sol Pro is not that great anymore, considering the token vs output balance.

What's worse, when reading the Google outputs, I have learned to ignore almost any website besides a select few, one of which is Reddit. Which is very weird thinking about it, but it worked in the past.

Oh yeah, I'm still amazed how good the current iteration of models are for coding (I have a fear it's too good to be true - so will get taken away..). Exactly a year ago I switched from GPT 5 to Gemini just because the coding with R language was terrible; and even with Python it kept forgetting and mixing basic stuff. Gemini at the time had much longer context window and was miles ahead on R syntax.

Current experience of just leaving a Codex Agent chug until a stable solution is completed is still mind blowing to me.

I recently used it at grocery stores in a foreign country. Photographed the whole aisle and told it to find Y (detergent, softener, glue, sour cream, whatever), at the same time recommend the best Y for whatever reason. Worked marvelously, including the cases where the object wasn't present and it told me there was nothing useful.

I asked then, can you crop the exact image of how does the item look like and where is it in the aisle - did that perfectly as well.

I will add that all frontier models were fine with such tasks from the early 2024's.

AI did a fixed amount of guesses, didn't yield anything. It probably documented the tries, outcome, and some numbers hinting at why they failed. So the user could have prompted "continue", or "try again with previous outcome in mind, generate new ideas and test them" and it would probably yield the same result.

They're sampling slope and bias for the line from U[0,1] and U[0,100], the expected values of which will be 0.5 and 50. That's why they get what they get.

They say that randomness is any positively sloped line with any positive bias. So they assume dependence, just a very high variance of it. It's incorrect as they miss the negative half of the parameters - it would then correctly yield zeroes, for the presumed independence.

The fact that given X and Y random and independent, that Y-X is correlated with X doesn't disprove the Dunning Kruger. It in fact proves that Y = 1 X is a poor predictor, and the true model is Y = 0 X. In other words, perceived ability (of the human) cannot predict the actual test scores. Which is exactly what DK claims, but to a very extreme effect.

Note that, if there is actual signal (plus noise), e.g., if Y = X + eps; so the actual score is exactly the perceived score plus some added variation, the (Y-X)~X will be uncorrelated. In such case, there will be no DK effect, because the users are good at predicting their actual test scores, plus some constant variation.

By the way, convex hull permits extrapolating past the training data. LLM won't invent a new word that could not be defined by a sequence of known words. Just if it's meaningless and fully random/hallucinated, the new knowledge won't work with other known information blocks (breaks convexity).

Add negative relationship and negative bias, and the random data will act as intended - hovering randomly around 50%.

Now it's much more clear. The simulated data tries generating the true relationship between actual and perceived scores from 0.0 to 1.0, and bias in self-reporting from 0% to 100%.

So the output graph should be the average of all these data generating processes, yielding perceived relationship around 0.5 and bias around 50%, with some high variation.

If you have the access, run their Shiny code with these values, and you will see the published plot.

I'd argue that this demonstration is much weaker than "making original no more meaningful than random". It's more that the "simulated 50% bias and 0.5 true correlation looks similar to what DK published", which is also far fetched given the data generation they did.

Note: true random (what they were going for) would cover negative relationships, yielding the random true relationship around 0; and if they wouldn't correct the sign of Bias, it would also average at around 0; yielding a realistic "random" with the slope hovering about 50% for any percentile.

It's clear that the perceived curve will be differently sloped, as no one will evaluate themselves as the topmost or the bottommost percentiles, so the edges will be biased.

And if in both cases we draw differences between perceived and actual, we will get the same curve that everyone knows, biased or not.