Until the big labs decide to start adding synthetic "I don't know" outcomes to their data sets, I've been thinking that the best way to evaluate uncertainty is to have a separate LLM monitoring the conversation and asking it to classify if the agent is overstating its confidence. On the other hand I've also noticed that most models will tell you they don't know something if you specifically include it in the prompt, eg "if you don't know the answer, just say so" and/or "be clear about any gaps in your knowledge that would reduce the confidence of your response" etc. but even with the big frontier models I have noticed some quality degradation if I throw too many instructions into the system prompt. I have a little more faith in harness-level engineering than in praying to the token generation gods.

That said, there is a completely different form of "uncertainty" in which the LLM tends to place very high trust in its own prior outputs as well as user provided inputs. Again if you look at the thinking traces, these models will try very very hard to rationalize the inputs they are given, falling back to the possibility of user error only after working through several alternative possibilities, maybe even investigating data or source code in the process. And if your context is big enough, the model might just completely miss when pieces of information conflict.

Case in point, I hit an error in a SQL query today because it turned out I was trying to do something that wasn't supported by the query engine. I pasted the error message and a bit of background info into my Claude Code session with Sonnet 5 High, it worked on a response for an unexpectedly long amount of time, including consulting the advisor model, and then came back with an explanation of the mistake I made in my query. Except it turned out I pointed it to the wrong file, and there wasn't a mistake in that file. It had completely taken for granted that the pasted error output was a real error and went on some wild goose chase.

Part of why the current gen models feel so smart is that they're getting better (via CoT and training) at recognizing when something is wrong and then back up to reassess. So it's easy to forget that it really is just token prediction, and (pending the next big advancement) there's only so much you can do with that.

You might want to do a human editing pass on the copy. This reads like it came right out of your Claude Code session.

That said, the duo of Opus 5 and Sonnet 5 do a fantastic job at fully automated work, and Claude Code still stands head and shoulders above the rest.

But I agree, the GPT models are so much simpler to work with, they have so much less personality and fewer quirks. They also are a little less aggressive about triple checking every little assumption immediately in a stack of 30 tool calls (but I haven't used 5.6 Sol yet so maybe that's not true anymore).

What actually happened is that AI-assisted training of LLMs (mind you -- stuck on that same old transformer architecture, with nothing resembling a medium-term memory, and fine-tuning still doesn't really work as a form of "learning") is right now giving us ~linear-ish progress... beecause we already hit the slowdown in the S curve of what human researchers can achieve.

We are also energy-constrained in ways that the exponential forecasts have no answer for other. Compute is not getting more efficient fast enough to achieve anything close to exponential growth driven by growth in computing power. There are social, moral, and political limits on the amount of energy we can dedicate to AI training in any given time period. And it's not just energy, we're short on RAM wafers and water for cooling and eventually we're going to hit other limits on various minerals and elements of the supply chain etc. etc. So even if it were true that R&DEffort == RateOfProgress without resource constraints, we're likely not going to see that exponential progress because we also need corresponding exponential scifi-scale progress in computing efficiency, cooling, and, energy delivery.

Basically this kid made a bunch of very clever and grand predictions which were always kind of ridiculous and no, they have not come to pass in 2026 and we are not at all on track to achieve AGI in the next 12 months, despite what Altman keeps trying to make the public think.

Oh yes they are! They vote to approve his cabinet picks, they campaign harder against left and progressive primary challengers than against the other party, they engage in the same insider trading as the other party, they take money from many of the same PACs and lobbying groups (Israel, AI, military-industrial complex), and...

> If you want to win in the general election, you want as many "normal" Democratic candidates as possible.

...and they continue to stick to the old strategy of "be centrist and try to flip upper-class urban moderate conservatives". This has been a proven consistent failure for 10 years now. Of course one can't expect every progressive and DSA leftist to win in November. But it's also not the 90s anymore. People still want to drain the swamp, as it were, and they don't see any hope in the establishment. They thought Trump was their anti-establishment man, but now, finally, after 10 years, they see what he really is, and there's an opportunity to really give the people what they want. And the best you can offer is more proven-failed centrism? Hell no. Schumer and Pelosi and Wasserman-Schultz and all the rest are complicit in the existential threat we face today. Primary them all out and send them to the dung heap of history.

The only challenge will be that the White House and complicit state governments are moving to disenfranchise as many people as possible before then, and likewise the White House will almost surely claim election fraud if Congress flips, and try to invalidate or even interfere with vote-counting.

1. the US president has previously threatened a war of conquest in Greenland, a territory of a NATO nation

2. the executive chair of the firm that is doing the illegal not-yet-drilling -- and also a top stakeholder -- is friends with said US president and donated to his campaigns.