For decision-making with neural networks, we instead need the older idea of estimators of the predictive uncertainty with constraints in the feature-representation space (over training/support of the estimator), as with Similarity-Distance-Magnitude estimators: https://pypi.org/project/reexpress-sdm/

The tricky thing with the neural networks is that the output logits are in effect a highly lossy compression of the epistemic (reducible) uncertainty, so even if the target calibration quantity is well-specified, it can be difficult to obtain in practice. A side-effect of this is that estimates in the high probability regions are not particularly stable under even modest co-variate shifts, which is a real problem if the estimates are being used for decision-making in a multi-step search graph that can lead to branches that are unlike what the model/estimator saw at training/calibration (if not altogether out-of-distribution). Here are a couple papers that describe how to approach those challenges:

[1] Similarity-Distance-Magnitude Activations. In Findings of the Association for Computational Linguistics: ACL 2026, pages 22037–22057, San Diego, California, United States. Association for Computational Linguistics.

[2] Introspectable, Updatable, and Uncertainty-aware Classification of Language Model Instruction-following. In Proceedings of the ACM Conference on AI and Agentic Systems (CAIS '26). Association for Computing Machinery, New York, NY, USA, 1259--1269.

That's only true in a practical sense if you can actually rely on the probabilities estimated by the model. That's a non-trivial problem for multiple reasons, among them: 1. What is the particular quantity you seek to estimate (marginal, approximately conditional, etc.)? 2. What is the reference class for that quantity? 3. What method are you going to use to estimate that quantity? 4. What is the error in your method to estimate that quantity (e.g., as via accounting for the effective sample size)?

A further practical challenge is that the output logits of neural networks are in effect a highly lossy compression of the epistemic (reducible) uncertainty. Even if your estimates are well-calibrated (for some definition of well-calibrated) on in-distribution data using the output logits, those estimates can be grossly uncalibrated in the presence of covariate shifts, and the logits themselves are not reliable signals of such shifts, nor of being out-of-distribution. Informally, the output logits themselves do not encode a good sense of what they do[n't] know.

Additionally, ideally the probability estimates are interpretable in the sense that there is some instance-wise connection to the training/calibration data. If the estimates are being used for decision-making, you need to be able to post-hoc audit the estimates to be able to modify the data for future decision-making, if needed.

Growing evidence in ML/NLP/Stats from the last few years is that with neural networks, as a starting point for constructing reliable estimates of the predictive uncertainty, we need to control for metric-learner signals over the support/training set (e.g., the L^2 distance to the nearest training instance and depth-matches into training). Once you have that, then you can choose your desired quantity of interest (e.g., class- and prediction-conditional accuracy at least some given value). Concretely, here's a tutorial (along with Apache-2.0 code) that steps through a simple, illustrative example: https://reexpressai.github.io/reexpress_sdm/tutorials/gettin...

More context is in the link, but at a high-level from an engineering perspective, just as dense vector matching is used by RAG for information retrieval, we can also use dense vector matching in this way to estimate the predictive uncertainty, getting around the limitations of the output logits.

Comments URL: https://news.ycombinator.com/item?id=49915210

Points: 1

# Comments: 0

For higher-risk settings, you can always turn the crank to be more conservative (i.e., more stringent parameters and/or requiring a larger sample size in the highest probability and reliability data partition).

For classification tasks, this follows directly. For generative output, this comes into play with the final verification classifier used over the output.

Hopefully in 2024 we can get at least one of the benchmarks to move to assessing non-parametric/distribution-free uncertainty for selective classification, reflecting more recent CS/Stats advances that should be used in practice. Working on it.

And I agree that the leaderboards don't currently reflect the quantities of interest typically needed in practice.

Comments URL: https://news.ycombinator.com/item?id=38880175

Points: 1

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=38852637

Points: 1

# Comments: 0

Perhaps counter-intuitively, and contrary to common wisdom, LLMs can in fact be transformed to generate very reliable uncertainty estimates (i.e., "knowing what they do and don't know" by assigning a probability to the output).

Getting there is a bit complicated, with vector matching/databases, prediction-time data dependencies, complicated inference, and multiple models flying all over the place.

We've made it simple and efficient to use in practice with an on-device, no-code approach. Common document classification tasks can be handled with the on-device models (up to 3.2 billion parameters). Additionally, you can add these capabilities to another LLM (e.g., for QA or more complicated tasks) by connecting your existing model by simply uploading the output logits into the app. For example, if you're using an on-device Mistral AI model, or cloud-based genAI model, just upload the output logits into the app.

Would be great to get feedback. Also, if you have another use case with a scale that doesn't fully fit into the on-device setting, happy to discuss and collaborate for your setting.

And if anyone finds this interesting and wants to get involved more in building reliable AI, let us know!

(Note that an Apple silicon Mac is required; ideally M1 Max or better with 64gb of RAM. You train the model yourself, which requires labeled data. The tutorial 1 video has a link to sentiment data in the JSON lines format; it's a good place to start: https://github.com/ReexpressAI/Example_Data/blob/main/tutori...)

Comments URL: https://news.ycombinator.com/item?id=38649799

Points: 2

# Comments: 0

> "[...] [AI] will act trustworthy, but it will not be trustworthy. We won’t know how they are trained. We won’t know their secret instructions. We won’t know their biases, either accidental or deliberate. [...]"

I agree that this is true with standard deployments of the generative AI models, but we can instead reframe networks as a direct connection between the observed/known data and new predictions, and to tightly constrain predictions against the known labels. In this way, we can have controllable oversight of biases, out-of-distribution errors, and more broadly, a clear relation to the task-specific training data.

That is to say, I believe the concerns in the essay are valid in that they reflect one possible path in the current fork in the road, but it is not inevitable, given the potential of reliable, on-device, personal AI.

(I do agree with the overall sentiment of the TC article, although as noted by others below, there's some mashing of terminology in the article. E.g., I, too, associate GOFAI with symbolic AI and planning.)

There's another dimension, too, not mentioned in the article: Even with general purpose LLMs, for production applications, it's still required to have labeled data to produce uncertainty estimates. (There's a sense in which any well-defined and tested production application is a 'single-task' setting, in it its own way.) One of the reasons on-device/edge AI has gotten so interesting, in my opinion, is that we now know how to derive reliable uncertainty estimates with the neural models (more or less independent of scale). As long as prediction uncertainty is sufficiently low, there's no particular reason to go to a larger model. That can lead to non-trivial cost/resource savings, as well as the other benefits of keeping things on-device.

There's also another related sense for which we want routing across models for efficiency reasons in the local setting, even for tasks for the same input modalities:

First, attempt prediction on small(er) models, and if the constrained output is not sufficiently high probability (with highest calibration reliability), route to progressively larger models. If the process is exhausted, kick it to a human for further adjudication/checking.