For higher-risk settings, you can always turn the crank to be more conservative (i.e., more stringent parameters and/or requiring a larger sample size in the highest probability and reliability data partition).

For classification tasks, this follows directly. For generative output, this comes into play with the final verification classifier used over the output.

Hopefully in 2024 we can get at least one of the benchmarks to move to assessing non-parametric/distribution-free uncertainty for selective classification, reflecting more recent CS/Stats advances that should be used in practice. Working on it.

And I agree that the leaderboards don't currently reflect the quantities of interest typically needed in practice.

Comments URL: https://news.ycombinator.com/item?id=38880175

Points: 1

# Comments: 1

Comments URL: https://news.ycombinator.com/item?id=38852637

Points: 1

# Comments: 0

Perhaps counter-intuitively, and contrary to common wisdom, LLMs can in fact be transformed to generate very reliable uncertainty estimates (i.e., "knowing what they do and don't know" by assigning a probability to the output).

Getting there is a bit complicated, with vector matching/databases, prediction-time data dependencies, complicated inference, and multiple models flying all over the place.

We've made it simple and efficient to use in practice with an on-device, no-code approach. Common document classification tasks can be handled with the on-device models (up to 3.2 billion parameters). Additionally, you can add these capabilities to another LLM (e.g., for QA or more complicated tasks) by connecting your existing model by simply uploading the output logits into the app. For example, if you're using an on-device Mistral AI model, or cloud-based genAI model, just upload the output logits into the app.

Would be great to get feedback. Also, if you have another use case with a scale that doesn't fully fit into the on-device setting, happy to discuss and collaborate for your setting.

And if anyone finds this interesting and wants to get involved more in building reliable AI, let us know!

(Note that an Apple silicon Mac is required; ideally M1 Max or better with 64gb of RAM. You train the model yourself, which requires labeled data. The tutorial 1 video has a link to sentiment data in the JSON lines format; it's a good place to start: https://github.com/ReexpressAI/Example_Data/blob/main/tutori...)

Comments URL: https://news.ycombinator.com/item?id=38649799

Points: 2

# Comments: 0

> "[...] [AI] will act trustworthy, but it will not be trustworthy. We won’t know how they are trained. We won’t know their secret instructions. We won’t know their biases, either accidental or deliberate. [...]"

I agree that this is true with standard deployments of the generative AI models, but we can instead reframe networks as a direct connection between the observed/known data and new predictions, and to tightly constrain predictions against the known labels. In this way, we can have controllable oversight of biases, out-of-distribution errors, and more broadly, a clear relation to the task-specific training data.

That is to say, I believe the concerns in the essay are valid in that they reflect one possible path in the current fork in the road, but it is not inevitable, given the potential of reliable, on-device, personal AI.

(I do agree with the overall sentiment of the TC article, although as noted by others below, there's some mashing of terminology in the article. E.g., I, too, associate GOFAI with symbolic AI and planning.)

There's another dimension, too, not mentioned in the article: Even with general purpose LLMs, for production applications, it's still required to have labeled data to produce uncertainty estimates. (There's a sense in which any well-defined and tested production application is a 'single-task' setting, in it its own way.) One of the reasons on-device/edge AI has gotten so interesting, in my opinion, is that we now know how to derive reliable uncertainty estimates with the neural models (more or less independent of scale). As long as prediction uncertainty is sufficiently low, there's no particular reason to go to a larger model. That can lead to non-trivial cost/resource savings, as well as the other benefits of keeping things on-device.

There's also another related sense for which we want routing across models for efficiency reasons in the local setting, even for tasks for the same input modalities:

First, attempt prediction on small(er) models, and if the constrained output is not sufficiently high probability (with highest calibration reliability), route to progressively larger models. If the process is exhausted, kick it to a human for further adjudication/checking.