Hacker News: _jonas

New comment by _jonas in "About AI Evals"

_jonas — Thu, 03 Jul 2025 20:48:37 +0000

Evals are critical, and I love the practicality of this guide!

One problem not covered here is: knowing which data to review.

If your AI system produces say 95% accurate responses, your Evals team will spend too much time reviewing production logs to discover different AI failure modes.

To enable your Evals team to only spend time reviewing the high-signal responses that are likely incorrect, I built a tool that automatically surfaces the least trustworthy LLM responses:

https://help.cleanlab.ai/tlm/

Hope you find it useful, I made sure it works out-of-the-box with zero-configuration required!

New comment by _jonas in "A.I. Is Getting More Powerful, but Its Hallucinations Are Getting Worse"

_jonas — Wed, 07 May 2025 23:31:30 +0000

You might be thinking of LLM as-a-judge, where one simply asks another LLM to fact-check the response. Indeed that is very unreliable due to LLM hallucinations, the problem we are trying to mitigate in the first place.

TLM is instead an uncertainty estimation technique applied to LLMs, not another LLM model.

New comment by _jonas in "A.I. Is Getting More Powerful, but Its Hallucinations Are Getting Worse"

_jonas — Tue, 06 May 2025 03:08:22 +0000

This is why I built a startup for automated real-time trustworthiness scoring of LLM responses: https://help.cleanlab.ai/tlm/

Tools to mitigate unchecked hallucination are critical for high-stakes AI applications across finance, insurance, medicine, and law. At many enterprises I work with, even straightforward AI for customer support is too unreliable without a trust layer for detecting and remediating hallucinations.

Prevent LLM Hallucinations with Trust Scoring in Nvidia NeMo Guardrails

_jonas — Tue, 22 Apr 2025 18:52:33 +0000

Article URL: https://developer.nvidia.com/blog/prevent-llm-hallucinations-with-the-cleanlab-trustworthy-language-model-in-nvidia-nemo-guardrails/

Comments URL: https://news.ycombinator.com/item?id=43765128

Points: 2

# Comments: 0

New comment by _jonas in "Cursor IDE support hallucinates lockout policy, causes user cancellations"

_jonas — Thu, 17 Apr 2025 19:42:31 +0000

Exactly, that's why my startup recommends all LLM outputs should come with trustworthiness scores:

https://cleanlab.ai/tlm/

New comment by _jonas in "Cursor IDE support hallucinates lockout policy, causes user cancellations"

_jonas — Thu, 17 Apr 2025 19:39:46 +0000

My startup is working on this fundamental problem.

You can try out our early product here: https://cleanlab.ai/tlm/

(free to try, we'd love to hear your feedback)

New comment by _jonas in "Cursor IDE support hallucinates lockout policy, causes user cancellations"

_jonas — Thu, 17 Apr 2025 19:34:49 +0000

I see this fallacy often too.

My company provides hallucination detection software: https://cleanlab.ai/tlm/

But we somehow end up in sales meetings where the person who requested the meeting claims their AI does not hallucinate ...

New comment by _jonas in "ChatGPT Search"

_jonas — Sun, 03 Nov 2024 02:07:19 +0000

Has anyone run any meaningful benchmarks of this vs. google vs. perplexity?

New comment by _jonas in "Ask HN: Local RAG with private knowledge base"

_jonas — Sun, 03 Nov 2024 02:03:25 +0000

This one looks pretty good, haven't tried it yet though: https://github.com/QuivrHQ/quivr

New comment by _jonas in "Embeddings are underrated"

_jonas — Sun, 03 Nov 2024 02:02:08 +0000

It's fun to try and guess what semantic concepts might be captured within individual dimensions / pairs of dimensions of the embeddings space.

New comment by _jonas in "Brute-Forcing the LLM Guardrails"

_jonas — Sun, 03 Nov 2024 02:00:51 +0000

Curious to learn how much harder it is to red-team models that use the second line of defense of an explicit guardrails library that checks the LLM response in a second step. Such as Nvidia's Nemo Guardrails package.

New comment by _jonas in "Get me out of data hell"

_jonas — Sun, 03 Nov 2024 01:58:16 +0000

I'm excited for LLM applications that can setup, monitor/validate, and optimize data pipelines at scale. Seems possible soon given that SQL and most data records aren't intended to be human-friendly

New comment by _jonas in "SimpleQA"

_jonas — Sun, 03 Nov 2024 01:55:27 +0000

It's easier to find the data now, I've run some benchmarks on it. Great to see OpenAI open-sourcing datasets like this!

New comment by _jonas in "SimpleQA"

_jonas — Sun, 03 Nov 2024 01:51:17 +0000

Here are some benchmarks I ran that compare the precision/recall of various LLM error-detection methods, including logprobs and LLM self-evaluation / verbalized confidence:

https://cleanlab.ai/blog/4o-claude/

These approaches can detect errors better than random guessing, but there are other approaches that are significantly more effective in practice.

New comment by _jonas in "Detecting when LLMs are uncertain"

_jonas — Sun, 03 Nov 2024 01:35:32 +0000

There is however a subfield of statistical ML of model uncertainty quantification. I've developed a product by applying to it to LLMs that can score the trustworthiness of any LLM response. Like any ML-based product, my tool is not perfect, but it can detect incorrect LLM responses with pretty high precision/recall across applications spanning RAG / Q&A, data extraction, classification, summarization, ...

I've published extensive benchmarks: https://cleanlab.ai/blog/trustworthy-language-model/

You can instantly play with an interactive demo: https://tlm.cleanlab.ai/

New comment by _jonas in "To Believe or Not Believe Your LLM"

_jonas — Thu, 06 Jun 2024 16:35:35 +0000

To try out an existing product that quantifies LLM uncertainty (accurately incorporating both aleatoric & epistemic uncertainty), you can try this Trustworthy Language Model I built (after similar research):

https://tlm.cleanlab.ai/

TLM is an API you can use to quantify uncertainty of any LLM model: https://help.cleanlab.ai/tutorials/tlm/

Benchmarks showing these estimates more reliably detect bad answers & hallucinations than logprobs, LLM-as-judge, Selfcheck-GPT, etc: https://cleanlab.ai/blog/trustworthy-language-model/

New comment by _jonas in "Train and Deploy Reliable Models on Messy Real-World Data with a Few Clicks"

_jonas — Mon, 24 Jul 2023 16:11:58 +0000

New feature alert: Auto-train & deploy reliable ML models (more accurate than fine-tuned OpenAI LLMs) on messy real-world data — all in just a few clicks!

Common reasons companies struggle to quickly get good ML models deployed and generating business value: messy data full of issues, a need to explore many ML models to train a good one, and infrastructural challenges serving predictions from the model. Now you can handle all of this in minutes using Cleanlab Studio.

For classifying product reviews, the deployed Cleanlab Studio model is more accurate than OpenAI LLMs fine-tuned on the same data. Producing this model merely required a few clicks in the platform which automatically: detect/correct issues in the dataset to produce a better version, identify and train the best ML model for this particular data, and deploy it for serving predictions in an application. Each of these steps typically requires significant code and effort from a team, but not if you use Cleanlab! Within hours, our cutting-edge AutoML with Foundation models produces highly accurate models for almost any dataset.

Cleanlab Studio allows you to rapidly turn raw image/text/tabular data into reliable ML model deployments, by automating all of the necessary steps. No other tool makes the full end-to-end pipeline this easy and performant!

Details on how we achieve this via novel Data-Centric AI techniques and benchmarks of model performance are in our new blogpost: http://cleanlab.ai/blog/model-deployment/

Train and Deploy Reliable Models on Messy Real-World Data with a Few Clicks

_jonas — Mon, 24 Jul 2023 16:11:57 +0000

Article URL: https://cleanlab.ai/blog/model-deployment/

Comments URL: https://news.ycombinator.com/item?id=36850272

Points: 5

# Comments: 1

Supervised learning on tabular data with numeric, categorical, and text columns

_jonas — Sat, 08 Oct 2022 00:08:21 +0000

Article URL: https://github.com/sxjscience/automl_multimodal_benchmark

Comments URL: https://news.ycombinator.com/item?id=33128502

Points: 2

# Comments: 0

Show HN: Accurate image classification in 3 lines of code with AutoGluon

_jonas — Sun, 08 Mar 2020 00:47:56 +0000

AutoGluon is an easy-to-use AutoML toolkit for deep learning that allows you to automatically leverage state-of-the-art techniques. Writing barely any code, we recently used AutoGluon to achieve around top 10% ranks in four Kaggle image classification competitions:

https://medium.com/@zhanghang0704/image-classification-on-kaggle-using-autogluon-fc896e74d7e8

To learn how to use AutoGluon for your own image classification problems, see our tutorial here: https://autogluon.mxnet.io/tutorials/image_classification/kaggle.html

Beyond image classification, AutoGluon also makes it easy to get started with object detection, as well as prediction tasks involving tabular/text data instead of images. If you're already a deep learning practitioner, AutoGluon helps you automatically tune your own custom models.

AutoGluon is open-source and available on GitHub: https://github.com/awslabs/autogluon/

Comments URL: https://news.ycombinator.com/item?id=22515426

Points: 4

# Comments: 0