My testing with this technique indicates that method they seem to be using (rag with an instruction to check sources) will reduce the confabulation rate for citations from the base rate 50-60% for regular models (e.g. regular Claude) to 5-15% (depending on how they implemented it). On the one hand this is way better. On the other hand it's just good enough that your spot check will look good and your work will still contain hallucinations (which is probably worse than obviously bad).

Getting to zero confabulation would require a different process. (stand-alone validation engine running in parallel in real-time which is hard but not impossible.)

This paper was not retracted for citation fraud, but a citation check gives it a zero score (desk reject)

"Overall Score 0 / 100 High Risk

Feedback Summary The citation analysis identified significant issues verifying the existence of cited references and references with potentially weak topical alignment.

Document-Level Findings 3 Source location and matching issues We were able to verify 117 of 144 references against sources. 1 not found, 26 potential mismatches.

One or more references exhibit patterns associated with higher citation risk, which can arise from citation errors or automated reference generation."

Current AI tools generate citations that LOOK real but ARE fake. This might not be solvable inside the LLM. If anyone could do it, it'd be OpenAI. (OK maybe I'm giving them too much credit, but they have a crap-ton of money and seem to show a real interest in making their AI better)

If it can't be done in the LLM we can't trust LLMs basically ever. I suppose there's a pretty big loophole here. Doing it outside the LLM but INSIDE the LLM product would be good enough.

The first AI tool to incorporate that (internal citation and claim checking) will win because if the AI can check itself and prevent hallucinated garbage from ever reaching the user we can start to trust them and then they can do everything we've been promised. Until that day comes we can't trust them for anything.

https://groundedai.company/veracity/publishing/

What can be done to solve it (while not perfect) is pretty powerful. You can force feed them the facts (RAG) and then verify the result. Which is way better than trusting LLMs while doing neither of those things (which is what a lot of people do today anyway). See the recent 5 cases of lawyers getting in trouble for ChatGPT hallucinating citations of case law.

LLMs write better than most college students so if you do those two things (RAG + check) you can get college graduate level writing with accurate facts... and that unlocks a bit of value out in the world.

Don't take my word for it look at the proposed valuations of AI companies. Clearly investors think there's something there. The good news is that it hasn't been solved yet so if someone wants to solve it there might be money on the table.

Configure it fresh from install using ansible. Prefer ansible built-ins when possible. If you're calling shell scripts you're probably doing it wrong. Create named user accounts and create keys for that user. grant your named user sudo nopasswd and disable password login. Named users are preferable for security audit purposes. ("Shit who logged in at 2PM on tuesday and deleted critical resources?", "dunno it was the 'ubuntu' user", "great I wish we logged in as ourselves so we at least had some way of knowing who's account got compromised") Learn how to troubleshoot ssh login problems (-v), did you know that you can't log in if the permissions on the authorized keys file are wrong?

Enable and configure unattended-upgrades. Learn how UFW works and learn how to show iptables rules. Learn how to check which services are running, which are listening, which tcp connections are active. Learn how to write a systemd start/stop script (it's like 5 lines, super easy) Learn how to tail and grep logs to diagnose problems. Learn how to use find and grep.

When using ssl you'll probably use let's encrypt and you'll probably want to renew automagically with certbot and dns verification.

Create an elk stack, ship some logs to it. (maybe loki too) Hold off on k8s for now. It is advanced and you're wasting your time and your most precious resource (your ability to work through frustration) if you beat your head against it too early. When you learn k8s assume the cluster is already set up and learn to deploy a single service (there's minikube and k3s for this sort of thing). Anyone running k8s already has a lot of k8s skills (or they shouldn't be using k8s to begin with). Contribute in other ways. If they don't have a lot of k8s skills and they're using it 1) they don't need a junior, they need a senior and 2) they need to get onto something simple and easy like ecs. And if they already screwed up that bad they probably need to move to heroku. (but i digress)

Run through some command line practice, learn awk, sed, the general way commands are structured (command, options, arguments). Learn to chain commands. Learn vi (did you know you can call arbitrary shell commands?). Learn how to replace lines in config files with sed, learn how to concatenate content to the end of a file, learn how to do that for a file owned by root (hint, look into tee)

Learn how to deploy code with github actions

Learn Terraform Take the free online AWS solutions architect training courses and begin taking practice exams. The other clouds are clones. Azure with a preference for active directory, ghoogle with a preference for unnecessary complexity and pedantic bullshit.

Skip configuring a mail server, if you're doing that you're probably doing it wrong. (hint, you'll be hitting a mail sending service. Several of the small ones have a free tier which is great for learning). You'll need some more info here but you can search it up.

When learning bash scripting focus on readability and maintainability. The Google style guide is a great reference. https://google.github.io/styleguide/shellguide.html In bash there are 5 ways to do something, the best way is the most readable way. The person who thanks you may very well be yourself.

While we're on the topic, Google's SRE book is free online and epic. Learn the section on postmortems backwards and forwards. If you got a job and only brought the google postmortem template and culture with you you could improve nearly every company in the world. Same with improving their code testing and deployment. Same with doing simple cloud security and cost accountability tasks (like reviewing network perimeter and cost control suggestions in trusted advisor, finding unpatched servers and patching them, resolving dependbot vulnerability notices, etc.)

We haven't put a lot of marketing into the service, maybe we should do that.

Post some contact info in your bio, maybe someone here is looking for a mentee.

Comments URL: https://news.ycombinator.com/item?id=33682861

Points: 10

# Comments: 5

If you take over-the counter supplements and follow the instructions you're unlikely to overdose.

https://summit.groundedai.company/vitamin-d/

One weird thing I noticed was that Vitamin D supplementation, while it improves outcomes in terms of cancer, does not seem to improve risk of cardiovascular disease. Maybe it's just that people who get sufficient Vitamin D from sunlight are more likely to get outdoors and exercise ;-) But it made me wonder if there's some additional benefit of sun exposure, some vitamin we haven't discovered yet.