https://artificialanalysis.ai/evaluations/omniscience

> measures knowledge reliability and hallucination. It rewards correct answers, penalizes hallucinations, and has no penalty for refusing to answer.

This is so useful because it makes you actually trust a models output. A high score on benchmarks is not as useful because a model overtrained to always answer will give confidently wrong responses. But this index measures how often it is correct while penalizing wrong responses so that a high score means you can trust this model more and when it doesn't know it is more likely to tell you that it really doesn't know rather than making shit up.

Fable also performs a lot better than opus 5 here which correlates very strongly with perceived strength despite the models performing similarly on e.g. DeepSWE

Astra is a big jump from sol and performs the same or slightly better than fable here.

Well kind of, I wouldn't be surprised to see that some things marketed as agents are actually good old deterministic software.

Luckily for Meta agents are not yet as much into doomscrolling as humans are.

> Brain electrical activity was recorded in 36 university students as they were handwriting visually presented words using a digital pen and typewriting the words on a keyboard

kinda funny.

It's also not clear from this study that there is actually any benefit to "more brain activity". Of course doing more complex motor tasks requires more brain activity but nobody guarantees that this helps with remembering things.

Ofc I can just fork gemini-cli if I want my own version but like I don't think that's what OP meant by build your own thing and customize it.

I haven't really found a good way to solve this other than:

1. Produce an initial PR fulfilling all the requirements I knew at the start

2. Chat with the model about any weird snippets I notice and talk through alternatives

3. Simplify anything that I think is overengineered or plain unncessary

Sometimes I restart all over with more precise requirements but then it sometimes makes different mistakes/takes different shortcuts.

In practice the earlier I review the better the end result imo, so /goal seems very unproductive to me?

Tbf as long as that chance is low enough it doesn't matter in practice, but I have definitely seen the classifier approve things that were questionable, and I've also seen it decline things that were obviously okay.

The nudge to think about both "tokenization as variable" as well as actual tokens consumed per task is still good.

Also finally closed the first real customer on it recently!

I want to get through a large chunk of the open issues the next few weeks and then spend some time building agentic capabilities for it. I believe a central place to configure database access for your dev team without having to share passwords and with sensible review policies should also help e.g. if claude needs to access production data to validate a premise.

Still have to figure out the right UX though not sure the agent should have the exact same review requirements that a human does. Maybe it needs to be configurable separately