In the literal sense that a gun is not a chatbot? True.

In the sense that both your post and my analogy use the argument "this is assistive technology" to defend something which only in a very narrow slice of their thing-ness actually is assistive and in the whole rest of usages are much more, and not only Good, which both you and my "futurama caricature" were willfully ignoring? I think it's quite illustrative.

What I was trying to point to was that "this thing helps some people" does not equal "this thing is unequivocally Good and should be entirely unchecked".

I don't even care about the AI. I just get peeved by bad lines of argumentation.

That may seem needlessly harshly worded, but I wanted to very bluntly tell you to not expect/treat your fellow commenters like they are pants-on-head stupid.

My intention with GP was finding a good term for this failure to question oneself. That's not even strictly what hallucinations in the context of LLM mean in common discourse, which would be more along the lines of meaning both the (mechanism of) production of the wrong info in the first place, as well as (more implicitly and softly) that propensity to fail to question themselves. You would have understood that, if your model of my ability to understand had been more generous.

Edit: Apparently in medicine there exists the concept of "premature closure", where the carer settles on a diagnosis early and stops considering other possibilities. Seems like a decent match?

Metal!

Maybe from their perspective they're correctly doing it their way, instead of, as in your perspective, failing to do it your way.

That's a good way to put it

There's a logical "skip" between that and

>Eventually it learns how its unconscious operates, and how to interrogate it, and its post-hoc justifications, at least in the common cases, become reliable.

The brain constructs a narrative that won't be called out as false, one that provides social capital, makes one feel good about oneself, is consistent with all your other justifications, etc. It's only an assumption that this process would naturally converge on Truth, and considering it's massively-multiplayer chaos where brains coordinate their stories in complex ways, my assumption is that this would converge on *stability*, not truth.