So the fallacy of the whole LLM field is the belief that language has some intrinsic meaning. Or if you mix the artifacts of language in some very smart way, the meaning will emerge. But it doesn't work if meaning occurs before the word. The text in books has no reasoning, it was authors. The machine shuffling the text fragments does not have a meaningful thought. The engineer which devised a shuffling machine had some meaningful thought, the users of the machine have same thoughts, but not the machine itself. To put it another way, if there was an artificial system capable of producing meaningful thoughts, it is not a presence of language which produces a proof, it's communication. Communication requires an agent (as in "agency") and an intent. We have neither in LLM. As to the argument that we ourselves are mere stochastic parrots - of course we can produce word salads, or fake mimics of coherent text, it is not a proof that LLM IS the way our minds work. It is just a witness to the fact language is a flexible medium for the meanings behind - it can just as well be used for cheating, pretending, etc.