Pretending that saying "we don't have much space" and crying like a baby when someone say "well you may not have plenty of 10 miles squares areas, but you can put a 4 miles square large reactor in one of them, and it will be better than having to build 10 1 miles square small reactors", that's being the pedantic one: you cannot complain that normal people understand normally what "we dn't have much space" means.

> part of the point of SMRs is to be able to have them in space-constrained places

Yes, but this is not a problem that exists in real life. It helps in some scenarios, but it is not the main practical issues that people have.

> that's the appeal! google and meta aren't looking at them

Google and Meta are not looking at SMR _because they don't have enough space_. This is not true at all: if you look at their projects, they have plenty of space.

They are looking at them because they want to generate a small quantity of electricity for their own usage. They want their own small reactor. But it does not invalidate that these small reactors are less efficient than big reactors: they are just happy to pay 2X dollars for a reactor they fully own than to pay X dollars for the same quantity of energy for a share of a big reactor, because it is more difficult to manage if you have to find partners and make sure everyone is agreeing.

Well, if you have 1000 places of 1 square mile and 0 space of 4 square miles, the available space is 1000 square miles. If you have 100 places of 4 square miles, the available space is 400 square miles.

You cannot say that the first sentence means the same thing that the second sentence, and you cannot say that "there is not enough space" is only something you can say in the first-sentence situation and not in the second-sentence situation.

Maybe what you meant to say is not "there is not enough space", but "there is plenty of small space but not a lot of large space" (which I doubt is true in the real world: space occupancy is usually regrouped in dense areas, leaving non-dense areas).

> if i cannot fit something large ... i only have so much physical space to work with

First, the idea that, for a domestic power plant, you only have limited space, seems very unrealistic. The real world is not a submarine or a 7/11: you want your power plant at the periphery of cities, not squeezed between 2 buildings in the middle. There is only disadvantage of doing so: you cannot distribute high power lines from the middle of the city safely, you probably need facilities to deal with the fuel, probably need water for cooling, probably need a security perimeter as you have around any typical factory, the cost of the square meter is more expensive, ...

But secondly, you need the power plant to produce some power. If your country needs X GWh, and you need either 1 large power plant of 4 square miles or 10 SMR of 1 square mile and you just have few places where you can put a power plant, the "the unit itself is more compact" does not matter . I only have so much physical space to work with. If the surface needed to get X GWh using SMR is too big, it's too big.

> you cannot fit a reactor from three mile island into a submarine

Yep. Similarly, you cannot fit a SMR in a bicycle. But how is that relevant? In real life, domestic power plant do not have the constraints of being in thigh places (on the opposite, it is better for a power plant to be in regions that also happen to not have thigh places).

But, yes, I get it is how it is sold. Just that even sold like that, people with common sense should say "wait a minute, that's obviously not that simple".

If a big nuclear reactor takes 10x more space but has 20x more capacity, then it means not having much space favors the big nuclear reactor rather than building 10 small ones that will take twice more space.

(and same for the time)

I've just read https://jamesfbaker.substack.com/p/why-the-ai-renaissance-ke... (and the Rich Sutton take on AI creativity, too) that explains exactly the opposite of you, but backed by real studies and real facts, instead of "that looks 'beyond training data' to me, so I will pretend it is, even if I have no objective ways to know if it is indeed the case".

For example, you just moved the definition problem to "novel". Do you even realise that?

You are claiming that there is an understanding because the model is able to do something in a novel situation where only deeper understanding of the situation will allow it to perform that well. The big problem is that you have no idea if this situation is "naturally easy to reach" or not.

For example, a system that is fitted on the electrostatics Coulomb's law will build, internally, a set of equations to generate realistic predictions. And then you take this system and put it in a totally novel situation: classical gravitational problems. Well, this system will be able to generate realistic predictions there too, because Newton's law works with equations that have the same form.

When you are discussing with a LLM in a "novel" subject, how do you know the LLM cannot directly use the fit and complex equations that it has created for the "non-novel" situations it has been trained on? For example, the LLM has been trained on "Pride and Prejudice and Zombies" and tons of other mash-ups. Even if asking a story of Keanu Reeves and the Supreme Court looks "novel" to you, it does not mean the generated text was not in fact super easy to generate based on the patterns that the LLM has seen in tons of examples.

Honestly, this whole conversation just convinced me that too many people who claim "GenAI does understand" are way above their head on the subject. If you want to continue talking, just talk to a LLM. Plenty of people have done so and convinced themselves they were geniuses when in fact they were not at all. Yet another example that LLM has no understanding, as it is very very often failing to distinguish between correct ideas and "things that look correct but that someone with real understanding will not encourage".

My point is that you can have the same result with a representation that "closely matches the topology of the relationships being modelled". For example, a representation that "allows relationships between tokens but yet does not care about the meaning or concept not useful to form convincing sentences".

And therefore, it means that you can have convincing text without needing a "representation's topology closely matching the topology of the relationships being modelled", and therefore, according to your own definition: no understanding.

2. It is not true I'm avoiding that. I have answered very clearly.

1) GenAI are not trained to get the higher representation of the world, but to get the best convincing sentence generation. This does not require a full world understanding. Worse, once a convincing sentence generation is reached, there is no gain by getting a better world understanding: the training mechanism that pushes into the correct direction stops and therefore it can go into any direction at all.

2) High compactness does not equal best solution. Even humans don't used "high compactness" when doing basic arithmetic, but use "by heart multiplication table". Being compact is useless if it comes with high complexity each time you need to recompute the output.

3) Very very good approximation can reach higher compactness anyway. Your Bezier curves is a good example: real physical phenomenons are almost never the result of a Bezier equation. A Bezier curve did not understood the phenomenon. When it comes to GenAI, it can "fit" the reality with very close precision with several representations, but the majority of the representation corresponds to an incorrect "understanding" of the reality.

Another example: if I throw a ball in the air, the motion will be at first order a quadratic equation, plus correction due to friction, wind, ... If I just "train" something for "throw a ball", this system may fit a quadratic function plus corrections, but they will achieve the same result with Bezier curves, or Fourier series, or additive Gaussian, or ... But the "understanding" is that the ball is influenced by gravity, which leads to a quadratic equation. The system does not understand that. It has no reason to understand that. And it has no reason to prefer a quadratic equation fit rather than a Bezier fit, on the contrary, the Bezier fit will be more realistic (as the quadratic equation is just the first order approximation).

If you want to understand a paper plane trajectory, it is a complex system, and you probably need plenty of parameters to describe the gravity, the wind at each position and each time, the shape of the plane at each time, ... But you can describe the trajectory with just few parameters using a Bezier curve. Train on plenty of paper plane trajectories, and you will have a system that can give you a very realistic paper plane trajectory based on Bezier curve. And yet, your system has no understanding of the paper plane trajectory: it does not know what are the mechanisms that make the paper plane goes up or down. It just creates a realistic trajectory without knowing why this trajectory is realistic, just that this trajectory makes sense based on the other trajectories it has seen.

3. This argument seems to go against your thesis. You are saying that humans, who "understand" + are not even able to have as much conversation as LLM, have way too much neurons. What are these neurons even for then? You are explaining that LLM are just "something different", a reduced mini-version of a brain, and yet you are also saying that they are able to do the complex things the brain do.

Another way of seeing it, is that LLM are "dropping" things that they don't need to create convincing sentences, such as "understanding the token". They just "get the Bezier curve fit of the relationship" instead of understanding the real mechanisms and concepts.

It's like your Bezier curve example: a system that just creates a realistic paper plane trajectory based on "typical Bezier curve observed during training" will need way less "neurons" than a system that needs to understand the whole aerodynamism of the paper plane.

4. I argue this the same way I say that a system that describe a paper plane trajectory based on best Bezier curves did not understood the mechanism behind how a paper plane trajectory works. I am not saying "I define 'understanding' as what humans do", I am saying that creating convincing sentences does not require understanding, the same way that generating realistic paper plane trajectories does not require understand gravity, Navier-Stokes equations and Brownian motions.

The Bezier curve paper plane trajectory predictor system I have mention, do you think it has understanding of gravity? of Navier-Sotkes? of Brownian motions?

No, it has not. You can open this system. It just has Bezier curve for plenty of examples, and thanks to that, it knows that one trajectory is realistic and another is unrealistic. And at some point, it is also able to give realistic trajectories in brand new situations it has never trained on.

You keep saying "what I observe with GenAI can only be the result of 'understanding'" without providing any proofs at all. Just few beliefs.

You just say "look at this behavior, that's the proof". I truly don't think it is: nothing proves that this behavior requires 'understanding'. And nothing you provided helps: all you provided are impressive behaviors and then the unsubstantiated conclusions "and this behavior can only be done with understanding".

At the same time, there are too much clues showing that such behavior does not require understanding, even if it _looks_ incredibly clever:

1. GenAI does not understand (after the training phase) things that humans don't understand. If GenAI had the capacity of building an understanding during training, then there is no reason this understand will coincide with human understanding.

2. Optimisation does not always lead to "understanding". Human brains choose to optimise "learning multiplication table by heart" rather than building a pocket calculator inside the neurons.

3. Human brains, that have "understanding", are working fundamentally differently from GenAI (flow of thoughts, intrinsically intertwined memory and compute, optimised for world-model treatment rather than token treatment, ...). It is an unsubstantiated jump to simply conclude AI has "understanding", while it can be the result of fundamental differences.

4. "Basic" LLM are surprisingly good at creating convincing sentence and yet there are situations where it is blatantly clear they did not understood anything. More advanced SOTA are based of refinement of "basic LLM", and therefore the "sentence construction that is done without understanding" is still used, and impair the SOTA model to build a full understanding.

> Another way to put this, is deep learning models are able to learn higher-order relationships directly, not be memorizing and interpolating across learned points or regions.

It's exactly what I'm saying: deep learning models are very good at learning complex relationships. Such as "I don't know what 'Paris' is, I don't have any understand of what a city is in reality, but when the token Paris is associated with these other tokens in this complex order, even if I never saw it before, I have learnt the complex relationships and therefore I'm able to build a series of token".

They are very good at learning complex relationship that allows them to choose the correct combination even if they did not "understand" the content of the correct combination.

I understand that it is impressive: those relationships are very complex and very numerous (there are billions of them). It is easier to do anthropomorphism and conclude that the AI has "understood".

But again, the main problem is that you just pretend, without any proof, "no, I cannot believe that, I refuse to believe that".

(and, by the way, I personally think that AI (SOTA but also even "basic LLM") do have 'rules' that correspond to some kind of understanding of basic mechanism. I think they have basic "world models". But these world models are optimised "to write text" rather than to "understand the world", and therefore the large majority of AI output is just not-understood token chains)

The way a LLM works is by creating a space of N dimensions, N being the number of token. This space contains all the possible combinations. The LLM will find the best combination, but will not scan the whole space. To find the best combination, it will minimize the loss function, which is low when the output corresponds to the target. By doing so, it will not explore the combination that "goes in the wrong direction", and therefore it is not true to say that increasing the space as a scale S corresponds to increasing the difficulty of running the model by a scale S.

Because of that, while the combination space scales like combinatorial, the model does not. A model with 2 weights (or rather tokens, but the number of weights should be at least the number of tokens) corresponds to 4 combinations (AA, AB, BA, BB can indeed be described by 2 binary weights of value "A" or "B"). A model with 3 weights corresponds to 9 combinations. A model with 4 weights corresponds to 16 combinations. ... A model with N weights corresponds to N to the power N combinations. The number of combination increases a lot, and yet the number of weights increase linearly.

In SOTA, we have billions of weights. That is a model that contains a very very very very big number of combinations, something so big that it is difficult to understand for a human. It will not try all of these combination one by one, the gradient descend method will help it finding the best combination without having to do so.

So, yes, SOTA are finding "the best combination" amongst an impressively huge number of combinations, yet without having to "scale like combinatorial".

> To prove me wrong (as a thought experiment), choose a lower order model, any kind you can imagine that would qualify as modeling without understanding. Demonstrate it can do anything close. That it could possibly scale to the human corpus with just a trillion parameters.

Yes. Easy. A SOTA LLM does that. It is a modeling without understanding. It does not understand, it finds the best patterns. And when you put it in a new situation, it uses these patterns to create a new text, without truly understanding the content of the text. And if you ask an additional question, it will use the previous text as context, and create a new text that, as it has been trained to, will be consistent with the output that has been given.

Your assertion "you can prove me wrong" is a circular reasoning: you start saying "if a model can do a text that looks realistic to me, then it means it has understanding. To prove me wrong, give me a text that looks realistic to me and has no understanding". Well, I cannot do that, because for you, if it looks realistic, it has to have understanding.

> If it the number of parameters goes up far too fast, then that can't be the way deep learning solves the problem with a trillion, or a few billion, either.

The combination space grows as N to the power N. So, a trillion parameters is not "just 1000 times bigger" than a billion parameters, but more than 1000 to the power of one billion bigger (the exact value is often even bigger than that). Do you realise the size of the combination space? That is 1 followed by 3 times one billion zeroes.

> What we do know, because combinatorial is too resource intensive, is we are not just combinatorial either.

I think you don't understand how LLM works: the find the best combinations in a incredibly huge parameter space, but don't need to explore the whole space, just the 1-dimension manifold that is the curve that follow the gradient descend within this huge combination space.

There are plenty of clues that SOTA don't "understand". For example, did you notice that SOTA happens to understand what human understand, and don't understand what human don't understand. If indeed the way SOTA works would be by "discovering the true mechanism", it means that it would discover with equal probability mechanisms that humans have already noticed and mechanisms that humans have not already noticed yet. For example, humans know that the Standard Model of particle physics is incomplete, and there are plenty of texts and books about that that the SOTA learnt about. Yet, SOTA did not "understood" the underlying mechanism that explain particle physics. It does not really know what an electron is by "making sense of what this object does", it only knows it as "a language word that can be used in some context in a specific way".

And, sure, SOTA is helping with new discoveries, but the way it does it is by using "reasoning" approach. If indeed SOTA creates its own understanding when learning the human language, then it should have the new discovery after the learning, without using any "reasoning" approach, because it would be something that it has already understood.

This just means "simpler representations are not enough", not "good representations cannot be complex combinatorial combinations" (complex enough that it is very different to see them for a human).

> REASON 2

Are you saying that I believe that the only way to get human-like text is by doing a near-infinite one-to-one mapping? This is obviously not the case.

You can do, for example, a GAM time-series forecast. This can have a relatively low number of weight, and still return very sensible prediction, and yet not capture the real understanding of the phenomenon they will predict. For example, it does not understand causality, just correlation.

> REASON 3

That is like saying "I've built and algorithm that is able to do 10 + 27, but there is an infinite list of number, so it is impossible for this algorithm to do 23113454453 + 1233253245". That is not true, you just decompose into (53+45), (44+32), ... and add rules to combine these elements together.

It is what is happening with AI: there is enough data to get "some pattern" in the language. Just the patterns, not the understanding of the language itself. And this pattern can be reproduced in plenty of different places.

> REASON 4

This argument is contradicted by "basic LLM" or even simpler model that are performing surprisingly well. Less than SOTA, but if your argument is true, CNN or ARIMAX could never provide better than a coin toss.

> REASON 5

Your example is a good place where the AI will _combine_ patterns learnt from different place. It will pick characteristics of each of your scenarios, and mix them together. The result will look realistic, but it is still applying learnt pattern together.

Also, you did not answered about my human arithmetic, and all your reasons are contradicted by my example there. Humans DO maths partially because they "learnt by hearth" some pattern rather than apply the understanding of fundamental arithmetic. If "answering very well based on pattern" was not a good strategy, or was necessitating infinite weights, or was making it impossible to use these patterns in novel situation, how do you explain that human can even do that themselves? As soon as we admit that humans do "some pattern some times", than we have to admit that there is a continuous spectrum and admit that it allows output that looks realistic being the result of pattern rather than understanding.

By the way, I just saw a new article reaching HN: https://news.ycombinator.com/item?id=48410427 , and it is indeed explaining similar things, and illustrates that the best way for SOTA to deal with arithmetic is by "not understanding it". And yet, when you use one of those SOTA, you would be able to argue each one of your "REASON" to pretend that the model did understood arithmetic.

How do you even know it is the case?

How do you know the output is not the result of combinatorial interactions?

How do you even know that the "sensible" response on unlikely combination is not the result of a simple recipe that "make the response sounds sensible"? Either you, yourself, have some expertise on the subject, and therefore the combination does probably exist in the AI training data, or you don't and you have no idea if the response is sensible or is the usual smooth talk that everyone could come up spending 2 or 3 hours googling on the subject and crafting something sensible.

Worse, you are saying that the model "understand", which means that it discovers the underlying mechanism that drive the output. This "understanding" is a set of equation that link different concept, that explain how one concept affects another concept. So, it is "combinatorial interaction". Not a simple linear one, but guess what, LLM are designed to introduce non-linearity.

Even when AI are able to find new solution of math problem, the result is, like when done by humans, by using existing basic tools to build more complicated ones.

> It took topological transforms, reminiscent of how we compute (dendrite-soma-axon, tensor-sum-nonlinear), and then they lept several orders of magnitude ahead of any alternative.

And yet, the LLM elements that are "similar" or "analogue" to how the human brain works are very small. The human brain has thoughts "flowing", while LLM can only work "by step". The human brain is able to learn on a very reduced dataset, while LLM need more data that a human will ever be able to analyse, even less store. The human brain has "memory" and "context" intrinsically intertwined with how it works, while you can decouple these from the LLM. ...

Finally, here is a good contradiction of having you in one side saying that AI is mimicking the human brain and it is why it works well and on the other hand saying that AI will find the lowest minimum and that this minimum is "understanding how the phenomenom works" rather than "repeating by hearth what it was told during training".

As a human, when you mentally compute 6 times 7, what do you do? Do you do: "6 follows 5, which follows 4, which follows 3, ... and 7 follows 6, which follows 5, ... so we have (1 + 1 + 1 + 1 + 1 + 1) times (1 + 1 + 1 + 1 + 1 + 1 + 1), which is 1 + 1 + 1 + 1 + ..."? I guess you probably don't, you just remember the most helpful element you remember by heart. For example, you remember by hearth that 6x7 is 42. Or you remember that 3x7 is 21, and therefore 6x7 is the double, 42. Or you remember that 7x7 is 49, and therefore 6x7 is 42. Or even have a "feeling" from a mixture of all these (6x7 is somewhere around 40 because 5x7 feels like being around 30 and 7x7 feels like being around 50, and if I think of number in the 40 that "feels" like they are from the 7-multiple-table, I remember 42).

Same thing when a human does 324x42: the majority of humans will decompose it in "simpler" multiplication that they remember by hearth and, and only then, they will combine them. It is a good example of how the brain optimise: by balancing the trade-off of "using memory" and "using understanding": basic operations use memory, but of course it is inefficient to use memory for all numbers, in which case it will use a combination of both.

The way human do basic math operation is not purely by "understanding" arithmetic, it is by relying on what they remember from their training. At the same time, humans know how arithmetic works, and they will use it when relevant. Yet, the human brains prefer to rely on some "learnt by hearth" elements. This is in contradiction with your assertion that optimisation will always lead to "understanding" and that human brains is optimizing the same way AIs do.

This is only one example with numbers, but of course it works with plenty of other things. This is also exactly why humans get "the wrong idea" on plenty of phenomenon, that are then described as "counter-intuitive".

The reason "by hearth" is part of a good strategy rather than "purely understanding" is because there is a trade-off between "memory" and "compute", in both the human brain and AI: it is easier (and therefore a stronger attractor during the optimisation of the process of "getting the correct answer") to do the faster operation "retrieve from memory" than to do the slower operation "retrieve the theory from memory, compute the first step, store it in the short term memory, compute the second step, store it in the short term memory, compute the final answer by adding the first step answer and the second step answer".

The comment you answer to says that their experience is that AI and the human brain are not analogous and that AI is good to store large amount of knowledge and repeat it (or extrapolate based on pattern on the large amount of knowledge), but bad at understanding the code as a human does. Which explains why a human is more efficient when reacting on a thing that don't have a lot of documentation (on which the AI built its knowledge).

Humans are bad at storing large amount of knowledge, and this is why we need supervisor for human.

AI are bad to understand new stuff, they need to be able to connect the new stuff with a lot of examples they have been trained on (it does not mean the stuff is "identical", but it means "connected"), and this is why we need supervisor for AI.

We need supervisors for both human and AI, but for different uncorrelated reason.

First, the "13+7" is an analogy. In this analogy, "13+7" is not the real question you ask, it represents _any questions_, not just arithmetic.

But secondly, did you even noticed that in my example, the system answer CORRECTLY "13+7"? So, in my example, the thing I'm talking about and I argue does not "understand" is Claude, even if it is able to answer correctly.

My point is: the "basic LLM" part is creating a mechanism that answer without understanding (as demonstrated for example by ChatGPT failing arithmetic), and the fine-tuning or the harness is just hiding the lack of understanding by adding ad-hoc correction on the residuals. And because it is on the residuals, it looses the logical links (13+7 -> 20 is "logical", it corresponds to the math logic, it corresponds to what you get when you add 13 stones and 7 stones together. The residual is "14 -> 20", which has no meaning in itself)

The ad-hoc correction is either: 1. by training the model so it learns by heart, without understanding, that the symbols "13+7" should lead to "20", 2. or by training the model to use a pocket calculator without understanding arithmetic so it can do it itself.

You can prove that the model does not understand it very simply. Let's take the normal fine-tuned model M1. Now, let's go back to the pre-tuned version, and fine-tune it so it answer "21" to the question "13+7", and use an harness that does "sum(x, y): return x+y+1". This is model M2. M2 will fail to answer "13+7" correctly, it will say "21". And yet, M2 has been trained exactly the same way M1 was. If it is true that the additional tuning "add understanding", M2 will not be possible, it will say "error, error, do not compute, you try to train me to say that 13+7 is 21, but it does not make logical sense to me". But it does not happen: the pre-tuned model has no idea that 13+7=20 is more logical than 13+7=21, and the additional tuning is just helping him returning a more correct answer while still having no idea where this answer comes from.

This feels a bit like whataboutism.

It also feels like people don't listen to each others.

For example, reading the previous comment, it feels like the thing that reduce the enthusiasm was that at first GenAI looks like it was "reading, understanding and using its own knowledge to answer the problem", but as soon as it is a ore niche or a more complex situation, GenAI looks like it "does not understand the code, just does the equivalent of a StackOverflow search and try to apply the solutions that it found there, and this is why it felt like it understood the code before".

It does not at all means that GenAI is not terribly useful. And even better than humans in some situations.

But it feels that answering "same with humans" is missing this point: that's the opposite, humans usually try to understand the code and are bad at covering a very large range of very well documented subjects. That's the "uncanny valley" they talk about: they assumed GenAI performance on a subject X is due to a "human-like" approach, and it feels very strange when this impression falls apart.

I'm happy both ways:

Either you say that a pocket calculator understands arithmetic, and that LLM understand language, which is something trivial. If a pocket calculator understands arithmetic, than previous substitutes to calculators, such as an abacus, do too. In this case, a word dictionary also understand language. And it is basically what Chiang's article says: the LLM don't understand language more than a word dictionary does. If you disagree with Chiang, it looks like you do only because you don't understand what he is saying, or somehow are not mature enough to realise that Chiang may use a different definition of "understanding" than yours in a fully legitimate way, like everyone is always doing when talking about plenty of subject.

Or you pretend that a pocket calculator understanding of arithmetic is somehow different than the one of an abacus or other obviously inanimate object who are obviously not thinking.

Have you read the article in question. It is saying that for one continuous thought, the brain will use different part of the brain to do different thing. It does not say that there is a "loop controler" anywhere. On the contrary, it illustrates that there is no loop controller: there is not special brain function that control this loop, this loop is "how the brain works", and LLM don't do that, they are incapable to do that, it is not how they work.

> Is your argument that, because they're external to the Llm, rather than integrated, they don't count, not even in a practical sense?

No, my argument is that the nature of the brain and the nature of the LLM are very different, as different as a real paper plane and a video game paper plane. Some characteristics (for example, awareness) that exist in the brain cannot exist in the LLM because these characteristics are the result of the nature of the thing in question.

The problem is not that you build a system by integrating 2 things together. The problem is that they are different "things", they are different machines, they function, fundamentally, differently. They may produce the same output, but when you say "the brain has the characteristic X, the LLM produce the same output, so the LLM also has the characteristic X", it is logically inconsistent.

Planes are built as a system combining 2 things: a motor and some wings. But they are fundamentally different from a bird. They just don't "work" the same. It is not the same mechanism.

> you should know that nobody has a practical use for plain LLMs these days

That is totally irrelevant. My point is about the nature of the LLM, and the fact that it is stupid to see the same output and to conclude that they have the same characteristic. It is like saying "Birds are flying in the air and are alive. Planes are flying in the air, so I guess they are alive".

> LLM don't have a some self contained loop, like we do, sure. Who cares though. The actual AI system that we use every day definitely do.

No, you miss the point. The problem is not that "you can just add an external loop". The problem is that the brain is a system that works without such control loop. The thoughts are flowing (and they may flow to different brain functions, like explained in the article you quote). It is part of how the system works. Having a system that contains 2 things, one that does one computation and one that control the loop is not equivalent to another system where you cannot decouple the "flowing of the thought" from the "thinking machine".

OP says: LLMs never think when not invoked.

What you said: I have example where, sometimes, human think when invoked.

That's the difference: human brains are intrinsically different because they are built to be able to think without being invoked, even if there are situations where they think when invoked.

There are tons of obvious examples of human thinking without being invoked. Just take a bath and you will see :)

In real life, the paper moves "by itself". It does not need an external loop that update its position in a loop manner.

In the video game, you need an internal loop, a step-by-step tick, that update the plane position based on its current position and its momentum. And this is why a video game paper plane is not a real object. It is a very good simulation, it looks like it, but it is missing some intrinsic properties that we expect from a real object.

Yet you can analyse the paper plane trajectory and see it as a Markov chain, with quantified step-by-step progress (for example one position point every 0.1 second). The same way you can look at your though process and identify a step-by-step progression. But it does not mean that it works like that intrinsically, it does not mean that the paper plane "jumps" from position point at time T1 to position point at time T1+0.1 second.

For the human brain, there is no "loop centre" in the brain. There is no one (to my knowledge) who got a brain injury and suddenly were unable to keep a single line of thought without having someone else having to feed them the previous thought in order to feed the next thought.

In the brain, the fact that the previous thought feeds the next thought is "how it works", it is intrinsic, it is by design. And this mechanism of thoughts feeding the next thoughts is what creates "consciousness" or "awareness": self-reflection is based on the fact that thoughts are intrinsically linked together, that they "flow" continuously, without needing an external system to update them.

You cannot take away the "loop" part of the paper plane so that it suddenly would be unable to move on its own once thrown away.

Now, you can always say "well, the paper plane in the video game is a very good simulation, it does not matter if it is a real object or not", and that is fair enough. But in this discussion, some people have arguments to support that this property matters, that it is one condition for consciousness or awareness.