This all seems orthogonal to reasoning, but also who is to say that somewhere in those billions of parameters there isn't something like a model of goals and emotional state? I mean, I seriously doubt it, but I also don't think I could evidence that.

The problem is that these sorts of things were thought to require some sort of understanding of general intelligence, when in practice you get solve them pretty well with algorithms that clearly aren't intelligent and aren't made with an understanding of intelligence. Like, if you time travel back 100 years and told them that in the future a computer could beat any grandmaster at chess, they might consider that a form of AGI too. But we know with hindsight that it isn't true, that playing chess doesn't require intelligence, just chess prowess. That's not to say that GPT4 or whatever isn't a step towards intelligence, but it's ludicrous to say that they're a significant advancement towards that goal.

the words "based on" are doing a lot of work here. No, we don't know what sort of stuff it learns from its training data nor do we know what sorts of reasoning it does, and the link you sent doesn't disagree.

Moreover, are you sure that e.g. loving people in non-algorithmic? We can already make chatbots which pretty convincingly act as if they love people. Sure, they don't actually love anyone, they just generate text, but then, what would it mean for a system or even a human to "actually" love someone?