Largely so from a programming perspective it becomes a simplified version of Einstein's Riddle that I showed the class, doing in a similar way.

https://theintelligentbook.com/willscala/#/decks/einsteinPro...

Where at each step, you're just eliminating one or more possibilities from a cell that starts out containing all of them.

Queens has fewer rules to code, making it more amenable for students.

https://theintelligentbook.com/supercollaborative/#/challeng...

(Albeit I made mine simulate how things like VS Code look while you're doing it a bit more)

From a quick read, they are essentially asking people to predict whether they'll be better/worse off next year, and then correlating that with whether they were right in the next year's financial data.

Their study period looks like it's the period immediately post-GFC (2009 to 2021). As I understand it, the economy has performed better for those in professional roles, who are likely to score well on the tests of cognitive ability, than it has for lower-skilled jobs. And of course in the middle of that period, there's the Brexit vote that was pretty uneven in terms of who thought the economy (and their circumstances) would go well or badly.

It's an interesting data set, but I don't think I'd be game to make the same "causative" conclusions that they do ("Our findings suggest that these supposed consequences of optimism bias, may be a side product of the true driver, low cognitive ability").

LLMs do not directly model the world; they train on and model what people write about the world. It is an AI model of a computed gestalt human model of the world, rather than a model of the world directly. If you ask it a question, it tells you what it models someone else (a gestalt of human writing) is most likely say. That in turn is strengthened if user interaction accepts it and corrected only if someone tells it something different.

If we were to define that as what "understanding" is, we would equivalently be saying that a human bullshit artist would have expert understanding if only they produced more believable bullshit. (They also just "try to sound like an expert".)

Likewise, I'm not convinced that we can measure its understanding just by identifying inaccuracies or measuring the difference between its answers and expert answers - There would be no difference between bluffing your way through the interview (relying on your interviewer's limitations in how they interrogate you) and acing the interview.

There seems to be a fundamental difference in levels of indirection. Where we "map the territory", LLMs "map the maps of the territory".

It can be an arbitrarily good approximation, and practically very useful, but it's a strong ontological step to say one thing "is" another just because it can be used like it.

https://theintelligentbook.com/thinkingaboutprogramming/#/de...

(programmable lander)

If you're doing URL routing, you're at the mercy of your web framework

If you've got a data structure coming in, you're probably turning it into types in the programming language you used fast, and you're more quickly into the land of plain old programming, rather than routing configuration in a web framework.

Although that sounds large, it's not in corporate terms. Charging the customer 2.3 to 3 times an employee's salary seems about standard. And universities are notorious for casual using work to an underpaid "precariate".

That suggests there isn't a whole lot of margin for technology to capture in the first place.

After which, you find that degree-awarding is a regulated activity in most countries, offered by public institutions subsidised by government loan and contribution schemes.

Tech has had a massive influence on higher education in the last ten years, but it seems to go via universities buying more systems and services rather than "University of the Unregulated Start Up" suddenly sweeping the public-controlled market for education.

E.g. an academic at a prestigious university has a healthy supply of able PhD students, post-docs, a "research environment" that will make applying for grants that bit easier, etc. Their publication and citation numbers will quickly diverge from their identical twin who has is less well resourced. Likewise, the PhD students at a prestigious university are more likely to be attached to well-funded grants, collaborators who have well-tuned paper mills, etc.

Academia is quite a social game. Network effects (which are one part of prestige) are a strong influence.