All businesses seek to survive, and will use human goodwill until it is not needed anymore. Everyone who thought that Google was opening up the web out of the generosity of their hearts will be shocked when they "feel" nothing when that is taken, because ultimately a company cannot "feel" anything at all, so the OP headline is a silly proposition.

I feel like people grasping straws on the shrinking limitations of AI systems are just copying the "god of the gaps" fallacy

What is preventing AI from continuing to improve until it is absolutely better than humans at any mental task?

If we compare AI now vs 2022 the difference is outstandingly stark. Do you believe this improvement will just stop before it eclipses all humans in everything we care about?

In certain domains there are no ceilings to return on compute. For example, offensive/defensive cybersecurity, it will be an arms race of who has more compute to patch up against more sophisticated attacks, and who runs more sophisticated attacks

>Americans will get to enjoy their rusty infrastructure and polluted air,

America has very high air quality and fairly good infrastructure given their population density.

1. Quantization: bigger models can be compressed and retain most of their quality, but run with a far smaller compute footprint

2. Moore's law: chips still are scaling so you can run more compute cheaper as it improves

3. Open source competition: if models from Anthropic or OpenAI get too expensive people will opt to use open source Chinese models, which would reduce demand and thus reduce prices down to a more reasonable equilibrium

This will basically become true for everything.

1. Native understanding of most languages

2. Full knowledge of everything in the company's knowledge base

3. Is never rude, annoyed, lazy

4. Can work 24/7

Of course developers could just move up the “next level of abstraction” and become managers of agents who write the code, but eventually AI becomes a better manager of agents than even the best humans, at which point there is no contribution a human can make that an AI model or system of models couldn’t do better.

The average person may not be an interesting model for getting the most out of life in a short time in any particular place, but the top 0.1% of people measured by the texture, quality and interesting-ness of their lives exceeds any metric of "noteworthy events per hour" by a factor of 100.

I asked the question to the default version of ChatGPT and Claude and got the same "Walk" answer, though Opus 4.7 with thinking determined that it was a trick question, and that only driving would make sense.

All the limitations you are describing with respect to LLM's are the same as humans. Would a human tripping up on an ambiguously worded question mean they are always just faking their thinking?