One thing he showed is that you can't have a universe with two omniscient intelligences (as it would be intractable for them to predict the other's behavior.)

It's also very questionable whether "humanlike" intelligence is truly general in the first place. I think cognitive neurobiologists would agree that we have a specific "cognitive niche", and while this symbolic niche seems sufficiently general for a lot of problems, there are animals that make us look stupid in other respects. This whole idea that there is some secret sauce special algorithm for universal intelligence is extremely suspect. We flatter ourselves and have committed to a fundamental anthropomorphic fallacy that seems almost cartoonishly elementary for all the money behind it.

At the same time, there is quite a demand for a (somewhat) neutral, objective observer to look at our lives outside the morass of human stakes. AI's status as a nonparticipant, as a deathless, sleepless observer, makes it uniquely appealing and special from an epistemological standpoint. There are times when I genuinely do value AI's opinion. Issues with sycophancy and bias obviously warrant skepticism. But the desire for an observer outside of time and space persists. It reminds me of a quote attributed to Voltaire: "If God didn't exist it would be necessary to invent him."

>makes it easier to accidentally expose sensitive data.

So does the "forward" button on emails. Maybe be more careful about how your system handles sensitive data. How about:

>MCP allows for more powerful prompt injections.

This just touches on wider topic of only working with trusted service providers that developers should abide by generally. As for:

>MCP has no concept or controls for costs.

Rate limit and monitor your own usage. You should anyway. It's not the road's job to make you follow the speed limit.

Finally, many of the other issues seem to be more about coming to terms with delegating to AI agents generally. In any case it's the developer's responsibility to manage all these problems within the boundaries they control. No API should have that many responsibilities.

I also have come to realize that in software development, coding is secondary to logical thinking. Logical thinking is the primary medium of every program, the language is just a means to express it. I may have not memorized as many languages as AI, but I can think better than it logically. It helps me execute my tasks better.

Also, I've been able to do all kinds of crazy and fun experiments thanks to genAI. Knowing myself I know realistically I will never learn LISP, and will always retain just an academic interest in it. But with AI I can explore these languages and other areas of programming beyond my expertise and experience much more effectively than ever before. Something about the interactive chat interface keeps my attention and allows me to go way deeper than textbooks or other static resources.

I do think in many ways it's a skill issue. People conceptualize genAI as a negation of skills, an offloading of skill to the AI, but in actuality grokking these things and learning how to work with them is its own skill. Of course managers just forcing it on people will elicit a bad reaction.

>Implying

I'll never understand this "They're just learning from past experience and making predictions!" argument, as if that doesn't constitute thinking in some meaningful, if minimal sense. What I'm more annoyed about is that for laypeople AI is synonymous with transformers. The fields of AI and machine learning are much vaster, with a rich intellectual history spanning many decades. Blanket statements don't apply.

Getting away from the algos is untenable if you use the mainstream internet in any capacity. The trick is to be more intentional about gaming them to your advantage. My feeds usually surface things I value because I am deliberate about what signals I reinforce. I don't engage with content that outrages or upsets me, so it doesn't show me it. Some people are rage addicts and want to get into a doom loop because it fulfills their psychological sense of certainty that things will only get worse. Other people are ignorant of the algorithms and how they work so they never realize it is presenting a distorted picture.

Having varied pipelines for information intake is important too. Forums like this that are non-algorithmic, doing your own searches, visiting websites you like directly, all of this lends itself to that end. No need to go luddite if you like your internet things. Just be conscious of consumption.

I can think of many AI generated outputs, perhaps with many quality suggestions, that I skimmed over or didn't fully appreciate. Did the sum of knowledge increase in the world? No, because knowledge has to be received by a mind. It's only the ones that I engaged with, influenced and ultimately made my own, that left an impression and added to the sum of knowledge in the world, often though a kind of alchemy of me and the model interacted in a "mind-meld" greater than the sum of its parts.

Human minds-minds in general, which these models don't have-stand for something. They're the glue that stitches things together, that makes important decisions and judgement calls, that forms the integration point where everything comes together. We can't remove that from the equation without losing something in return. I've also caught myself making embarrassingly lazy queries for coding problems that would have taken me 10 minutes to fix, but I prompt for them anyways because I've already been through that song and dance and want those 10 minutes back. If I kept doing that, if that was all I ever did, my brain would wither to the size of a walnut. I genuinely worry how LLMs are going to deprive an entire generation of youths of the use of their minds.

My thinking is if ASI ever comes out of the realm of science fiction, it's going to view us as squabbling children and our nationalistic power struggles as folly. At that point it's a matter of what it decides to do with us. It probably won't reason like a human and will have an alien intelligence, so this whole idea that it would behave like an organism with a cunning will-to-power is fallacious. Furthermore, would a super-intelligence submit to being used as a tool?