So maybe the big revolution is that we won't have "big software" that tries to cater to all kinds of users but "small software" that is tailored perfectly to the needs of a single user.

Then again maybe that's where all software is heading, if AI can produce features on demand at infinite space there might no longer be specialized software, or at least not nearly as much, as people will just code their own solutions tailored perfectly to their use case.

On the weekend I wrote a full-fledged 3D planner where I modeled my whole house based on plans and written descriptions I handed to Claude, it runs in the browser, produces high quality renderings, has path tracing for realistic lighting and materials and builds real objects like windows based on specifications and blueprints, so you can actually see how stuff looks like (unlike with AI images where proportions are always off). For me it's way better than SketchUp and it works completely dialogue based and allows me to experiment with furniture, window styles, doors, colors etc. in a super intuitive way. There is software I could use for this but why bother at this point? This whole thing took me a few hours and maybe 10-20 € in AI credits to build. Why try to use open-source software where I need to wait weeks for a reply from a maintainer?

So I think these projects fundamentally misunderstand where software engineering is heading, they are trying to hold back the flood but that will only lead to them being left in the dust once people realize they no longer have to wait for maintainers to update their sanctioned software timeline for you. That said, I think it's also ok and maybe even beneficial to be very protective of your feature set. There will (at least for now) still be room for complex core software that can serve as a basis for AI modification, so from that perspective it might be good to keep the core really tight and fully audited. My little software project only exists because three.js and other libraries provide most of what I need as LLMs will not write everything from scratch in such a short amount of time.

1: https://simpsonswiki.com/wiki/The_Homer

Overall it's a good thing, I think we will see the big studios loose footing in the gaming market even more, with the whole Microsoft restructuring blood bath we can already see it, big studios and brands still have their exclusives for now but I imagine they will lose market share overall since there's just so much exceptional stuff being produced by indies, and a good indie game can be just as much fun as the most expensive AAA title.

So, while this is great and I think LLMs will accelerate materials science the main bottleneck isn't having enough promising candidate materials, I think we have a backlog of at least a few hundred candidates that are worth pursuing. Maybe some money that goes to data centers would better go into CVD machines.

But yeah I noticed how they play with your emotions, it's a bit creepy. Whenever it asks a hook question and I tell it "Actually I did X and Y" it will invariably go like "Wow that is such a brilliant and well thought out strategy, almost a stroke of genius!" (not literally but you get the idea). I guess they found out that people really like having their ego stroked, the other providers are also doing that to some degree (I assume since no one likes an unfriendly AI that makes you feel stupid) but Google really cranked that setting to 11.

That got me thinking that programming might be like that in the near future, it's still an impressive skill for a human to learn and master and it takes tremendous effort and the craft itself is beautiful, but economically it will just not matter anymore. And in the same way as the music industry today is bigger than it ever was and there's tremendous money in it but 95-99 % of highly talented musicians will never earn a decent wage with it, the software industry will be bigger than it ever was but the money will be highly concentrated in the hands of a few players. Maybe we will still have some rockstar programmers than can masterfully command legions of agents or that win the love of the crowd for their unique work, but most of us will be enjoying this craft as a creative hobby and will do other things for a living.

Then again, coming back to the music parable, for a long time "amateurs" can make music with software tools without having any talent or even knowing musical notation or anything about tempo. Does that make them musicians? So take solace in the fact that these people aren't programmers and never will be, they're just pushing buttons on a synth board watching the "music" that comes out. Mastering a skill, regardless of what it is, will still matter to people, it will just not be worthwhile economically for most of us. There are so many things in that category, running for example isn't worthwhile for most of us in an economic sense as you have to be at the top 0.1 % of runners to make a living with it, still it has tremendous benefits and brings enjoyment.

I guess there's no reason to believe these models can't be as smart as a great software architect / engineer or team of such people that build an elegant and maintainable software solution over many years together based on customer feedback, then again the models are appallingly bad at some forms of reasoning, I mean they will "understand" something once you make them aware of it like e.g. a flaw in the software architecture, but when asking them to audit the code and check for issues they will often have a blind spot to finding such problems. It's interesting, like they have very high ability but very little awareness or self-directed thinking outside of the prompts they receive.

And maybe let's not only hear the opinion of two or three Fields level mathematicians with blogs, 99 % of the worlds mathematicians in academia might profit from these tools as they might partially close the gap between them and the world elite, making creativity and tenaciousness more important than having the right neocortical structure allowing you to outperform 99.9 % of other humans at keeping context in your head and making predictions, AI can do that better now with the right prompts.

As a simple experiment, try giving AI a high level goal for your software and let it iterate on it by just repeatedly prompting it to continue, it will happily churn forever on the goal, turning the codebase into a useless spaghetti mess with very high probability, and growing it more and more without ever cutting anything back. That's what happens without human intervention regarding system state and manipulation. The main issues here are most prompts that are extremely underspecified ("fix the issue with the buttons on the main page") so AI will ingest context data it likely generated itself in a previous step and assumptions from its own training data, then act on that to produce a new state. Think of it like a random walk, the AI makes a small step in one random direction to achieve a goal, that brings the system to a new state which is now the basis for the next step, and so on. If there's no (or not enough) corrective action that pulls the system back to a known good reference state it will keep wandering in random directions.

That's the main issue, people have a hard time steering recursive, probabilistic systems, especially when they never look at the output of the system after each step and correct it. And let's be real, if you examine AI generated output in great detail after each iteration you're often better off writing the code yourself, so I would argue that the promised speed up of agentic development can only be realized if you stop inspecting every output of the system. And it seems we still haven't figured out how to specify the steering instructions that keep a system close to a given ideal state that allow unsupervised, recursive work on most codebases. I think some codebases are by themselves better suited for this as they provide a more rigid harness for AI development and exist in the training data (e.g. CRUD apps using RoR), whereas complex software that doesn't use rigid frameworks is at much higher risk of destruction by AI as there's no reference point in the training data that would hold the AI back from randomly walking to a garbage state.

And that's why people have such different views on agentic software development, some work on codebases that are better represented in the training data and so have great success using agentic tools on them, others work on software that isn't represented so well so AI does poorly on it. I don't think it's an issue with quality management, from my own experiments no amount of hand-written rules or system prompts will keep AI from destroying a codebase for which it doesn't have a strong idea how the code is supposed to look from its own training data in the first place. As another experiment, try giving AI strict rules about how to change code or introduce new features, it will always find a way around them or appropriate them in a maliciously funny way that you haven't anticipated. That's also an artefact of the training process, these systems aren't designed to say no or do nothing, they produce outputs to achieve goals and they will bend your rules to the greatest amount possible if it helps with goal fulfilment.

I recently tried writing a paper with Claude and it was an absolute disaster, I spent hours (days?) instructing it about writing style and pointing out anti patterns to avoid, but I couldn't get it to even produce simple sentences, it would always add unnecessary lead up sentences, put the most important information at the end of the sentence, use the typical "It's A, not B!" or "B, not A!" type sentences. In the end I gave up and edited everything manually. Makes me wonder how AI can be so smart that it poses a human-level extinction threat but can't seem to even write a simple paper based on facts and information you spoon feed it. I now think it's an intelligence illusion due to the training data and optimization process being hidden from us, essentially it keeps working better and better because we invested massively in optimization of specific use cases like coding, where users contributed billions of training samples that are part of the LLM model. The same is true for text-based workflows and others, the sampling density of the training space is getting much better due to the massive use of AI everywhere so the models extrapolate better between the different instances, but I'd wager they would still miserably fail to generalize to things that are outside of the most common training use cases now. That's why I am also very skeptical about recursive self improvement of these systems, look at what happens when you let agents work recursively / in a loop now, they just keep piling more garbage onto garbage and choke on their own output. I have observed it in my paper writing as well, you feed input into the AI system, the system produces output, the next paper iteration works on that output but the AI doesn't differentiate properly between it's own output and your original input, that pollutes the next output which is then used as input again, eventually the system just churns on its own hallucinated/fabricated outputs until the result is complete garbage that no amount of steering will fix. The same is true for most vibe coded software I built with AI, it holds together decently initially, but the more AI code and decisions accumulate the more the system operates on its own outputs and keeps piling more output on it. More than anything we really need a way to keep system data accurately tagged, i.e. clearly mark human input from AI output and keep AIs churning on output data that it produced itself but treats as input.

I find people that are the "unconscious incompetence" stage of software development absolutely love them because they produce output that looks fine to them and they can't find anything wrong with it because they simply lack the ability to perceive quality. For a contact widget that might be acceptable as it will be used a few dozen times so who cares if it breaks, but let's not pretend you can actually build a compliant e-signing or invoicing software without having domain expertise and investing a ton of editing/revision time.

Put more bluntly, a domain expert that, as of currently, refuses to use LLMs can easily start using LLMs in their work, an idiot who uses LLMs cannot easily become a domain expert. Now let's think about who you would rather hire or who do you think will have the better future career outlook. Pretty damning how many people here are willing to reject the idea that domain expertise or ability to write program code yourself is a waste of resources.

So I'm greatly excited what AI will bring about in physics, more so than in math, because in physics it's clear that our fundamental theories are missing a big piece of the picture, and given how easily AI crunches through Millenium prize problems I think it's possible that AI will come up with a viable grand unified theory uniting quantum mechanics and gravitation, or produce new predictions in other areas. There's enough contradictory or unexplained observational data available to make a ton of progress on the theory side I think. Exciting times ahead!