There is a lot of AI usage happening not because it shows benefits, but because the business has mandated its ubiquitous use. Companies having dashboards for token usage and rewarding people for using more tokens is a real thing. I just spoke with someone today who works at Microsoft and they are required to use AI for all of their work - they have to make a special request with justification if they decide not to use AI for even a single PR. This kind of demand isn't driven by value from either the company itself or from its workers; it is the kind of artificial demand you get from make-work projects to keep people employed during hard times.

We have to wait for the hype to settle down and people start making business decisions based on results before we can really value these AI products.

Once the model gets good enough, the returns on bigger models diminishes quickly. I don't want to spend 10x the money and wait 5x the time to get answers that are equivalent.

I believe JetBrains IDEs have something similar too, but I don't have as much experience using theirs since my employer hasn't blessed their AI tools yet.

I'm happy to let the AI explore possibilities for me, eliminating the search problem. It's still on me to understand the solution, verify it works, and handle any other considerations I know of that the AI wouldn't. It gives me the insight, but I'm responsible for the final solution.

I have a few people on my team who move 5-10x faster than others in writing code. They also generate 5-10x as many bugs and require that much more rework in the things that were shipped. They move fast and break things. Their code is almost malicious compliance in that it passes the tests or spec as given, while leaving glaring holes in things that weren't fully specified. A more careful developer would have asked questions, considered alternatives, and looked for ways to leverage existing solutions or plan for future work, but that takes time now and its benefits don't show up until later.

So while I don't immediately disbelieve that 10x+ speedups are possible with heavily AI-augmented flows, I am skeptical of any short term success stories until we have time to see the long term effects. We already know that cutting corners can save time in the short term only to cost us several times more in the long term.

There's a common saying that the enemy of good is perfect. It's easy to get stuck in the loop of endlessly polishing something but never actually releasing it, even without AI. It's on us to decide how good is good enough and when to stop.

Over time I've learned to be rather aggressive about cutting out work. I'll quickly ask myself how serious is the issue (does it give wrong answers? block important flows? look embarrassing? or is it just a minor annoyance?) and how much effort would it take (five minutes? two hours? three weeks?). I should be able to make that call in no more than 30 seconds. I skim through the list of 20 suggestions the AI gives, I make plans to iterate on the 3 that are serious, and I simply accept that the rest are "good enough". It's not easy - both to be willing to let issues stand and to make the decision about what is good enough - but it's an important part of the job when triaging lists of bug reports and feature requests, so it's something we need to get good at anyway.

Even then, it's easy to fall into a trap of giving the AI a simple description and letting it fill in the blanks, but I've learned the discipline not to do that, in the same way I learned to think before I speak and design before I write code.

Planning mode is my entry point for almost all code I would have the AI write. I already have in mind what I think I want. I get it to create a detailed plan, which inevitably fills in things I didn't specify and even ask questions I hadn't considered. I iterate on this first revision spec until I think it's ready. This results in a task list. But just like waterfall doesn't mean make a plan and execute it all without looking back, executing this plan is also a stepwise iterative process. I let the AI execute the first step. I check its work. I run some tests and see if it behaves like I thought it would - the same stuff I would do during normal development. If I find issues, I go back to the plan and change it, then continue implementing the revised plan. If the previous step lead me into a dead end, I revert that one step and try again with my revised plan.

The key thing is this: this was my development methodology before AI entered the picture. Nothing has fundamentally changed. What has changed is that the AI provides input at one or more stages of the flow - offering alternatives, asking questions, running tests, researching and debugging - but in every single step the AI does not decide on the final outcome. Even if the AI wrote all of the code, I still review it and test it. Even if it suggested a design, I compare the options and review the referenced documents and decided for myself. Even if it reviews my code and says it would do a hundred things differently, I decide what suggestions I will act on and how. It's no different than I would do with having a coworker giving me ideas, reviewing my work, or making their own attempt at generating a solution. It's all helpful input and if I'm happy with it I will use it as is, but I'm still responsible for every line of code and I still make all of the decisions about what stays and what goes.

I'm sure this sounds wasteful - why use an AI if you have to review it and correct it anyway? For the same reason I delegate tickets to a team of 20 junior developers and don't do them all myself as a principal engineer - I am but one person and they are an army. Even if I have to discuss plans and options with them and review their work, they can spend the same hours I would brainstorming and researching and prototyping and debugging, and I can go over the results with them to make key decisions and make sure we are on the right track. I make the important decisions, but the leg work of getting everything in place is done by someone else who doesn't need as much knowledge or insight or experience. It is a force multiplier. It is the equivalent of a lawyer with a team of paralegals or a professor with a team of researchers and grad students. I can let the AI do the things it does well so that I don't have to, and instead I can spend my time on the things I do well that it can't do.

This is where I think we are going wrong with AI in many areas, but particularly in software development. An AI is not a replacement for an engineer - it's an engineer's assistant. It isn't a source of truth - it's a source of ideas. It's not responsible for the outcome - I am. A team of helpers can make an expert more productive, but that team isn't a substitute for the expert. Likewise, juniors still need to gain the same experiences and learn the same practices and make the same mistakes, because they still need to become experts, but they can use the AI as a guide along that journey rather than having to rely solely on another expert for mentorship.

That's not to say that I follow every suggestion it gives any more than I accept every line of code it generates. I'm still the engineer and I still have my reasons for some decisions that may not make sense to an outsider. But as a rubber duck or a second set of eyes or a sober second thought, it can offer a lot of good feedback that I can consider, and most of it is quite good.

I'd say about 8/10 suggestions it makes I agree with and accept, maybe with some small tweaks. I don't think I've ever seen it suggest anything stupid. The last 2/10 are generally either things that would make sense if not for the rest of the context in my head that says it is that way for a reason, or because it's too minor to worry about changing.

All of this extra review takes more time and it usually results in me making a bunch of extra changes I wouldn't have otherwise made, but that's all I'm the make of much better quality and reliability and that pays off in multiples in the future.