This happens if the model does not have good context regarding what is being built, why and for whom.

For one of my side projects, my CLAUDE.md has a line that says "the users of this product will be non-technical people who find technology scary." And you know what? Claude always steers me away from complexity when working on that project.

This is a pretty shitty thing to say, and if you think about it for even a few minutes, you'll understand why.

New features have actually been remarkable. For me they are almost always in one of two categories: things I've always wanted (but never actually bothered communicating to the devs), and things I didn't know I needed but did. The product team is clearly very tuned in.

All in all, I'd say their approach is working.

Yeah. OpenAI basically said the same thing with regards to having Astra available to their team internally over the summer.

Bro come on.

This is completely normal if you haven't written your own custom skill with instructions on what types of issues the AI should emphasize, which ones would be considered nits and which ones aren't a problem at all.

In our repos we use a classification system: blocker, should-fix and nit. Each one has specific definitions, criteria and examples encoded in the skill file. When the time comes to review a PR, agents invoke the skill, and frankly do a stellar job. A human then reads each finding, asks the AI follow-up questions and makes the final decision in terms of whether the finding goes in the PR review.

The reason I know this works is that we have one guy on the team who does not use this skill, and blindly throws his agents at PRs. And the results are exactly as you describe.

This, I find completely unbelievable. Because we had (emphasis on had) seasoned engineers on the team who similarly eschewed AI tools and insisted on doing everything by hand in the manner you describe, well after the rest of the team adopted AI. Yet when it came to code reviews, even in parts of the codebase they were familiar with, the bugs they found often came down to nits, bike-shedding and opinion-based feedback (that they usually tried to frame as objective fact). They would also disagree with almost every AI finding, arguing that it was an unrealistic scenario or an edge case not worth worrying about.

Fundamentally, I don't think humans are going to be capable of providing high quality feedback on PRs authored by AI agents unless those PRs are fairly small in lines of code and volume. It's just way too much information and context for one person to keep in their head. I read a statistic that said the average lines of code a senior engineer can read and provide good feedback on is about 400 per hour, and that number goes down the more time they spend doing code reviews. So, to anyone who insists on trying to keep up with AI, I say: good luck.

Nah. There are benchmarks. They are free to look at. And they paint a very clear picture.