I think those kinds of questions would be more interesting than the raw win/loss ratios.

This is what it seems like so far for software. The speed limit has changed. It’s interesting to explore what that means. Extending the analogy, long ago you could be a race car driver part time; that would be pretty laughable for an F1 driver. It’s a very specialized skill and the optimal athlete is quite different for driving a car than for running, for example. That feels like what I’m seeing in software.

The most effective folks are much faster and better with these tools than average. It’s a different skill set.

I think that formal verification makes Math much closer to code than many other fields. But I’m eager to see the first mathematicians that are dramatically more proficient with the new tools than average.

It was something I had never considered. Not for a moment. Books were sacred to me. Cut one? Blasphemy.

But he was right. And so I cut it. And it fixed all of my complaints.

What I learned is that some folks have a different mindset; they see the world as something to be shaped around themselves, rather than seeing themselves as something that should be shaped to fit the world. That’s been a useful lesson sometimes.

Long story short, I think we’re in a phase where the organizational value function is lagging behind the tech. A “proof of concept” used to be correlated with “proof of work” and some amount of domain understanding, but I think now what we need is a focus on “proof of understanding” or else you’re probably just wasting tokens on a baby version of the problem. A decent proxy right now is that if you have zero external dependencies then your solution is probably a toy.

Honestly, when I ran this interview I didn’t care much about the specifics of what you memorized beforehand. I care if you can read and write code a bit. I care more whether we can have a productive conversation. If you learn something new from me or the problem, how does that look and feel? If I make a mistake, how do you react? Are we able to communicate technical ideas to each other? Are we able to productively work through conflict?

We’re not computing many medians day-to-day, but we’re doing all those other things constantly.

The more pernicious effect I’ve been seeing is that we’re living in the golden age of LLMs, but eventually that’ll fade. Tokens are subsidized and cheap, model capabilities leap forward regularly, and there’s competition driving it all. But even now there’s stories about frontier models suddenly becoming less capable, or providers switching to usage-based billing, and new model releases feel a bit more sluggish and less dramatic. (Fable/Mythos notwithstanding.)

Eventually the models are going to settle into a rut of being just “good enough” to earn a living rather than all this hoopla. A lot of people will be re-hired. And we’ll do it all again for the next wave.