Reverse engineering can take several forms for me: it started with writing parts of systems from scratch, and now (due to academic training) it oftens looks like "thinking about how to explain the system to a group of colleagues", which is one degree of separation from writing the system from scratch, but is much faster - and seems to work well when working with LM agents on code (because you have to confront your ignorance)

part of the training in academia is how to know when you don't understood the thing, so this ends up working well for me now.

One trick: you can force yourself to understand something better by making a presentation and imagining you're going to be grilled by someone smarter than you. This is kind of the feeling of pressure that academic paper reviews give (in an ideal world) and it translates when you think about explaining something to someone else.

I started practicing this for the stuff I'm interested in ... and the result is that I generate a lot of ideas. 99% of them are garbage.

Before AI, a conversation with colleagues would help me weed out the garbage, and possibly give me a new perspective ("here's something surprising about what you just said!")

Now, AI plays a helpful role in helping me weed out the garbage for the subset of ideas that I can test empirically -- and sometimes it does give me a new perspective, but with significantly more noise than my colleagues, who would take me right to the point that would be useful for me to see.

I've never felt like I needed a second brain ... because if I had such a thing, I think I would be keeping the 99% garbage around -- instead I just let biology GC it.

RIP to those poor humans. I can't imagine having your brain melted by Fablespeak as your FTJ.

Of course, all I’m saying is that you need to refine your sample! For instance: the allocation architecture is not correct, and one has to run a bunch of performance investigations and resolve it.

My responses are intending to convey that I don’t believe this is possible, no matter how good LMs get — and it seems like we are in agreement.

I don't believe the problem is "ah, you didn't write the spec clearly enough" -- which is why I'm asking about your own experiences.

> If you instead spend a day writing a proper specification, then ask the agent to spend a week implementing that, you'll need zero tools and skills afterwards to clean it up, because there won't be any misunderstandings, assumptions or other things, just code fulfilling what the specification says.

Are you claiming that you reliably get good software out of agents using this process?

My experiential claim is that I've followed this process for complex pieces of software (and, often, taking significantly longer than a day -- more like several weeks, as if I was writing an academic paper) and they still get it wrong ... even with an "academic paper level spec"

Models: only the best, Fable 5 max, Sol 5.6 max, yada yada -- I'm not convinced this process works for "serious endeavors" (for instance: let's build a novel accelerator compiler or let's build a new incremental linker) where "serious" here means "something where I'll likely be spending multiple weeks working on it"

What is the most complicated thing you’ve built with LM agents? Have you done it with a single spec? How novel was it?

This comment is so laughably “you’re holding it wrong” I can’t respond to you seriously.

> If you instead spend a day writing a proper specification, then ask the agent to spend a week implementing that, you'll need zero tools and skills afterwards to clean it up, because there won't be any misunderstandings, assumptions or other things

The set of software that has followed this process is measure zero.

To correct those behaviors, you're going to write tools and skills, and that's going to help, but it is still clear that you are fighting the distribution (today).

> A stochastic search process with an executable optimization objective over space of programs S can only maintain or improve the objective

This is superoptimization. We've known this since the 80s (Massalin, STOKE is more recent: https://github.com/StanfordPL/stoke) The only novelty is that the proposer is now way better with LMs.

Further, there's a large number of reasons for software written by agents to be slow:

- LMs still don't do data or hardware-oriented design well out of the box, and therefore if you're engaging in any sort of serious novel work, beyond porting an extremely well-understood program with extremely well-understood workloads, you're going to be spending hours tracking down bad allocation decisions (c.f. why TigerBeetle doesn't use agents), which are often the root of evil (before you'd reach for anything further)

- The knobs you'd need to get serious performance are nearly unreachable in languages which LMs are good at (even Rust requires a discipline that the default language doesn't enforce). When you drop into the lower realms, you're trading consumption context for access to these levers. The levers are also "soft": you find yourself writing a bunch of skills, and tools to try and enforce the discipline.

The reality is to get performant code (quickly) out of an agent, you need to know how to write performant code (and you need to know how to surface the information that you'd use to create a verifier for such a thing to the agent), which 99% of developers do not know in 2026.

Sure, agents can teach you how to do this -- but it's one of these things where iykyk.

Experience: I've poured 10s of billions of tokens into Zig with the best agents and I have the time and space to try these things.

If you want to start learning the discipline, I'd recommend matklad's + TigerBeetle blog -- as well as hardware-oriented design.

Why do you think that Anthropic wants fewer tokens inputted and outputted?

Am I to believe the creators, knowing full well that the source will, as Boris Cherny put it in a recent interview, be deleted and rewritten from scratch at the release of the next big model?

Further: I'm responding to content in the blog post itself:

> Until pretty recently, the tools you wrote code with were a flat fee (or free). Your editor cost the same whether you fixed one test or fifty that afternoon, so an individual task didn't really have a price of its own.

I find this type of prose ridiculous. It conveys "this is the way things are now, get used to it".

Does that make sense?

My contention is that we should be building towards less churn, not more. I'm aware that some churn is the cost of engaging in any sort of enterprise, but I'm deeply suspicious of an AI company inserting themselves between me, and the tasks I wish to do with my device -- with a completely opaque system that I can't really "learn".

- I'm happy to learn how to use tools efficiently

- I like to be able to inspect my tools

- I'm against tools changing underneath me

Are you against any of these points?

Oh yes, Claude will do all sorts of different things -- it depends on how you use it! You should totally learn all of these little finicky things ... because now completing your tasks cost money. It's not "free" anymore haha like when you used your old text editor, what are you a grandpa?

Oh, and those things will definitely change, as we (the priests of Claude) are vibe coding the system you use to do your little "tasks" ... right, you can't see how it works ... the code is not available. It's all good, just trust us -- we're totally looking out for you.

I mean it is utterly ridiculous to talk around this model of development. There are so many walls between you and doing the thing you want to do.

Agents are great, but the notion of "best tricks" for how to best use an opaque costful tool which will, by all odds, be completely different in a few months time is quite funny.

You know what won't change? A fucking text editor. Or your pi config, or a local model you run and trust.

Really couldn't recommend it more.

And it's a bit of a shame: by virtue of their position (their embedding in the fabric of venture capitalism), it seems like they can only make a subset of things -- what they can make is dictated enormously by capital, as they are engines of capital.

Not sure the point I'm trying to make, I just find it amusing.

Perhaps the point is that it might be more worthwhile to give independent creators a billion dollars to play around with agent swarms if we want to keep diversity in the evolutionary algorithm that is the software industry high.

Wondering if there are modeling or analysis modalities that don’t fit cleanly into MLIR concepts (understand that abstract interpretation on the IR comes with a unique set of concerns)

"It actually made something that worked. But when I tried using it, I found I didn’t like it. I need to wait for a Claude that has the taste ..."

Yes, I am familiar with that experience.