One thing I will add: I actually don’t think it’s wrong to start out building a vibe coded spaghetti mess for a project like this… provided you see it as a prototype you’re going to learn from and then throw away. A throwaway prototype is immensely useful because it helps you figure out what you want to build in the first place, before you step down a level and focus on closely guiding the agent to actually build it.

The author’s mistake was that he thought the horrible prototype would evolve into the real thing. Of course it could not. But I suspect that the author’s final results when he did start afresh and build with closer attention to architecture were much better because he has learned more about the requirements for what he wanted to build from that first attempt.

But aren’t there still plenty of opportunities for building ML models beyond LLMs, albeit a bit less sexy now? It’s not like you can run a business process like (say) AirBnB’s search rankings or Uber’s driver marching algorithms on an LLM; you need to build a custom model for that. Or am I missing something here? Or is that point that those opportunities are still there, but the pond has shrunk because so much new work is now LLM-related? I buy that.

But some works transcend the specific details of their historical origins and authorship and contain ideas that echo down the centuries. Locke’s ideas were instrumental in founding the United States and feed into much of modern liberalism. And I can read Hobbes here today in the 21st century and still find the pessimistic core of his book powerful and relevant, even while ignoring much of the book because it’s full of the parochial concerns of 17th century England. That was really what I was getting at: not “this is the exact meaning of these works in the 17th century”, but “here is the tension of ideas these books bequeathed to us.”

I would say Hobbes in particular is a complex and difficult and frankly eccentric thinker; don’t make the mistake of believing you understand him; he is weird. If you really want to grok the guy in the context of his culture and historical moment, you should just read Quentin Skinner. That’s hardcore intellectual history though; for the basics I’d just go for the clear and brief and informative Oxford Hobbes: A Very Short Introduction.

* He thinks everyone should be compelled to worship in the state-sanctioned religion

* Censorship of publications, teaching, etc. is necessary because ideas can be dangerous.

* Separation of powers (e.g. between executive, legislature, judiciary) is bad; he wants a single unitary sovereign with unlimited power.

* The sovereign is above the law

* Resisting a tyrannical sovereign is bad

...and that's why I'd pick Locke over Hobbes. And I think most of us would too.

The great debate of political philosophy coming out of the 17th century was between Hobbes (anarchy is horrible, humans aren’t nice to each other, best to give up your freedoms to a strong sovereign/state for protection) and Locke (liberty is best, people are reasonable, limit government). I will say that like most of us I probably side more with Locke but as a pessimist about human nature I find Hobbes’s argument fascinating too.

I did consider MoEs but decided (pretty arbitrarily) that I wasn’t going to count them as a truly fundamental change. But I agree, they’re pretty important. There’s also RoPE too, perhaps slightly less of a big deal but still a big difference from the earlier models. And of course lots of brilliant inference tricks like speculative decoding that have helped make big models more usable.

On the LLM Architecture Gallery, it’s interesting to see the variations between models, but I think the 30,000ft view of this is that in the last seven years since GPT-2 there have been a lot of improvements to LLM architecture but no fundamental innovations in that area. The best open weight models today still look a lot like GPT-2 if you zoom out: it’s a bunch of attention layers and feed forward layers stacked up.

Another way of putting this is that astonishing improvements in capabilities of LLMs that we’ve seen over the last 7 years have come mostly from scaling up and, critically, from new training methods like RLVR, which is responsible for coding agents going from barely working to amazing in the last year.

That’s not to say that architectures aren’t interesting or important or that the improvements aren’t useful, but it is a little bit of a surprise, even though it shouldn’t be at this point because it’s probably just a version of the Bitter Lesson.

[0] https://en.wikipedia.org/wiki/Mach%27s_principle

I will add though, on 2 and 3, during most of the coding I do in my day job as a staff engineer, it’s pretty rare for me to encounter deeply interesting puzzles and really interesting things to learn. It’s not like I’m writing a compiler or and OS kernel or something; this is web dev and infra at a mid sized company. For 95% of coding tasks I do I’ve seen some variation already before and they are boring. It’s nice to have Claude power through them.

On system design and architecture, the problems still tend to be a bit more novel. I still learn things there. Claude is helpful, but not as helpful as it is for the code.

I do get the sense that some folks enjoy solving variations of familiar programming puzzles over and over again, and Claude kills that for them. That’s not me at all. I like novelty and I hate solving the same thing twice. Different tastes, I guess.

That sounds good for many of us (and don’t we all like to think we’re top candidates here on HN…) but is there any data to back this up? Or it just anecdata (not to dismiss anecdata, still useful info).

Incidentally, I would recommend trying implementing speculative decoding yourself if you really want to understand LLM inference internals (that, and KV caching of course). I tried it over the Christmas holidays and it was a wonderful learning experience. (And hard work, especially because I forced myself to do it by hand without coding agent assistance.)

It’s right there. You can go and see it any time, doing the things you don’t think it’s capable of doing. Just a little curiosity is all you need.

If you think, hey but people had a “job” in 1700, and they had a “job” in 1900, think again. Being a peasant (majority of people in Europe in 1700) and being an urban factory worker in 1900 were fundamentally different ways of life. They only look superficially similar because we did not live the changes ourselves. But read the historical sources enough and you will see.

I would go as far as to say that the peasant in 1700 did not have a “job” at all in the sense that we now understand; they did not work for wages and their relationship to the wider economy was fundamentally different. In some sense industrialization created the era of the “job” as a way for most working-age people to participate in economic life. It’s not an eternal and unchanging condition of things, and it could one day come to an end.

It’s too early to say if AI will be a technology like this, I think. But it may be. Sometimes technologies do transform the texture of human life. And it is not possible to be sure what those will be in the early stages: the first steam engines were extremely inefficient and had very few uses. It took decades for it to be clear that they had, in fact, changed everything. That may be true of AI, or it may not. It is best to be openminded about this.