As an example, if you chat with an LLM it can sort of explain concepts it learned in its training, because it created some internal representation of them. However it’ll never be able to explain things about your project unless you tell it (then it can simply repeat it, which is the same RAG does, the LLM is just spitting it out without any deep understanding of it).

You can try to cram as much context to it as possible and overload it with extra information, or you can understand this difference and separate a task that you know the LLM can handle because you’ll be able to leverage its knowledge and judge the final work yourself.

But this distinction needs to be more clear, I think. The LLM doesn’t have any deep understanding of anything you’re creaming into the context window the same way it does with concepts it learned during training from millions of examples and building some internal model of the concepts and their relationships.

I kinda [wrote about this before](https://dielsonsales.github.io/2023/05/11/starting-a-new-blo...) but the takeaway is that writing forces you to structure your thoughts, giving them a final shape and enables you to notice flaws or gaps in your own thinking process. It’s impossible to do this without writing.

Every time you work in a complex software project, you have all the context in your head about what you must take into consideration. LLMs don’t, you have to explain every little detail to them, but there’s no telling where it stops: do you have to explain X to it or is it in the training already?

When you try to shape how a LLM should behave and what it should know, you end up writing a Bible of relevant context that’s increasingly difficult to maintain as well, often with outdated or contradictory information, and it is still free to ignore things you have written and proceed however it wants.

That’s why so many people only feel comfortable giving the LLM some limited task to do, because you can judge if that specific task needs just enough context that it can handle by itself.

As you said, building a script that only you use personally or a very simple thing that just accomplishes one task and it’s easy to test require almost no engineering, and an LLM can often build those with very little downsides.

If your operating system was regenerated every day slightly differently and with certain things working and others not, you’d quickly revert to the lower predictable abstraction.

Or not, because other 7 billion agents were also waiting for it.

So maybe the project is sort of maintainable, as long as people maintain Vite.

What I mean is, it seems they try to tune them to a few certain things, that will make them worse on a thousand other things they’re not paying attention to.

One of these days I had a doubt about something related to how pointers work in Swift and I tried discussing with ChatGPT (don’t remember exactly what, but it was purely intellectual curiosity). It gave me a lot of explanations that seemed correct, but being skeptical and started pushing it for ways to confirm what it was saying and eventually realized it was all bullshit.

This kind of thing makes me basically wary of using LLMs for anything that isn’t brainstorming, because anything that requires knowing information that isn’t easily/plentifully found online will likely be incorrect or have sprinkles of incorrect all over the explanations.

What would really be useful is a very similar prompt should always give a very very similar result.