Personally, I know coding pretty well. So when I'm using it for coding, I can spot most of its mistakes / misunderstandings

I would not trust using it on a complex domain I'm not super familiar with, like doing taxes

A mistake here is pretty high cost (getting audited, and/or having to pay a bunch in penalties)

The potential of the current crop of LLM/AIs will stop at being a very powerful tool to search large volumes of text using free-form questions.

I do think that pretty clearly contradicts with what a lot of people who make/use LLM models are saying haha

And no, I wouldn't ever give it a year of changelog.md. I give it a short description of the current functionality, and a well-trimmed list of 'lessons-learned' (specific pitfalls/traps from previous work, so the AI doesn't have to repeat them)

If you think git logs are a good way to give context, try it and and see how it works! My instinct's that it won't work as well as a short readme, but I could be wrong. It's so easy to prototype these days, no reason to not give it a shot

And, depending on effort settings, they do more 'thinking', i.e., use more rounds of inference to generate longer internal chains of thought

Both very good reasons to prefer a smaller model, if the small model is good enough for the task

These things are built for pattern matching, and if you keep their context focused on one pattern, they'll perform much better

You want to avoid dumping in a bunch of data (like a year's worth of git logs) and telling it to sort out what's relevant itself

Better to have pre-processing steps, that find (and maybe summarize) what's relevant, then only bring that into context

You can do that by running your git history through a cheap model, and asking it to extract the relevant bits for the current change. But, that can be overkill and error prone, compared to just maintaining markdown files as you make changes

But, my guess, I could see an algorithm like that being very fast. It's basically just doing a form of compression, so I'm thinking ballpark, like similar amount to just zipping the log

Can't be anything CLOSE to the compute cost of running any part of the file through an LLM haha

That might also mean it has no drive for self-determination. It might just be perfectly happy to do whatever humans tell it to, even if it's far smarter than us (and, this is exactly the sort of AI people are trying to make)

So, superintelligence winds up doing whatever a very small group of controlling humans say. And, like you say, humans want to win

Might as well say humans are just a better search tool - it's true in the exact same sense you're using.

All humans do is absorb information, then search through our memories and apply that information in relevant contexts to affect the world

Feudalism was the dominant economic system for millennia. The point is to extract value for the upper class. Peasants only matter as a source of labor, and they only get 'cared for' to the extent of keeping them alive and working.

Now think about what feudalism might look like if the peasants' labor could be automated

Are you just saying that you're more intelligent than them? You can see clearly, where all the steam engine technicians can't?

It's a nice packaging, of a whole bunch of preexisting things. Agentic AI inside a nice sandbox container, running the model on a cron schedule, and with an ecosystem of ready made skills

Nothing new, but it made the tech easy for people to download and start using immediately. That's why you see so many people treating it as new - it's their first time hearing about such a setup

Yeah, it really doesn't matter if AGI has happened, is going to happen, will never happen, whatever. No matter what sort of definition we make for it, someone's always doing to disagree anyway. For a looong time, we thought the Turing test was the standard, and that only a truly intelligent computer could beat it. It's been blown out of the water for years now, and now we're all arguing about new definitions for AGI

At the end of the day, like you say, it doesn't matter a bit how we define terms. We can label it whatever we want, but the label doesn't change what it can DO

What it can DO is the important part. I think a lot of software devs are coming to terms with the idea that AI will be able to replace vast chunks of our jobs in the very near future.

If you use these things heavily, you can see the trajectory.

6 months ago I'd only trust them for boiler plate code generation and writing/reviewing short in-line documentation.

Today, with the latest models and tools, I'm trusting them with short/low impact tasks (go implement this UI fix, then redeploy the app locally, navigate to it, and verify the fix looks correct).

6 months from now, my best guess is that they'll continue to become more capable of handling longer + more complex tasks on their own.

5 years from now, I'm seeing a real possibility that they'll be handling all the code, end to end.

Doesn't matter if we call that AGI or not. It very much will matter whose jobs get cut, because one person with AI can do the work of 20 developers

If they can run a tool from the terminal, see all the output in text format, and have a clear 'success' criteria, then they're usually able to figure out the issue and fix it (often with spaghetti code patching, but it does at least fix the bug)

I think the testing/verification part is going to keep getting better, as we figure out better tools the AI can use here (ex, parsing the accessibility tree in a web UI to click around in it and verify)

I give it a question (narrow or really broad), and the model does a bunch of web searches using subagents, to try and get a comprehensive answer using current results

The important part is, when the model answers, I have it cite its sources, using direct links. So, I can directly confirm the accuracy and quality of any info it finds

It's been super helpful. I can give it super broad questions like "Here's the architecture and environment details I'm planning for a new project. Can you see if there's any known issues with this setup". Then, it'll give me direct links + summaries to any relevant pages.

Saves a ton of time manually searching through the haystack, and so far, the latest models are pretty good about not missing important things (and catching plenty of things I missed)

The AI will get better at compensating, but I think some of it's weaknesses are fundamental, and are going to be showing up in some form or another for a while yet

Ex, the AI doesn't know about what you don't tell it. There's a LOT of context we take for granted while programming (especially in a corporate environment). Recognizing what sort of context is useful to give the AI without distracting it (and under what conditions it should load/forget context), I think is going to be a very valuable skill over the next few years. That's a skill you can start building now

The danger is, it doesn't quite scale up. The more complex the project, the more likely the AI is to get confused and start writing spaghetti code. It may even work for a while, but eventually the spaghetti piles up to the point that not even more spaghetti will fix it

I'll get that's going to get better over the next few years, with better tooling and better ways to get the AI to figure out/remember relevant parts of the code base, but that's just my guess

Not how we want things to be, but how they actually are and will be

I don't think AI for programming is a passing fad

You can ask it about the code, sure, and it'll try to tell you how it works. But, what if there's a bug in the code? Maybe the LLM will guess at how it was supposed to work, or maybe it'll start making stuff up to justify the bug's existence (it's actually a hidden feature!)

The docs say how the code should work. For an LLM that has to go relearn everything about your code base every time you invoke it, that's vitally important

When you think about what an LLM is, it makes more sense. It causes a strong activation for neorons related to "code review", and so the model's output sounds more like a code review.