Similarity search is rarely that relevant when working on codebases. Our biggest learning actually came from an accidental technical bug where indexing had been returning zero hits for quite some time, and we barely noticed any performance degradation. In fact, the agent seemed to be working better than ever.

After discovering the bug, we did some rigorous A/B testing and realized it was best to drop indexing altogether. There could still be a case for it in extremely large codebases with a lot of docs e.g. excel or several text files.

Comments URL: https://news.ycombinator.com/item?id=49476449

Points: 1

# Comments: 1

After doing some rigorous A/B testing, we dropped indexing. For coding, I think the reason is that a repo is already searchable. Imports, call sites, file and test names, grep gives you cheap yet reliable version of what indexing would do, and the agent can read around a hit to verify it. Chunked retrieval hands the model something that looks right, and it tends to trust that instead of going to look for the actual source. Another thing that I noticed was the most intelligent models like Opus 5 and Fable ignored chunks anyway most of the time for some reason. Possibly perhaps they are trained around not trusting similarity checks for codebases.

Extremely large codebases with docs feel different. You can’t grep for a concept you can’t name. That’s the case where I’d still use retrieval.

(I work on TheGitAI, for disclosure.)

Comments URL: https://news.ycombinator.com/item?id=49436326

Points: 2

# Comments: 0