Comments URL: https://news.ycombinator.com/item?id=49729799

Points: 2

# Comments: 0

IMHO the value of these nuggets is that you understand what you're doing and why; opening yourself to large amounts of non-default behavior likely will end up in a less than pleasant setup.

Not taking into account that different users might disagree on what is convenient and what is not, which is basically the point of making things configurable.

Comments URL: https://news.ycombinator.com/item?id=49726537

Points: 1

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=49525257

Points: 1

# Comments: 0

Though I believe it's more of a direction for rust-analyzer rather Rust Glancer, at least for now.

Data is written at the end of indexing phase, during write we hold a lock, and the codebase is aware that offloaded data can be corrupted/outdated (mostly relevant for cache).

Query processing works on top of transaction, and the transaction holds data loaded from files for the duration of the query, and frees once the query is dropped.

There is a ton of tricky parts there, but I guess that's the gist of it.

If you're interested, here are the relevant parts of the source code: https://github.com/rust-glancer/rust-glancer/tree/main/crate... https://github.com/rust-glancer/rust-glancer/tree/main/crate... https://github.com/rust-glancer/rust-glancer/blob/main/crate... https://github.com/rust-glancer/rust-glancer/blob/main/crate...

As for the size -- the Rust Glancer artifacts for Rust Glancer itself currently take 225mb. And that's proportial to the project size only, e.g. it doesn't grow over time on its own.

I certainly enjoy working with humans much more than working with LLMs. Working with humans is usually "how can we get this done together", while working with LLMs is pretty much is "how can I make this model do what I need", which is something I don't like. An personal persistent thought I have now is not to transfer my relationship with LLMs to my relationship with human beings, and I don't really like it.

This is when you do full reindexing.

> Is that only an incremental indexing with normally low RAM usage?

Yup, on save you only update the crates that were invalidated.

And note that on dirty buffers Rust Glancer doesn't do reindexing at all: it reuses the last available analysis, plus it does syntax-based shallow overlay that is sufficient to be useful but doesn't necessarily detect semantic changes. It's a tradeoff, but this tradeoff makes Rust Glancer competitive in terms of latency with rust-analyzer without compromising RAM and while keeping your CPU cool.

> it completely reindexes the single file that’s saved

This is true, though I have to mention that change in the file might invalidate its reverse dependencies, which can make the partial analysis bigger than just one file/crate, but it's still very fast in practice.

So all in all -- yeah, LLMs can be good _domain experts_ when you build an LSP, but a) I wouldn't trust them blindly, and b) the quality of code is still very much your responsibility.

So I don't deny the idea, just stating that it's probably _significantly_ more complex to implement than it sounds.

The difference, however, is that with Rust Glancer you don’t need full reindexing often, so it probably compensates for that to a degree.

It can use storage when running build scripts/expanding proc macros, or when running flycheck diagnostics. In both cases, it’s because it runs cargo and it writes artifacts to the target dir. And if features do not align between “common” cargo commands and configuration rust analyzer has, it can lead to conflicts and even more increased storage size (because you end up having effectively 2 sets of artifacts).

But all of that does not apply to rust glances, since it does not build code for you (even cargo diagnostics are disabled by default).

Rust Glancer analysis artifacts are not that big (it’s basically stuff that would otherwise be loaded to memory), and Rust Glancer cleans garbage so that it does not accumulate over time, so it should be fine.

Several things here: 1. We don't need all the information (project can have 1000+ dependencies, while query might only care about the current open file), so the amount of information we load is smaller. 2. Most of the time IDE does not actually do any queries, so if you switch to browser/Slack, you don't pay the tax. 3. Since data is loaded to the disk, after initial indexing restarting no longer consumes that much ram, and you get reindexing for free. 4. Besides offloading, I implement quite a bit of memory optimizations (some of which are covered in docs: https://rust-glancer.github.io/docs/development/MEMORY.html ), so it's a combination of factors.

With things like trait solving I am not reinventing the wheel, and use official tooling (Chalk). Even though now the new solver is recommended, Chalk still does its job and lets me not to worry about potentially the most complex part of the machinery.

In places that seem to be underdocumented, it's always possible to: 1) look into sysroot implementation for clues 2) look into compiler sources 3) hijack stuff from rust-analyzer

I am lucky to not be the first guy who does a Rust LSP, so it's not that fundamental of a research, and much more of just an implementation :)

Some comments on the thoughts post

> I think that part can perhaps be made lazy (but not incremental!) with little overhead?

I am still thinking about making stuff lazy, since with non-incremental approach it can introduce more lags than would be perceived comfortable, but what I do right now is that I prioritize open buffers (so the stuff user needs gets processed faster), and everything else is indexed in background. I have some thoughts about lazy approach, but before I'll try them, I want to work on the quality of analysis first.

> Would be interesting to compare memory usage with Rust Rover. Net of the IDE GUI itself, I would expect RR to be more compact.

I've received a few comments about RR already, and, to be honest, I've never tried it (somehow I never got along with JetBrains IDEs) -- but will look into it.

> One potential approach here is to pull the Sorbet trick, where you don’t run meta programming at all, and instead have a plugin interface to “explain” the effects of what that would have done.

Funnily, that's exactly (well, mostly) the idea I have in mind and want to try out. Tentatively planned for Rust Glancer 0.3.0 (0.2.0 will be mostly about more complete indexing/functionality and editors support). In short, I don't want to have random code execution in the LSP itself (even diagnostics are disabled by default), but it's quite possible that we don't need that for proc macros.

> Try changing this option and see if it helps?

I have tried both editor and server watcher options, didn't really feel the difference, but can't say that I performed a high quality investigation. I certainly noticed that vs code is not very good at properly reporting external changes (it misses a lot of them), and the server watcher was tricky to get right (and yeah, it has quite a bit of platform-specific quirks; which is one of the reasons I don't feel comfortable providing a server for Windows yet -- I have no machine to test it).

> This still seems to me to be the lowest-hanging watermelon here — split the world into arcy-pointy incremental tip of the iceberg, and mostly read-only, on disk, compact, dark, moist breeding ground for supply chain attacks.

This would be awesome! And I'd be really happy to see that change making Rust Glancer redundant; while ability to experiment is cool, I think that unified tooling is ultimately better for the language.