These diffs already exist (at least for some languages) but aren't yet integrated into the standard tools. For example, if you want a command line tool, you can use https://github.com/Wilfred/difftastic or if you are interested in a VS Code extension / GitHub App instead, you can give https://semanticdiff.com a try.

If the code needs to be reformatted, this should be done in a separate commit. Fortunately, there are now structural/semantic diff tools available for some languages that can help if someone hasn't properly split their formatting and logic changes.

The language specific logic does not end with the parsers though. The core of SemanticDiff also contains language specific rules that are picked up by the matching and visualization steps. For example, the HTML module might add a rule that the order of attributes within a tag is irrelevant. So it all comes down to writing a generic rule system that makes it easy to add new languages.

The leaves are the ones that end up being highlighted in the diff, but the inner nodes play an important role as well. We try to preserve as much of the code structure as possible when mapping the nodes. A developer is unlikely to change the structure of the code just for fun. A mapping with a larger number of structural changes is therefore more likely to be incorrect.

> - for diffing, it feels more acceptable to restrict the matching to be monotonous on the leaves since it's difficult to visually represent moves if you can detect them. For merging, supporting moves is more interesting as it lets you replay changes on the moved element,

We use a pipeline based approach and visualizing the changes is the last step. For some types of changes we don't have a way to visualize them yet (e.g. moves within the same line) and ignore that part of the mapping. We are still trying to get the mapping right though :)

We upstreamed a few bug fixes for tree-sitter itself. The grammars were a bit more complicated because we were just using them as a starting point. We patched tree-sitter, added our own annotations to the grammars and restructured them to help our matching algorithm achieve better results and improve performance. In the end there was not much to upstream any more.

Using a well tested parsing library, such as Roslyn for C#, and writing some code to integrate it into our existing system aligned more with our goals than tinkering with grammars. Context-sensitive keywords in particular were a constant source of annoyance. The grammar looks correct, but it will fail to parse because of the way the lexer works. You don't want your tool to abort just because someone named their parameter "async".

We try to really get the diff quality nailed down before going after merges. We don't have merge functionallity in SemanticDiff yet.

The main issue we have with tree-sitter is that the grammars are often written from scratch and not based on the upstream grammar definition. Sometimes they only cover the most likely cases which can lead to parsing errors or incorrectly parsed code. When you encounter parsing errors it can be difficult to fix them, because the upstream grammar is structured completely different. To give you an example, try to compare the tree-sitter Go grammar for types [1] with the upstream grammar [2]. It is similar but the way the rules are structured is somewhat inverted.

We use separate executables for the parsers (this also helps to secure them using seccomp on Linux), and they all use the same JSON schema for their output. This allows us to write the parser executable in the most appropriate language for the target language. Building all them statically and cross-platform for our VS Code extension isn't easy though ;)

[1]: https://github.com/tree-sitter/tree-sitter-go/blob/master/gr... [2]: https://go.dev/ref/spec#Types

Tree sitter grammars are primarily written to support syntax highlighting and often use a best effort approach to parsing. This is perfectly fine for syntax highlighting, since the worst that can happen is that a few characters are highlighted incorrectly. However, when diffing or modifying code you really want the code to be parsed according to the upstream grammar, not something that mostly resembles it. We are currently in the process of moving away from tree-sitter and instead using the parsers provided by the languages themselves where possible.

GumTree is good at returning a result quickly, but there are quite a few cases where it always returned bad matches for us, no matter how many follow-up papers with improvements we tried to implement. In the end we switched over to a dijkstra based approach that tries to minimize the cost of the mapping, which is more computationally expensive but gives much better results. Difftastic uses a similar approach as well.

[1]: https://semanticdiff.com/

As author of SemanticDiff, I am obviously a bit biased. But Wilfred, the author of difftastic, found the analysis to be "pretty even-handed" [1], so I think it should be somewhat fair.

[1]: https://x.com/_wilfredh/status/1764424652611318146

I'm sorry you didn't have a good experience testing the tool. If it doesn't work / makes things worse than a standard diff, that's definitely considered a bug. It is probably something specific to your code and not a general issue. It would therefore be great if you could open an issue [1] or support ticket [2], ideally with some sample code, so we can take a look. Thanks in advance!

[1] https://github.com/Sysmagine/SemanticDiff/issues [2] support@semanticdiff.com

Comments URL: https://news.ycombinator.com/item?id=40987674

Points: 2

# Comments: 0

Comments URL: https://news.ycombinator.com/item?id=40917931

Points: 2

# Comments: 0

We are working on https://semanticdiff.com/ which detects basic semantic changes like converting a literal from decimal to hex or reordering keys within JSON objects. It is not a command line utility but a VS Code extension and GitHub App. You can check out https://semanticdiff.com/blog/semanticdiff-vs-difftastic/ if you want to learn more about how it works and how it differs from difftastic.

I think a system that parses the code and uses verifiable rules to hide invariant changes would have a much easier time being adopted. I may be biased since I work on SemanticDiff (think difftastic but for VS Code and GitHub) and have implemented such rules myself. There have been several times where I've thought about implementing a new rule, only to find out that some obscure edge case would not be handled correctly. So I don't see how LLMs could handle these cases correctly in the near future.

https://app.semanticdiff.com/django-money/django-money/pull/...

It doesn't make a huge difference, but it filters out changes like the added line break in "if value: value = str(value)" nicely. I haven't announced the project yet, but maybe someone will find it useful :-)

Comments URL: https://news.ycombinator.com/item?id=27530074

Points: 178

# Comments: 63

Comments URL: https://news.ycombinator.com/item?id=6964346

Points: 63

# Comments: 15